DEV Community: Patryk Kozłowski

9 software development trends for 2023

Patryk Kozłowski — Fri, 03 Mar 2023 09:39:10 +0000

Undoubtedly, this year will be marked by the world’s crisis, which won’t leave out software development. Thus, a lot of trends will be directly related to the current state of economics, which affects the way we build applications but also applies to the popularity of particular programming languages. To learn more, keep on reading!

1. A broadened pool of developers

Anyone who keeps a weather eye on the market can notice a significant reduction in development teams in tech companies worldwide. For the first time in years, software developers are either fired or placed on the bench, awaiting the next project to come.

It is an unprecedented situation that, without a doubt, will affect the market in the long run.

If you’ve decided to build your app now, it will be easier to find developers or even cherry-pick them. And, surprise, surprise, their services might be slightly less expensive.

2. The end of overemployment

Let’s call a spade a spade, the pathological situation of overemployment has come to an end. Developers juggling two or more full-time jobs had become a widespread and well-known problem. And often, being aware that a particular developer is indispensable, companies used to turn a blind eye to the situation.

However, as a result of fewer developers needed on the market, overemployment won’t be an accepted practice anymore. What is more, programmers might be unable to find two simultaneous jobs or will simply get cold feet about deciding on more than one full-time employer.

We published an extensive article on overemployment in April last year, so we won’t repeat ourselves here.

3. AI-assisted development

We know automation perfectly well from production processes in which industrial robots take over repetitive tasks. However, thanks to the rapid development of artificial intelligence, machines have also started to take on highly-skilled and creative work. Of course, they aren’t perfect nor infallible yet, so we use them mainly to support or speed up the work, but the hopes are high.

Just a few examples of the systems used in IT:

ChatGPT: a chatbot mimicking human speech; we can also use it to write and debug software. It has been trained with reinforcement learning on vast amounts of data. This one is a real game-changer on the market!
Codex: a part of ChatGPT, it translates natural language to code, which means you can issue commands in English, and it will try to understand your intentions. It is compatible with over a dozen of programming languages, such as Python, JavaScript, and PHP;
Nvidia Canvas: creates realistic images using artificial intelligence;
GithubCopilot: AI pair programmer that suggests code and whole functions, trained on billion of publicly available lines of code;
CodeQL: an engine conducting semantic code analysis, particularly helpful in finding weak points.

4. Technologies on the rise and fall

Although it isn’t the latest technology in software development, we observe that Node.js is still in high demand. No wonder that, according to the Stack Overflow Survey, together with React.js, it is the most common web technology developers use.

On the other hand, there are far fewer projects written in PHP and more PHP developers available. From our perspective, the global crisis might be the reason. PHP is a technology with a relatively low entry threshold, quite easy to set up and learn. Thus, it was eagerly used by small start-up companies, which lived mainly off investors’ money. In times of crisis, external financing is most frequently reduced or cut off completely.

5. Microservices

Microservices describe a relatively new software development trend where applications are composed of smaller, independent parts – services. They communicate with each other over APIs but don’t depend on one another, which is the case of monolithic architecture. When each app process is run as a service, it can be deployed, scaled, and updated separately according to the current needs. If one function fails, it also doesn’t affect the others. Moreover, each service can be written in a different programming language, so you can employ different devs and teams to work on one product.

When teams work independently on smaller parts of the software, they simply do it faster in terms of development cycles and reduced time-to-market of new features. However, if you don’t want to build services from scratch, platforms like AWS offer off-the-shelf building blocks for storage, logging, messaging, monitoring, DevOps, and many more.

6. Nearshoring/Offshoring

We’re fully aware that nearshoring and offshoring aren’t anything new in the software development world. But still, we’ve decided to include them in the list of the upcoming trends in the software industry, and here is why.

As we mentioned at the beginning of this article, 2023 trends are affected by the world’s crisis. Nowadays, more than ever, companies bend over backward to cut costs wherever possible. However, instead of halting their projects completely, they prefer to look for savings elsewhere. Thus, businesses show a particular interest in developers and development companies from other (meaning cheaper) parts of the world.

If you’re also rooting around for a reliable contractor, we recommend the blog post explaining how to verify a nearshore development company.

7. Increase of computing power for AI

Artificial intelligence is at the forefront of computing development, and multiple solutions are specifically designed for that very purpose. However, it works the other way round, too; AI growth is directly linked to the increase in processing power.

To mirror human reasoning and skills, artificial intelligence algorithms have to process large amounts of data. Thus, computers increase their CPU, GPU, and microchip capacity at full tilt to cope with the growing demand for AI applications. In effect, it will reinforce one of the software industry trends for machine learning, computer vision, and natural language processing applications.

8. Bioinformatics

There’s an enormous and unprecedented demand for biological data analysis and interpretation, impossible to satisfy with human endeavors only. As a result, there’s a growing need for software that can do the job effortlessly and accurately. The field of bioinformatics deals with these issues, combining computer science, biology, and mathematics.

Bioinformatics is used on a large scale in clinical research, drug discovery and design, gene sequencing, and gene therapy. Therefore, specialists in the field and software companies building AI systems will be in high demand in pharmaceutical companies and biomedical laboratories in 2023. Thus, among trends in software, we can find bioinformatic apps.

You can learn more about data analysis in microbiology labs from our case study.

9. Autonomous things

The emergence of autonomous things (AuT) marks the dawn of a new technological era. The most significant advancement is that computers, as separate entities with no human guidance, are carried into our physical environment and share it with us for the first time.

Autonomous things, such as cars and shuttles, industrial robots, self-navigating military or delivery drones, home robotics, surgical robots, and cashier-less shops, operate so well thanks to the more effective AI algorithms, increase in computing power, and inbuilt sensors that finally have become less expensive.

Creating software for autonomous things will be one of the trends in application development and a challenge for the next few years, particularly in the field of the safety levels they need to provide.

Wrap-up

We’ve gone through software development trends for 2023 that covered four different areas: programmers, development technologies and tools, business issues, and, finally, technological advancements that will revolutionize our lives. Hopefully, the broad range of provided forecasts will give you an idea of the direction in which software development is heading. And, more importantly, a handful of prompts on how to make the most of it. See you soon!

Tips on how to reduce software development costs

Patryk Kozłowski — Fri, 03 Mar 2023 09:23:09 +0000

It shouldn’t surprise anyone that cutting costs in business is a tremendously timely topic in times of a worldwide crisis. At the same time, no company wants to inhibit their development or slow down processes, which might result in a high churn rate. Thus, considering the situation, looking for cost reduction can become the most clever maneuver. Today, we will show you how to do it in our pet area – software development. As they say, a penny saved is a penny earned.

Might also interest you: 9 software development trends for 2023

How much does it cost to create an app?

The cost of developing software depends on so many aspects that it would be impossible to calculate without knowing your project inside out. Given app type, its functionalities and their complexity, technologies, data storage, security, third-party integrations, QA, development process, and deadlines – they all have to be taken into account. The thing we can do, though, is to show you in detail how to reduce custom software development costs.

How to reduce the costs of software development?
Drastic times call for drastic measures. But as a matter of fact, we wouldn’t call savings an extreme move, just reasonable and worth considering no matter the circumstances. So, without further ado, let’s see how to reduce IT costs in the following five areas.

1. Costs to create an app

Development incurs the lion’s share of the digital product cost. Thus, that’s the first area we’ll investigate.

Find off-the-shelf solutions

We might be putting the cat among the pigeons by saying that, but first, ascertain if there are no ready-made solutions on the market that you could implement easily. It may be that modifying your processes to use an off-the-shelf software instead of developing one from scratch, and paying through the nose, is a more favorable option for your company

Use ready-made blocks

There’s nothing wrong with using off-the-peg components, libraries, cloud services, and open source, as long as they do the job. The approach can significantly reduce development time and, as a result, the costs.

Build an MVP

If you want to build your application by the book, start with a minimum viable product or at least cut down on its functionalities and utilize reusable components. It isn’t trumped-up advice, for we had a client that wanted to build an immaculate product from the start, and their budget ran out before we had the simplest version of the system ready. Just remember, a bird in the hand is worth two in the bush, also in software development.

Choose popular technology

By choosing popular programming languages, you can achieve significant technology cost reduction. The fact that a solution is widespread translates into providers’ competitiveness, more ready-made solutions available, and the freedom to change your development company if the cooperation turns out ineffective. A pro tip in crisis reality is to be mindful of what kind of developers are laid off most often and consider these technologies first.

2. Labor cost

When the budget is unlimited, indigenous in-house developers are a favored option. However, if you aim at cost reduction in IT, you should also consider other options.

Offshore and nearshore

Take into account cooperation with a nearshoring or offshoring company. The development services will be more cost-effective, and you won’t have to foot the office, vacation, sick leaves, and other perks. Also, you will pay for what is delivered, either in time and materials (T&M) or fixed fee system.

Automate developers’ work

Whenever it’s possible, automate the development tasks. It applies to, for example, automated testing, continuous development and integration, pair programming systems, such as GithubCopilot, or code analyzing tools that can help in weak point detection, such as CodeQL.

3. Fixed costs

Fixed costs are predictable, but the problem is we have to cover them even if our app doesn’t earn. That’s why, cost reduction opportunity assessment will be indispensable in the case of any product. Below you can find some of the areas worth analyzing in the first place.

Third-party solutions, such as development tools and libraries, often involve substantial monthly/yearly fees. We’re far from unequivocally stating that building dedicated tools will be less expensive, but it’s always worth calculating and comparing with extraneous costs.

Infrastructural costs

Each element of the system like on-premise or cloud servers, CDN, data storage, third-party integrations, payment gateways, APIs, domain DNS, and SSL costs. Let’s not forget about infrastructure monitoring. You won’t be able to cut these expenses off completely, but over the years, we’ve learned that there’s always room for optimization. One of the ways can be cloud migration or cloud optimization.

4. Your time

Last but not least, your time is beyond price, isn’t it? And it really makes a difference whether you spend it on defining the system features or trying to handle the team. Thus, a skilled product owner who knows the ropes and will take responsibility for the product backlog, goal delivery, and maximizing product value will be an invaluable part of your team. Acting as an intermediary between business and development, they will take an awful lot of communication off your shoulders.

This way, you can focus on what is most relevant at your end: business development, monetization, or promotion, so your system becomes profitable as early as possible. But don’t get us wrong, a dedicated system will never be created without your active involvement, though hiring an experienced proxy will certainly help you focus attention on landmark issues.

Wrap-up

Instead of answering the question How much does it cost to make an app?, we choose to serve you with concrete tips on cost reduction, simply because you can implement them no matter the type of system you’re developing.

Just remember that haste makes waste, so before you roll up your sleeves, devote considerable time to analyzing and planning. Cost savings at that stage may bring real trouble later on.

In drawing things to a close, we’re always there to help you with app price estimates, so don’t hesitate to contact us. Thanks for making it to the end, and see you in the next entry!

Enterprise cloud computing. How to choose the cloud and migrate there in 6 steps?

Patryk Kozłowski — Fri, 03 Mar 2023 09:13:22 +0000

Enterprise cloud computing can be seen as one of the most transformative technologies in the last decade. It offers higher scalability, elasticity, and business stability, and allows companies for significant savings compared to building systems from scratch and maintaining physical servers.

What is an enterprise cloud? The definition

We used to think about the cloud as servers accessed over the internet, bought from external providers, mainly for data storage. Back then, it was treated as an alternative to physical servers, which were, and still are, costly and involved a lot of maintenance.

However, the essence of the cloud for enterprise nowadays is managed services offered by the cloud providers. They can cover extensive developer tools – libraries of configurations, automations, and runbooks that can be used freely to build cloud applications. The cloud’s scalability and relatively low cost have become a game changer for enterprises worldwide.

You can find the above apps under the terms: infrastructure as a service (IaaS) and software as a service (SaaS).

A hybrid cloud

We won’t delve into this topic in the article, but it’s worth mentioning that there’s also a mixed environment called a hybrid cloud. In this case, the apps run in public and private clouds, and on-premise servers. The tasks are performed by clouds working together, sharing data or processes. A hybrid cloud is a popular approach as it doesn’t make you dependent on a single solution or provider, which, in turn, helps you minimize the risk of app failure. It also offers greater flexibility to use solutions that work best for you in a given scenario.

How to choose the cloud?

It’s worth noting that all cloud providers offer more or less the same services. However, it doesn’t mean you can toss a coin when choosing one. This is a landmark decision for your business, and the devil is in the detail. Thus, here are the factors we recommend considering:

The popularity of the provider
There are three leading enterprise cloud providers, namely

(data source: Synergy Research Group for Q3 2022)

On the whole, we wouldn’t look further than the Big Three. They have data centers located all over the world, which translates into seamless access to content and services. In addition, they belong to esteemed global technological companies, so that you can be sure of the security of their solutions. In NeuroSYS, we partner with both Amazon and Microsoft; these are the clouds we recommend to our clients in most typical cases.

However, we’d like to highlight that some scenarios require a different approach to data storage, such as the one we’ve taken in our bacteria counting project.

Service availability
Having collected all the requirements, you may discover that there are particular services your system requires for proper functioning. They might not be available in every cloud, or their prices may vary dramatically. The services may be connected with integration tools, business applications, computing, databases, storage, security, and many more.

Team competencies
Typically, developers and DevOps specialize in particular enterprise clouds or just prefer one over another. For example, having a team that is experienced and certified in AWS might be a legitimate reason to choose a specific provider. This way, your team knows what to do straight away, which clearly saves you a lot of time and money.

Data centers vicinity
Although we mentioned earlier that the Big Three hold worldwide data centers, you can look for a provider with a data center closest to your end or target users. The edge locations’ proximity allows for smoother content synchronization, which translates into faster and more reliable services. You’ll find the (almost) exact points on the providers’ websites, such as Google Cloud Points of Presence.

Budget
The cost of enterprise cloud service varies depending on the provider and your individual needs. The final amount can greatly affect your project’s be or not to be. Keep in mind that when you choose the cloud provider for the first time, you can use various accelerating programs that enable using some of the services for free or almost free. AWS Migration Acceleration Program is one of the examples. Also, reserving a given capacity a few years in advance can reduce the price even down to 50%.

We always recommend contacting local support in the early going. They can help you gather requirements and collate their services with your context. Oftentimes it results in significant cost savings. We’re also there to help you; just book a one-hour consultation, it’s free of charge.

Legal/investor prerequisites
When you receive external funding, your investors can have the final say in many matters, such as choosing external providers. They might even want to avoid the Big Three because of personal anxieties. Also, when you develop and maintain, for example, governmental and administration systems, the applicable country law can forbid data to be stored outside of the country or the EU. In such a case, you need to find an enterprise cloud provider that operates locally.

How to migrate a system to the cloud step-by-step?

Cloud migration can be a large undertaking if you plan to move the whole company infrastructure. Taking into account the high complexity of such projects and many possible paths based on multiple decisions, we will show you the migration process of one system only.

Collect requirements and needs
First, you have to gather essential information, such as:
Choose the cloud in enterprise
Only by knowing the requirements can you start looking for a cloud provider. We explored the issue in the previous section, but let us highlight a few things.

First, it is a relatively simple task if you plan to move one system only. However, when you want to migrate all company systems, it can take months or even years. It is a landmark decision, and you can’t take it lightly. Here, you’ll have to carry out a thorough analysis of systems and company strategy. It doesn’t mean, though, that you can’t change your mind later on if your cloud of choice appears not to be serving the purpose. In such a case, backing off will be cheaper if it’s a representative cloud.

Second, we reckon that service availability plays the most prominent role in the decision making, i.e., the services offered by the enterprise cloud providers, that should perfectly fit your system. It’ll be much easier to migrate all the components in such a case.

Choose partners Now it’s time to decide with whom you’ll go through the process. Usually, companies don’t have enough internal resources, know-how, and experience to migrate systems swiftly on their own, keeping the risk minimal. Thus, it is vital to find a technological partner offering cloud computing consulting services, ideally specializing in the cloud you’ve already chosen.

Although we do migrations for our clients as a standard service, when our developers were busy working on commercial projects, we have also partnered with a third party to move our Nsflow platform to the cloud. You can read about the whole cloud migration process in our blog post.

Plan migration
You can schedule your cloud migration after you determine what, where, and with whom. Running the test version is an essential part of the process. It’s not a dry run, as you do it with all the real data to verify if everything is correct and works well. Then, an important step, you have to conduct security and stress tests. To learn more, read the article covering 7 test types to ensure software quality.
Migration
If the test migration was successful, here comes the real one. You move the system with all its components and data to the cloud and test it thoroughly once again.
Maintenance
Just because the system works well in the cloud and you’ve achieved your goals doesn’t mean you can rest on your laurels. Cloud computing for enterprise requires regular attendance, maintenance, and optimization. System updates are essential for your system to run smoothly, but also, when new cloud services come into life, as you can achieve significant savings by their introduction. Last but not least, only regularly maintained systems remain attack-proof.

Recently, thanks to enterprise cloud management for our globally operating client, with thousands of system users, we were able to save a substantial amount of money. We achieved that by cloud optimization and introducing new AWS services.

A wrap-up

We’ve shown you the way when it comes to migrating single systems to the cloud, and what to consider when choosing the enterprise cloud. However, if you plan to move the whole organization to the cloud, you’ll need way more time to prepare the process; thus, defining the steps can be challenging. So, do not hesitate to book a free consultation with one of our experts, and we’ll show you how to move to the cloud as smoothly as possible, and point out the business gains you can expect.

How long does it take to develop an application? How to make it faster?

Patryk Kozłowski — Fri, 03 Mar 2023 08:59:02 +0000

We were surprised to discover how many people ask how long it takes to develop an application. What is more, we have even stumbled across online calculators that promised to count the time span at least roughly.

We’re afraid we’ll have to disappoint you because there is no way to estimate it precisely, automatically, and at the same time in a split second. App development depends on many factors, such as business goals and a current market situation. Working on an application involves risk analysis and agile response to changing conditions. But worry not; we won’t leave you empty-handed. Instead of telling you how long it takes, we will show you how to do it faster.

How to develop an application faster?

Below, we throw some light on six time-saving methods we find particularly useful in app development. They apply to different areas, including the very product, how it’s made, and how the team works.

Six methods that will help you develop an application faster
Develop MVP first
You might have already heard this before, but building a minimum viable product first is a great way to kick off. Not only because it allows you to validate your idea but also because you can improve the working product early on. And “working” is the keyword here. We like to see an MVP as a kind of insurance, for one of its roles is to protect you against failure.

What can help you with building a minimum viable product, is defining and implementing the simplest process path, excluding the edge cases. Then, you can move on to the MoSCoW method, developing only the Must have features. We described this approach in detail in the article on effective planning in Scrum.

MoSCoW method in MVP building

Minimize customization
It might be a truism to say that custom software development takes a lot of time. Well, unfortunately, that’s the state of affairs. Thankfully, we are familiar with strategies on how to minimize programming time, such as:

Using ready-made components as often as possible, using custom development just to connect them
Utilizing reusable components, so you develop them once and put them into action repeatedly in your app
Ultimately, finding off-the-shelf software and applying slight modifications if possible.
Employing the above methods may involve changing the way we work to adjust to ready-made solutions. Still, sometimes it’s worth a try.

Minimize (inessential) communication

While open and efficient communication is essential for app development, it can also take the lion’s share of your people’s time. The following strategies will help you minimize time spent on communicating while maintaining productivity on a desired level:

Create a communication plan.

To avoid meetings that could have been emails, create a set of communication channels and guidelines, and designate people responsible for communication. Use real-time communication tools like Slack, Mattermost, or Google Chat whenever questions arise. Instead of sending your messages to all team members, target particular groups of stakeholders.
Be transparent.
Use project management tools that everyone can access, to keep track of the task progress and milestones, and avoid unnecessary questions. Keeping the team on the same page saves a lot of time.
Define tasks precisely.
While breaking the project into digestible bits, use a why-what-how approach. Make sure that everyone knows what to do and who’s responsible for task delivery to avoid misunderstandings. This way increases the probability you meet the estimated time for app creation.
Document your work.
Use software development documentation tools. This way, your developers, system administrators, testers, and other technical stakeholders will know where to find essential information on the app, not bothering others.
We’re aware it may seem tangled. But the good news is, we can help you structurize communication in web development projects. Have a look at our consultations, which are free of charge.

Work in sprints

Long-term goals are, by definition, hard to achieve, and the workload involved becomes challenging to estimate accurately. Such situations often lead to discrepancies between the initial calculations and final results. That’s why we recommend working in sprints, which produces predictable results and cyclical growth of functionalities. With the planned sprints list, the timeline for app development becomes more controllable and achievable.

Moreover, clear sprint goals affect productivity. Being aware of short deadlines, the team members are more energized to complete their tasks in time. Achieving smaller but frequent objectives is highly motivating as everyone likes to celebrate quick successes.

Outsource

Although it might be a favored option, having an in-house development team isn’t the fastest way to get things done. You need to recruit every single team member, introduce them to your processes, and make them understand each other to finally work as a well-oiled machine. Consider their holiday and sick leaves, which might hold off development even further.

Thus, outsourcing is a common way of kick-starting your project. You hire a whole dev team that is used to working as a team, and if someone is absent for a longer time, it is your contractor that is responsible for filling in.

If you already have an internal team that isn’t big, skilled, or effective enough, there’s a third way out: a mixed development team. It consists of both in-house and outsourced programmers who band together to achieve your business goals.

Track time

According to the Pareto principle, 80% of the app can be developed in 20% of the total estimated time. Also, we’ve heard on the grapevine ;), that some programmers like to polish details that are not that essential in terms of the app operation. Thus, it is crucial to focus on key endeavors, leaving out those that bring the lowest business value. Tracking time devoted to particular tasks will allow you to hunt down both groups of tasks and allocate your resources accordingly.

Moreover, keeping a tally of working hours will help avoid time-consuming micromanagement. After a few sprints with accurate time-tracking, you should finally know the answer to How many hours does it take to develop an app? quite precisely.

Wrap-up

Time is money; we have no doubts about that. So, instead of answering the tricky question: How long does it take to make an app? we choose to provide you with some handy tips on how to make it faster. We reckon they are altogether more valuable because you can apply them no matter what digital product you’re building.

At a close, we’re always here to give you a hand with development time estimates, so don’t think twice and contact us. Thanks for reading, and see you in the following write-up!

The most effective big data tools and techniques

Patryk Kozłowski — Tue, 28 Feb 2023 11:32:07 +0000

Data: the 21st century’s most valuable resource. But do we really know how to read and comprehend it? Or do we know the way to gather, process, and analyze mass data to make it useful? And useful is the keyword here.

This article is based on an internal project carried out by our team of data scientists, machine learning researchers, and R&D software engineers, who analyzed medical articles from various sources in real time. We don’t want to focus on the project itself, but we will delve into details of big data tools and technologies they used, such as machine learning libraries, monitoring tools, and data management software.

You will also learn more about Spark, MLib, Spark NLP, Kubernetes and its six distributions, Prometheus, Grafana, and Cassandra. There’s a lot to cover here, so without further ado, let’s get started!

What is big data?

First, it would be a good idea to go back to the roots, defining what big data is because we need to be able to differentiate it from the standard data we gather. A 5V model explains it in a straightforward way.

Big data can be characterized by the following:

huge volume: we gather hundreds of terabytes or petabytes
high velocity: data flow and analysis take place at great speed
data variety: we collect multimodal data (numerical, text, audio, and video), that can be structured and unstructured
veracity: data is accurate and can be validated
business value: there’s a practical purpose to collecting and analyzing given information
The first three are the must-have criteria; without meeting them, we can’t say data is big.

From data silos to data lakes

In business reality, big data is commonly used in the context of analytical pipelines that gather and examine data for the purpose of business intelligence.

Even within a single company, analytical systems are often created independently – in different branches, countries, and at different times. Thus, they frequently turn out to be incompatible and challenging or even impossible to integrate. This situation is called data silos.

In response to problems generated by the above mentioned phenomenon, a new, high-level approach has emerged: data lakes. According to this technique, we gather all data in one place, regardless if the records are structured or not. Furthermore, we assume that all pipelines will have independent ETLs (extract-transform-load), so they’ll be able to pick whatever they need from the data lake. Unlike data silos, the lakes allow for easy pipeline integration, support data mining, and prepare us for future data application that might be hard to predict at a given moment.

How to process big data?

To make use of big data, we need to collect it and make calculations with it. We perform both activities in a distributed manner.

For storage, we use distributed file systems like HDFS (Hadoop Distributed File System) and databases like Apache Cassandra that support scalability and integrity between the nodes.

We carry out distributed computing on computing clusters. To manage distributed calculations, we usually use algorithms like MapReduce and frameworks like Hadoop or, more often, Apache Spark.

Big data analytics tools: A summary

It wouldn’t be possible to go through all tools for big data analytics in one blog post. Thus, we will focus on the ones we used in the project that classified medical publications. Here they are:

Apache Spark: a multi-language big data analysis engine
Apache Kafka: a distributed event streaming platform and data pipeline source
MLlib, Spark NLP: libraries used for machine learning, both compatible with Spark
Apache Cassandra: a distributed database
Kubernetes: a big data software for orchestration and dockerization
Prometheus: monitoring and alerting toolkit for Kubernetes
Grafana: a visualization system that uses data from Prometheus
In the next part of the article, we explain how the above tools work.

The pipeline

Below, you can see the pipeline we’ve implemented:

First, the data simulators feed Kafka, the event streaming platform, and the static database (Mongo) that Spark reads from
In the Spark stream, there are a few stages of data processing; the first one being text and feature extraction
Then, depending on the version, we use one or more ML models, e.g., for named entity recognition, embeddings calculation, classification based on embeddings (MLlib trained in this particular case)
The results are sent to Cassandra and predictions are sent back to Kafka
Based on the above, we can update the results in Prometheus and the dashboard in Grafana

How to analyze big data: Apache Spark

Apache Spark is a widely-used engine for data analysis and processing and executing machine learning models on clusters. It goes well with programming languages like Python, Scala, SQL, Java, and R, and integrates with various frameworks, providing high system scalability. These include tools we commonly use in our projects, such as TensorFlow, PyTorch, mongoDB, Kubernetes, and Elasticsearch. What’s essential, Spark works on structured and unstructured data.

A driver program that executes parallel operations on a cluster is essential to Spark applications. A group of elements partitioned across the cluster is called a resilient distributed dataset (RDD); we will use the term often in the rest of the article.

Structured Streaming in Spark: How does it work?

RDDs are static, which means their content doesn’t change while we execute operations. Thus, we need a tool to work on continuous data streams. For this purpose, we use Structured Streaming, a Spark processing engine that updates the computation results as streaming data arrives. Together with Continuous Processing, end-to-end latencies can be as low as 1 ms.

The idea is to consider a data stream as a simple, unbounded table that changes with time. We can run the same operations on this table as we do on an RDD: aggregate and filter data, define functions to calculate column values, and join with RDDs. Stream-specific operations are also possible, such as windowed aggregations or joining streams.

Spark Machine Learning Library (MLlib) and Spark NLP

Spark allows us to view data streams and send them to the external data source in three ways:

Append: we send only the new rows; it’s a default mode
Complete: we send all table rows
Update: we send only the rows that have been updated
How does it work under the hood? First, we use the existing RDD machinery, so every stream is divided into small batches (static micro RDD tables). Then, Spark performs operations on them, and finally, they are converted back to one stream. A slight latency may occur in the process; however, a new experimental method also allows for proper real-time mode.

big data spark streaming big data
Data sources and sinks
When it comes to data sources, there aren’t many of them built in Spark. The existing ones include the following:

Kafka
Sockets
Files
However if you plan to use other data sources, you’ll find relevant Spark plugins for most of them.

Spark Machine Learning Library (MLlib) and Spark NLP
When discussing tools for big data analytics, we cannot fail to mention Spark MLlib. We use it, as the name suggests, for machine learning. It is a highly convenient tool for simpler models, like classification, regression, clustering, decision trees, and recommender systems.

However, the drawback is, we can’t use it with our own models, so we have to utilize an additional library like Spark NLP or Spark Deep Learning. For article classification, we use modern language models based on transformers like RoBERTa. On the other hand, Spark Deep Learning supports deep convolutional networks handy in, for example, Computer Vision.

What is Kubernetes?

Kubernetes is a big data software created by omnipresent Google and constantly developed. Kubernetes, also known as K8s, can automatically scale instances, verify whether the given containers are working, and, if necessary, destroy faulty ones and create new, healthy containers in their place. The process is called self-healing. In addition, this orchestrator allows for automatic zero downtime, new release deployment, and rollbacks to the previous working app versions.

K8s is widely used by large organizations. Though, it might be too time-consuming and problematic to set up for smaller companies. Hence, the latter most often use a ready-made version of Kubernetes installed in the cloud.

Although Kubernetes consists of many parts, we can distinguish two primary ones:

Control plane: ensures the app is always in the state that meets our intention; there can, or even should be, more than one control plane
Worker nodes: virtual or physical machines responsible for executing containers that, in turn, are placed into pods; there can be one (a recommended option) or more containers in one pod
Kubernetes distributions
We can distinguish various Kubernetes distributions – software packages with prebuilt K8s versions. We’ll explain the six core in short.

K8s
It’s a full version of Kubernetes. K8s allows for numerous modifications, and the sky’s the limit of what it can do. But the truth is, it’s pretty heavy and resource-consuming. We can’t imagine using it anywhere else than in production because of its long-drawn-out setup. Every component has to be picked and installed separately, which takes time. But, of course, the problem doesn’t apply to the cloud K8s version.

K3s
It’s a lighter version of Kubernetes with minimum hardware requirements. All components are bundled into a single binary, which makes it easy to install and configure. It allows you to create clusters that run in a virtual machine (VM) and supports helm chart management. You can use K3s in various environments, particularly in resource-constrained ones like IoT apps or edge devices.

K3d
It is a lightweight wrapper for K3s in Docker instead of VM. It creates multi-node K3s clusters that can be set up on a single machine.

microK8s
MicroK8s allows for a fast Kubernetes installation, with K8s services provided in a single package. It’s known for its low-maintenance and low- or even zero-ops infrastructure. Its lightweight distribution is often used in the cloud, for local builds, and IoT devices.

miniKube
MiniKube is the most popular kind of Kubernetes distribution used for local testing. We use it for local cluster development on a computer or CI rather than on the whole infrastructure.

Kind
Kind is a Kubernetes in Docker. We use the tool for development and testing. Kind works similarly to miniKube; the only difference is that it takes advantage of Docker as cluster nodes.

Monitoring tools: Prometheus and Grafana

Performance and task monitoring, as well as visualizing, is an integral part of working with high data volumes; for that purpose, we can use ready-made software for big data analytics. The main goal is to examine tasks performed within the pipeline, such as a live preview of the inferences from the ML models, prediction statistics, etc. Secondly, we analyze computer performance in real-time to find possible errors. This way, we can swiftly react with upgrades or extra resources.

We can recommend two tools that do the job and get on pretty well with each other:

Prometheus
Prometheus is a monitoring and alerting toolkit considered a standard for Kubernetes. It collects time-series data, stores it, builds queries, and generates alerts. Custom metrics allow for prediction statistics gathering.

Moreover, cAdvisor and nvidia-smi-exporter can be used for easy export of load metrics like RAM, GPU VRAM, and CPU utilization that can be directly pulled by Prometheus. System overload can be defined based on the aforementioned load metrics and monitored using alerts sent to services like pager duty, gmail, or slack channel. Although Prometheus offers built-in dashboarding, its use is limited. Thus, we prefer to use a more advanced tool, described below.

Grafana
Grafana is a visualization system that creates dashboards based on data collected by Prometheus. It presents information in the form of graphs, charts, heatmaps, tables, and many more (see all visualization options available). You can also share the dashboards with your colleagues and manage user access.

As you can see, the two solutions are essentially complementary. In the most straightforward words, Prometheus collects metrics, and Grafana transforms them into visual representations.

Data management: Apache Cassandra

We’re coming to an end with the subject of data governance. For this purpose, we’ve chosen Cassandra. It is a distributed, multi-node data management system that easily integrates with Spark and various programming languages.

Cassandra has been designed particularly for multiple data center deployments. In Cassandra, all nodes have the same rank; thus, we won’t find a single point of failure (SPOF), which works in favor of the system’s reliability.

Asynchronous replication across multiple data centers translates into low-latency operations, which means an ability to withstand data center outages irrespective of the type of storage you use: a public/private cloud or on-premise infrastructure. Cassandra has been tested on 1,000-node clusters to ensure its stability.

The system has introduced the Cassandra Query Language (CQL), but if you’re familiar with SQL we use to handle data, you shouldn’t worry about CQL use.

Last but not least, here’s the team that worked on our internal project and tested all the big data technologies and tools we explored for you (kudos for your work!):

Daniel Popek, Data Scientist
Paweł Mielniczuk, Data Scientist
Mikołaj Patalan, Machine Learning Researcher
Tomasz Cąkała, Machine Learning Researcher
Jakub Jaszczuk, R&D Software Engineer

Authors