DEV Community: Valentin Kulichenko

Ignite 3 Alpha: A Sneak Peek into the Future of Apache Ignite

Valentin Kulichenko — Tue, 26 Jan 2021 19:23:55 +0000

What Is Ignite 3?

Apache Ignite has existed for more than six years. During those years, Ignite evolved incredibly and, with thousands of deployments worldwide, became a top-5 project of the Apache Software Foundation. The SQL engine became more comprehensive, page-memory architecture and the persistence layer were introduced, and many features were added. These advancements make Ignite an extremely powerful tool that is suitable for a wide variety of use cases, from basic caching to complicated, multi-component data integration hubs that power services and applications that are used by millions of people every second of every day—and where a minute-long outage can cost millions of dollars.

With Ignite 3, we leap forward with a revamped codebase that supports a modern, modular architecture. The intuitive configurations, APIs, and Ignite behaviors provide a speedy, flexible developer experience so that even developers new to Ignite can quickly take advantage of the power of the distributed database.

Major Enhancements Planned for Ignite 3

Dynamic Configuration

Ignite 3 has a new configuration engine that provides a clear separation between static and dynamic parameters. The engine introduces an API that can update most Ignite configuration parameters in runtime, without requiring you to follow special parameter-update procedures or do cluster-node restarts.

For example, imagine that you create a table in Ignite with the WAL synchronization mode set to fsync (every update needs to be flushed to disk). Then, in production, you realize that, for the sake of performance, you want to relax the synchronization mode for the table and allow the operating system decide when to flush the changes from memory to disk. With Ignite 3, you to change that configuration on the fly—via simple parameter updates.

The API is also exposed via specialized REST endpoints so that it can integrate with external tools.

In addition, the legacy Spring XML format has been replaced with a much more compact and easy-to-use HOCON format, which is fully compatible with JSON and Java Properties formats. You can read about HOCON on GitHub.

For more technical details refer to the Ignite Enhancement Proposal #55.

Schema-Driven Architecture

Ignite 3 enforces a strong one-to-one match between tables and their schemas. The role of a schema is to unequivocally describe the data that is stored in its table. Therefore, any entry that is inserted into a table must be compliant with the table's schema. Also, any update that is made to the schema is reflected in the data.

Thus, schema-level consistency is maintained at all times, at both the object level and the SQL level. The SQL level is controlled by an application developer, and the object level is defined and updated automatically during the serialization of records. Nevertheless, the strong match enforcement results in clear and consistent behavior. In addition, although caches are essentially schema-less and can simultaneously store multiple data types, matching ensures consistency for data of the same type. This elegant, schema-driven architecture empowers developers to work easily with data across the breadth of an Ignite database.

To make the behavior even more predictable, a unified API for schema updates is being introduced. DDL statements and external tools function on top of the unified API, so users can always get intuitive results.

For technical details, refer to Ignite Enhancement Proposal 54.

Unified CLI Tool

Probably, in regard to developer experience, a unified CLI tool is the most important addition planned for Ignite 3. The tool is responsible for all operations related to the Ignite cluster and serves as a single entry point. Among many other capabilities, the tool enables you to do the following:

Easily install and upgrade Ignite
Manage Ignite optional modules and external dependencies without the need to move JAR files and folders manually
Start and stop nodes
Generate configuration stubs and Docker images for painless deployments on the cloud
Connect to clusters in order to update configurations and schemas, run SQL queries, and so on

The new CLI tool transforms the developer experience. It dramatically improves the usability of Ignite and makes Ignite much friendlier to newcomers.

For technical details, refer to Ignite Enhancement Proposal 52.

Get a Sneak Peek: Download Ignite 3 Alpha

Ignite 3 development began only three months ago, but some tangible results have already been produced. To give users a sneak peek into how Ignite will look in the future, the community decided to publish an alpha release. Now, anyone can download and play with the current alpha version of the CLI tool. To download, go to the Ignite website.

The current alpha build does not represent a functional product. It does not incorporate cluster discovery, data storage, compute capabilities, or any of the other core Ignite features that we are used to. However, it demonstrates the major mechanics of how you will interact with Ignite in the future.

Here are some highlights of the first Ignite 3 Alpha release:

Single-file download (Instead of downloading a huge ZIP file that has hundreds of files and a complicated structure, you get a single script, the CLI tool, which you can use for all further operations.)
Ability to install core Ignite artifacts and external dependencies via Maven
Ability to manage locally running nodes (start, stop, inspect)
Ability to connect to a cluster, acquire the current configuration, and update some of the dynamic parameters

The best way to get a good feel for the alpha build is to review the Getting Started Guide.

For a quick demonstration of the alpha build, watch the Apache Ignite 3.0 Alpha video series.

Your feedback is greatly appreciated! If you want to share your ideas, wishes, or requests or if you notice any issues, please do not hesitate to write a message to the Apache Ignite developers mailing list. This dedicated thread is the best place to share your thoughts.

Recommended Resources

Apache Ignite — More Than Just a Data Grid

Valentin Kulichenko — Mon, 27 Apr 2020 19:04:18 +0000

Many of us are familiar with data grids, which are known to be a great way to improve the performance of various applications. Oracle Coherence, Gigaspaces, Hazelcast, Apache Geode — these are all excellent examples.

The central concept behind almost any data grid is a distributed in-memory hash table. The “in-memory” part provides ultra-low latency for data access, while the “distributed” part offers virtually unlimited scalability, and therefore virtually unlimited throughput.

Typically, data grids also come with processing capabilities, as well as mechanisms to collocate computation with the data. As a result, you get a powerful combination of distributed data storage and distributed processing engine, which allows you to fully leverage the collective CPU and RAM power of multiple computers to accomplish your tasks.

Is Apache Ignite a Data Grid?

The short answer is: “YES!”
A bit longer answer is: “Yes, but it is also so much more.”

“Apache Ignite is a horizontally scalable, fault-tolerant distributed in-memory computing platform for building real-time applications that can process terabytes of data with in-memory speed.” — ignite.apache.org

Apache Ignite was initially created and developed as a data grid. Today, however, it incorporates some interesting and unique features that significantly widen the applicability of the technology. It is far beyond our common understanding of a data grid.

Here are some of the use cases that you can use Ignite for.

High-Performant Greenfield Applications

Data grids usually run on top of existing legacy data sources (e.g., relational databases), and serve as acceleration mechanisms for existing applications.

But in today’s age of digital transformation, there are tons of newly built applications that require scalability and performance from day one. Sure, you can go the “old way”, using one of the proven RDBMSs as the system of record with one of the data grids on top for better performance. This approach has multiple implications, though:

Such architecture is still limited. Mainly, data grids are efficient at read-intensive workloads, but whenever you need to write something, you have to go to the underlying database. Therefore, the database still bounds the scalability and performance of updates.
In-memory storage is volatile, which means that every time you restart your data grid cluster, you make the data unavailable. You then need to reload the data from the underlying database, which might take quite a bit of time, increasing possible downtimes up to unacceptable values.
This approach is generally overcomplicated and even a little “hacky”. If you have to deploy two independent data storages to serve a single purpose, then something is wrong. There must be a better way!

And there is a better way. Apache Ignite implements a concept that we call “Durable Memory”. It relies on page-memory architecture where any page can be stored not only in memory but on disk as well — you just need to enable the Native Persistence option. Ignite’s persistence is scalable, supports transactions, SQL, and all other APIs. Its tight integration with the in-memory layer makes data access entirely transparent for applications (i.e., an application can read or update the data at any point in time even if it’s not yet loaded into RAM).

The addition of the out-of-the-box persistence layer transforms Ignite into a memory-centric database. Essentially, this creates a durable system of record with scalability and performance characteristics of an in-memory data grid — a perfect tool for performance-sensitive greenfield applications.

HTAP and Real-Time Analytics

We are all used to a proposition that transactional processing (OLTP) and analytical processing (OLAP) are supposed to be done by separate systems. I’m sure you’ve seen a diagram similar to this:

The idea is to execute operations in an operational database and then execute an ETL process that propagates changes to an analytical database. You can use the latter for analytics, machine learning, deep learning, etc.

Many companies have used this architecture for a long time. Nowadays, many of those companies start questioning the approach and looking for alternatives. The issue is that ETL happens periodically — every several hours in the best case, but usually overnight, or even over a weekend. Such a process creates a time lag between a change in the operational database and a corresponding reaction of the analytical system. This lag is not compatible with the growing demand for real-time analytical processing.

Imagine that your credit card is used by someone else without your consent. Your bank has all those complicated fraud detection algorithms in place that can notify you that something is wrong. Real-time processing ensures that you get the notification within minutes or even seconds, rather than after several hours.

The only way to achieve such an immediate reaction is to unify the two databases into a single store, capable of running hybrid transactional/analytical processing (HTAP).

HTAP = OLTP + OLAP

There are a couple of things that make Apache Ignite suitable for HTAP.

Ignite follows “single storage, multiple workloads” approach, which means that you can store all your data once in a single cluster, and then work with this data via all available APIs, whether it’s key-value, transactions, compute, SQL or machine learning.
Ignite’s Native Persistence allows storing a superset of data on disk without affecting the availability of the data. Let’s say your total dataset is several petabytes, but you only have 100GB of data that is updated and accessed frequently. It’s not reasonable to put everything in memory, and with Ignite, you don’t have to. The majority of the data can be stored on disk only, and it will still be fully available to the applications when needed.

Demand for real-time analytics calls for a significant shift in how we architecture backend systems. There is still a lot to figure out, but Apache Ignite already provides an incredible combination of features to support HTAP workloads.

Digital Integration Hubs

One of the biggest challenges that IT architects currently face is that the data often spreads across multiple systems. Within a single company, you will easily find relational databases, NoSQL databases, mainframes, data lakes, SaaS solutions...

Different APIs, different data models, different everything.

More and more regularly, such companies feel a need for aggregation between multiple application’s data to get deeper insights into what is happening in the business. IT infrastructures are often not ready for this.

The solution is to create a Digital Integration Hub.

“Digital Integration Hub is an advanced application architecture that aggregates multiple back-end system of record data sources into a low-latency and scale-out, high-performance data store… The high-performance data store is synchronized with the back-end sources via some combination of event-based, request-based, and batch integration patterns.” — Gartner

Essentially, you put the data from multiple sources into a single scalable in-memory platform and provide a unified API to various applications that you have.

Apache Ignite appears to be an excellent foundation for such a solution. I’ve already mentioned the Native Persistence that allows Ignite to act as a system of record, as well as its ability to work with HTAP workloads — both become important when you build a data integration hub.

Besides, you need a good set of integration components with databases, streaming platforms, and SaaS products. Here are some of such integrations that Ignite provides out of the box:

The CacheStore interface for pluggable synchronization with relational, NoSQL databases, or big data stores like Hadoop.
Multiple streaming connectors.
Integration with Apache Spark.

Back in the day, distributed caches used to be the prominent way to improve performance. While new use cases and requirements kept coming up, many of those systems evolved into what we now refer to as “in-memory data grids”.

Similarly, in-memory computing platforms like Apache Ignite is the next stepping stone in the evolution of distributed systems. Data grids are still vital and are going to remain so for many years to come, but we need more comprehensive frameworks to deliver on the latest modern requirements.

The demand for real-time processing will only keep growing, and I’m excited to watch how this trend unfolds. I also welcome everyone to become a part of it by joining the Apache Ignite community: https://ignite.apache.org/community/contribute.html

Apache Ignite on Kubernetes: Things to Know About

Valentin Kulichenko — Fri, 17 Apr 2020 18:49:18 +0000

Kubernetes is becoming more and more popular these days. No wonder! It’s an impressive technology that helps a lot in deploying and orchestrating applications in various cloud environments, both private and public.

I work a lot with Apache Ignite and get questions about how to use it together with Kubernetes almost daily. Ignite is a distributed database, so utilizing Kubernetes to deploy it seems appealing. Besides, many companies are currently in the process of transitioning their whole infrastructures to Kubernetes, which sometimes includes multiple Ignite clusters, as well as dozens of Ignite-related applications.

I consider Apache Ignite to be very Kubernetes-friendly. First of all, it’s generally agnostic to where it runs. It is essentially a set of Java applications, so if you can run JVMs in your environment, then you will most likely run Ignite without any issues. Second of all, Ignite comes along with Docker images and detailed documentation on how to use them in Kubernetes: https://apacheignite.readme.io/docs/kubernetes-deployment

At the same time, there are several pitfalls specific to the Ignite+Kubernetes combination that I would like to share with you. Here are the questions that I will try to give answers for:

Cluster Discovery — How to achieve it in Kubernetes?
Stateless and Stateful Clusters — What is the difference, and how it affects the Kubernetes configuration?
Thick Clients vs. Thin Clients — Which ones should you use when running in Kubernetes?

Cluster Discovery

When you deploy Apache Ignite, you need to make sure that nodes connect to each other and form a cluster. This is done via the discovery protocol: https://apacheignite.readme.io/docs/tcpip-discovery

The most common way of configuring discovery for Ignite is to simply list the socket addresses of the servers where nodes are expected to run. Like this:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class=”org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
          <property name="addresses">
            <list>
              <value>10.0.0.1:47500..47509</value>
              <value>10.0.0.2:47500..47509</value>
              <value>10.0.0.3:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>

This configuration, however, does not work in Kubernetes. In such a dynamic environment, where pods can move between underlying servers, and where addresses can change during restarts, there is no way to provide static configuration.

In order to tackle this, Ignite provides the Kubernetes IP Finder, which utilizes a Kubernetes service API to lookup a list of currently running pods. With this feature, all you need is to provide the name of the service to use for discovery. All nodes that run in the same namespace and are pointed to the same service, will discover each other. The configuration becomes very straightforward:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class=”org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
          <property name="namespace" value="ignite"/>
          <property name=”serviceName" value="ignite-service"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

Stateless and Stateful Clusters

There are two different ways how Ignite can be used: as a caching layer on top of existing data sources (relational and NoSQL databases, Hadoop, etc.), or as an actual database. In the latter case, Ignite uses the Native Persistence storage, which is provided out of the box.

If Ignite acts as a cache, all data is persisted outside of it, therefore Ignite nodes are volatile. If a node restarts, it starts as a brand new one, without any data. There is no permanent state, in which case we say that the cluster is stateless.

On the other hand, if Ignite acts as a database, it is fully responsible for data consistency and durability. Every node stores a portion of the data both in memory and on disk; in case of a restart, you lose only in-memory data. On-disk data, however, must be preserved. There is a permanent state that has to be maintained, so the cluster is stateful.

In the Kubernetes world, this translates to different controllers that you should use for different types of clusters. Here is a simple rule:

For stateless clusters, you should use the Deployment controller.
For stateful clusters, you should use the StatefulSet controller.

Following this rule will help you to not overcomplicate the configuration for stateless clusters, and at the same time, make sure that stateful clusters behave as expected.

Thick Clients vs. Thin Clients

Ignite provides various client connectors that serve different purposes and are quite different in the way they are designed. You can read about them here: https://dev.to/vkulichenko/apache-ignite-client-connectors-variety-4lop. Specifics of the Kubernetes environment add additional considerations to the topic.

Kubernetes does not expose individual pods directly to the outside world. Instead, there is usually a load balancer that is responsible for accepting all external requests and redirecting them to the pods. Both Ignite server nodes and client applications can run on either side of the load balancer, which affects the choice between thick and thin clients.

Typically, you will see one of these three scenarios.

Scenario 1: Everything in Kubernetes

By “everything”, I mean both the Ignite cluster and the application talking to this cluster. They run in the same namespace of the same Kubernetes environment.

In this case, there are no limitations whatsoever — you are free to use any type of client connector. Unless there is a specific reason to do otherwise, I would recommend sticking with thick clients here, as it’s the most efficient and robust option.

Scenario 2: Cluster Outside of Kubernetes

This scenario is very typical for companies that are in progress of transitioning existing clusters and applications from more traditional bare metal deployment to Kubernetes deployment.

Such companies usually prefer to transition applications first, as they are much more dynamic — updated frequently by a large number of individual developers. Proper orchestration is much more critical for the applications than for Ignite clusters, which can run for months or even years without disruption, and are usually maintained by a small group of people.

Here we have a significant limitation: thick clients are not compatible with this scenario. The reason is that a server node can try creating a TCP connection with a particular thick client. Such an attempt will most likely fail due to the load balancer in front of the Kubernetes cluster.

If you find yourself in this situation, you currently have two options:

Use thin clients or JDBC/ODBC drivers instead of thick clients.
Move the Ignite cluster into the Kubernetes environment (effectively transitioning to scenario #1, which doesn’t have these limitations).

Good news though: Apache Ignite community recognizes the impact of this limitation and is currently working on improvements that would allow using thick clients in this kind of deployment. Stay tuned!

Scenario 3: Applications Outside of Kubernetes

This scenario is the opposite of the previous one — Ignite cluster runs within Kubernetes, but applications are outside of it.

Although I’ve seen it a couple of times, this scenario is scarce compared to the other two. If this is your case, I would strongly recommend you to consider moving the applications to Kubernetes, since you already use it.

From Ignite perspective, this type of deployment allows only for thin clients and JDBC/ODBC drivers. You can’t use thick clients in this scenario.

What I’ve described above are the main complications that you might stumble upon when working with Apache Ignite in Kubernetes environments.

As a next step, I would suggest taking a look at this GitHub repo where I uploaded full configuration files that can be used to deploy a stateful Ignite cluster in Amazon EKS: https://github.com/vkulichenko/ignite-eks-config

And of course, always feel free to refer to the Ignite documentation for more details: https://apacheignite.readme.io/docs/kubernetes-deployment

Apache Ignite: Client Connectors Variety

Valentin Kulichenko — Fri, 10 Apr 2020 18:01:46 +0000

If you have worked with Apache Ignite, you have probably noticed that there are quite a few different client connectors that you can use. Client node, thick client, thin client, JDBC driver, ODBC driver, REST API, … All this might get a little confusing.

“Which type of client do I need to use in MY application?” — you would likely ask yourself.

So let me break this down. Below are all the client connectors available for Apache Ignite out of the box, as well as some best practices around when you should (or should not) use them.

Thick Client (a.k.a. Client Node)

Thick client is basically a regular Ignite node, which runs in client mode.

The difference between client nodes and server nodes is logical rather than physical. Clients do not participate in caching, compute, service deployment and other similar activities, but under the hood, they work in (almost) exact same way as servers. Mainly, this means that a client node utilizes the same discovery and communication mechanisms, which brings both advantages and disadvantages.

Pros

A thick client is a part of topology, so it’s aware of partitioning configuration as well as data collocation techniques you applied to your caches. Whenever a thick client sends a request to read or update a cache entry, it goes directly to the node where this entry is stored — this is very efficient from scalability and performance standpoint.
Thick clients provide a full set of APIs available in Apache Ignite. There are certain features (e.g., near caches or continuous queries) that are available ONLY in thick clients.

Cons

As always, high efficiency and rich functionality come with a price.

First, a thick client is pretty heavy (hence the name) — in some cases it might consume quite a bit of CPU and RAM. For example, if you execute a SQL query, the client is actually used as a reducer that accumulates sub results from different nodes and makes the final calculations. Again — this is good for performance, but this also requires more CPU/RAM power. If your application is running on a device with limited resources, you would most likely want to use other options.
Second, it is required that every thick client has full connectivity with every single server node. Moreover, since a thick client is just a regular node, it is possible that a server node will initiate a TCP connection with a client node. If there is a firewall, NAT or a load balancer between the cluster and the application, it will be really hard or even impossible to use thick clients. Basically, this means that thick clients are better used in the same physical environment in which server nodes run.
Finally, keep in mind that think clients are available only for JVM languages, .NET languages, and C++. For other platforms, you will have to use other options.

Thin Client

Thin clients are actually much closer to what we usually think about when talking about clients. A thin client is very lightweight, initiates a single TCP socket connection with one of the nodes in the cluster, and communicates via a simple binary protocol. It also follows a very straightforward request-response pattern.

Pros

Thin clients consume minimum resources and are very undemanding from the network communication standpoint. This makes them ideal for remote applications, running outside of the cluster’s data center behind NAT or a firewall. They also work perfectly fine on devices with limited resources. IoT is a good example of a use case for thin clients.
The binary protocol used by thin clients is language and platform agnostic, which means that you can run a thin client virtually anywhere. Currently, Ignite provides implementations written in Java, .NET, C++, Python, NodeJS and PHP, which already exceeds what we have for thick clients. And it’s fairly easy to add new languages to this list.

Cons

A thin client is not collocation-aware. It’s typically connected to only one node in the cluster, which is used as a gateway — all requests from the client go through this server node. This creates an obvious bottleneck. Some implementations support load balancing between two or more nodes, but even in this case scalability is more limited than with thick clients.
Thin clients provide limited API. Currently, you can only create/destroy caches, execute key-value operations and SQL queries. Compute Grid and Service Grid APIs are likely to be added in the future (contributions are welcome!). However, advanced features that require push notifications from servers to clients (near caches, continuous queries) or sophisticated threading models (data streaming) will most likely never be implemented for thin clients.

JDBC and ODBC Drivers

JDBC and ODBC are standard SQL APIs for JVM and C++ compatible languages respectively. Ignite provides implementations for these APIs out of the box:

Architecturally, they work exactly like thin clients — connecting to one of the nodes and using that node as a gateway. Because of that, SQL drivers share advantages and disadvantages with thin clients. There are a couple more key differences I would like to mention though.

Pros

JDBC and ODBC are industry standards, which means that the provided drivers enable easier integration with existing SQL-oriented tools. As an example, you can use them to transparently connect BI tools like Tableau or Pentaho to Apache Ignite.

Cons

The limitation here is obvious — you get only SQL API from this type of client connector. In my experience, the vast majority of use cases require more features if you want to get the most from Apache Ignite in terms of scalability and performance. Unless you don’t have a choice other than to use standard JDBC/ODBC, I would absolutely recommend looking at other options.

REST API

Apache Ignite also comes with the REST API, which allows performing basic operations like reading or updating cache entries, executing SQL queries, looking at some of the metrics, etc.

In its current implementation, this API is not really suitable for any performance-sensitive purposes, so I usually avoid using it in production for applications of any type.

However, the REST API might sometimes be useful during development or testing — to quickly check a particular entry value or a SQL query result, for example.

Choosing The Right Option

Okay, there are several options for client connectors, and each of them has its own pros and cons. So, which one should you go with for a particular application? Here is a simple algorithm that will help you to make the choice.

If the application is deployed in the same environment where server nodes run, and as long as there is full network connectivity between the application and every server node (i.e. no firewall or NAT), use the thick client. Generally speaking, thick clients should be your first choice as they are the most efficient and provide the most functionality.
If the application is remote and/or runs on a device with limited resources, use the thin client. This is your second choice.
Finally, if the application must use standard JDBC or ODBC API (e.g. it’s a BI tool like Tableau), use the corresponding SQL driver.

That’s it! Not that confusing anymore, right? 🙂