DEV Community: Eugene R.

Aerospike Trino Connector - Chapter Two

Eugene R. — Mon, 06 Jun 2022 18:07:28 +0000

Photo by Danny Sleeuwenhoek on Unsplash

Speed up your SQL queries using Aerospike secondary indexes

With Aerospike Server 6.0, we have enhanced secondary index queries to allow for querying by partition, throttling, etc. The Trino connector now leverages this functionality to accelerate your secondary key queries at scale.

What is a secondary index?

A secondary index is a data structure used to quickly locate all the records in a namespace, or a set within it, based on a bin value in the record. When a value is updated in the indexed record, the secondary index automatically updates. You can retrieve records whose indexed value matches specified criteria using a secondary index query.

A query in Trino against a non-primary key bin that does not have a secondary index would invariably trigger a set (table) scan leading to an inefficient query. You can imagine the performance penalty that you would incur when dealing with a billion record table. Creating a secondary index (sindex) on a high-cardinality bin can help significantly speed up your Trino queries.

As shown in Figure 1, Aerospike secondary indexes are stored in DRAM for fast look-up, built on every partition in each cluster node, and are co-located with the primary index. Each secondary index entry contains only references to records local to the node.

Figure 1: Secondary index query

How do I get started?

Creating a secondary index on a bin in Aerospike is easy. You can use an Aerospike tool, such as asadm, or the API to dynamically create and remove indexes based on bins and data types you want to index. For an indexed bin, updating the record to include the bin updates the index. See how to create a secondary index for more information.

The Trino connector automatically analyzes the query and applies a secondary index to it, if one is available. If you know the cardinality of the secondary indexes, you can declare which sindex should be used for your query using the sindex_name session property. The __sindex table is created for each schema and provides details on available secondary indexes. You can change its name using the aerospike.index-table-name configuration property.

Here is an example of secondary indexes that we’ll see in action later:

We recommend that you provide the sindex to use in the query. The statement below shows an example of declaring a sindex to use:

set session aerospike.sindex_name=idx1;

If you do not declare a sindex for your query when multiple sindexes are available, the connector will pick the sindex with the highest lexical order from the list of available secondary indexes.

Will this really improve my Trino SQL query performance?

Prior to supporting secondary indexes, each query that was not a primary key search ended up with a full table scan on the Aerospike side. To illustrate my point, I will use a large (~20M records) data set to compare query response times with and without a secondary index.

A query for a particular pkup_datetime:

With a string secondary index:

Without a secondary index:

Another example of a range query on the id column:

Using a numeric secondary index:

Without a secondary index:

The response time is in seconds and we can see that the performance gain is significant. The primary index (PI) query was ~80 times slower than the secondary index (SI) query.

Summary

Aerospike secondary indexes can make a huge performance difference to your SQL queries. Make sure you read the official documentation regarding the compatibility and installation.

And finally, process your data faster than ever with the new Aerospike Connect for Presto!

Continuous Integration with Aerospike on GitHub

Eugene R. — Wed, 21 Apr 2021 09:54:36 +0000

Aerospike is a highly available and scalable NoSQL distributed database used in production to provide blazingly fast performance at a Petabyte scale.

As a software engineer developing with Aerospike, you probably needed integration tests, and you already have your own way to set up an Aerospike cluster for this.

In this short post I will show you how to add an Aerospike server to your development workflows using the GitHub Action in seconds.

As a GitHub user, you may already know what GitHub actions are. They were launched in late 2018 and set a new standard for CI/CD on GitHub.

Getting Started

The first thing you need is to have your repository hosted on GitHub. Better understand the GitHub Actions if you’re new to them.

My Aerospike GitHub action repository is here, and it is also published on the GitHub marketplace.

Now what’s left is to see some

Usage examples

You can set up either Aerospike Community Edition or Enterprise Edition. What makes the difference is the feature-key-file parameter. Specify it to have an Aerospike Server Enterprise Edition.

Aerospike Community Edition

To set up an Aerospike Community Edition server with the default configuration

- name: Set up Aerospike Database
  uses: reugn/github-action-aerospike@v1

Aerospike Enterprise Edition

To set up an Aerospike Enterprise Edition server by specifying the port, the server configuration and the feature key files

- name: Set up Aerospike Database
  uses: reugn/github-action-aerospike@v1
  with:
    port: 3300
    config-file: '.github/aerospike/aerospike.conf'
    feature-key-file: '.github/aerospike/features.conf'

Make sure to commit the configuration and the feature key files in your repository and provide the path, they must be in the same directory.

Now you have your Aerospike server running on Docker and ready for integration tests.

A complete list of supported inputs is available here.

That was fast as I promised. Both the read and the setup. Hope you found this useful, and don’t forget to subscribe to our newsletter to get updated with Aerospike weekly news.

Skyhook: a Redis-compatible interface to Aerospike Database

Eugene R. — Tue, 20 Apr 2021 16:21:39 +0000

Aerospike is a highly available and scalable NoSQL distributed database used in production to provide blazingly fast performance at Petabyte scale. Production deployments of Aerospike are almost always serving their data from NVMe drives, though it’s possible to run namespaces in memory.

Redis is a well loved key-value data store that is widely used by developers, which was designed to run single node (single-threaded originally) and in-memory.
This article might be interesting to developers who are using Redis and looking for more scalable and highly available alternatives, or maybe already have some applications running on Aerospike.

Migrating to a different technology is always a challenging process. You need to train engineers, rewrite the codebase, and set up a production cluster to take over. Using Skyhook, you can move applications to Aerospike first, then come back to rewrite them as Aerospike native, or if you’re satisfied with their performance, keep them as they are.
Or maybe you’re looking to expose your Aerospike data to external Redis-based applications? We would love to hear how you would use this project.

Skyhook to the rescue

We, as a company, observed this need and came up with the highly performant bridge service, which acts as a fully-fledged (with commands support limitation) Redis server, bridging the clients’ commands to Aerospike clusters.

Technical details

Skyhook is a standalone server application written in Kotlin, which projects Redis protocol commands to an Aerospike cluster using the Aerospike Java client under the hood. The server supports a single namespace and set configuration, where the incoming commands will be applied. This project uses Netty as a non-blocking I/O client-server framework.

Netty, is a highly performant JVM asynchronous event-driven network application framework. Its customizable thread model and native transport support forge an incredibly performant network communication layer.

What’s left is to translate Redis commands to Aerospike commands, and here come the async capabilities of the Aerospike Java Client.

Installation and configuration

You will need JDK8 and an Aerospike Server version >= 4.9 because Skyhook uses the new scan capabilities which are available starting from that version.

Clone aerospike/skyhook or grab a prebuilt executable JAR from the releases to get started.

To build the service from the sources:

./gradlew clean build

This command will build an executable JAR file.

Skyhook is configured using a configuration file and it’s in YAML format. You will need to specify it using the -f flag.

To run the server:

java -jar skyhook-[version]-all.jar -f config/server.yml

An example configuration file can be found in the config folder under the project repository.

Here are the current configuration properties available.

Property name	Description	Default value
hostList	The host list to seed the Aerospike cluster.	localhost:3000
namespase	The Aerospike namespace.	test
set	The Aerospike set name.	redis
bin	The Aerospike bin name to set values.	b
redisPort	The server port to bind to.	6379
workerThreads¹	The Netty worker group size.	number of available cores
bossThreads	The Netty acceptor group size.	2

¹ workerThreads property is used to configure the size of the Aerospike Java Client EventLoops as well.

workerThreads and bossThreads are the Netty thread pool sizes and can be easily fine-tuned for optimal performance.
After the server is up and running, any Redis client can connect to it as if it were a regular Redis server.

For the test purposes you can use redis-cli or even the nc (or netcat) utility:

echo "GET key1\r\n" | nc localhost 6379

Benchmarking

From the very beginning, the project was built with an attention to performance, and the results speak for themselves.

Running this Redis benchmark:

redis-benchmark -t set -r 100000 -n 1000000

We had the following results on local Skyhook with a single-node default configuration Aerospike cluster on Docker:

Summary:
throughput summary: 13453.88 requests per second
latency summary (msec):
avg min p50 p95 p99 max
3.627 0.864 3.263 5.583 7.463 1012.223

While those were the results for the Redis node running on Docker:

Summary:
throughput summary: 14967.15 requests per second
latency summary (msec):
avg min p50 p95 p99 max
3.275 0.608 2.935 5.087 6.679 1006.591

You can see that they are really close. More than that, we can multiply the number of the Skyhook nodes and significantly improve performance working with the same aerospike cluster. For instance, you can use the Round-robin DNS or any popular load balancer like HAProxy. This will not require any specific configuration since Skyhook is completely stateless.

Summary

The project is constantly evolving, having more and more commands being supported. You can find the coverage of the supported Redis commands in the repo readme.

It will be really great to hear from you about the experience of using it. Also, your contributions to the project are very welcome.

If you encounter a bug, please report it or open an issue stating the commands you would like to be supported.

Deploy Aerospike and Trino based analytics platform using Docker

Eugene R. — Mon, 19 Apr 2021 18:15:35 +0000

Aerospike Connect for Presto was released earlier this year to address the need to convert data stored in Aerospike into valuable insights quickly using the Trino (formerly known as PrestoSQL) distribution of Presto. Based on the feedback that we received from our users, we are pleased to announce that you can now deploy the Presto connector along with Trino quickly using Docker.

Here are the salient features of our Docker based solution:

Supports multiple deployment modes depending on your use case:
- Standalone - both coordinator and worker are launched within the same container, which when coupled with Aerospike launched within a separate container can help you with rapid prototyping. Consider trying it out with the Aerospike single-node Enterprise edition to test the waters.
- Distributed - coordinator and several workers are launched within separate containers across a Trino cluster to offer the scale needed for production workloads.
Provides the flexibility to your CI pipeline for building a single image that consists of the connector and the Trino version of your choice so that you can launch the combined image later using the simple docker run command.
Allows you to easily configure the connector container using either environment variables or using bind-mounts to mount your connector configuration files, schema JSON files along with the Trino configuration files.

Build Presto and Connector images

Aerospike does not provide pre-built Trino images, hence you need to build a Trino and the connector image by choosing the versions of your choice, so that you can run it using a docker command. The Trino version is available on the Trino release page and the connector versions can be found on the Connector release page. Here are the steps:

git clone https://github.com/aerospike/trino-aerospike.docker
Change directory to trino-aerospike.docker
docker build . -t trino-aerospike

The above command will build TRINO version 351 and the latest connector version. If you plan to use a different version of Trino and/or the connector, you can pass the TRINO_VERSION and the CONNECTOR_VERSION as build arguments with the docker build command. For example, use the below command to build TRINO version 353:

docker build --build-arg TRINO_VERSION=353 . -t trino-aerospike-353

Configure and run the image

There are a couple options available to deploy the connector by modifying the docker run command.

Option 1: Launch Trino along with the connector in standalone mode

Standalone mode i.e. Trino coordinator and worker run on a single node mode, is suitable for testing purposes. For example, if your Aerospike server is running on port 3000, you can use the following command to run the connector container on a Mac. See the configuration section for more information on how to configure the connector container.

docker run --rm -p 8080:8080 -e AS_HOSTLIST=docker.for.mac.host.internal:3000 --name trino-aerospike trino-aerospike-353

Based on our testing, you can launch a Trino cluster with the connector in the standalone mode with defaults in under 5 minutes.

Option 2: Launch along with the connector in distributed mode

Distributed mode i.e. a single coordinator and multiple workers, is usually preferred for production deployments. See the configuration section for more information on how to configure the connector container.

Start a Trino coordinator.

docker run --rm -p 8080:8080 -e TRINO_NODE_TYPE=coordinator -e AS_HOSTLIST=docker.for.mac.host.internal:3000 --name trino-aerospike-coordinator trino-aerospike-353

Start a Trino worker, specify the TRINO_DISCOVERY_URI to be the URI of the Trino coordinator. For example,

docker run --rm -e TRINO_NODE_TYPE=worker -e AS_HOSTLIST=docker.for.mac.host.internal:3000 -e TRINO_DISCOVERY_URI=http://example.net:8080 --name trino-aerospike-worker1 trino-aerospike-353

Replace example.net:8080 to match the host and port of the Trino coordinator. This URI must not end in a slash. Note that the above command will only run a single docker container instance of the worker. If you would like to spin up multiple worker instances, create a bash script and include the below command as many times as the number of workers desired, for automation. However, you would have to change the name of the worker instance e.g. trino-aerospike-worker-2, and so on.

docker run --rm -e TRINO_NODE_TYPE=worker -e AS_HOSTLIST=docker.for.mac.host.internal:3000 -e TRINO_DISCOVERY_URI=http://example.net:8080 --name trino-aerospike-worker2 trino-aerospike-353

You can configure the Trino connector using either environment variables or mounting configuration files to the container. For more information on how to configure the Trino connector and launch the Trino and connector containers, visit our documentation website.

Once you have successfully launched the Presto and the aerospike connector containers, you can generate insights from 100’s of TB of data stored in Aerospike using the Trino CLI, create insightful dashboards using Tableau, or manipulate data using Presto and Python libraries via the PyHive Python interface. If you would like to quickly try out our Docker based solution with a single node Aerospike Enterprise Edition, please follow the instructions here.

Aerospike as a Backend Storage for HashiCorp Vault

Eugene R. — Mon, 05 Apr 2021 17:48:50 +0000

Vault 1.7.0 is released, and it includes my contribution to support the Aerospike database as backend storage. See the release notes for more details.

From now on, Aerospike users can store their sensitive data using Vault almost seamlessly.

I will not talk about the benefits of using Vault and will jump into the installation and configuration details right away.

Setup

First thing you’ll need is to install Vault if you haven’t done this yet.
Make sure that the Vault binary is available on the PATH. See this page for instructions on setting the PATH on Linux and Mac. This page contains instructions for setting the PATH on Windows.

Verify the installation worked by opening a new terminal session and checking that the vault binary is available.

$ vault version
Vault v1.7.0 (4e222b85c40a810b74400ee3c54449479e32bb9f)

Configuration

Outside of development mode, Vault servers are configured using a file. The format of this file is HCL. Let’s configure our Aerospike cluster to be the Vault’s backend storage:

storage "aerospike" {
  hostname = "localhost"
  port = "3300"
  namespace = "test"
  set = "vault"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

You can find more information about the Aerospike backend configuration here.

To start the server:

vault server -config aerospike_backend.hcl

The Vault server is up and running on the default 8200 port.

Now open a new terminal window and go through the guide to initialize the Vault server.
It is a little bit cumbersome with all those unseals and login, but you’ll figure it out.

After the successful login, we need to enable a version 1 kv store:

vault secrets enable -version=1 kv

Usage

Now is the time to try things out.

$ vault kv put kv/my-secret my-value=s3cr3t
Success! Data written to: kv/my-secret

$ vault kv get kv/my-secret
====== Data ======
Key         Value
---         -----
my-value    s3cr3t

Your first secret was successfully stored and retrieved from Aerospike using Vault!

In this short introduction blog post, we covered the setup of Vault using Aerospike as a storage backend.

The Aerospike backend supports both CE and EE and doesn’t expose all the configuration properties available. We will work to include those in future releases.

I hope you are excited about this new Vault capability. Please let us know if you encounter any issues using it.

Introducing Aerospike JDBC Driver

Eugene R. — Mon, 15 Mar 2021 14:24:49 +0000

One of the most common ways to interact with databases on JVM is the JDBC API.

JDBC stands for Java Database Connectivity, which is a standard Java API for database-independent connectivity between the Java programming language and a wide range of databases.

Aerospike is a fast and durable No-SQL database. It has its own Java client, but this article will introduce you to a supplementary way of interacting with it using SQL.

Get yourself a hot cup of tea or coffee (for the true Java look and feel) and we will dive into the details of the Aerospike journey into the JDBC world.

Introduction

The Aerospike JDBC driver requires Java 8 and is compliant with JDBC 4.2. Also, Aerospike server 5.2+ is required, because the driver uses the new Filter expressions.

The first release of the JDBC driver supports the following SQL statements:

SELECT
INSERT
UPDATE
DELETE

You can also add WHERE conditions and LIMIT your queries. JOINS, ORDER BY and GROUP BY are not supported.
From the very beginning, the driver was designed to support operations that could be done using the regular Java client, without UDFs and other compute and memory hungry components. So, the original intention is to keep it small and easy to start, without workarounds to support features that aren’t native to the Aerospike Database.

The driver doesn’t support SQL functions as well as the Aerospike collection data types (CDTs).

Getting Started

Install the Aerospike JDBC driver and add the location of it to your classpath.
You can take the JAR file from the releases, add a Maven dependency, or build it from the sources.

The Aerospike JDBC driver is statically registered in the AerospikeDriver class. So the only thing required is to load this class.

Class.forName("com.aerospike.jdbc.AerospikeDriver").newInstance();

The next thing you’ll need to do is to specify the JDBC URL. The URL template is:

jdbc:aerospike:HOST[:PORT][/NAMESPACE][?PARAM1=VALUE1[&PARAM2=VALUE2]

For example the jdbc:aerospike:localhost URL will connect to the Aerospike database running on a local machine and listening on the default port (3000). The jdbc:aerospike:172.17.0.5:3300/test URL connects to the test namespace on the Aerospike database running on 172.17.0.5:3300.

After the initial setup let’s see a simple usage example of it:

try {
    String url = "jdbc:aerospike:localhost:3000/test";
    Connection connection = DriverManager.getConnection(url);

    String query = "select * from ns1 limit 10";
    ResultSet resultSet = connection.createStatement().executeQuery(query);
    while (resultSet.next()) {
        String bin1 = resultSet.getString("bin1");
        System.out.println(bin1);
    }
} catch (Exception e) {
    System.err.println(e.getMessage());
}

JDBC Client tools

You can browse and manipulate data in Aerospike with any of the available SQL client tools using the JDBC driver.

There are a number of multiplatform and free database tools available like DBeaver, SQuirreL, and others.

Here are the steps to configure the DBeaver SQL Browser with the Aerospike JDBC driver:

Database -> Driver Manager -> New Fill in settings:
- Driver Name: Aerospike
- Driver Type: Generic
- Class Name: com.aerospike.jdbc.AerospikeDriver
- URL Template: jdbc:aerospike:{host}[:{port}]/[{database}]
- Default Port: 3000
Click the Add File button and add the JDBC jar file.
Click the Find Class button.
Click OK.

Create a connection:

Database -> New Database Connection
Select Aerospike and click Next.
Fill in the connection settings
- Host and Port
- Database/Schema: the namespace you are connecting to
- Username and Password if you have security turned on in Aerospike Database Enterprise Edition
Click Finish.

Now you can open an SQL editor and query your Aerospike cluster:

Summary

The Aerospike JDBC driver is in its very early stages. It will be great if you could try it and give us some feedback. Any contributions to the project are very welcome.

Check out my previous Aerospike SQL series if you haven’t done this yet.

And don’t forget to subscribe to the Aerospike developer blog to get updated with our latest news.

Aerospike Data Browser

Eugene R. — Mon, 01 Feb 2021 16:19:32 +0000

Aerospike is an extremely fast and reliable data store that supports millions of transactions with incredibly low read and write latency. SQL is the lingua franca for data analysis. The key challenge here is that Aerospike is a NoSQL database, hence you cannot use standard SQL tools to analyze data stored in it. Aerospike offers AQL for lightweight administrative tasks for database maintenance. However, our developer community has been asking for a tool that would enable them to quickly browse or discover data stored in the Aerospike clusters visually along with the ability to run ANSI SQL queries. This has provided the impetus for our effort to create a data browser.

There are a few open-source data browsers such as Clairvoyance that can help you view data stored in Aerospike however, but none of them are feature-complete or actively maintained.

Enter Aerospike Data Browser

Presto is a highly parallel and distributed SQL query engine. We recently announced the GA of the Aerospike Connect for Presto, which can be used to run ANSI SQL queries to perform in-place, on-demand analytics on massive amounts of data in the Aerospike database via Presto.

So, it was a logical next step for us to leverage Presto and the Presto connector for our data browser design.

Aerospike Data Browser is basically a stack that consists of Quix, Presto, and the Aerospike Connector for Presto, and is dockerized. Figure 1 depicts an under the hood view. The Quix UI provides a DB Explorer and a SQL editor, in addition to a notebook manager for managing your notebooks. Presto exposes a JDBC interface to Quix and uses the Aerospike Connector to translate SQL queries into API calls to the DB. Building a stack with the aforementioned components for a desktop installation is not trivial by any means. Presto can scale to 100’s of nodes for a large scale deployment, but we wanted to limit the data browser to a single Presto instance that would run both the coordinator and worker in the developer's desktop environment. Our initial size of the Presto docker image was over 2GB, which was not acceptable. Hence, we stripped out all but the Aerospike connector from the plugin directory. Similarly, we had to downsize the Quix connector. Finally, we got the compressed docker image size under 1GB. We also made a design decision to default to schema inference so that a user that does not know the schema apriori is not left out.

Figure 1 - Under the hood view

Why should you care?

The data browser enables you to:

View namespace/set/schema of data stored in Aerospike, without the need to know the schema apriori, across multiple Aerospike clusters.
Run ANSI SQL queries for data exploration, including aggregate functions.
Visualize data.

It can be installed very easily using docker. See the Github repo for details on installation, configuration, and usage.

See it in action

The DB explorer component of the UI allows you to browse the database and list catalogs, namespaces, and sets that are available as shown in Figure 2. Note that a catalog corresponds to an Aerospike cluster name, schema to a namespace, and table to an Aerospike set name. See SQL syntax page for examples and more information.

Also, there is an option to save your queries to a notebook for future use.

Figure 2 - DB Explorer

Supported SQL statements:

SHOW CATALOGS
SHOW SCHEMAS
USE
SHOW TABLES
DESCRIBE
SELECT
INSERT INTO
EXPLAIN

Figure 3 depicts the result of a query that was written in the query editor built into the UI.



SELECT *
FROM aerospike.test.covid_stats
LIMIT 1000

Figure 3 - Query Editor

You can also visualize the data using the charting capabilities built into the UI as depicted in Figure 4.

Figure 4 - Visualizer

Finally, you can use aggregate functions as depicted in Figure 5.

Figure 5 - Using aggregate functions

What’s next?

If you find your data in the Aerospike database staring at you waiting to be quickly discovered, look no further. Use the Aerospike Data Browser to browse through the namespaces and sets to understand the schema of the data. Further, you can run supported ANSI SQL queries against it to generate quick insights. And finally, create some cool visualization. So, try it out by downloading the Aerospike Community edition if you don’t have an existing Aerospike deployment, and follow the instructions in the data browser repo.

Keep in mind that the data browser is not an analytics tool and should not be used for complex analysis of large datasets. You should consider using the Aerospike Connect for Presto for analytics use cases.

Spoiler alert! We are working on the Aerospike JDBC driver, which is lightweight, more aligned with the Aerospike server functionality, and easily droppable in any JDBC data browser application. Stay tuned!

Subscribe to the Aerospike Developer Blog for interesting blogs and other info.

Authentication and Authorization using Aerospike REST Client

Eugene R. — Thu, 30 Jul 2020 11:19:04 +0000

In this article we will explain and give some examples of the authentication and authorization flows with the Aerospike REST client.

If you’re new to the Aerospike REST client please read the introduction blog post by Robert Marks.

Starting from version 1.0.0, the Aerospike REST Client has support for a user and password configuration used to login to the Aerospike cluster. Database administrators need to create these credentials first with the required permissions.

It’s important to mention that the Access Control feature is available only in Aerospike Enterprise Edition.

This setup works perfectly, but what if we want to support multiple users to enable multi-tenancy?

In this case, customers might establish multiple Aerospike REST client clusters, authenticated with different credentials, and route the traffic accordingly. But this setup will require tedious installation operations and will tend to waste resources most of the time.

Starting withAerospike REST client v1.6.0, multi-user authentication is supported. So how does it work? As we already know, the REST client uses the Java client under the hood. Using the single authentication configuration, the REST client shares a singleton authenticated Java client for all requests.

The multi-user feature introduces an LRU based Java client cache. To start using it, customers will need to send Aerospike login credentials using Basic access authentication.

For example, using the default admin:admin credentials, send the Authorization: Basic YWRtaW46YWRtaW4= header with your requests to make authenticated queries.

The REST client will seek for a properly authenticated Java client in the LRU cache and if such does not exist, it will create a new one using the given username and password.
If the cache capacity is exceeded, the least recently used client will be dropped.

To use the new feature, set the following configuration variables:

aerospike.restclient.requireAuthentication - Set this boolean flag to true to require Basic Authentication on each request.
aerospike.restclient.pool.size - Represents the max size of the authenticated clients LRU cache (default value: 16). Please note that an oversized client cache will consume a lot of resources and affect performance.

Refer to the Installation and Configuration REST client page for installation, configuration and running manual.

All this is good but we should talk about different approaches to setting up a multi-user environment, right? Let’s move to the next one.

Kubernetes

To deploy the Aerospike REST client in Kubernetes please use the official Helm chart.

The installation and configuration details are available in the readme file, but still, here are the first steps we need to get started.

Add Aerospike repository

helm repo add aerospike https://aerospike.github.io/aerospike-kubernetes-enterprise

Install the chart

helm install rest-client aerospike/aerospike-rest-client --set config.hostname=<aerospike_hostname>

We can configure Ingress and route the traffic to different K8s deployments. Kubernetes will manage the resources for us. The multi-user header based authentication is available here as well though.

Using a proxy server authentication middleware

This approach changes the way we handle auth flows and shifts it to a third party software layer instead of using the Aerospike cluster Access Control (hey there Community Edition folks...)

To get things up and running we need the following:

A proxy server. It can be Traefik, Envoy, Nginx or any other proxy that we like with the auth middleware support.
An authentication and authorization software to handle the auth requests (like Auth0 or any other OAuth provider).
The Aerospike REST client itself.

The proxy will require authorization from the auth middleware and if the check passed will route the request to the REST client. See the following example implementing this architecture.

I’ll paste here the architecture diagram just in case.

Also, you may find the REST client gateway example using an API Gateway and Auth0 to be useful.

In this article we’ve listed a number of approaches on how to use the authentication and authorization with the Aerospike REST client and have seen some examples.

I hope it was interesting and beneficial at the same time. If you have any questions regarding the Aerospike REST client, don’t hesitate to open an issue under the GitHub repository or contact our team on the https://discuss.aerospike.com/.

Dealing with Predicate Expression Filters in Aerospike REST Client (Part 2)

Eugene R. — Sun, 26 Apr 2020 16:02:56 +0000

In the previous part we spoke about using simple Predicate Expression Filters with the Aerospike REST Client. In this post I want to dive deeper and show how special filter operators could come in handy.

LAST_UPDATE
Asserts the last update time before processing the transaction.

Parameters:

comparison operator
time in seconds since the epoch time (1970-01-01)

Example:
LAST_UPDATE(>=, 1577880000)

Java equivalent:

PredExp.recLastUpdate()
PredExp.integerValue(1577880000000000000) //in nano
PredExp.integerGreaterEq()

VOID_TIME
Asserts the record’s expiration time before processing the transaction.

Parameters:

comparison operator
time in seconds since the epoch time (1970-01-01)

Example:
VOID_TIME(>=, 1577880000)

Java equivalent:

PredExp.recVoidTime()
PredExp.integerValue(1577880000000000000) //in nano
PredExp.integerGreaterEq()

DIGEST_MODULO
Asserts a modulo against the record’s digest. The 160 bits (20 Bytes) digest field is generated by hashing the set name and the user key using the RipeMD-160 hashing function.

Parameters:

modulo divisor
comparison operator
expected remainder value

Example for digest(key) % 3 == 1:
DIGEST_MODULO(3, ==, 1)

Java equivalent:

PredExp.recDigestModulo(3)
PredExp.integerValue(1)
PredExp.integerEqual()

STRING_REGEX
Asserts that the regular expression matches the string valued bin.

Parameters:

string bin name
regular expression

Example:
STRING_REGEX(str, [0-9]*)

Java equivalent:

PredExp.stringBin("str")
PredExp.stringValue("[0-9]*")
PredExp.stringRegex(RegexFlag.NONE)

LIST_ITERATE_OR
Creates a predicate expression to match any item in a list value bin.

Parameters:

list bin name
comparison operator
value

Example:
Find records where any list item v = "hello" in list bin x.
LIST_ITERATE_OR(x, ==, hello)

Java equivalent:

PredExp.stringVar("v")
PredExp.stringValue("hello")
PredExp.stringEqual()
PredExp.listBin("x")
PredExp.listIterateOr("v")

LIST_ITERATE_AND
Creates a predicate expression to match all items in a list value bin.

Parameters:

list bin name
comparison operator
value

Example:
Find records where all list elements v != "world" in list bin x.
LIST_ITERATE_AND(x, !=, world)

Java equivalent:

PredExp.stringVar("v")
PredExp.stringValue("world")
PredExp.stringUnequal()
PredExp.listBin("x")
PredExp.listIterateAnd("v")

MAPKEY_ITERATE_OR
Creates a predicate expression to match any map key.

Parameters:

map bin name
comparison operator
value

Example:
Find records where any map key k = 7 in map bin m.
MAPKEY_ITERATE_OR(m, ==, 7)

Java equivalent:

PredExp.integerVar("k")
PredExp.integerValue(7)
PredExp.integerEqual()
PredExp.mapBin("m")
PredExp.mapKeyIterateOr("k")

MAPVAL_ITERATE_OR
Creates a predicate expression to match any map value.

Parameters:

map bin name
comparison operator
value

Example:
Find records where any map value v > 100 in map bin m.
MAPVAL_ITERATE_OR(m, >, 100)

Java equivalent:

PredExp.integerVar("v")
PredExp.integerValue(100)
PredExp.integerGreater()
PredExp.mapBin("m")
PredExp.mapValIterateOr("v")

MAPKEY_ITERATE_AND
Creates a predicate expression to match all map keys.

Parameters:

map bin name
comparison operator
value

Example:
Find records where all map keys k < 5 in map bin m.
MAPKEY_ITERATE_AND(m, <, 5)

Java equivalent:

PredExp.integerVar("k")
PredExp.integerValue(5)
PredExp.integerLess()
PredExp.mapBin("m")
PredExp.mapKeyIterateAnd("k")

MAPVAL_ITERATE_AND
Creates a predicate expression to match all map values.

Parameters:

map bin name
comparison operator
value

Example:
Find records where all map values v > 500 in map bin m.
MAPVAL_ITERATE_AND(m, >, 500)

Java equivalent:

PredExp.integerVar("v")
PredExp.integerValue(500)
PredExp.integerGreater()
PredExp.mapBin("m")
PredExp.mapValIterateAnd("v")

At the end you can glue those with logical operators (and | or) mixing with simple filters, Base64-encode the whole expression and send it as a GET parameter.

KVS_URL = 'http://localhost:8080/v1/kvs'

# In this example I put all metadata checks first, because Aerospike doesn't have an
# optimizer that knows how to shuffle predicates around more efficiently. This way, if the
# data is on an SSD, we can save on reading the record.
exp = b'(c >= 11 and not c < 20) or LAST_UPDATE(>=, 1577880000) or DIGEST_MODULO(3, ==, 1)'
encoded_exp = base64.b64encode(exp)

predexp_uri = '{base}/{ns}/{setname}/{userkey}'.format(
    base=KVS_URL, ns=namespace, setname=setname, userkey=userkey)

response = requests.get(predexp_uri, {"predexp": encoded_exp})
print(response.json())

To summarize, we’ve seen that Predicate Expression Filters are a very powerful tool and learned how to use them with the Aerospike REST Client.

If you still haven’t tried Aerospike REST Client and want to do so, you can find it here.

Dealing with Predicate Expression Filters in Aerospike REST Client (Part 1)

Eugene R. — Thu, 23 Apr 2020 17:36:16 +0000

If you're already an Aerospike user, you’ve likely developed with one of the client libraries available on GitHub, and might be familiar with Predicate Expression filters, which became available for all transactions starting from Aerospike Server v4.8.

I want to show how you would use PredExp filters with the REST client.

So what is the PredExp filter and why do we need it? Predicate expression filters are applied to scan, query and key-value results on the server. Meaning, if an expression evaluates to false, the transaction is ignored.

The REST client uses the Java client under the covers, where PredExp filters should be declared as an array in postfix notation. This form is hard to read and not intuitive. Having that in mind, how can we send such a construction using the REST API?

To solve this issue, I created a sort of DSL in infix notation, which uses well known logical operators and supports a special filters syntax.

A lot of theory till now, let's jump to some examples.

Here’s an example of a Java client PredExp filter:

PredExp.integerBin("bin2"),
PredExp.integerValue(126),
PredExp.integerGreater(),
PredExp.integerBin("bin2"),
PredExp.integerValue(140),
PredExp.integerLessEq(),
PredExp.and(2),
PredExp.integerBin("bin2"),
PredExp.integerValue(360),
PredExp.integerEqual(),
PredExp.or(2)

Which actually expresses the following expression:

(bin2 > 126 and bin2 <= 140) or bin2 == 360

Isn't the latter more concise and intuitive?

What is left, is to Base64 encode it, and send it as a GET parameter with the request.

Want the bin value to be a String? Just wrap it with double quotes like this. For non-numeric values we can omit the quotes:

(bin2 == "126" and bin2 != "140") or bin2 == ten

Before I wrap up, let's talk about some PredExp use cases.

Let's say you have a counter, but you only want to increment it if the value is under a certain number (for example 100).
Don't alter the record if its expiration time is above or under some particular value (here you'll need to write a special filter VOID_TIME which is not covered in this post).

This was a very simple example of how to use Predicate Expression Filters in the Aerospike REST Client. In my next post, I plan to write about special filter operators and their syntax.

If you still haven't tinkered with the Aerospike REST client, you can find it here:
https://github.com/aerospike/aerospike-client-rest

Take care, stay tuned and continue to Aerospike. Till next time.