DEV Community: Pieter Humphrey

Blasting Off into Stargate using HTTPie

Pieter Humphrey — Tue, 01 Nov 2022 21:57:41 +0000

Author: Kirsten Hunter

As a DataStax Developer Advocate, my job is to help our amazing teams provide you with the best possible experience with Cassandra and our products.

Datastax Astra is built on Apache Cassandra. In addition to great documentation, Astra offers a robust free tier that can run small production workloads, pet projects, or just let you play—all for free, no credit card required. Cassandra can be tricky for hardcore SQL developers, because it uses a different slightly different query language (CQL), but when you get Astra, Stargate is there to let you interact with your data through APIs. Our open source Stargate product provides REST, GraphQL, and schemaless document APIs in addition to native language drivers. If you like them but don’t want to use our products, that’s fine! It’s completely open source and you can implement it on your own system.

One of the things I noticed when I arrived at DataStax is that our tutorials rely heavily on curl for performing commands against our APIs. I prefer HTTPie, a similar tool designed for REST API interaction.So I created an authentication plugin for the HTTPie tool that stores your variables and lets you make requests like:

http:/v2/schemas/keyspaces

I’ve put together a Katacoda scenario for you to see how everything works. If you want to implement it locally, here are the instructions:

The first secret to this is an rc file (~/.astrarc), which keeps track of your DB, region, username, and password, and auto-refreshes your token. You can have as many sections in this file as works for you—it’s just an INI-style configuration file.

The second secret is the HTTPie configuration file, in ~/.config/httpie/config.json

My configuration is what you see here; I have the “fruity” color scheme, my default auth-type is astra, and the section of the .astrarc file is “default”.

{
"default_options": [
  "--style=fruity",
  "--auth-type=astra",
  "--auth=default:"
]
}

My goal here is to make it so you can bounce around between REST and GraphQL queries, get nicely formatted JSON results, and perhaps use jq to pare them down—but mostly I want for the tool to get out of your way.

Interested? At this point httpie-astra requires python 3.5, but if you want me to make it support 2.7, please just let me know in the Datastax Community or on Discord.

You can get it through github: git clone https://synedra-datastax/httpie-astra

Or with pip: pip3 install httpie-astra

Once you’ve got it installed, get your environment set up by making a simple call like:

http –auth-type astra -a default: :/v2/schemas/keyspaces

This will give you instructions to get all your variables set up.

Let’s take a look at an example from our Astra REST documentation.

The curl command looks like this:

curl --request GET \
--url https://${ASTRA_CLUSTER_ID}-${ASTRA_CLUSTER_REGION}.apps.astra.datastax.com/api/restv2/schemas/keyspaces/${ASTRA_DB_KEYSPACE}/tables/products
\
--header 'accept: application/json' \
--header 'x-cassandra-request-id: {unique-UUID}' \
--header "x-cassandra-token: ${ASTRA_AUTHORIZATION_TOKEN}"

Then it gives you back a single line JSON doc.


{"data":{"name":"products","keyspace":"tutorial","columnDefinitions":[{"name":"id","typeDefinition":"uuid","static":false},{"name":"created","typeDefinition":"timestamp","static":false},{"name":"description","typeDefinition":"varchar","static":false},{"name":"name","typeDefinition":"varchar","static":false},{"name":"price","typeDefinition":"decimal","static":false}],"primaryKey":{"partitionKey":["id"],"clusteringKey":[]},"tableOptions":{"defaultTimeToLive":0,"clusteringExpression":[]}}}

The httpie command already knows a lot of this information, so the call is much simpler. I’m using the config.json I described above to set my auth-type and config section. I’ll show this output as a screen shot because it’s really easy to understand:

We’re on the job, getting the Stargate examples and Astra API examples to include HTTPie tabs, but in the meantime if you’re having fun with this, please let me know in the community or discord, and let’s make this rock for you!

Serverless Storage for Your Node.js Functions with Astra DB

Pieter Humphrey — Tue, 27 Sep 2022 16:03:18 +0000

Did you ever wish you had persistent storage for your serverless functions? Storage that was as easy as an idiomatic API call in your favorite language? What if you could even handle JSON data with no upfront schema definition? Functions as a service (FaaS) are excellent containers for business logic. With functions, you can:

Run backend code without managing any infrastructure whatsoever.
Run short-lived code that starts FaaST, runs and shuts down when complete or unused.
Run your code in a specific framework or library of your choice.
Trigger a function based on events that are defined by the FaaS provider.

That’s great, but what happens to the application state? Yes, you could run an in-process cache, session store, or use the modest filesystem allocated to the function. But these workarounds will be as short-lived as the function, so it’s not what most people consider persistent storage.

Using APIs and or Drivers, you can get data out from your functions and into a database, but not many databases employ data API gateways or offer easy ways to surface a fluent data access layer in APIs like REST, GraphQL or gRPC. Perhaps you have JSON data and you simply want a document-style NoSQL option that skips defining the schema upfront – just JSON and go.

So let’s assume getting to your database is easy with APIs, drivers, and schemaless JSON for a moment. Why then, would a serverless, autoscaling database be the preferred choice for persistent storage for your serverless, autoscaling functions? First off, it’s important to understand that serverless and FaaS are not quite the same thing. The key thing that makes FaaS and Serverless DBaaS so great together is autoscaling.

If you've invested in DBaaS and FaaS, you're probably not interested in managing infrastructure. Running serverless functions that have the potential to autoscale up rapidly is dangerous if connected directly to a back-end database that cannot auto-scale.

Scaling up the application dynamically could put an unpredictable and increasing load on a data service or database that doesn’t use a similar (auto) scaling mechanism. The industry spent years trying to solve this problem in the application server era with connection pools and database connection conservation techniques. Most of those techniques are antithetical to FaaST-startup-and-terminate style serverless functions because there’s no connection to preserve!

Pairing an autoscaling application tier to a data tier that doesn’t autoscale drags down a system that could have otherwise been fully automated. Manually running a terraform script, or worse, waiting for an operations ticket to be created and serviced for scaling the database instance up or down would kill 50% of the automation value between application (function) and database. No Bueno.

So how can we wire together an autoscaling DBaaS like DataStax Astra DB, with serverless functions without having to write a ton of REST services to expose the database functions you need? Let us show you how!

In this livestream playback, Alex Ellis and Stefano Lottini show you how to pair FaaS with a serverless, autoscaling DBaaS for end-to-end automated scaling.

To get the most out of the event replay, be sure to check out the OpenFaaS blog first!

Learn More:

Follow the DataStax Tech Blog for more developer stories. Check out our YouTube channel for tutorials and here for DataStax Developers on Twitter for the latest news about our developer community.

Baeldung Series Part 2: Build a Dashboard With Cassandra, Astra and CQL – Mapping Event Data

Pieter Humphrey — Tue, 13 Sep 2022 17:44:29 +0000

1. Introduction

In our previous article, we looked at augmenting our dashboard to store and display individual events from the Avengers using DataStax Astra, a serverless DBaaS powered by Apache Cassandra using Stargate to offer additional APIs for working with it.

In this article, we will be making use of the exact same data in a different way. We are going to allow the user to select which of the Avengers to display, the time period of interest, and then display these events on an interactive map. Unlike in the previous article, this will allow the user to see the data interacting with each other in both geography and time.

In order to follow along with this article, it is assumed that you have already read the first and second articles in this series and that you have a working knowledge of Java 16, Spring, and at least an understanding of what Cassandra can offer for data storage and access. It may also be easier to have the code from GitHub open alongside the article to follow along.

2. Service Setup

We will be retrieving the data using the CQL API, using queries in the Cassandra Query Language. This requires some additional setup for us to be able to talk to the server.

2.1. Download Secure Connect Bundle.

In order to connect to the Cassandra database hosted by DataStax Astra via CQL, we need to download the “Secure Connect Bundle”. This is a zip file containing SSL certificates and connection details for this exact database, allowing the connection to be made securely.

This is available from the Astra dashboard, found under the “Connect” tab for our exact database, and then the “Java” option under “Connect using a driver”:

For pragmatic reasons, we're going to put this file into src/main/resources so that we can access it from the classpath. In a normal deployment situation, you would need to be able to provide different files to connect to different databases – for example, to have different databases for development and production environments.

2.2. Creating Client Credentials

We also need to have some client credentials in order to connect to our database. Unlike the APIs that we've used in previous articles, which use an access token, the CQL API requires a “username” and “password”. These are actually a Client ID and Client Secret that we generate from the “Manage Tokens” section under “Organizations”:

Once this is done, we need to add the generated Client ID and Client Secret to our application.properties:

ASTRA_DB_CLIENT_ID=clientIdHere
ASTRA_DB_CLIENT_SECRET=clientSecretHere

2.3. Google Maps API Key

In order to render our map, we are going to use Google Maps. This will then need a Google API key to be able to use this API.

After signing up for a Google account, we need to visit the Google Cloud Platform Dashboard. Here we can create a new project:

We then need to enable the Google Maps JavaScript API for this project. Search for this and enable this:

Finally, we need an API key to be able to use this. For this, we need to navigate to the “Credentials” pane on the sidebar, click on “Create Credentials” at the top and select API Key:

We now need to add this key to our application.properties file:

GOOGLE_CLIENT_ID=someRandomClientId

3. Building the Client Layer Using Astra and CQL

In order to communicate with the database via CQL, we need to write our client layer. This will be a class called CqlClient that wraps the DataStax CQL APIs, abstracting away the connection details:

@Repository
public class CqlClient {
  @Value("${ASTRA_DB_CLIENT_ID}")
  private String clientId;

  @Value("${ASTRA_DB_CLIENT_SECRET}")
  private String clientSecret;

  public List<Row> query(String cql, Object... binds) {
    try (CqlSession session = connect()) {
      var statement = session.prepare(cql);
      var bound = statement.bind(binds);
      var rs = session.execute(bound);

      return rs.all();
    }
  }

  private CqlSession connect() {
    return CqlSession.builder()
      .withCloudSecureConnectBundle(CqlClient.class.getResourceAsStream("/secure-connect-baeldung-avengers.zip"))
      .withAuthCredentials(clientId, clientSecret)
      .build();
  }
}

This gives us a single public method that will connect to the database and execute an arbitrary CQL query, allowing for some bind values to be provided to it.

Connecting to the database makes use of our Secure Connect Bundle and client credentials that we generated earlier. The Secure Connect Bundle needs to have been placed in src/main/resources/secure-connect-baeldung-avengers.zip, and the client ID and secret need to have been put into application.properties with the appropriate property names.

Note that this implementation loads every row from the query into memory and returns them as a single list before finishing. This is only for the purposes of this article but is not as efficient as it otherwise could be. We could, for example, fetch and process each row individually as they are returned or even go as far as to wrap the entire query in a java.util.streams.Stream to be processed.

4. Fetching the Required Data

Once we have our client to be able to interact with the CQL API, we need our service layer to actually fetch the data we are going to display.

Firstly, we need a Java Record to represent each row we are fetching from the database:

public record Location(String avenger, 
  Instant timestamp, 
  BigDecimal latitude, 
  BigDecimal longitude, 
  BigDecimal status) {}

And then we need our service layer to retrieve the data:

@Service
public class MapService {
  @Autowired
  private CqlClient cqlClient;

  // To be implemented.
}

Into this, we're going to write our functions to actually query the database – using the CqlClient that we've just written – and return the appropriate details.

4.1. Generate a List of Avengers

Our first function is to get a list of all the Avengers that we are able to display the details of:

public List<String> listAvengers() {
  var rows = cqlClient.query("select distinct avenger from avengers.events");

  return rows.stream()
    .map(row -> row.getString("avenger"))
    .sorted()
    .collect(Collectors.toList());
}

This just gets the list of distinct values in the avenger column from our events table. Because this is our partition key, it is incredibly efficient. CQL will only allow us to order the results when we have a filter on the partition key so we are instead doing the sorting in Java code. This is fine though because we know that we have a small number of rows being returned so the sorting will not be expensive.

4.2. Generate Location Details

Our other function is to get a list of all the location details that we wish to display on the map. This takes a list of avengers, and a start and end time and returns all of the events for them grouped as appropriate:

public Map<String, List<Location>> getPaths(List<String> avengers, Instant start, Instant end) {
  var rows = cqlClient.query("select avenger, timestamp, latitude, longitude, status from avengers.events where avenger in ? and timestamp >= ? and timestamp <= ?", 
    avengers, start, end);

  var result = rows.stream()
    .map(row -> new Location(
      row.getString("avenger"), 
      row.getInstant("timestamp"), 
      row.getBigDecimal("latitude"), 
      row.getBigDecimal("longitude"),
      row.getBigDecimal("status")))
    .collect(Collectors.groupingBy(Location::avenger));

  for (var locations : result.values()) {
    Collections.sort(locations, Comparator.comparing(Location::timestamp));
  }

  return result;
}

The CQL binds automatically expand out the IN clause to handle multiple avengers correctly, and the fact that we are filtering by the partition and clustering key again makes this efficient to execute. We then parse these into our Location object, group them together by the avenger field and ensure that each grouping is sorted by the timestamp.

5. Displaying the Map

Now that we have the ability to fetch our data, we need to actually let the user see it. This will first involve writing our controller for getting the data:

5.1. Map Controller

@Controller
public class MapController {
  @Autowired
  private MapService mapService;

  @Value("${GOOGLE_CLIENT_ID}")
  private String googleClientId;

  @ModelAttribute("googleClientId")
  String getGoogleClientId() {
    return googleClientId;
  }

  @GetMapping("/map")
  public ModelAndView showMap(@RequestParam(name = "avenger", required = false) List<String> avenger,
  @RequestParam(required = false) String start, @RequestParam(required = false) String end) throws Exception {
    var result = new ModelAndView("map");
    result.addObject("inputStart", start);
    result.addObject("inputEnd", end);
    result.addObject("inputAvengers", avenger);


    result.addObject("avengers", mapService.listAvengers());

    if (avenger != null && !avenger.isEmpty() && start != null && end != null) {
      var paths = mapService.getPaths(avenger, 
        LocalDateTime.parse(start).toInstant(ZoneOffset.UTC), 
        LocalDateTime.parse(end).toInstant(ZoneOffset.UTC));

      result.addObject("paths", paths);
    }

    return result;
  }
}

This uses our service layer to get the list of avengers, and if we have inputs provided then it also gets the list of locations for those inputs. We also have a ModelAttribute that will provide the Google Client ID to the view for it to use.

5.1. Map Template

Once we've written our controller, we need a template to actually render the HTML. This will be written using Thymeleaf as in the previous articles:

<!doctype html>
<html lang="en">

<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />

  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css" rel="stylesheet"
    integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6" crossorigin="anonymous" />

  <title>Avengers Status Map</title>
</head>

<body>
  <nav class="navbar navbar-expand-lg navbar-dark bg-dark">
    <div class="container-fluid">
      <a class="navbar-brand" href="#">Avengers Status Map</a>
    </div>
  </nav>

  <div class="container-fluid mt-4">
    <div class="row">
      <div class="col-3">
        <form action="/map" method="get">
          <div class="mb-3">
            <label for="avenger" class="form-label">Avengers</label>
            <select class="form-select" multiple name="avenger" id="avenger" required>
              <option th:each="avenger: ${avengers}" th:text="${avenger}" th:value="${avenger}"
                th:selected="${inputAvengers != null && inputAvengers.contains(avenger)}"></option>
            </select>
          </div>
          <div class="mb-3">
            <label for="start" class="form-label">Start Time</label>
            <input type="datetime-local" class="form-control" name="start" id="start" th:value="${inputStart}"
              required />
          </div>
          <div class="mb-3">
            <label for="end" class="form-label">End Time</label>
            <input type="datetime-local" class="form-control" name="end" id="end" th:value="${inputEnd}" required />
          </div>
          <button type="submit" class="btn btn-primary">Submit</button>
        </form>
      </div>
      <div class="col-9">
        <div id="map" style="width: 100%; height: 40em;"></div>
      </div>
    </div>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js"
    integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf" crossorigin="anonymous">
    </script>
  <script type="text/javascript" th:inline="javascript">
    /*<![CDATA[*/
    let paths = /*[[${paths}]]*/ {};

    let map;
    let openInfoWindow;

    function initMap() {
      let averageLatitude = 0;
      let averageLongitude = 0;

      if (paths) {
        let numPaths = 0;

        for (const path of Object.values(paths)) {
          let last = path[path.length - 1];
          averageLatitude += last.latitude;
          averageLongitude += last.longitude;
          numPaths++;
        }

        averageLatitude /= numPaths;
        averageLongitude /= numPaths;
      } else {
        // We had no data, so lets just tidy things up:
        paths = {};
        averageLatitude = 40.730610;
        averageLongitude = -73.935242;
      }

      map = new google.maps.Map(document.getElementById("map"), {
        center: { lat: averageLatitude, lng: averageLongitude },
        zoom: 16,
      });

      for (const avenger of Object.keys(paths)) {
        const path = paths[avenger];
        const color = getColor(avenger);

        new google.maps.Polyline({
          path: path.map(point => ({ lat: point.latitude, lng: point.longitude })),
          geodesic: true,
          strokeColor: color,
          strokeOpacity: 1.0,
          strokeWeight: 2,
          map: map,
        });

        path.forEach((point, index) => {
          const infowindow = new google.maps.InfoWindow({
            content: "<dl><dt>Avenger</dt><dd>" + avenger + "</dd><dt>Timestamp</dt><dd>" + point.timestamp + "</dd><dt>Status</dt><dd>" + Math.round(point.status * 10000) / 100 + "%</dd></dl>"
          });

          const marker = new google.maps.Marker({
            position: { lat: point.latitude, lng: point.longitude },
            icon: {
              path: google.maps.SymbolPath.FORWARD_CLOSED_ARROW,
              strokeColor: color,
              scale: index == path.length - 1 ? 5 : 3
            },
            map: map,
          });

          marker.addListener("click", () => {
            if (openInfoWindow) {
              openInfoWindow.close();
              openInfoWindow = undefined;
            }

            openInfoWindow = infowindow;
            infowindow.open({
              anchor: marker,
              map: map,
              shouldFocus: false,
            });
          });

        });
      }
    }

    function getColor(avenger) {
      return {
        wanda: '#ff2400',
        hulk: '#008000',
        hawkeye: '#9370db',
        falcon: '#000000'
      }[avenger];
    }

    /*]]>*/
  </script>

  <script
    th:src="${'https://maps.googleapis.com/maps/api/js?key=' + googleClientId + '&callback=initMap&libraries=&v=weekly'}"
    async></script>
</body>

</html>

We are injecting the data retrieved from Cassandra, as well as some other details. Thymeleaf automatically handles converting the objects within the script block into valid JSON. Once this is done, our JavaScript then renders a map using the Google Maps API and adds some routes and markers onto it to show our selected data.

At this point, we have a fully working application. Into this we can select some avengers to display, date and time ranges of interest, and see what was happening with our data:

6. Conclusion

Here we have seen an alternative way to visualize data retrieved from our Cassandra database, and have shown the Astra CQL API in use to obtain this data.

All of the code from this article can be found over on GitHub.

Building Scalable Streaming Applications with DataStax Astra Streaming

Pieter Humphrey — Thu, 18 Aug 2022 18:16:58 +0000

Astra Streaming enables developers to build streaming applications on top of an elastically scalable, multi-cloud messaging and event streaming platform powered by Apache Pulsar. DataStax Astra Streaming is currently in beta, and we’ll be releasing a full demo soon. In the meantime, this article will walk you through a short demo that will provide a great starting point for familiarizing yourself with this powerful new streaming service.

Here’s what you will learn:

How to create an Astra Streaming tenant, complete with namespaces, topics, and sinks.
How to produce messages for a topic that make use of serialized Java POJOs.
How to store topic messages in a DataStax Astra database.

To illustrate, we’ll use Astra Streaming to replicate the streaming object tracking information normally provided by the Federal Aviation Authority (FAA) to each airport. This stream reports the location of every piece of equipment at the airport (planes, fuel trucks, aircraft tow tractors, baggage carts, etc).

Let’s start building!

With Astra Streaming, not only can you feed information into the streaming “pipe”, but you can also store those events into an Astra database for later analysis. This allows us to view our object tracking information in two ways:

Where is everything located right now?
Where has a specific object been located historically? This is useful for tracking the paths of the object over time.

To build our streaming pipeline for tracking objects in real-time and historically, we’ll need to build the following:

An Astra Streaming tenant with a single topic object_location
A Java producer that will publish events to the object_location topic.
An Astra database with two tables that get data from the object_location topic.

Figure 1: All the moving parts we’ll need to build for this demo.

In our object tracking example, a single topic will feed data into two different tables. The object_location table records only the most recent known location for an object while the object_location_history records all locations that an object has been located at any given time. The location history data is useful for different types of analyses, such as analyzing the flow of different objects through the airport terminal.

This approach is not only applicable to object tracking, it can be used for any use case that requires the ability to see both real-time streaming data and historical data, for example, tracking stock prices where one table holds the current stock price while another table holds the historical stock prices.

Create the database

Now back to our object tracking example. Our first step will be to create the database. This is a very simple database with only two tables, which we’ll create in a keyspace called airport to keep things simple. The tables in the airport keyspace is object_location which tracks where every object is at the moment (well, really, the last known location), and object_location_history which tracks the location of the object over time with the most recent update listed first.

If you are following along with your own Astra instance, simply create a database with the keyspace airport and then run the database/create.cql file to create your tables.

You can create your Astra Streaming on one cloud provider even if your Astra database is hosted by another. However, you’ll get better performance if they are both hosted by the same cloud provider and in the same region.

Create a custom role

While it is possible to create an access token that grants you access to all of your databases, I highly recommend creating database-specific tokens based on custom roles. On more than one occasion, I’ve accidentally leaked security tokens into GitHub (errors that I corrected within minutes). The only thing that’s saved my bacon is the fact that the token was restricted to a single database. If you’re not familiar with the process for creating a token, I’ll show you how to do that in this section.

After you have created the database, click on the “down arrow” next to your organization name, as shown in the red square in the following image.

Figure 2: Accessing your organization settings.

This will bring up a menu. Click on the Organization Settings menu item. Once the page loads, click on the Role Management menu item on the left side of the page and press the Add Custom Role button. Give your role a meaningful name. As you can see in the following image, I named my custom role airport-demo. Then you can start selecting the permissions for your role. Since this role will be specific to a database built for demo purposes, I tend to be pretty liberal with my permissions. Set your permissions to suit your needs and scroll down to access the rest of the page.

Figure 3: Define the permissions for your custom role.

Select the keyspace permission and table permissions as appropriate. I like to enable all of the APIs for my databases, so I usually select them all. The most important step occurs at the very bottom where you select the single database for which this role applies.

Figure 4: Finishing the custom role definition.

When you are satisfied with your role configuration press the Create Role button.

Generating your database token

Now you can create a security token that is specific to your customer role and database. Select the Token Management menu item. Then select the custom role you created earlier and press the Generate Token button.

Figure 5: Generating a token.

You’ll need to exercise some caution here because the dialog box that pops up will never be displayed again. You’ll need this token information in your source code to connect to the database from Astra Streaming. So, you might want to press the Download Token Details button to download a CSV file with your token.

Figure 6: This information will never be shown to you again! Take this opportunity to download it.

Create the Astra Streaming components

Now we’re going to shift gears and create the Astra Streaming components. Here we will create a tenant, namespace, and a topic.

Create the Astra Streaming tenant

An Astra tenant is the top-level object for streaming. You can think of a tenant as akin to an application or a database. Create a new streaming tenant in the Astra DB web console and name it airport-events. When the tenant is fully created and running you will see a small green dot to the left of its name and the dashboard for the tenant will show up in your browser, as shown in the following image.

Figure 7: The airport-events tenant dashboard.

Create the Astra Streaming namespace

This step is optional because there is a default namespace created for you when you create a tenant. However, I like to keep things organized and isolated so I strongly recommend that you create a namespace for the airport-demo. Click on the Namespaces tab.

Figure 8: Creating a namespace in Astra Streaming.

Set the namespace to airport and press the create button. It’s just that easy!

Create the Astra Streaming Topic

Our next step is to create the topic for our object location events. Click on the Topics tab in the dashboard. By default, you will see both the new airport namespace and the default namespace listed in the dashboard. Click the Add Topic button in the airport namespace to create the new topic.

Figure 9: Creating a topic in the airport namespace.

You only need to specify the name of the topic, object-location as shown in the next image.

Figure 10: Creating the object-location topic in the airport namespace.

Press the Save button. At this point, we have a topic on which we can publish events. However, those events don’t go anywhere just yet. Next, we will create two “sinks’’ that will consume the events and store them in a database. A “sink” in streaming terms is either an Astra DB or an ElasticSearch instance. For this article, we will use the Astra DB to store our events.

Create the Astra DB sinks

The mechanism that Astra Streaming uses to store events to a database is a “sink”. We will need to create two sinks, one for each of our tables.

Create the object-location sink

Our first sink will store the event on the object_location table. This table is different from the object_location_history table in that it does not have the ts (timestamp) field. Click on the Sinks tab and then press the Create Sink button.

Figure 11: Creating the sink for the object_location table in our Astra DB.

In Step 1 of the wizard, select the fields as shown in the following image.

Figure 12: Step 1-Create the object-location sink.

Be sure to select the object-location topic in Step 1 of the wizard.

Figure 13: Step 2-Select the topic.

Next, you need to provide the connectivity information for your database. All of the information is important, but the database token is probably the most critical piece here. After you have pasted your token in, press the TAB button to exit the token field. This will prompt the Astra website to inspect your database and table and generate the field mappings, as you will see next.

Figure 14: Step 3-Specify the database you want to use.

The field mapping is done automatically for you. Notice that the automatic mapping only concerns itself with the fields in the table you have specified. There is no schema for the overall topic yet because we haven’t sent any messages over the topic (we will get to that in a little bit). I have yet to find a condition where the automatic mapping is incorrect, but it never hurts to check twice! Also, you can now expand the area for the object_location schema and view the details there as shown in the following image.

Figure 15: Notice the field mapping is automatically generated for you.

Press the Create button to create the sink.

Create the objloc-history sink

Now to create our second sink, the one that will capture information into the object_location_history table. You will perform essentially the same steps that you did for the first sink, with some key differences:

Sink Name: objloc-history (names are limited to 18 characters)
Topic: Pick the object-location topic again. It will feed both of our tables!
Table Name: object_location_history

This time when you enter the database token and TAB out of the field, the mapping will appear a little differently as shown below.

Figure 16: Notice the field mapping is slightly different for this sink.

You see here the ts or timestamp field (a Java long data type) is included in the mapping. Press the Create button to create this sink.

Create a Java producer

Things that generate messages on a topic are called “producers” in Apache Pulsar (and by extension in Astra Streaming). We need to create a producer that will send messages to the object-location topic. In fact, we don’t want to send simple string messages like many of the Pulsar, “Hello World” level examples do. We want to send an object that can be stored in database tables.

If you take a look at the Java code in the folder for this demo in GitHub, you will see several files. The main entry point is the App.java file. It’s a pretty simple file that just instantiates a Flight object and causes the Flights run() method to be invoked every second. The interesting work is in the Flight class.

The Flight class is designed to be a producer. It produces messages on the object-location topic each time the run() method is invoked. The constructor of the Flight class takes care of creating the PulsarClient connection and then the Pulsar topic producer. The most important thing to note here is the use of a JSONSchema based on the ObjectLocation class. This tells Pulsar the exact schema of the object that is being sent. Pulsar will expect the message to match the specified JSON Schema. If the message does not match the schema exactly, you will receive an error message.

public Flight(String flightID, String aircraftType) {
    try {
         // Initialize our location
         Date now = new Date();
         objLoc = new ObjectLocation(flightID, aircraftType, 0.0, 0.0,
now.getTime());
         // Create client object
         client = PulsarClient.builder()
         .serviceUrl(BROKER_SERVICE_URL)
         .authentication(
              AuthenticationFactory.token(Credentials.token)
         )
         .build();


         // Create producer on a topic
         producer = client.newProducer(JSONSchema.of(ObjectLocation.class))
         .topic("persistent://" + STREAM_NAME + "/" + NAMESPACE + "/" + TOPIC)
         .create();
    } catch(Exception ex) {
         System.out.println(ex.getMessage());
    }
  }

No messages are sent to the topic until the run() method is invoked. Here is the run() method implementation:

public void run() {
    // Send a message to the topic
    try {
         producer.send(objLoc);
         System.out.println(objLoc.toString());
         Date now = new Date();
         updatePosition(objLoc);
         objLoc.setTs(now.getTime());
    } catch(PulsarClientException pcex) {
         pcex.printStackTrace();
    }
}

The producer.send(objLoc) takes a native Java POJO that watches the schema expected and sends it over the topic. Note that you don’t have to serialize your object. The Pulsar libraries are smart enough to take care of that for you! Also, the very first time you run this code (which we will do next), Astra Streaming will record the schema for the message type. You can view that schema by navigating to your topic and clicking on the Schema tab, as shown next.

Figure 17: Viewing a topic schema.

Seeing it in action

If you load the project up in an editor like VS Code you can run the App class to see the application in action. When you do, you will see output like the following:

Figure 18: Events generated by the producer.

From the output above, we can see that the producer is generating events/messages on our topic. Now let’s check our database tables to see the data that was recorded. I’m going to use the CQLShell window on the Astra website to keep things simple. Let’s start by looking at the object_location table.

Figure 19: There should be a single record in your object_location table.

Remember, the purpose of this table is to record the last known location of an object, a Boeing 737 in this case. Your X and Y coordinates will vary depending on when you stopped the application from creating messages.

Now let’s take a look at our object_location_history table:

Figure 20: Our object_location_history table’s data.

Try it yourself!

As you can see, making real use of Astra Streaming is easy to do. Despite the many screenshots and the level of detail provided here, building this application requires just a few simple steps:

Create a Database

Create the object_location table
Create the object_loction_history table.
Create the custom role (optional)
Generate a token for the database

2.Create a Streaming Tenant

Create the airport namespace
Create the object-location topic
Create the object-location sink
Create the objLoc-history sink

3. Create a Java Topic Producer

That’s all there is to it. Now you have a recipe for sending and receiving event objects via Astra Streaming and storing them in an Astra DB. Try this code yourself by creating your free Astra account at https://astra.datastax.com (no credit card required). Your Astra account will work for both Astra DB and Astra Streaming. When you sign up, you’ll get $25.00 worth of free credits each month in perpetuity! That’s around 80GB storage, and 20M read / write ops. There’s never been a better time to start building streaming applications, and now with Astra Streaming, it’s never been easier.

Follow the DataStax Tech Blog for more developer stories. Check out our YouTube channel for tutorials and here for DataStax Developers on Twitter for the latest news about our developer community.

Resources

Deploy a Netflix Clone with GraphQL and DataStax Astra DB

Pieter Humphrey — Thu, 14 Jul 2022 21:19:01 +0000

This is part three of the DataStax app development workshop series, which guides you through fundamental technologies like Node.js, React, Netlify, and JavaScript to kickstart your app development portfolio. The full series is available on YouTube. In this post, we’ll guide you through building a Netflix clone backed with DataStax Astra DB and accessed using the Stargate GraphQL API.

If you’ve been following our full app development series, you should now have a good grasp of how to build a to-do list web app and how to run a TikTok clone. With these two apps under your belt, you’re ready to take your app development a step further with the final app of the series: a Netflix clone.

In this workshop, we’ll walk you through a simple Netflix homepage running on Astra DB. In the process, you’ll learn how to interact with the database using GraphQL, as well as how to implement infinite scroll and paging. The technologies we’ll be using in this workshop are the following:

GitPod is our cloud-based IDE that integrates directly with GitHub.
Netlify to deploy our Netflix clone to production across a global content distribution network (CDN).
Astra DB is our serverless database based on Apache Cassandra®.
Stargate GraphQL API to create, read, update, and delete app data.

Here’s a quick breakdown of the Netflix app you’ll be working with:

Figure 1: Diagram of Netflix clone app.

In short, the Netflix clone is a Node.js application divided into two parts: the user interface (built using React) and GraphQL, which interacts with the Astra DB database. You’ll learn how to create the database, build tables and then import data using GraphQL. You’ll also get a tour of the React code and learn to deploy the app using Netlify — all within two hours.

To get you started, let’s dig into the latest addition to your tech stack: GraphQL.

Understanding GraphQL

GraphQL is an open-source language used to fetch data from an API into your application. Originally developed by Facebook in 2012, this flexible query language is designed to pull data more efficiently than REST APIs when building mobile and web apps.

Typically, REST APIs need to load data from multiple URLs and make several round trips with heavy payloads. GraphQL, in contrast, fetches data from multiple sources all within a single request. This makes it ideal for mobile, IoT devices, and apps running on slow internet connections.

It also allows you to fetch the exact data you need (without over-fetching), which saves on bandwidth and improves the performance of your app. Another advantage is there’s no need to maintain versions with GraphQL. You can add new fields and deprecate older ones so you can evolve your API without breaking existing queries.

Overall, GraphQL makes APIs more flexible and developer-friendly — but there’s a catch. For developers who are already familiar with REST APIs, switching to GraphQL can present quite the learning curve. It can also add some complexity to API management and server-side maintenance, since GraphQL shifts most of the data query work onto the server to make the client-side easier.

Simplifying GraphQL with Stargate

Stargate is an open-source data gateway that makes it simple to query any Cassandra database using GraphQL types, queries, and mutations. When you add the Stargate GraphQL API to a Cassandra deployment, it scans the database and automatically creates HTTP endpoints with GraphQL queries and mutations for the objects it finds.

You can also use the API to create new database tables directly, and there’s a built-in GraphQL Playground servlet for you to easily prototype and tinker around with your mutations and queries.

Figure 2: GraphQL Playground in a nutshell.

With the Stargate GraphQL API, you can get the data you need into your apps — directly from Cassandra. And, since part of the mission with Stargate is to make Cassandra easier for every developer, we’ve made it available in DataStax Astra by default. So if you have a free Astra database, then you can use the Stargate GraphQL API, which is precisely what you’ll learn how to do in this workshop.

Set up and deploy a Netflix clone in 50 minutes

To wrap up our exciting app workshop series, we’ll hand you a sample Netflix clone app pre-built in React and guide you as you learn to do the following:

Create a free database with Astra DB
Learn about GraphQL and create your data model
Insert the dataset using GraphQL
Deploy your application to production with Netlify

Here’s the architecture we’ll be using in this workshop:

Figure 3: Netflix clone app architecture.

To start, make sure you’ve signed up for your free Astra DB account, then jump into the workshop video on YouTube.

For the source code, slides, exercises, and the step-by-step guide go to our DataStax Developers repo on GitHub. Lastly, if you get stuck or simply want to chat with our community, join our Discord.

And with that, we conclude our summer workshop series and the third app in your budding frontend development portfolio. To keep learning about open-source technologies and how to use them, simply register for any of our free DataStax Workshops to get started on your next big app.

Explore more tutorials on our DataStax Developers YouTube channel and subscribe to our event alert to get notified about new developer workshops. For exclusive posts on all things data: Cassandra, streaming, Kubernetes, and more; follow DataStax on Medium.

Resources

How to Build and Deploy a Serverless Game with DataStax Astra DB, JAMStack, Stargate, and Netlify

Pieter Humphrey — Tue, 12 Jul 2022 17:48:58 +0000

In one of our many free tutorials on DataStax Developers YouTube channel, we walked you through how to build and deploy a serverless game — BattleStax — an online party game that you can enjoy with your friends.

BattleStax is implemented as a JAMStack app that uses Stargate, Netlify, DataStax Astra DB, and GitHub to demonstrate how to build and deploy an application using modern, scalable architectures. In this post, we’ll break down the video to help you quickly create your own BattleStax game using React and Redux — implemented with a CI/CD pipeline, global content delivery network (CDN), and Apache Cassandra®.

The technologies we will take you through are:

React for the frontend
Redux for state management
Stargate and Astra DB for the backend
Netlify for deployment

Even if you’ve never heard of JAMStack or worked with JavaScript, we have everything preloaded for you in the cloud so you can follow along with the exercises, information, and solutions provided. We hope that by the end of the tutorial, you will have new knowledge on how to work with JAMStack.

Figure 1: A screenshot of the global scale, hit party game you will get to deploy publicly.

What is JAMStack?

JAMstack is the new standard architecture for the web, and JAM stands for JavaScript API Markup. It takes all the modern technologies we have and puts them together into a single platform to deploy an application from start to production. Because it has full continuous integration, it wins in security, scalability, performance, maintainability, and portability.

Using Git workflows and modern build tools, pre-rendered content is served to a CDN and made dynamic through APIs and serverless functions. Technologies in the stack include JavaScript frameworks, Static Site Generators, Headless CMSs, and CDNs.

Figure 2: What is JAMStack, www.JamStack.org?

JAMstack uses static pages to deliver content rather than rendering content at runtime. Traditional servers create a website, load it up on the server, and render it in real-time. In the case of JAMStack, you actually pre-render all of the pages, serving up static content. But it still offers dynamic, serverless functions, by clicking up to backend microservices. Microservice architecture enables the rapid, frequent and reliable delivery of large, complex applications.

Essentially, we’re pre-building when we build and pre-rendering all of these pages, and pushing them to various devices in the CDN. If you’re not familiar with CDN, one way to understand it is that it takes your content and distributes it across edge servers automatically as part of their system.

So instead of having a single web server where your site is hosted, and users connecting to those servers to get to your site, your content has been spread around the globe on this network, making delivery instantaneous.

If you want to understand more, check out our JAMStack guide on GitHub.

Why are we using Apache Cassandra?

Cassandra is a distributed, NoSQL database that is used by the vast majority of Fortune 100 companies. Facebook, Instagram, and Netflix all use Apache Cassandra. If you go on your phone and open an app, chances are, you’re already using it.

Apache Cassandra helps companies process large volumes of fast-moving data in a reliable, scalable way. Known for its performance at scale, Cassandra is regarded as the Porsche Lamborghini of the developer world, but can be finicky if you don’t know how to run it properly. Cassandra is essentially indefinitely scalable. There is no master node and Cassandra is a peer-to-peer system. You can actually have two-thirds of your cluster down and still be up and available to facilitate operations and write and read data.

DataStax Astra DB, built on the best distribution of Cassandra, provides the ability to develop and deploy data-driven applications with a cloud-native service, without the hassles of database and infrastructure administration. By automating tuning and configuration, DataStax Astra radically simplifies database operations.

Astra DB is basically Cassandra in the cloud. What is really helpful is that, from the development standpoint, even using just the free tier, you can load it up, hook up your Cassandra database with Astra and Netlify, and make your app work. And if you decide to go open source and run your own Cassandra, you can do that too with the same driver and code.

Tutorial Overview

You will find step-by-step instructions with screenshots all on GitHub. We’ve pre-baked the IDE and configurations and dependencies so you don’t have to download anything and you can complete the workshop in the cloud. Just launch GitPod right out of your code and jump into it.

Begin by following the instructions in our BattleStax repo on GitHub to setup and deploy your first app, create your BattleStax repository and set up your Netlify account.

Here is an overview of the four labs we will walk you through on GitHub.

Bootstrapping: Setup and deploy your first app
Lab 1 — Introduction to Netlify \ Expose your “hello world” API
Lab 2 — What are DataStax Astra and Stargate?\ Implementing a Serverless Data API
Lab 3 — What are Redux and React? \ Create client state with Redux
Lab 4 — Bind Redux to the User Interface

Ready? Let’s go!

Lab 1: Intro to Netlify + Expose your “hello world” AP

Let’s talk about Netlify and what it can do for you (or, feel free to jump ahead to exposing your “hello world” API)

You may already have used Netify’s Build, Package, deploy and host features to set up and deploy your empty application. Netlify can also configure your builds and manage atomic deploys.

Figure 3: Netlify can configure builds and deploy sites.

Netlify lets you deploy serverless Lambda functions without an AWS account, and with function management handled directly within Netlify. This means you can create serverless functions in your application that you can access seamlessly in your local environment, or via a global CDN (once deployed). You can do this without an actual server to deploy code to.

Remember, we’ve created static pages, and we’re going to generate those static pages. That’s where these Netlify functions come in, creating serverless functions.

If you want to dig in and learn more, click here for detailed documentation: what can Netlify do for you.

Let’s get coding. You have two options:

Option A: Cloud-based GitPod (recommended)
Option B: Local development environment

We recommend Option A. Gitpod is a cloud based IDE based on Eclipse Theia very similar to VSCode. You need to authenticate with your GitHub account, and GitPod will initialize your workspace, building the solution. Click here to launch Gitpod to initialize your environment.

Alternatively, you can use your laptop, which is explained in the documentation.

Lab 1 will cover how to:

Lab 2: DataStax Astra DB and Stargate + Implementing a Serverless Data API

We’ve covered Astra DB briefly in the section above and now we’ll introduce Stargate, an open-source data gateway to abstract Cassandra-specific concepts from app developers and support different API options. It is deployed between client applications and a database to provide an abstraction layer that can be used to shape your data access to fit your application’s needs.

Figure 4: Stargate is a framework used to customize all aspects of data access.

Let’s implement a CRUD API in Astra DB. In Lab 2 of the BattleStax tutorial, we will:

Create an Astra DB database to store game documents
Create test cases to check that our API call is working correctly
Build the API call to Astra DB to create a game document, based on the requirements from our test

Figure 5: With Document API, save and search schemaless JSON documents in Cassandra

Lab 3: Intro to Redux and React + Creating a client state with Redux

React is a fast, component-based, front-end JavaScript library. React typically runs in your browser and renders single-page web application user interfaces. Redux is a JavaScript library that is used mostly for application state management. Redux maintains the state of an entire application in a single immutable state tree (object), which can’t be changed directly. When something changes, a new object is created (using actions and reducers).

React and Redux work together by letting you build components that react to changes of the application state. Components affected by a state change are re-rendered with the new data. Components also dispatch actions, for example when a button is clicked.

Figure 6: How React and Redux works together.

Learn more by reading our Redux and React documentation here.

In Lab 3, we will:

Build out the game slice boilerplate code by implementing one reducer, exporting an action and a selector
Run tests to try out the functionality of our game slice

We will cover how to:

Lab 4: Bind Redux to the User Interface

In this last and final step, we’ve already built a UI with React, but now we need to connect it to our game state.

In Lab 4 we will:

Build out the NewGame.js component by connecting it to redux.
Build a test to try out the functionality of NewGame.js

We will cover how to:

Figure 7: Hooks in React to extract state information from Redux store.

Summary

Figure 8: BattleStax final display.

If you’ve followed all the steps in the GitHub tutorial, by this point, you would have:

Deployed an application through a CI/CD pipeline with GitHub Actions
Learned how to use serverless functions that are globally available via Netlify
Used these backed by the top NoSQL distributed database Apache Cassandra on DataStax Astra DB

And it was possible to do all this without touching a server, deploying back-end code, or needing to talk to your IT friends. Enjoy creating your next viral JAMStack app!

Resources

Building Reactive Java Applications with Spring Framework

Pieter Humphrey — Wed, 29 Jun 2022 02:47:02 +0000

In one of our many free tutorials on DataStax Developers YouTube channel, we walked you through how to build a reactive implementation of Spring PetClinic in Apache Cassandra® using Spring WebFlux. The full series is available on YouTube.

If you’re a Java developer who uses the Spring ecosystem, you’ve probably seen the Spring Pet Clinic. In this workshop, we will walk you through a new reactive implementation of the Pet Clinic backend that uses Spring WebFlux and Apache Cassandra® (via DataStax Astra DB).

The cloud-native database-as-a-service built on Cassandra fits the highly concurrent, non-blocking nature of our reactive application. We’ll do all of our work in the cloud with Gitpod’s open-source, zero-install and collaborative development environment.

Figure 1: A REST API using the Spring framework.

The steps we will take you through are:

Setting up Astra database
Theory: synchronous vs. asynchronous vs. reactive programming
Using the Gitpod platform
Introducing the Spring Boot and WebFlux frameworks
Reviewing backend code
Starting frontend application

Setting up your free Astra database

Figure 2: Datastax Astra DB Benefits.

Follow the step-by-step instructions to create your Astra database, and once your database is ready, you can copy your credentials over to GitHub.

Synchronous vs. asynchronous vs. reactive programming

Most Java developers use synchronous programming. When you initiate a session, you execute quickly and you will get a response. Then you send the parameter to the API, and the driver will create a query. You bind the parameter that simply maps the parameter to the query. You execute the query and get back an object called a ResultSet.

Figure 3: Synchronous queries weaknesses.

The issue with synchronous programming is that you need to wait. It can take a lot of time if you are querying for a lot of data, or a big cluster. Although synchronous programming is very simple, it can block communication. This means that nothing else in the application proceeds until the result from the query is returned. The application blocks for the entire round trip, from when the query is first sent to the database until the results are retrieved and returned to the application.

The advantage of synchronous queries, however, is that it is simple to tell when a query completes, so the execution logic of the application is easy to follow. However, synchronous queries cause poor application throughput.

Figure 4: Asynchronous queries weaknesses.

An asynchronous query executes call does not block for results. With asynchronous queries, you will put the parameter in to prepare your queries, bind the parameters to your queries, and execute the query.

But instead of waiting for the result, the customer and the driver will immediately give you an object called a completion stage. A future is returned from the asynchronous execute call. A future is a placeholder object that stands in for the result until the result is returned from the database. Depending on the driver and feature set of the language, this future can facilitate asynchronous processing of results. This typically allows high throughput. However, because you can process the result only when you get a callback, this can create sync resistance.

Figure 5: The Reactive Manifesto.

Now, coming to the reactive manifesto, the programming offers what is missing with the synchronous-only stack by achieving a responsive service. This means that the system always responds in a timely manner and stays responsive even in the face of any failure. It also offers a scale-out dynamic.

When it comes to huge volumes of data or multi-users, we often need asynchronous processing to make our systems fast and responsive. In Java, a representative of old object-oriented programming, asynchronicity can become really troublesome and make the code hard to understand and maintain. So, reactive programming is especially beneficial for this ‘purely’ object-oriented environment as it simplifies dealing with asynchronous flows.

Figure 6: Reactive Queries.

Using the GitPod platform

Gitpod is an open-source Kubernetes application providing prebuilt, collaborative development environments in your browser. Gitpod provides step-by-step screenshots to launch and build the Spring PetClinic Reactive backend application, created by our special workshop guest, Moritz Eysholdt from TypeFox.

Figure 7: When Gitpod finishes building the app, a new tab will open in your browser showing the following.

Introducing the Spring Boot and WebFlux frameworks

With its simple abstraction, Spring Reactor is a popular framework for Java from Spring developers. Spring Framework is a Java platform that provides comprehensive infrastructure support for developing Java applications. Spring enables you to build applications from “plain old Java objects” (POJOs) and to apply enterprise services non-invasively to POJOs.

Figure 8: What Spring can do.

We will also introduce Spring Boot, a tool that makes developing web applications and microservices with Spring Framework faster and easier. Spring Boot can create stand-alone Spring applications (without deploying WAR files), simplify your build configuration, and automatically configure Spring and 3rd party libraries. Spring Boot also has production-ready features such as metrics, health checks, and externalized configuration.

At the heart of the Spring Framework are two fundamental features: inversion of control (IoC) and dependency injection (DI).

Inversion of control essentially transfers the control of objects to the Spring Framework. Unlike traditional programming, where custom code makes calls to a library, IoC allows the framework to control the flow of a program. This keeps Java classes independent of each other for increased modularity and extensibility.
Dependency injection is a pattern in Spring that’s used to implement IoC, where the Spring container is in charge of “injecting” objects into the right dependencies. This allows for the loose coupling of components and shifts the responsibility of managing components onto the container.

Figure 9: How Spring Boot works.

Reviewing backend code

Let’s have a look inside the main component spring-petclinic-reactive to see which libraries and frameworks have been used.

Figure 10: Understanding the architecture.

Spring-boot: Spring Boot makes it easy to create stand-alone, production-grade Spring-based Applications that you can “just run”. We take an opinionated view of the Spring platform and third-party libraries so you can get started with minimum fuss. Most Spring Boot applications need minimal Spring configuration.
Spring-Security: Spring Security is a powerful and highly customizable authentication and access-control framework. It is the de-facto standard for securing Spring-based applications. Spring Security is a framework that focuses on providing both authentication and authorization to Java applications. Like all Spring projects, the real power of Spring Security is found in how easily it can be extended to meet custom requirements.
Spring-WebFlux: Spring sub-framework to create Reactive Rest Endpoint.
Spring-Actuator: Expose Endpoints to expose metrics to third party systems. Examples are health, infos, jmx,and prometheus.
Spring-Test: Enabled unit testing and mocking with Spring configuration and beans.
Spring-Cloud: Spring Cloud provides tools for developers to quickly build some of the common patterns in distributed systems (e.g. configuration management, service discovery, circuit breakers, intelligent routing, micro-proxy, control bus, one-time tokens, global locks, leadership election, distributed sessions, cluster state). Coordination of distributed systems leads to boilerplate patterns, and using Spring Cloud developers can quickly stand up services and applications that implement those patterns. They will work well in any distributed environment, including the developer’s own laptop, bare metal data centers, and managed platforms such as Cloud Foundry.
SpringFox (Swagger): Annotation-based rest documentation generation and test client generation (swagger-ui).

To understand the underlying data model implemented in Apache Cassandra, check out our Gitpod guide.

Starting frontend application

Figure 11: Frontend of Spring PetClinic.

Once you configure and run the application, you can also test CRUD with Swagger as a hands-on exercise as follows:

Figure 12: Test CRUD with Swagger.

And with that, we conclude our workshop on a reactive implementation of Spring PetClinic with Cassandra. To keep learning about open-source technologies and how to use them, simply register for any of our free DataStax Workshops to get started on your next big app.

Resources

How to Create a Fullstack App Using NuxtJS, NestJS, and Datastax Astra DB (with a Little Help from Github Copilot)

Pieter Humphrey — Wed, 29 Jun 2022 02:46:30 +0000

If you want to create a full-stack application, complete with dynamic data retrieved from a cloud database by an API, then watch this tutorial created by Eddie Jaoude.

Building a full-stack application can be daunting because you have to not only think about how the frontend will display the data but where the data will come from and where it’s stored. But it’s not as hard as you might think to get the basics of a full-stack application up and running.

In his tutorial, Eddie shows you how to do it in less than an hour using NuxtJS with VuetifyJS for the frontend, NestJS to create RESTful APIs, and DataStax’s Astra DB for the cloud database service. Also, you’ll use Github Copilot as your AI-powered pair programmer.

In this tutorial, you’ll learn how to:

Create a basic frontend using VuetifyJS.
Use an API to retrieve and save data.
Retrieve data from a cloud database and display it in the application.
Use Github Copilot to help you code faster with autocompletion.

Let’s recap the key technologies you’ll be using briefly.

NuxtJS

NuxtJS is a framework for creating VueJS applications. It was first released in 2016 to build upon applications that use VueJS. NuxtJS handles the server-side and client-side distribution so you can focus just on the application development.

Some features of NuxtJS are:

Server-side rendering
Static site generation
Meta Tags
Automatic routing and code splitting

The result of these improvements is faster applications, improvements to SEO due to server-side rendering, and a helpful start-up wizard that lets you select different UI frameworks, linting tools, and testing frameworks.

→ nuxtjs-nestjs-copilot 11
→ nuxtjs-nestjs-copilot npm init nuxt-app ui

Create-nuxt-app v3.7.1 
? Project name: ui
? Programming language: JavaScript
? Package manager: Npm
? UI Framework: Vuetify.js
? Nuxt.js modules: Axios - Promise based HTTP client
? Linting tools: (Press <space> to select, <a> to toggle all, <i> to invert selection)
? Testing Framework: None
? Rendering mode: Single Page App
? Deployment target: Server (Node.js hosting)
? Development tools: 
> ● jsconfig.json (recommended for VS Code if you’re not using typescript)
  ○ Semantic Pull Requests
  ○ Dependabot (For auto-updating dependencies, GitHub only)

Figure 1: NuxtJS configuration wizard.

All of this means you can get working on your universal or single-page application much faster because of the speed at which you can set it up.

NestJS

NestJS is a framework developed to rapidly build server-side applications. NestJS is built from NodeJS and ExpressJS and uses progressive JavaScript. It fully supports TypeScript and combines principles of:

NestJS APIs are exposed so you can take advantage of a selection of third-party modules, extending your applications with more features without having to code them yourself.

Eddie will show you how to take advantage of ExpressJS with NestJS to create a RESTful API to retrieve and save data from a cloud database.

DataStax Astra DB

If you are looking for a database that can scale fast, has a dynamic schema for unstructured data and flexible data models then choose a NoSql database. In this tutorial, Eddie is using Apache Cassandra®.

Rather than having to set up and manage the database manually, he has chosen a fully managed version through Astra DB. This is an autoscaling DBaaS, built on Cassandra. It handles all of the configuration and management of your cloud databases, so you can spend more time building your applications.

Figure 2: DataStax Astra DB dashboard.

Astra DB uses Stargate APIs so you can interact with data using GraphQL, CassandraQL, REST, or Document API.

To follow along in the tutorial, you can sign up for a free Astra DB account to get up to 80 GB of free storage and 20 million read/write operations per month.

Github Copilot

Github Copilot offers a helping hand while you code. Powered by OpenAI, Copilot has been trained on billions of lines of public code to provide you with smart suggestions.

It can convert comments into code so you only need to provide the logic and Copilot will assemble it.

Copilot helps you avoid having to type the same thing over and over with its ability to identify and auto-fill repetitive code patterns from only a few examples. In addition to saving you time, with less manual coding, you’ll have fewer bugs due to typos in your code.

To help you with testing, you can import a unit test package and Copilot will suggest tests from your implementation code.

Figure 3: Github Copilot service.

If you’ve ever wanted to get into full-stack development, there’s no easier way to get started than by watching Eddie’s tutorial on YouTube. With his easy-to-follow example and the technologies described here, you’ll be able to create your very own full-stack application in under an hour.

You can find the source code for the tutorial on Eddie’s Github. If you want to learn more about DataStax and Astra DB, sign up for a free Astra DB account and then head over to the DataStax’s Developer Youtube Channel to see all the things you can do with these technologies. To learn about the other technologies mentioned here, just check out the resources we’ve provided below.

Follow DataStax on Medium for exclusive posts on Cassandra, Kubernetes, streaming, and much more.

Resources

Stargate gRPC: The Better Way to CQL

Pieter Humphrey — Tue, 14 Jun 2022 15:56:53 +0000

Stargate’s new gRPC API is so much more than just a feature release — it’s your official welcome to the “no drivers” future of Apache Cassandra.

Recently, we released gRPC as the newest API supported by Stargate, our API data gateway. On the surface, it would seem like the API doesn’t do very much; it receives CQL queries via the gRPC protocol, then passes those to Apache Cassandra® and returns the results. Sounds like a pretty modest feature release, right?

In reality, what the Stargate team has delivered is groundbreaking. Not quite a native driver and not quite a simple HTTP-based API, Stargate’s gRPC implementation represents a fundamentally new approach for applications interacting with Cassandra — an approach that’s more cloud-native than any driver, and more performant than any simple HTTP-based API.

So let me tell you why this approach is so important, and why this is such a revolutionary solution for a common developer problem.

Native drivers are not cloud-friendly

Functionality inside a native driver can be divided into two parts:

The query engine. This issue requests in a particular query language for a particular database, and receives responses to those requests that can then be used in application code. In our case, the query language is CQL, and the database is Cassandra.
Operational management. This includes tasks like connection pooling, TLS, authentication, load balancing, retry policies, write coalescing, compression, health checks, etc.

You’ll notice that most of those operational tasks are abstracted away from applications in cloud environments, and simply handled automatically on behalf of the application. For example, load balancing, health checks, and TLS termination are intrinsic to most cloud environments; even retries can be configured within the environment.

Put another way: in a well-designed microservices environment, network management tasks should live inside a service boundary and execute within an SLA defined in a service contract. There should be no need, and it would be a violation of microservices principles for an application to want to reach across that boundary and directly manipulate those operational tasks.

And yet, this is exactly what native drivers do.

This is not a mere architectural nicety. Building native drivers into an otherwise cloud-native, microservice-oriented application have real and negative consequences. Let’s dig a little deeper into why.

Native protocol drivers are expensive to maintain and require reimplementing the same complex functionality for different platforms (like Java, Go, Node, Rust). All that operational management forces developers to extend their skillset from application development in their preferred language to areas of systems operation, thus steepening the learning curve for native drivers.

More significantly, this co-mingling of concerns opens up a new vector that could trigger the need for a driver update. A configuration change in the network environment, for example, could require an update to the way every driver handles load balancing or connection pooling. Now your organization has to stop every application instance using that driver, apply the change within the driver, and restart all those application instances. Depending on the nature of the driver change, some changes in the rest of the application may also be required.

They also inject surprising brittleness into applications because of the network management overhead required, making it more likely that drivers, and therefore applications that use them, must be updated.

In sum, native drivers are:

Complex and present a steep learning curve
Hard to update and maintain
Speed bumps for developer velocity
A threat to application resilience

So we can safely say that native drivers are fast, which makes it easy to overemphasize raw performance, but the overall picture of performance and resilience is much more complicated.

HTTP-based APIs are a performance trade-off

The modern approach to application development is, in part, a rebellion against the burden of native drivers. Today’s application developers, particularly front-end developers, expect to interact with data through an HTTP-based API and rely on JSON as the primary method of structuring data.

We fully support this API-based approach on Stargate. This has several advantages:

Language agnosticism. Applications can be written in any language that can talk to an HTTP endpoint.
Separation of concerns between application environment and infrastructure environment. Precisely as should happen in a cloud-native context, all of the network management and operational overhead lives behind the API. Changes and updates there stay contained within that service boundary, removing this as an area of concern for application logic.
Resilience. The statelessness of HTTP constrains application design to avoid reliance on durable network connections, meaning applications designed in this manner are more resilient against the vagaries of network behavior.

Unsurprisingly, HTTP-based APIs have become the backbone of microservice applications for a cloud-native environment. But these benefits are not free. HTTP-based APIs is a slower way to query a database, for two reasons:

Networking. Native drivers talk “closer to the wire,” which significantly improves performance. The Java driver for CQL, for example, operates at Layer 5, whereas HTTP operates at Layer 7.
Data transformation. Databases don’t store JSON natively (even MongoDB relies on the WiredTiger storage engine when you drill down far enough). So some transformation has to happen to turn a JSON-oriented query into a native database query (CQL, in the case of Cassandra). The compute overhead of performing this transformation further slows performance.

And now, we have a dilemma. On one hand, HTTP-based APIs offer simplicity and language agnosticism that accelerates developer velocity, while also offering separation of concerns between application and infrastructure that improves application resilience. To put it simply, HTTP-based APIs are good cloud citizens, presenting and abiding by clear service boundaries.

On the other hand, while native drivers are a burden to developers and co-mingled concerns between development and operations negatively impact resilience, native drivers are just flat out more performant than HTTP-based APIs.

So, what to do?

Decomposing the driver

Stargate supports native driver calls, offering a CQL API through which to talk to Cassandra. This is essentially just a transparent proxy, and so CQL calls via Stargate remain highly performant. Let’s look at a simple architecture diagram of this part of Stargate. (See Figure 1.)

Figure 1: Simple Architecture of Native Driver and Stargate.

The fundamental problem is the co-mingling of concerns. Some of what lives inside the native driver should, in a cloud-native context, live behind an API and thus inside the API’s service boundary. So what if we looked at it this way instead? (See Figure 2.)

Figure 2: Decomposing the driver.

The real challenge is how to move that box that says “Network Management Tasks” across the service boundary into Stargate and behind an API. We’ll also have to do it in a way that honors the language agnosticism of APIs. Without that agnosticism, we have to maintain a different “box” of network management tasks for each language, even though those tasks are essentially the same across languages. We’d lighten the driver but make Stargate harder to maintain, and a good bit less cloud-friendly.

Enter gRPC

In 2008, Google developed, open-sourced, and released Protocol Buffers — a language-neutral mechanism for serializing structured data. In 2015, Google released gRPC (also open source) to incorporate Protocol Buffers into work to modernize Remote Procedure Call (RPC).

gRPC has a couple of important performance characteristics. One is the improved data serialization, making data transit over the network much more efficient. The other is the use of HTTP/2, which enables bidirectional communication. As a result, there are four call types supported in gRPC:

Unary calls
Client-side streaming calls
Server-side streaming calls
Bidirectional calls, which are a composite of client-side and server-side streaming

Put all this together and you have a mechanism that is fast — very fast when compared to other HTTP-based APIs. gRPC message transmission can be 7x to 10x faster than traditional REST APIs. In other words, a solution based on gRPC could offer performance comparable to native drivers.

Stargate gRPC

When you pull all of the network management tasks out of a driver, what you’re left with is a thin client library containing little more than the query engine. In our case, these CQL queries transit to a Stargate API endpoint via gRPC. (See Figure 3.)

Figure 3: Stargate’s gRPC implementation.

Behind that endpoint is what amounts to a CQL driver written in gRPC. In other words, it receives CQL calls on the API endpoint via gRPC, and then makes direct CQL calls to Cassandra. No data transformation is required because we’re using CQL end to end.

These client libraries are dramatically easier to write and maintain. Our original intent was to launch with client libraries for Java and for Go, since these are our two most requested languages. As it happened, adding new languages was so easy that we also included client libraries for Node.js and Rust.

These four — and perhaps more languages in the future - — represent a fully DataStax-supported way to make CQL calls from your application. We’ll continue to support our existing native drivers, and in those languages, the gRPC client libraries represent an additional, supported alternative. For languages like Go where DataStax does not have a supported native driver, the supported gRPC client library is now a great way to go.

Do more with Stargate gRPC

If your favorite language is not on our list, extending to a new language is not hard. From a protobuf file, you get a skeleton of the CQL calls you to need to make in your chosen language, and none of the operational overhead is required. You get that operational functionality out of the box with gRPC, and it lives inside of Stargate where it belongs in a proper cloud-native context.

Thanks to HTTP/2 and efficient data serialization, you’ll now get performance on par with native drivers combined with the simplicity of a thin client library, all within a context that plays nicely with the rest of your microservices.

To learn more, head over to Stargate’s Github. You can also find source code and examples on using Stargate gRPC API clients:

And, lastly, welcome to the “No Drivers” future of Apache Cassandra.

Follow DataStax on Medium for exclusive posts on all things Cassandra, streaming, Kubernetes, and more. To join the best developers around the world and stay in the data loop, follow DataStaxDevs on Twitter.

Resources

Curious to learn more about (or play with) Cassandra itself? We recommend trying it on the Astra DB free plan for the fastest setup.

Stargate — The official data API for DataStax Astra DB
What is Astra DB? — DataStax Astra Documentation
Stargate — GitHub
Using Stargate gRPC API clients
Say Goodbye to Native Drivers with Stargate gRPC API in Java

Aggregate Functions in Stargate’s GraphQL API

Pieter Humphrey — Thu, 19 May 2022 19:19:13 +0000

Thursday, June 3rd, 2021,a new release of Stargate was applied to Astra DB. It includes an exciting new feature: aggregate functions! If you’re not familiar with aggregate functions, they are functions that look at the data as a whole and perform a function like min(), max(), sum(), count() and avg().

Until now, aggregate functions were only available using cqlsh (the CQL Shell). However, with the Stargate 1.0.25 release they are now also available using the GraphQL API. In this blog entry, I’ll walk you through the process to get early access to this exciting new functionality in Stargate, and how to setup everything you need to test your own aggregate queries.

Using cqlsh to perform an aggregate query is pretty straight forward. Assuming you have an employee table with the following sales data:

id	name	sale	rtime
1	John	1000	2019-01-12T09:48:31.020Z
2	Mustafa	2000	2019-02-12T09:48:31.020Z
3	Krishna	2500	2019-01-12T09:48:31.020Z
1	John	2200	2020-01-12T09:48:31.020Z
1	John	2350	2021-01-12T09:48:31.020Z
2	Mustafa	3000	2020-02-12T09:48:31.020Z
2	Mustafa	300	2021-02-12T09:48:31.020Z
3	Krishna	1500	2020-01-12T09:48:31.020Z
3	Krishna	3600	2021-01-12T09:48:31.020Z

Now we want to find the highest sale number for employee 1, John. Our cqlsh query would look like the following:

select id, name, max(sale) as highest_sale from employee where id = 1 and name = “John”;

Would return a single record:

id name highest_sale

1 John 2350

Prerequisites

This blog tutorial assumes that you already have Docker and curl installed and configured on your machine. Alternatively, if you have a Astra account (they’re free) you can do your testing there.

Getting Stargate

If you are using Astra you can skip this part and go to the next section.

The main repository for the Stargate source code is on GitHub at https://github.com/stargate/stargate However, I recommend just using the Docker container that is already configured for testing. Assuming you have Docker installed already, just run this command:

docker run -d — name stargate \
-p 8080:8080 \
-p 8081:8081 \
-p 8082:8082 \
-p 9042:9042 \
stargateio/stargate-dse-68:v1.0.25 \
--developer-mode — cluster-name test \
--cluster-version 6.8 --dse --enable-auth

Your Docker instance of Stargate will load and start executing.

Preparing Your Test Environment

Next we need to create our keyspace and table. Then we will load our test table with test data. While it is possible to do almost all of this using GraphQL, I did most of it using the REST API since that is the API with which I’m most familiar.

Note: All of these URLs are designed for the Docker container running locally on your machine. If you are using Astra, adjust the URLs accordingly.

Once the Docker image is fully up and running, you will need to get authentication credentials for the Cassandra instance it contains. Use this curl command to get the authentication token:

curl -L -X POST 'http://localhost:8081/v1/auth' \
-H 'Content-Type: application/json' \
--data-raw '{ "username": "cassandra", "password": "cassandra" }'

Set the auth token as an environment variable for easy reuse

export authToken=”The token returned in the previous step”

Now run the following command to get a list of existing keyspaces. This is a good test to ensure you’ve set your authToken environment variable correctly:

curl -L -X GET 'localhost:8082/v1/keyspaces' \
--header 'accept: application/json' \
--header ‘content-type: application/json’ \
--header “X-Cassandra-Token: $authToken”

You should see the following output from the command:

[“data_endpoint_auth”,”system”,”system_auth”,”system_backups”,”system_distributed”,”system_schema”,”system_traces”]

Next we need to create our keyspace for our database. The following command will create the test keyspace:

curl -L -X POST 'localhost:8082/v2/schemas/keyspaces' \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--header "X-Cassandra-Token: $authToken" \
-d '{ "name": "test", "replicas": 1}'

Now we need to create our employee table in the test keyspace. This command is rather lengthy for a blog post so I recommend getting the create_table.sh file from the github repository at https://github.com/jdavies/blogs/blob/master/20210602_aggregate_stargate/create_table.sh

Now it’s time to load some data into our table. The easiest way to do this is to download the load_data.sh file from my GutHub repository (another blog-unfriendly script) and execute it via the command:

./load_data.sh

It contains the curl commands to load the records into your Docker database.

Once the data is loaded, let’s run a quick query to ensure that everything is as we expect. Execute the following command:

curl -L -X GET ‘localhost:8082/v1/keyspaces/test/tables/employee/rows’ \
--header ‘accept: application/json’ \
--header ‘content-type: application/json’ \
--header “X-Cassandra-Token: $authToken”

You should get all 9 rows of data back. They can be a little hard to read from the terminal. If you want to see a prettier version I suggest copying the resulting text and pasting it into an online JSON browser like jsonbeautifier.org. You should see the following 9 rows:

Now we are ready to get down to business!

Using Aggregate Queries

Since I’m new to GraphQL, it seems strange to me. However, once you get used to its way of doing things (like omitting commas in a JSON-esque data format) it’s pretty straightforward. Here is the curl command that will retrieve the highest sales year for employee 1, named John:

curl 'http://localhost:8080/graphql/test' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8080' \
-H "x-cassandra-token: $authToken" \
--data-binary '{"query":"query maxJohnSales {\n  employee(value: { \n    id: 1, \n    name: \"John\" }) {\n    values {\n      id\n      name\n      rtime\n      highest_sale: _int_function(name: \"max\", args: [\"sale\"])\n }\n  }\n}"}' --compressed

The payload is a little hard to read on the command line, so here it is in GraphQL format:

query maxJohnSales {
   employee(value: {
      id: 1,
      name: “John” })
   {
      values {
         id
         name
         rtime
         highest_sale: _int_function(name: “max”, args: [“sale”])
   }
}

If you examine the command, you will see how we included the max() aggregate command (named as “highest_sale”). Just like a cqlsh version of the call, the max() function is applied to the return values, not the select criteria. Your output should match the following:

How about searching for the highest sale of all time? Here’s how you do it:

curl 'http://localhost:8080/graphql/test' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8080' \
-H "x-cassandra-token: $authToken" \
--data-binary '{"query":"query maxJohnSales {\n  employee {\n    values {\n      id\n      name\n      rtime\n      highest_sale: _int_function(name: \"max\", args: [\"sale\"])\n }\n  }\n}"}' --compressed

By removing the “where” clause from the GraphQL statement (traditionally in the parenthesis after the table name), you can search the entire table, across all partitions. In cqlsh this is the equivalent of adding ALLOW FILTERING, which in general is regarded as a “bad thing” since it forces a full table scan across all partitions, which can be very slow. However, since aggregate functions are often used for reporting, it might be acceptable to do this for a few special queries.

ALLOW FILTERING isn’t necessarily a “bad thing”, but you have to understand what it does and use it sparingly if you want to keep your database performing at max speed! ALLOW FILTERING can come in very handy when we’re invoking a SELECT operation on a single partition (i.e. providing the full partition keys at the very minimum which is “id” in this “test.employee” table’s case).

Conclusion

That’s all there is to using aggregate functions in GraphQL using Stargate. Bear in mind this is GraphQL API-specific. It won’t work with the REST or Document APIs.

How DataStax Tracked Down a Linux Kernel Bug with Fallout

Pieter Humphrey — Thu, 17 Mar 2022 22:04:22 +0000

Sometimes as a developer, you run into a bug buried deep within the layers of your software stack. Chasing down the root cause requires not only curiosity, patience, and a healthy dose of tenacity but a willingness to try different tools and approaches. This post describes our challenges and ultimate success in tracking down a Linux kernel bug using Fallout.

Bugs come in all shapes and sizes and it’s not always clear at the beginning of a debugging session which one you’re currently chasing. Some bugs can be fixed up in a matter of minutes while others take weeks to nail down. And the really tricky ones require you to dig through multiple layers of your software stack, stressing the limits of your patience.

This article covers an example of the latter kind of bug. When one of the daily performance tests started timing out after 16 hours, it turned out that I had unknowingly stumbled onto an issue in the Linux kernel hrtimer code. Since I didn’t know it was a kernel bug when I started looking, I had to methodically dig deeper into the software stack as I homed in on the problem.

Fallout is an open-source distributed systems testing service that we use heavily at DataStax to run functional and performance tests for Apache Cassandra, Apache Pulsar, and other projects. Fallout automatically provisions and configures distributed systems and clients, runs a variety of workloads and benchmarks, and gathers test results for later analysis.

As you’ll see below, Fallout was instrumental in being able to quickly iterate and gather new data to validate and invalidate my guesses about the underlying bug.

The original bug report

At DataStax we run a variety of nightly functional and performance tests against Cassandra, and one morning we noticed that one of our write-throughput benchmarks failed to complete after 16 hours. Normally, this test would complete within four, so something was clearly wrong. Like all good engineers, we checked to see what had changed since the test last passed, and sure enough we found a commit in our git repository that could potentially explain the failure.

Since the most important thing after discovering a test failure is to get the tests passing again, we reverted the suspected commit and re-ran the tests. They passed. So with the tests now passing we figured that even if we hadn’t analysed the commit to understand how the test could be failing, we’d at least narrowed down the cause of the issue to a single commit. High-fives were exchanged and we all went back to work leaving the author of the reverted commit to understand the root cause of the failure.

Only the test failed again the next day, even with the reverted commit. Clearly we’d been too hasty in our diagnosis and hadn’t actually fixed anything.

But this begged the question: if the commit we reverted didn’t originally cause the test to fail, why had it suddenly started failing? It dawned on us that we were dealing with an intermittent failure and our task now was to figure out how to reproduce it more reliably.

Reproducing the failure

Intermittent failures are a pain to debug because they take so much longer to analyze when you can’t trigger the conditions for the bug on every test run. Sometimes there’s no telling how many runs you’ll need to hit it. Worse, you have to gather much more data to be confident that you’ve actually solved the bug once you apply your fix.

Almost all of the performance tests at DataStax run using Fallout, our distributed systems testing service. Tests are described in a single YAML file and Fallout takes care of scheduling tests on available hardware and collecting the results for later analysis. Our team figured out we could normally hit this bug at least once every five runs. We initially looked at trying to increase the reproducibility of this bug but simply couldn’t make the failure happen more regularly. Fortunately, there was one way we could reduce the time to trigger it — by running multiple tests in parallel. Since Fallout can automatically queue test runs and execute them once hardware becomes available, I fired up five runs of the same test and waited for the results to come in. Instead of having to wait up to 20 hours to see if I had hit the bug, I now only needed to wait around four hours. Still not ideal, but it meant that I could make progress every day.

But we were still using the “does the test run for longer than four hours” symptom to detect when we’d run into a problem. Finding a less time-consuming way to know when we’d triggered the bug required understanding the side effects of the bug at the Cassandra level. In other words, we had to know what Cassandra was doing to cause the test to timeout in the first place.

The side effects of the bug

Cassandra has a number of tools to understand what’s happening internally as it serves data to clients. A lot of this is tracked as metrics in monitoring tools such as Prometheus or Grafana which integrate with Fallout. Checking those metrics showed that request throughput (requests per second) dropped off to near zero whenever we triggered the bug. To understand what was happening on the server side, I waited for the bug to occur and then took a look at the output of nodetool tpstats.

Using nodetool tpstats for diagnosing bugs is much easier if you have a mental model of how requests are handled by Cassandra. Native-Transport-Requests threads are responsible for handling requests from clients and they are run by the SEPWorker mechanism which is the thread pool framework inside of Cassandra. Cassandra uses a pool of threads and wakes up threads when all the running threads are busy and there’s still work to be done. When there’s no work to do, threads are put to sleep again until more work comes in.

From the output of tpstats I could see that there was plenty of work to do because the number of pending requests was high but the number of active requests at any one time hovered around two or three. To put this into perspective, there were hundreds of requests and around 172 Native-Transport-Requests threads on the machine but most of those threads were sleeping instead of handling the incoming requests from clients. Something was preventing Cassandra from waking up more Native-Transport-Requests threads.

The SEPWorker infrastructure has a mechanism for only waking up the minimum number of threads necessary when work comes in. If the currently running threads finish processing all of their work they enter the SPINNING state which is part of the internal Cassandra state machine. In this state, threads will check for available work and then sleep for a short period of time if there isn’t any. When they wake up they do one more check for work and if there still isn’t any work they go to sleep until explicitly awakened by another thread. If there is work, then before processing it they check to see whether they took the last work item and wake up another thread to process the next one if there’s still outstanding work.

Knowing all of this and combining that knowledge with the tpstats output I decided to see what all of the Native-Transport-Requests threads were doing by looking at their callstacks with jstack. Most of the threads were sleeping just like I expected based on the tpstats output, and a few were busily executing work. But one had this interesting callstack:

"Native-Transport-Requests-2" #173 daemon prio=5 os_prio=0 cpu=462214.94ms elapsed=19374.32s tid=0x00007efee606eb00 nid=0x385d waiting on condition  [0x00007efec18b9000]
4   java.lang.Thread.State: TIMED_WAITING (parking)
5 at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
6 at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:357)
7 at org.apache.cassandra.concurrent.SEPWorker.doWaitSpin(SEPWorker.java:268)

This showed that Native-Transport-Requests-2 was currently in the SPINNING state and sleeping prior to checking for more work one last time before sleeping permanently. The only problem was, no matter how many times I ran jstack, this thread never exited parkNanos() which meant no other threads would ever be woken up to help process the backlog of work!

This finally explained why throughput dropped whenever we hit the bug. The next step was understanding how parkNanos() was behaving.

Understanding the parkNanos() implementation

First of all, I wanted to make sure that Cassandra wasn’t somehow passing a really big value to parkNanos() which could make it appear that it was blocked indefinitely when in actual fact it was just sleeping for a very long time. Now that I had more of an idea of what to look for, it was easy to verify the values passed to parkNanos() by printing a log message every time this code path was executed. All of the values looked within the expected range so I concluded that we definitely weren’t calling parkNanos() incorrectly.

Next, I turned my attention to the implementation of parkNanos(). parkNanos() uses the timeout feature of pthread’s conditional variables to sleep for a specified time. Internally, pthread uses the kernel to provide the timeout facility, specifically by using the futex() system call. As a quick recap, the futex() (or fast userspace mutex) system call allows application threads to sleep, waiting for the value at a memory address to change. The thread is woken up when either the value changes or the timeout expires.

The parkNanos() call was putting the Native-Transport-Requests thread to sleep expecting the timer to expire within 10 microseconds or so but it was starting to look like that never happened. Two questions came to mind: did the thread somehow miss the signal to wake up from the expiring timer or did the timer never actually expire? I couldn’t see anything obviously wrong with the code, and I didn’t want to deal with building a custom version of the Java SE Development Kit (JDK) to pursue that line further. Instead, I decided to switch gears and write a BPF script to explore the problem from the kernel side first

Tracing kernel timers with BPF

BPF is an in-kernel virtual machine that’s capable of dynamically running kernel code injected from userspace. The beauty of BPF is that you can run these scripts without needing to recompile your kernel or even reboot your machine. Among other things, BPF allows you to insert probes or hooks at the entry and exit of kernel functions, effectively allowing you to run your own kernel code. Tracing through the kernel’s futex code I could see that if I hooked my script into the return path of the internal functions, I could display the timer details when the thread returned from the futex call. The next trick was getting the stuck thread to return.

Signals are a handy way of interrupting sleeping application threads that are inside the kernel and you can often send a SIGSTOP, SIGCONT sequence without triggering any error paths in the kernel or your application. So by sending these two signals I could force the sleeping Native-Transport-Requests thread to return from the blocking futex() call, run my BPF script and print the timer details in my terminal window.

The main timer detail I was interested in was the expired value which tells you when the timer should have expired and awakened the sleeping thread. If the value was far into the future it seemed likely that the timer had been programmed incorrectly, but if it was in the past then we’d missed the wakeup or it had never been sent. I bundled my BPF script into a bash script that also took care of first sending the SIGSTOP and SIGCONT signals to the Native-Transport-Requests thread.

$ sudo ./futex 14469
Tracing pid 14469…
Sending SIGSTOP
55880730777795 expires

The expires value for high-resolution timers in the kernel is in nanoseconds since boot. Doing a quick bit of math (5880730777795/1000000000 / 60 = 98.012 or 98 minutes of uptime) I could tell that the expire value looked correct because it was within the four hours that the test usually took. And the real issue was that a wakeup had never been sent to the sleeping Native-Transport-Requests thread. At this point, I was convinced that our test was hitting a kernel bug.

How are timers implemented in the Linux kernel?

High-resolution timers, or hrtimers, is the in-kernel infrastructure responsible for programming the timer hardware in your CPU. At the application level, hrtimers are programming using system calls such as clock_nanosleep() and nanosleep(). Each CPU maintains its own tree of hrtimers grouped by clock ID (CLOCK_MONOTONIC, CLOCK_REALTIME, etc). Each tree is implemented using a red-black tree data structure. Timers are inserted into the red-black tree and sorted based on their expiration time. The kernel’s red-black tree implementation also maintains a pointer to the smallest entry in the tree which enables the kernel to lookup the next expiring timer in O(1).

I modified my BPF script to display the location in the red-black tree for the hrtimer our Native-Transport-Requests thread was waiting on and discovered that it never reached the leftmost spot. This explained why the thread was never awakened! This is also a violation of the way that red-black trees are supposed to work: it’s normally expected that the entries in the tree are in sorted order, smallest to highest, at all times.

Tracing the behavior of the kernel’s red-black trees

Tracing the workings of these red-black trees was beyond anything I could achieve with a BPF script. I resorted to writing a kernel patch to print a warning message and dump the contents of the hrtimer red-black trees whenever the auxiliary pointer to the leftmost entry no longer pointed to the timer that expired next.

10: was adding node=00000000649f0970
10: found node in wrong place last=0000000091a72b7e, next=00000000649f0970 (expires 208222423748, 208222332591)
10: last timer=0000000091a72b7e function=hrtimer_wakeup+0x0/0x30, next=00000000649f0970 timer function=hrtick+0x0/0x70
Printing rbtree
=========
node=0000000091a72b7e, expires=208222423748, function=hrtimer_wakeup+0x0/0x30
node=00000000649f0970, expires=208222332591, function=hrtick+0x0/0x70
node=00000000063113c0, expires=208225793771, function=tick_sched_timer+0x0/0x80
node=000000003705886f, expires=209277793771, function=watchdog_timer_fn+0x0/0x280
node=00000000e3f371a2, expires=233222750000, function=timerfd_tmrproc+0x0/0x20
node=0000000068442340, expires=265218903415, function=hrtimer_wakeup+0x0/0x30
node=00000000785c2d62, expires=291972750000, function=timerfd_tmrproc+0x0/0x20
node=0000000085e65b06, expires=86608220562558, function=hrtimer_wakeup+0x0/0x30
node=00000000049a0b4d, expires=100000159000051377, function=hrtimer_wakeup+0x0/0x30

Before I started this part, I wasn’t even sure that Fallout would allow me to boot into a custom kernel. Sure enough, it was trivial to do.

After rebooting into my custom kernel I saw the warnings triggering immediately on startup so I naturally assumed that something was broken with my patch. But after a closer look, I realized that it was possible to trigger my bug within two minutes of booting the VM! I no longer had to wait up to four hours to see if I had reproduced the issue. What was even better was that I was able to hit this warning every single time I booted the VM. With the reproduction time cut by around 99%, I quickly made progress understanding the kernel bug that caused the red-black tree to become inconsistent.

The real bug and the fix

The kernel bug underlying this whole ordeal was already fixed upstream in February this year in Linux 5.12 but not yet pulled into Ubuntu’s kernel. What was happening was that some of the kernel’s scheduler code was modifying the expiration time of a hrtimer it had previously inserted into the red-black tree without first removing it from the tree. This violated the red-black tree property that all entries in the tree need to be sorted by their expiration time and resulted in hrtimers not firing. This meant that threads waiting on those timers were never woken up.

But why had no one else hit this kernel bug? The reason is because of a little-known or documented scheduler tunable that we were using. We were using the kernel’s HRTICK feature in order to improve scheduler latency. Unfortunately, it’s not widely used and a number of bugs have been fixed over the years. What’s worse is that HRTICK fixes are not being backported to stable kernels. So Linux distributions need to take them in an ad-hoc fashion. Rather than waiting for this to happen or rolling our own kernels, we decided to disable the feature altogether to avoid any future multi-week kernel debugging.

Conclusion

It’s helpful when debugging these multi-layered issues to have a bag of tools and techniques you can use to understand the behavior of your app at various levels of the stack. Even if you don’t know how to instrument the code at a particular point (or don’t want to), remember how I didn’t want to build a custom version of the JDK? You can still infer the location of a bug by using tools to understand the behavior above and below a layer. And when you don’t know exactly how a piece of code works, having a mental model or a rough idea of what goes on inside can help you to make forward progress by skipping layers that are unlikely to contain the bug.

Secondly, there is no way I could have taken on this problem without a service to automatically deploy and provision virtual machines for running the test. Fallout was the key to unlocking this bug because even though the bug could only be reproduced intermittently, I managed to parallelize the test runs and reduce the time to trigger the issue. The whole investigation took several weeks, and that’s despite my being able to analyze test results to make a little bit of progress every day.

If any of this sounds like the kind of problem you’d enjoy working on, check out the open roles on the DataStax careers page. We’re always on the lookout for tenacious developers and software engineers!

Follow the DataStax Tech Blog for more developer stories. Check out our YouTube channel for tutorials and here for DataStax Developers on Twitter for the latest news about our developer community.

https://gist.github.com/mfleming/7c6cfd225c305c0e44cc7357cdc481f3

Resources

Curious to learn more about (or play with) Cassandra itself? We recommend trying it on the Astra DB free plan for the fastest setup.

Save hours on your setup of K8ssandra with the new config builder

Pieter Humphrey — Thu, 24 Feb 2022 19:36:00 +0000

Setting up K8ssandra in your workflow just got a whole lot easier. With the new Config Builder you can be running Apache Cassandra® on Kubernetes in a matter of minutes.

The purpose of K8ssandra is to make it easy to run Apache Cassandra® on Kubernetes. We recently took another big step in that direction by releasing the K8ssandra config builder.

Even if you’ve created thousands of nodes or integrated K8ssandra in your stack, you’ll probably want to give the config builder a try. Walk through the interactive wizard defining the shape of your cluster, resource requirements, and toggle features to fit your needs. This is great for smoothing out the on-ramp for a production ready environment of K8ssandra.

Especially if you’re new(er) to K8ssandra, this builder will help you save several hours getting started in your environment. Choose one of the pre-defined templates as a starting point with cloud-specific options preconfigured.

As of this writing you can take your pick between a custom configuration and templates for Google Kubernetes Engine (GKE), Azure AKS, Digital Ocean, Amazon EKS or a local setup of K8ssandra.

Write hundreds of lines of code with a couple of clicks

As you may know K8ssandra is a platform for production deployments of Cassandra on Kubernetes. This includes everything you want to run alongside them like your monitoring system, repair process (Reaper) and backups.

However, the default configuration file is hundreds of lines. So we realized that we could make it even easier. With the K8ssandra Config Builder all you have to do is set your preferred parameters like:

Limits for deployments
Storage requirements
Size and number of nodes
Racks and logical datacenters

When writing your own YAML file you need to know the right values and sane limits. But with the config builder you don’t need to worry about that. Once you have the file, you run helm install and get a cluster running.

The config builder adjusts your settings to keep you within the limits that make sense. So you can’t request 32 GB of Heap for Cassandra if you only have 16 GB of RAM available due to your quotas.

When you are finished (or even at a stopping point), you can click copy or share to get going yourself or to share the configuration with a colleague to collaborate.

Figure 1. A single click in the K8ssandra Config Builder will give you the 52 line YAML file for the setup of three nodes in Amazon EKS.

Sure, there might still be instances where you want to write the YAML file yourself. But if you get stuck on something, the config builder is the fastest way for you to find the solution.

We’ll keep the builder up to date so you can always come back and get exactly what you need.

What’s next for the K8ssandra Config Builder

Our goal is to give you a rich experience building anything in K8ssandra. So we’re working on new features to be released for the K8ssandra config builder in the near future. Near the top of our to-do list are:

Linking to the Config Builder from the K8ssandra install documentation to the specific YAML file template
Syntax highlighting the code and line numbers
User experience improvements.

Of course our priorities depend on the feedback we get from the users of the config builder.

So don’t hesitate to tell us what you think. You can do so by joining the K8ssandra Discord or K8ssandra Forum today. For exclusive posts on all things data, follow DataStax on Medium.. We recommend trying it on the Astra DB free plan for the fastest setup.

Resources

Try out the K8ssandra Config Builder
Join the K8ssandra Community
Kubernetes and Apache Cassandra: What Works (and What Doesn’t)
DataStax Discord: Fellowship of the (Cassandra) Rings
K8ssandra documentation
Google Kubernetes Engine (GKE)
Azure Kubernetes Service (AKS)
Digital Ocean cloud infrastructure
Amazon Elastic Kubernetes Service (EKS)