DEV Community: Jay Clifford

Tutorial: Modifying Grafana's Source Code

Jay Clifford — Mon, 07 Aug 2023 10:25:41 +0000

A story of exploration and guesswork.

So this blog is a little different from my usual tutorials…

A little background: I have been working with Jacob Marble to test and “demo-fy” his work with InfluxDB 3.0 and the OpenTelemetry ecosystem (If you would like to learn more I highly recommend checking out this blog).

During the project, we identified a need to enable specific Grafana features for InfluxDB data sources, particularly the trace to logs functionality. Grafana is an open-source platform, and one of its major advantages is the ability to modify its source code to suit our unique requirements. However, diving into the codebase of such a robust tool can be overwhelming, even for the most seasoned developers.

Despite the complexity, we embraced the challenge and dove headfirst into Grafana's source code. We tumbled, we stumbled, and we learned a great deal along the way. And now, having successfully modified Grafana to meet our specific project needs, I believe it's time to share this acquired knowledge with you all.

The purpose of this blog is not just to provide you with a step-by-step guide for tweaking Grafana's source code, but also to inspire you to explore and adapt open-source projects to your needs. It's about imparting a method and a mindset, cultivating a culture of curiosity, and encouraging more hands-on learning and problem-solving.

I hope that this guide will empower you to modify Grafana's source code for your projects, thereby expanding the horizons of what's possible with open-source platforms. It’s time to roll up your sleeves and venture into the depths of Grafana's code.

The problem

So our problem lies within the Trace visualisation of Grafana.

As you can see the visualisation performs rather well with InfluxDB except for one button which appears to be disabled:** Logs for this span**. This button is automatically disabled when our trace data source (in this case, Jaeger with InfluxDB 3.0 acting as the gRPC storage engine) has not been configured with a log data source. A log data source within Grafana is usually represented by default using the log explorer interface, common log data sources are; Loki, OpenSearch and ElasticSearch. So let's head across to the Jaeger data source and configure that…

Data sources can be navigated to via Connections -> Data Sources. We currently have three configured; FlightSQL, InfluxDB and Jaeger. If we open the Jaeger configuration and navigate to the Trace to Logs section we want to essentially be able yo select either InfluxDB or FlightSQL as our Data source.

Houston, we have a problem. It appears Grafana doesn’t recognise InfluxDB as a log data source. Fair enough; only recently has InfluxDB become a viable option for logs. So, what are our options?

We lie down, accept the issue, and hope that in the future this feature becomes generic enough to support more data sources.
Take action and make the change ourselves.

Well, by now you know what option we chose.

The Solution

In this section, I will summarize the steps I took to discover what changes needed to be made. How to implement the changes for your own data source and finally how to build your own custom build of Grafana OSS.

Discovery

So the first step is to understand where to even begin. Grafana is a huge Open Source platform with many components so I needed to narrow down the search. So the first thing I did was search the Grafana repository for signs of life.

As you can see I made this little discovery by using the keyword trace. Which led me to the directory TraceToLogs. This led me to this section of code within TraceToLogsSettings.tsx:


export function TraceToLogsSettings({ options, onOptionsChange }: Props) {

  const supportedDataSourceTypes = [

    'loki',

    'elasticsearch',

    'grafana-splunk-datasource', // external

    'grafana-opensearch-datasource', // external

    'grafana-falconlogscale-datasource', // external

    'googlecloud-logging-datasource', // external

  ];

This section of code seems to create a static list of data sources supported by the Trace to Logs feature. We can confirm this by some of the common suspects within the list (Loki, Elasticsearch, etc.). Based on this our first alteration to the Grafana source code should be to add our data sources to this list.

Now as the coding pessimist that I am, I knew this probably wouldn’t be the only change we needed to make but it's a good place to start. So I did the following:

I forked the Grafana repo

Cloned the repo:


git clone https://github.com/InfluxCommunity/grafana

Before I made those modifications I wanted to do some more searching to see if there are any changes I should be making. One line stood out to me in TraceToLogsSettings file:


  const updateTracesToLogs = useCallback(

    (value: Partial&lt;TraceToLogsOptionsV2>) => {

      // Cannot use updateDatasourcePluginJsonDataOption here as we need to update 2 keys, and they would overwrite each

      // other as updateDatasourcePluginJsonDataOption isn't synchronized

      onOptionsChange({

        ...options,

        jsonData: {

          ...options.jsonData,

          tracesToLogsV2: {

            ...traceToLogs,

            ...value,

          },

          tracesToLogs: undefined,

        },

      });

    },

    [onOptionsChange, options, traceToLogs]

  );

It was TraceToLogsOptionsV2. When I searched for places this interface was used I found the following entry.

So it appears we might also have work to do in the createSpanLink.tsx file. Within I found this section of code, so my question was what exactly is this code doing?

To cut a long story short the case statement essentially tells the trace visualisation to check what log data source has been defined (if any) and to define a query interface relevant to that data source. If the specified data source is not found within this case statement then the button is simply disabled which meant changing the original file won’t be enough as we suspected.

Okay, we have now completed our investigation. Let's move on to the code changes.

Modification

We have two files to modify:

TraceToLogsSettings.tsx
createSpanLink.tsx

Let's start with the simplest to tackle and go from there.

TraceToLogsSettings

So this file was relatively simple to change. All we needed to do was modify the static list of supported log input sources like so:


export function TraceToLogsSettings({ options, onOptionsChange }: Props) {

  const supportedDataSourceTypes = [

    'loki',

    'elasticsearch',

    'grafana-splunk-datasource', // external

    'grafana-opensearch-datasource', // external

    'grafana-falconlogscale-datasource', // external

    'googlecloud-logging-datasource', // external

    'influxdata-flightsql-datasource', // external

    'influxdb', // external

  ];

As you can see I have added two. I ran a quick build of the Grafana project to see how this affected our data source configuration (we will discuss how to build at the end).

Hey presto! We have a result. Now this still didn’t enable the button within our Trace View but we already knew this would requiremore work.

createSpanLink

Now, onto the meat of our modification. For the record, I am not a TypeScript developer. What I do know is that the file has a whole bunch of examples we can use to attempt a blind copy-and-paste job with a few modifications. I ended up doing this for both plugins but to keep the blog short we will focus on the InfluxDB official plugin.

My hypothesis was to use the Grafana Loki interface as the basis for the InfluxDB interface. The first included adding data source types:


import { LokiQuery } from '../../../plugins/datasource/loki/types';

import { InfluxQuery } from '../../../plugins/datasource/influxdb/types';

These are easy to locate when Grafana has an official plugin for your data source since it's embedded within the official repository. For our community plugin I had two options: define a static interface within the file or provide more query parameters. I chose the latter.

The next step was to modify the case statement:


   // TODO: This should eventually move into specific data sources and added to the data frame as we no longer use the

    //  deprecated blob format and we can map the link easily in data frame.

    if (logsDataSourceSettings && traceToLogsOptions) {

      const customQuery = traceToLogsOptions.customQuery ? traceToLogsOptions.query : undefined;

      const tagsToUse =

        traceToLogsOptions.tags && traceToLogsOptions.tags.length > 0 ? traceToLogsOptions.tags : defaultKeys;

      switch (logsDataSourceSettings?.type) {

        case 'loki':

          tags = getFormattedTags(span, tagsToUse);

          query = getQueryForLoki(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'grafana-splunk-datasource':

          tags = getFormattedTags(span, tagsToUse, { joinBy: ' ' });

          query = getQueryForSplunk(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'influxdata-flightsql-datasource':

            tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });

            query = getQueryFlightSQL(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'influxdb':

            tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });

            query = getQueryForInfluxQL(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'elasticsearch':

        case 'grafana-opensearch-datasource':

          tags = getFormattedTags(span, tagsToUse, { labelValueSign: ':', joinBy: ' AND ' });

          query = getQueryForElasticsearchOrOpensearch(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'grafana-falconlogscale-datasource':

          tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });

          query = getQueryForFalconLogScale(span, traceToLogsOptions, tags, customQuery);

          break;

        case 'googlecloud-logging-datasource':

          tags = getFormattedTags(span, tagsToUse, { joinBy: ' AND ' });

          query = getQueryForGoogleCloudLogging(span, traceToLogsOptions, tags, customQuery);

      }

As you can see I added two new cases:** influxdata-flightsql-datasource and influxdb. I then copied from Loki the two function calls within the case: getFormattedTags and getQueryFor. It appeared one I could leave alone (getFormattedTags) as this seemed to be the same for the majority of the cases. However,I would need to define my own getQueryFor function.

Let's take a look at the new getQueryForInfluxQL function that’s called in the influxdb case statement:


function getQueryForInfluxQL(

  span: TraceSpan,

  options: TraceToLogsOptionsV2,

  tags: string,

  customQuery?: string

): InfluxQuery | undefined {

  const { filterByTraceID, filterBySpanID } = options;

  if (customQuery) {

    return {

      refId: '',

      rawQuery: true,

      query: customQuery,

      resultFormat: 'logs',

    };

  }

  let query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE time >=${__from}ms AND time &lt;=${__to}ms';

  if (filterByTraceID && span.traceID && filterBySpanID && span.spanID) {

            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time &lt;=${__to}ms';

    } else if (filterByTraceID && span.traceID) {

            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND time >=${__from}ms AND time &lt;=${__to}ms';

    } else if (filterBySpanID && span.spanID) {

            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time &lt;=${__to}ms';

  }

  return {

    refId: '',

    rawQuery: true,

    query: query,

    resultFormat: 'logs',

  };

}

So there is quite a lot here, but let me highlight the important parts. First of all, I started with an exact copy of the Loki function. Then, I made the following changes:

I changed the return interface from ‘LokiQuery | undefined‘ to ‘ InfluxQuery | undefined‘ –the data source type we imported earlier.
Next, I focused on the return payload. After some digging within the InfluxQuery type file, I came up with this:
```
return {

    refId: '',

    rawQuery: true,

    query: query,

    resultFormat: 'logs',

  };
```
The InfluxDB data source had a hany parameter which allowed me to define the result format (usually metrics) and also I now knew the data source would be expecting a raw query rather than an expression.
Lastly, I had to define the queries which would run when the user clicked the button. These depended on what filter features the user had toggled within the data source settings (filter by traceID, spanID or both). So I modified the if statement defined within the Loki function and constructed static InfluxQL queries. From there, I then used the Grafana placeholder variables found within other data sources to make the queries dynamic. Here is an example:
```
if (filterByTraceID && span.traceID && filterBySpanID && span.spanID) {

            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time &lt;=${__to}ms';
```
Full disclosure, it took me a good minute to find out about the **>=${from}ms and <=${to}ms. This ended up being a brute force build and error case.

Building

Phew! We’re past the hard bit. Now onto the build process. I have quite a few years of experience with Docker, so this part was stress-free for me, but I imagine it could be daunting for new Docker users. Luckily, Grafana has some easy-to-follow documentation for the task. To paraphrase, these are the steps:

Run the following build command (this can take a while and make sure your docker VM has enough memory if using macOS or Windows)
```
make build-docker-full  
```
The build process produces a Docker image called: grafana/grafana-oss:dev . We could just use this image, but as a formality, I like to retag the image and push it to my Docker registry.
```
docker tag grafana/grafana-oss:dev jaymand13/grafana-oss:dev2

docker push jaymand13/grafana-oss:dev2
```

This way I have checkpoints when I am brute forcing changes.

There we have it! A fully baked Grafana dev image to try out with our changes.

The results and conclusion

So after investigating, making the changes, and building our new Grafana container, let's take a look at our results:

With our changes, the Logs for this span button now activates. We also have this neat little Log button that appears next to each span. A confession: the blue Logs for this span button currently only works within the Grafana Explorer tab, but the new Log link works within our dashboard. To quickly explain the differences; Grafana Dashboards are custom-built by users and can include 1 or many data sources with a variety of different visualisations. Data Explorers on the other hand provide an interface for drill-down and investigation activities like you see in the above screenshot. Still, it’s not a huge problem compared to how little we needed to change to get here.

And so, we've reached the end of our dive into the intricacies of modifying Grafana's source code. Over the course of this tutorial, I hope you've not only gained a practical understanding of how to customize Grafana for your specific requirements, but also an appreciation for the flexibility and potential of open-source platforms in general.

Remember, in the realm of open-source, there's no limit to how much we can tweak, adjust, and reimagine to suit our needs. I hope this guide serves you well as you delve deeper into your own projects, and that it brings you one step closer to mastering the powerful tool that is Grafana. For me, my journey continues as I now plan to add exemplar support to this OSS build. If you would like to try this out yourself you can find the OpenTelemetry example here.

Client Library Deep Dive: Python (Video Tutorial)

Jay Clifford — Thu, 03 Aug 2023 09:46:03 +0000

To round the first part of the series off I thought I would include a video tutorial using the new Python client for InfluxDB 3.0.

Reflection

Its been a wild ride so far...

Learning the ropes of Pyarrow, Apache Flight and Parquet. When I set out on this journey I didn't imagine that would mean building and maintaining a Client library for InfluxDB 3.0 as a Developer Advocate. I must admit it was a refreshing feeling acting like a full time developer again.

I am excited to continue this journey learning different parts of the Apache Ecosystem within the columnar space, as InfluxDB 3.0's use cases expand so do mine.

Contribute

I would love to hear your thoughts on the client library; use it in anger, submit feature requests, write examples or even build features. A community project is only ever as good as the community around it.

Here is the repo: "client lib repo"

Client Library Deep Dive: Python (Part 2)

Jay Clifford — Thu, 03 Aug 2023 09:30:45 +0000

Working with the new InfluxDB 3.0 Python CLI and Client Library

By Jay Clifford

Okay, we are back for Part 2! Last time we discussed the new community Python library for InfluxDB 3.0. Let's talk about a bolt-on application that uses the client library as the core of its development, the InfluxDB 3.0 Python CLI.

Python CLI

Okay, so following the same format as before, what were the reasons for building the CLI? Well, there are two primary reasons:

We wanted to give users a data browsing tool that leveraged the new Flight endpoint. Python gave us the opportunity to prototype fast before we invested work in a more robust CLI offering. It also allowed us to leverage some interesting data manipulation libraries that could extend the scope of the Python CLI.
We wanted a robust way to test the newly created InfluxDB 3.0 Python Client library, as you will see most of the tooling and functionality in use.

Install

Let's talk about the installation process because, I must admit, Python doesn’t provide the most user-friendly packaging and deployment methods unless you use it daily. I recommend installing the CLI in a Python Virtual Environment first for test purposes:


$ python3 -m venv ./.venv 
$ source .venv/bin/activate
$ pip install –upgrade pip

$ pip install influxdb3-python-cli

This set of commands creates our Virtual Python Environment, activates it, updates our Python package installer, and finally installs the new CLI.

If you would like to graduate from a Python Virtual Environment and move the CLI to your path, you can do so with a sudo install (You have to be careful here not to cause permission issues with packages):


sudo python3 -m pip install influxdb3-python-cli

Creating a CLI config

The first thing you want to do is create a connection config. This feature acts like the current InfluxDB influx CLI by saving your connection credentials for InfluxDB to use later.


influx3 create config \

--name="poke-dex" \

--database="pokemon-codex" \

--host="us-east-1-1.aws.cloud2.influxdata.com" \

--token="&lt;your token>" \

--org="&lt;your org ID>"

–name	Name to describe your connection config. This must be unique.
–token	This provides authentication for the client to read and write from InfluxDB Cloud Serverless or Dedicated. Note: you need a token with read and write authentication if you wish to use both features.
–host	InfluxDB host — this should only be the domain without the protocol (https://)
–org	Cloud Serverless still requires a user’s organization ID for writing data to 3.0. Dedicated users can just use an arbitrary string.
–database	The database you wish to query from and write to.

Config commands

Config commands also exist to activate, update, delete, and list current active configs:

```bash influx3.py config update --name="poke-dex" --host="new-host.com" ```	The update subcommand updates an existing configuration. The --name parameter is required to specify which configuration to update. All other parameters (--host, --token, --database, --org, --active) are optional.
```bash influx3.py config use --name="poke-dex" ```	The use subcommand sets a specific configuration as the active one. The --name parameter is required to specify which configuration to use.
```bash influx3.py config delete --name="poke-dex" ```	The delete subcommand deletes a configuration. The --name parameter is required to specify which configuration to delete.
```bash influx3.py config list ```	The list subcommand lists all the configurations.

Writing and querying

You can use the CLI to either directly call the application, followed by the commands you wish to run, or run it through an interactive REPL. I personally believe the REPL approach provides a better flow, so let’s demo some of the features.

Once you created your config you simply enter the following to activate the REPL:


influx3

Which leads to:


influx3

InfluxDB 3.0 CLI.

(>)

Query

Let’s first take a look at the query options. Within the REPL you have 3 query options: SQL, InfluxQL, and chatGPT (more on this later). Let’s drop into the SQL REPL and run a basic query against the Trainer data we generated in the previous blog:


InfluxDB 3.0 CLI.

(>) sql

(sql >) SELECT * FROM caught

Now I wouldn’t normally recommend querying without some form of time-based WHERE clause, but I wanted to highlight how the CLI can handle large datasets. It uses ‘mode = chunk’ from the Python Client Library to break large datasets into manageable Arrow batches. From there we have three options.

We can either hit **TAB **to see the next portion of data, if one exists.
Press F to save the current Arrow batch to a file type of our choosing (JSON, CSV, Parquet, ORC, Feather).
Press **CTRL-C **to return back to the SQL REPL.

Let’s take a look at the option 2:


| 3961 |       82 | Venusaur                  |        83 |   80 | 0003 |      12 |     7 |      80 | 2023-07-06 13:41:36.588000 | ash       | Grass    | Poison   |

| 3962 |       64 | Dratini                   |        45 |   41 | 0147 |       6 |     7 |      50 | 2023-07-06 14:30:32.519000 | jessie    | Dragon   |          | 



Press TAB to fetch next chunk of data, or F to save current chunk to a file

Enter the file name with full path (e.g. /home/user/sample.json): ~/Desktop/all-trainer-data.csv

Data saved to ~/Desktop/all-trainer-data.csv.

Here is a sample of the CSV file created:


"attack","caught","defense","hp","id","level","num","speed","time","trainer","type1","type2"

49,"Bulbasaur",49,45,"0001",12,"1",45,2023-07-06 14:30:41.886000000,"ash","Grass","Poison"

62,"Ivysaur",63,60,"0002",7,"1",60,2023-07-06 14:30:32.519000000,"ash","Grass","Poison"

62,"Ivysaur",63,60,"0002",8,"1",60,2023-07-06 14:30:38.519000000,"ash","Grass","Poison"

Once we reach the end of our dataset, it prompts us to press ENTER **to drop back into the SQL REPL. Just remember if you feel like you’re pressing **TAB **forever, you can always drop out of the query with **CTRL-C.

Now, let’s look at a more interesting example with the InfluxQL REPL:


(sql >) exit

(>) influxql

(influxql >) SELECT count(caught) FROM caught WHERE time > now() - 2d GROUP BY trainer

|    | iox::measurement   | time                | trainer   |   count |

|---:|:-------------------|:--------------------|:----------|--------:|

|  0 | caught             | 1970-01-01 00:00:00 | ash       |     625 |

|  1 | caught             | 1970-01-01 00:00:00 | brock     |     673 |

|  2 | caught             | 1970-01-01 00:00:00 | gary      |     645 |

|  3 | caught             | 1970-01-01 00:00:00 | james     |     664 |

|  4 | caught             | 1970-01-01 00:00:00 | jessie    |     663 |

|  5 | caught             | 1970-01-01 00:00:00 | misty     |     693 | 
 \
(influxql >) SELECT count(caught) FROM caught WHERE time > now() - 2d  GROUP BY time(1d),trainer ORDER BY time

|    | iox::measurement   | time                | trainer   |   count |

|---:|:-------------------|:--------------------|:----------|--------:|

|  0 | caught             | 2023-07-05 00:00:00 | ash       |     nan |

|  1 | caught             | 2023-07-06 00:00:00 | ash       |     625 |

|  2 | caught             | 2023-07-07 00:00:00 | ash       |     148 |

|  3 | caught             | 2023-07-05 00:00:00 | brock     |     nan |

|  4 | caught             | 2023-07-06 00:00:00 | brock     |     673 |

|  5 | caught             | 2023-07-07 00:00:00 | brock     |     180 |

|  6 | caught             | 2023-07-05 00:00:00 | gary      |     nan |

|  7 | caught             | 2023-07-06 00:00:00 | gary      |     645 |

|  8 | caught             | 2023-07-07 00:00:00 | gary      |     155 |

|  9 | caught             | 2023-07-05 00:00:00 | james     |     nan |

| 10 | caught             | 2023-07-06 00:00:00 | james     |     664 |

| 11 | caught             | 2023-07-07 00:00:00 | james     |     157 |

| 12 | caught             | 2023-07-05 00:00:00 | jessie    |     nan |

| 13 | caught             | 2023-07-06 00:00:00 | jessie    |     663 |

| 14 | caught             | 2023-07-07 00:00:00 | jessie    |     144 |

| 15 | caught             | 2023-07-05 00:00:00 | misty     |     nan |

| 16 | caught             | 2023-07-06 00:00:00 | misty     |     693 |

| 17 | caught             | 2023-07-07 00:00:00 | misty     |     178 |

We will save this one as a Parquet file for later.

Write

Moving on from using the CLI for querying, let’s talk about the write functionality. Now, this feature set isn’t as fleshed out as I would like it to be but it covers the basics. We can drop into the write REPL and write data to InfluxDB using line protocol like so:


(influxql >) exit

(>) write

(write >) caught,id=0115,num=1,trainer=brock attack=125i,caught="KangaskhanMega Kangaskhan",defense=100i,hp=105i,level=13i,speed=100i,type1="Normal" 1688741473083000000

Next let’s have a look at the write_file feature. For this we need to drop out of the REPL entirely and use flag commands when calling ‘influx3’. Let’s load our count results into a new table:


(write >) exit

(>) exit

Exiting …

influx3 write_file --help

usage: influx3 write_file [-h] --file FILE [--measurement MEASUREMENT] --time TIME [--tags TAGS]

options:

  -h, --help            show this help message and exit

  --file FILE           the file to import

  --measurement MEASUREMENT

                        Define the name of the measurement

  --time TIME           Define the name of the time column within the file

  --tags TAGS           (optional) array of column names which are tags. Format should be: tag1,tag2

influx3 write_file --file ~/Desktop/count.parquet --time time --tags trainer --measurement summary

Here is the result:


(influxql >) SELECT count, trainer, time  FROM summary

|    | iox::measurement   | time                |   count | trainer   |

|---:|:-------------------|:--------------------|--------:|:----------|

|  0 | summary            | 2023-07-05 00:00:00 |     nan | ash       |

|  1 | summary            | 2023-07-05 00:00:00 |     nan | brock     |

|  2 | summary            | 2023-07-05 00:00:00 |     nan | gary      |

|  3 | summary            | 2023-07-05 00:00:00 |     nan | james     |

|  4 | summary            | 2023-07-05 00:00:00 |     nan | jessie    |

|  5 | summary            | 2023-07-05 00:00:00 |     nan | misty     |

|  6 | summary            | 2023-07-06 00:00:00 |     625 | ash       |

|  7 | summary            | 2023-07-06 00:00:00 |     673 | brock     |

|  8 | summary            | 2023-07-06 00:00:00 |     645 | gary      |

|  9 | summary            | 2023-07-06 00:00:00 |     664 | james     |

| 10 | summary            | 2023-07-06 00:00:00 |     663 | jessie    |

| 11 | summary            | 2023-07-06 00:00:00 |     693 | misty     |

| 12 | summary            | 2023-07-07 00:00:00 |     148 | ash       |

| 13 | summary            | 2023-07-07 00:00:00 |     180 | brock     |

| 14 | summary            | 2023-07-07 00:00:00 |     155 | gary      |

| 15 | summary            | 2023-07-07 00:00:00 |     157 | james     |

| 16 | summary            | 2023-07-07 00:00:00 |     144 | jessie    |

| 17 | summary            | 2023-07-07 00:00:00 |     178 | misty     |

Experimental feature (ChatGPT)

So with chatGPT and OpenAI being all the rage these days, I looked to see if their Python package could benefit the CLI. Interestingly it does… Because InfluxDB has been open source since its inception, chatGPT has become pretty well-versed in building InfluxQL queries. Take a look at this example:


(chatgpt >) give me a list of the top 10 caught with an attack higher than 100 from caught

Run InfluxQL query: SELECT * FROM caught WHERE attack > 100 LIMIT 10

|    | iox::measurement   | time                       |   attack | caught                    |   defense |   hp |   id |   level |   num |   speed | trainer   | type1    | type2   |

|---:|:-------------------|:---------------------------|---------:|:--------------------------|----------:|-----:|-----:|--------:|------:|--------:|:----------|:---------|:--------|

|  0 | caught             | 2023-07-06 13:09:36.095000 |      110 | Dodrio                    |        70 |   60 | 0085 |      19 |     1 |     100 | jessie    | Normal   | Flying  |

|  1 | caught             | 2023-07-06 13:09:36.095000 |      125 | Pinsir                    |       100 |   65 | 0127 |       6 |     1 |      85 | brock     | Bug      |         |

|  2 | caught             | 2023-07-06 13:10:53.995000 |      130 | CharizardMega Charizard X |       111 |   78 | 0006 |       6 |     1 |     100 | brock     | Fire     | Dragon  |

|  3 | caught             | 2023-07-06 13:10:53.995000 |      150 | BeedrillMega Beedrill     |        40 |   65 | 0015 |      12 |     1 |     145 | jessie    | Bug      | Poison  |

|  4 | caught             | 2023-07-06 13:10:53.995000 |      102 | Nidoking                  |        77 |   81 | 0034 |      20 |     1 |      85 | gary      | Poison   | Ground  |

|  5 | caught             | 2023-07-06 13:10:53.995000 |      105 | Primeape                  |        60 |   65 | 0057 |      16 |     1 |      95 | misty     | Fighting |         |

|  6 | caught             | 2023-07-06 13:10:53.995000 |      120 | Golem                     |       130 |   80 | 0076 |       8 |     1 |      45 | ash       | Rock     | Ground  |

|  7 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |       5 |     1 |      50 | brock     | Poison   |         |

|  8 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |      19 |     1 |      50 | james     | Poison   |         |

|  9 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |      16 |     2 |      50 | james     | Poison   |         |

This feature currently only uses ChatGPT 3.5 and requires an OpenAPI token. If you would like instructions on how to use this feature, check out this part of the README.

Future hopes

The future is bright for the Python CLI as our development team pushes forward with tooling for InfluxDB 3.0. For now, the scope is to keep it as a bolt-on tool for Python developers and those who want an easily extendable CLI. Here is my current laundry list for the project:

Feature	Status
Improve OpenAI functionality: Upgrade to chatgpt 4 Add call functions Extend to SQL	TO DO
Find a better way to package and distribute. Currently looking into Pyinstaller as an option.	TO DO
Extended write functionality.	TO DO
Provide post query exploration support (Pandas functions)	TO DO
Integrate delta sharing	TO DO

Wrapping up

So there you have it, Part 2 done and dusted. I really enjoyed writing this blog series on both the Python Client Library and CLI. Having such a heavy hand in the inspection makes writing about them far more exciting and easy. I hope these blogs inspire you to join our new community-based libraries and tooling. If you want to chat about how to get involved, you can reach me via Slack or Discourse.

Client Library Deep Dive: Python (Part 1)

Jay Clifford — Thu, 03 Aug 2023 09:17:22 +0000

Working with the new InfluxDB 3.0 Python CLI and Client Library

By Jay Clifford

Community Client libraries are back with InfluxDB 3.0. If you would like an overview of each client library then I highly recommend checking out Anais’s blog on their status.

In this two-part blog series, we do a deep dive into the new Python Client Library and CLI. By the end, you should have a good understanding of the current features, how the internals work, and my future ideas for both projects. From there my hope is that it gives you the opportunity to contribute to, and have your say in their future.

In this post (Part 1), we will focus primarily on the Client Library because it underlies the Python CLI.

Python client library

So, let's start off with the Python client library. The scope was simple: build a library that could write to and query InfluxDB 3.0. Because the write endpoint didn’t change inInfluxDB 3.0, we could bring forward much of the functionality from the V2 library, such as batch writes, data parsing, point objects, and much more. However, on the query side of things, we had to completely remake it. We wanted to focus on the capabilities of Arrow Flight and bring support for both SQL and InfluxQL-based queries. PyArrow also opened up better ecosystem support for libraries such as Pandas and Polars, but I’ll have more on this later.

Let's build a simple Python application together that writes and queries InfluxDB 3.0.

Install

To install the client library (I recommend making a Python Virtual Environment first):


$ python3 -m venv ./.venv \
$ source .venv/bin/activate \
$ pip install –upgrade pip

$ pip install influxdb3-python

This set of commands creates our Virtual Python Environment, activates it, updates our Python package installer, and, finally, installs the new client library.

Creating a client

In this section, we import our newly installed library and establish a client. I also discuss some configuration parameters and the reasoning behind them.

Let's create a main.py file with the following code:


from influxdb_client_3 import InfluxDBClient3, Point

import pandas as pd

import numpy as np

import datetime

client = InfluxDBClient3( token="",

    host="eu-central-1-1.aws.cloud2.influxdata.com",

    org="6a841c0c08328fb1",

    database="pokemon-codex")

This example shows a minimal configuration for the client. Like previous clients, it requires the following parameters:

token	This provides authentication for the client to read and write from InfluxDB Cloud Serverless or Dedicated. Note: you need a token with read-and-write authentication if you wish to use both features.
host	InfluxDB host — this should only be the domain without the protocol (https://)
org	Cloud Serverless still requires the users’ organization ID for writing data to 3.0. Dedicated users can just use an arbitrary string.
database	The database you wish to query and write from.

I recommend creating a client on a per-database basis, though you can update the _database instance variable if you only want to create one client.

Next, let's take a look at the advanced parameters of the client:

flight_client_options	This provides access to parameters for the flight query protocol. You can find configuration options here. Example.
write_client_options	This provides access to the parameters used by the V2 write client, which you can find here. Example.
**kwargs	Lastly, this provides access to the parameters used by the V2 client, which you can find here. Example. (gzip compression)

Let's continue our original example by discussing the write functionality.

Writing data

So now that we established our client, in this section we look at the different methods you can use to write data to InfluxDB 3.0. Most will be familiar to you as they follow the same ingestion method as V2.

Let's start off with basic point building:


# Continued from the Client's example \
 \
now = datetime.datetime.now(datetime.timezone.utc)

data = Point("caught").tag("trainer", "ash").tag("id", "0006").tag("num", "1")\

                                             .field("caught", "charizard")\

                                             .field("level", 10).field("attack", 30)\

                                             .field("defense", 40).field("hp", 200)\

                                             .field("speed", 10)\

                                             .field("type1", "fire").field("type2", "flying")\

                                             .time(now)

try:

    client.write(data)

except Exception as e:

    print(f"Error writing point: {e}")

In this example, you can see we build our line protocol using an instance of the Point class, which then translates into line protocol: \


Point,trainer=ash,id=0006,num=1 caught="charizard",level=10i,attack=30i,defense=40i,hp=200i,speed=10i,type1="fire",type2="flying" &lt;timestamp>

You can also format this as an array of points: \


data = []

# Adding first point

data.append(

    Point("caught")

    .tag("trainer", "ash")

    .tag("id", "0006")

    .tag("num", "1")

    .field("caught", "charizard")

    .field("level", 10)

    .field("attack", 30)

    .field("defense", 40)

    .field("hp", 200)

    .field("speed", 10)

    .field("type1", "fire")

    .field("type2", "flying")

    .time(now)

)

# Adding second point

data.append(

    Point("caught")

    .tag("trainer", "ash")

    .tag("id", "0007")

    .tag("num", "2")

    .field("caught", "bulbasaur")

    .field("level", 12)

    .field("attack", 31)

    .field("defense", 31)

    .field("hp", 190)

    .field("speed", 11)

    .field("type1", "grass")

    .field("type2", "poison")

    .time(now)

)

You can also write via dictionary encoding and structured data methods. One of my favorite ingest methods is via Pandas DataFrame.

Let's take a look at an example utilizing this method:


# Convert the list of dictionaries to a DataFrame

caught_pokemon_df = pd.DataFrame(data).set_index('timestamp')

# Print the DataFrame

print(caught_pokemon_df)

try:

    client.write(caught_pokemon_df, data_frame_measurement_name='caught',

             data_frame_tag_columns=['trainer', 'id', 'num'])

except Exception as e:

    print(f"Error writing point: {e}")

This example creates a Pandas DataFrame of our caught Pokemon for this session. We set the index of our dataframe to the timestamp of when the Pokemon was caught and then provide the dataframe plus the following write parameters to the ‘write()’ function:

data_frame_measurement_name	The name of the measurement you wish to write your Pandas DataFrame into.
data_frame_tag_columns	A list of strings containing the column names you wish to make tags.
data_frame_timestamp_column	Use this parameter to set the timestamp column if your index is not set to the timestamp.

Make sure to check out the full example here. You can also find a batching example here.

Writing data from a file

A much-requested feature of the previous client library was more ways to upload and parse different file data formats. Leveraging the utilities of PyArrow, we can now support the upload of files in the following formats:

CSV	Example here.
JSON	Example here.
Feather	Example here.
ORC	Example here.
Parquet	Example here.

Querying data

Now that we wrote some data into InfluxDB 3.0, let’s talk about how to query it back out. 3.0 provides a fully supported Apache Arrow Flight endpoint, which allows users to query using SQL or InfluxQL.

Let's first take a look at a basic time series query in both SQL and InfluxQL;


from influxdb_client_3 import InfluxDBClient3

import pandas as pd

client = InfluxDBClient3(

    token="",

    host="eu-central-1-1.aws.cloud2.influxdata.com",

    org="6a841c0c08328fb1",

    database="pokemon-codex")

sql = '''SELECT * FROM caught WHERE trainer = 'ash' AND time >= now() - interval '1 hour' LIMIT 5'''

table = client.query(query=sql, language='sql', mode='all')

print(table)

influxql = '''SELECT * FROM caught WHERE trainer = 'ash' AND time  > now() - 1h LIMIT 5'''

table = client.query(query=influxql, language='influxql', mode='pandas')

print(table)

As you can see in this example we used the same client to query both with InfluxQL and SQL. Let's take a quick look at the query parameters to see how they shape our returned result.

query	This parameter currently accepts the string literal of your SQL or InfluxQL query. We hope to add prepared statements to this soon.
language	This parameter accepts a string literal of either ‘sql’ or ‘influxql’
mode	There are currently 5 return modes: ‘all’: this returns all the data queried as a PyArrow Table. ‘pandas’: Returns all data as a Pandas DataFrame ‘chunk’: Returns a flight reader so a user can iterate through large queries in smaller sample sizes (see example) ‘reader’: Attempts to convert the stream to a RecordBatchReader ‘schema’: returns the query payload schema

Future hopes

Rome wasn’t built in a day, and there are plenty of quality-of-life improvements and new features to add. Here is a table outlining a few: \

Feature	Status
Merge the Write API from the V2 Client to remove the external library dependency.	In progress
Prepared Statements for queries	TO DO
Arrow table writer for InfluxDB	TO DO
Improve Polars support	TO DO
Integrate delta sharing	TO DO

Try it out for yourself

We built the foundations of what I hope will be a great community-driven client library for InfluxDB 3.0 in Python. My call to action is if you haven’t already done so, try out the library and put it through its paces. There are so many edge cases we might not be aware of and we won’t find those without community help. I am eagerly awaiting issues and feature requests.

OpenTelemetry Tutorial: Collect Traces, Logs & Metrics with InfluxDB 3.0, Jaguar & Grafana

Jay Clifford — Tue, 30 May 2023 12:58:47 +0000

By Jay Clifford & Jacob Marble

Here at InfluxData, we recently announced InfluxDB 3.0, which expands the number of use cases that are feasible with InfluxDB. One of the primary benefits of the new storage engine that powers InfluxDB 3.0 is its ability to store traces, metrics, events, and logs in a single database.

Each of these types of time series data has unique workloads, which leaves some unanswered questions. For example:

What schema should I follow?
How do I convert my traces to line protocol?
How does InfluxDB connect with the larger observability ecosystem?

Luckily this is where our work within OpenTelemetry comes into play. If you want to learn more about what OpenTelemetry is, we recommend this blog by Charles Mahler. However, the aim of this blog is to take you through a working example of OpenTelemetry and InfluxDB 3.0.

Running the demo

Let's start with the fun part by running the demo and then discussing the theory behind it. Here is the demo we are going to deploy:

This demo uses Hot R.O.D. to simulate traces, logs, and metrics. We then use OpenTelemetry Collector to collect that data and write it to InfluxDB 3.0. Finally, we use Grafana and Jaeger UI to visualize this data in a highly summarized view.

To run the demo you can use this KillerCoda demo environment.

Simply create an account and follow the tutorial to run through the demo without worrying about a local installation. For local installs, we have all the info you need in a GitHub repo, so check out the repository readme for up-to-date installation instructions.

Walkthrough

Let's run through the demo with Killercoda so you can see what to expect:

You will see the demo provisioning script ticking away on the first load. It shouldn’t take too long. You will know it finished loading once InfluxDB OTEL Demo appears.
If you don’t already have one, follow the steps to create a free InfluxDB 3.0 Cloud account for the demo.
Make sure to follow the instructions for creating a bucket in InfluxDB called otel and generating a read-and-write token for the new otel bucket.
Next, we provide credentials for InfluxDB 3.0 so the demo can write and query from our otel bucket. To do this, select the Editor tab within Killercoda and navigate to influxdb-observability/.env. This is where we update our connection credentials.
Note: Make sure to update INFLUXDB_ADDR to point to your InfluxDB Cloud region. Also, remove any protocols from the address (e.g., HTTPS://).
Once we finish updating the environment, we can start the demo. Simply click this command to spin up the demo:
```
docker-compose --file demo/docker-compose.yml --project-directory
```
Now we get to the fun part! Let's generate some traces. First of all, let's open the HotROD application. Within a local installation, this runs via localhost:8080. To access this address in Killercoda, follow the hyperlinks provided in the screenshot below.
From there we can start generating traces by clicking on the different buttons. Each one simulates ordering a car for the indicated task, which triggers the background service calls and creates the trace.
If you head over to your InfluxDB 3.0 Cloud instance, you can explore the otel bucket schema to see how we translate the OpenTelemetry data structure into InfluxDB’s line protocol, which consists of measurements, tags, and fields. (Note: We will cover line protocol more in-depth in the next blog.)
If we head back to our Killercoda instance and open up Grafana, we can explore our OpenTelemetry dashboard and configuration.

Grafana explained

This section breaks down some of the key features of the Grafana dashboard. Let's start with the data sources.

As you can see, we connected to two data sources – Flight SQL and Jaeger. Both sources pull data directly from InfluxDB 3.0, but we will discuss how they work and their differences in more detail in the next blog. For now, you just need to know the following:

Flight SQL – This is the direct SQL query interface for InfluxDB 3.0. It is great for general-purpose time series-based queries and metrics summaries.
Jaeger – This functions as the bridging interface for metrics, logs, and traces between InfluxDB 3.0 and Grafana visualizations.

Let's look at our dashboard again and map our data sources to the different panels.

We use Flight SQL to generate our general-purpose navigation and summary overviews:

Services queries InfluxDB for all unique services found within the otel bucket. We use the service names as data links for the rest of our visualization. For example, clicking redis filters my summary results and trace list to only include the service redis.
Duration returns the duration in nanoseconds of each spanID over the selected time period. As part of our SQL query, we convert this value to seconds to improve readability.
Error Rate calculates the service error rate, as a percentage, based on the number of errors flagged in the otel.status column.

We use Jaeger to drill into our traces with the following visualizations:

Service Latency Histogram creates a histogram based on the latency of traces within the service. This panel groups trace spans based on their detected latency range.
Traces provides a table of traces associated with the selected service. The table includes trace name, start time, and duration. Users can select TraceID’s to generate the next two visualizations: Relationships and Trace.
Relationships: After selecting a TraceID in the Traces panel, users can navigate span relationships within the trace using the span tree. We’ll have more on this in the next blog.
Trace: This panel allows users to navigate raw trace data for the selected TraceID and its associated log data.

Conclusion

We hope this tutorial provides you with a solid foundation to learn more about OpenTelemetry and how InfluxDB 3.0 will play a pivotal role in future observability solutions. Stick around for the next blog where we will delve into the theory of OpenTelemtry, breaking down each component in the demo architecture and discussing data schemas. Until then play with the demo, fork the repo, and see if you can apply these components to your own use case. If you have any questions please do not hesitate to reach out to us on Slack.

Querying InfluxDB IOx Using the New Flight SQL Plugin for Grafana

Jay Clifford — Mon, 20 Feb 2023 12:33:57 +0000

A quick start guide to installation, configuration, and usage

Grafana has been a staple visualization tool used alongside InfluxDB since its inception. With the release of InfluxDB Cloud powered by IOx, there is now a new way to integrate InfluxDB and Grafana: Flight SQL.

Two of our engineers, Brett and Helen, have been working hard to create a new Grafana plugin called Flight SQL. This open-source plugin allows users to perform SQL queries directly against InfluxDB IOx and other storage engines compatible with Apache DataFusion. This blog post provides a quick start guide to installing, configuring, and utilizing the data source plugin.

Note: If you are looking for an InfluxDB OSS tutorial with either Flux or InfluxQL, check out the following blog.

Installing the plugin

We are currently releasing the Flight SQL plugin for Grafana as an experimental package. To install it, check out the most up-to-date instructions here. Since the package is currently experimental, it is unsigned and requires you to explicitly approve the installation of the plugin (The instructions will help you through this).

Configuring the plugin

To configure the FlightSQL plugin check out the following documentation. My top tip here is to make sure you specify your host URL in the following manner:

us-east-1-1.aws.cloud2.influxdata.com:443

You will find that if you include any protocols or trailing paths after the port, the connection will timeout.

Once you complete the steps in the above documentation, you will receive the following connection success.

Quick Start examples

If you would like to skip some of the manual setups and are familiar with Docker, I also created a repository called InfluxDB-IOx-Quick-Starts. This repository contains a series of Telegraf and Grafana examples deployed using Docker-Compose. To utilize this repository, run the following instructions:

Clone the repository:

clone https://github.com/InfluxCommunity/InfluxDB-IOx-Quick-Starts.git

Create an environment file within the repository


cd InfluxDB-IOx-Quick-Starts && touch .env

Add the following environment variables to the file:

export INFLUX_HOST=<INSERT_HOST> 

export INFLUX_TOKEN=<INSERT_TOKEN>

export INFLUX_ORG=<INSERT_ORG>

export INFLUX_BUCKET=<INSERT_BUCKET_NAME>

Note: make sure to remember to only include the host:port like so:

us-east-1-1.aws.cloud2.influxdata.com:443

Save Changes.

Source your newly created environment file:
```
source .env
```
Now you can deploy both Telegraf and Grafana using docker-compose:
```
docker-compose -f system-monitoring/docker-compose.yml up -d
```

Once the quick start is running, you can access Grafana at http://localhost:3000. You can log in with the default username and password: admin and admin.

Dashboard overview

Now that we installed and configured the plugin, let's take a tour of an example Grafana dashboard using the Flight SQL data source. Linux System is a beloved dashboard with the InfluxDB Community, so we will use this dashboard as an example of how to convert Flux to SQL.

Global variables for Flight SQL

Before we move on to the query conversions, it is worth mentioning that there are a series of global Grafana variables you can use to make your queries dynamic and streamlined. Here is a non-exhaustive list:

Variable	Description
$__timeRange(time)	This variable allows you to dynamically set the query time range. For instance, if you selected to see the last 15 minutes of data from the drop-down you would see an equivalent example to the conversion column. Example Conversion: ’2023-01 01T00:00:00Z’ and time <= ’2023-01-01T01:00:00Z’
$__dateBin(time)	This variable is a dynamic short form for creating an interval window, and is useful for aggregation. The interval is set based on the dashboard specification. Example Conversion: date_bin(interval ’30 second’, time, interval ’1970-01-01T00:00:00Z’)
$__timeGroup(time, hour)	As we know ```dateBin``` is effectively a window period. This gives us a way to reduce the amount of data returned via aggregations or selectors. ```timeGroup``` differs by producing projections that can be grouped on. Example Conversion: datepart(’minute’, time),datepart(’hour’, time),datepart(’day’, time),datepart(’month’, time),datepart(’year’, time);

You can find a full variable list by clicking Show Query Help while constructing your query:

Get the list of hosts

The InfluxDB dashboard introduced a dashboard variable that allowed you to filter based on the hostname tag. Let's take a look at the conversion.

Flux	FlightSQL
import "influxdata/influxdb/v1" v1.measurementTagValues(bucket: v.bucket, measurement: "cpu", tag: "host")	SELECT distinct (host) FROM cpu WHERE $__timeRange(time)

Aggregation Window (Average)

One of the most commonly used functions within Flux is the aggregate window. This allows us to window our time series data by a specific interval (e.g. 30 seconds, 2 minutes, 1 year) and then perform an aggregator (mean, mode, median, etc.) or a selector (max, min, first, last, etc.) function on the windowed data points.

Flux	Flight SQL
from(bucket: v.bucket) \|> range(start: v.timeRangeStart, stop: v.timeRangeStop) \|> filter(fn: (r) => r._measurement == "cpu") \|> filter(fn: (r) => r._field == "usage_user" or r._field == "usage_system" or r._field == "usage_idle") \|> filter(fn: (r) => r.cpu == "cpu-total") \|> filter(fn: (r) => r.host == v.linux_host) \|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)	SELECT $__dateBin(time) , avg(usage_user) AS 'usage_user', avg(usage_system) AS 'usage_system', FROM cpu WHERE host='${linux_host}' AND cpu='cpu-total' AND $__timeRange(time) GROUP BY time

Calculating the derivative

Now let’s not beat around the bush here: not all time-based queries are simple within SQL. The derivative of a value with respect to time, also known as the rate of change of the value, is calculated by using the rules of differentiation. The derivative of a function gives us the rate of change of the function at a given point. This requires us to know the previous value as well as the current. Let's look at an example:

Flux

Flight SQL

from(bucket: v.bucket) |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r._measurement == "diskio") |> filter(fn: (r) => r._field == "read_bytes" or r._field == "write_bytes") |> filter(fn: (r) => r.host == v.linux_host) |> derivative(unit: v.windowPeriod, nonNegative: false)

SELECT time, (read_bytes_delta_v / delta_t_ns) * 1000000000 as read_bytes, (write_bytes_delta_v / delta_t_ns) * 1000000000 as write_bytes FROM ( SELECT (lag(read_bytes, 1) OVER (ORDER BY time)) - read_bytes as read_bytes_delta_v, (lag(write_bytes, 1) OVER (ORDER BY time)) - write_bytes as write_bytes_delta_v, (lag(cast(time as bigint), 1) OVER (ORDER BY time)) - cast (time as bigint) as delta_t_ns, time FROM diskio WHERE host='${linux_host}' AND $__timeRange(time) ) as sq

For these types of examples, we plan to provide workarounds like this where possible and slowly develop a series of custom SQL functions to handle these complex time series calculations.

Conclusion

I hope this blog post provides enough insight into the new Flight SQL plugin for Grafana to start trialing it. My call to action for you is to start testing the plugin and leaving your feedback within the plugin repository. This will help us improve the overall useability and feature set of the plugin. I would also like to take the time to thank Brett and Helen for their efforts in making an open-source plugin for Flight SQL.

Come join us on Slack and the forums. Share your thoughts — I look forward to seeing you there!

Intro to Py-Arrow

Jay Clifford — Mon, 06 Feb 2023 11:57:12 +0000

So by now, you are probably aware that InfluxData has been busy building the next generation of the InfluxDB storage engine. If you dig a little deeper, you will start to uncover some concepts that might be foreign to you:

Apache Parquet
Apache Arrow
Arrow Flight

These open-source projects are some of the core building blocks that make up the new storage engine. For the most part, you won’t need to worry about what’s under the hood. Though if you are like me and want a more practical understanding of what some of the projects are, then join me on my journey of discovery.

The first component we are going to dig into is Apache Arrow. My colleague Charles gave a great high-level overview, which you can find here.

In short:
“Arrow manages data in arrays, which can be grouped in tables to represent columns of data in tabular data. Arrow also provides support for various formats to get those tabular data in and out of disk and networks. The most commonly used formats are Parquet (You will be exposed to this concept quite a bit).”

For performance reasons, our developers used Rust to code InfluxDB’s new storage engine. I personally like to learn new coding concepts in Python, so we will be making use of the pyarrow client library.

The basics

In Apache Arrow, you have two primary data containers/classes: Arrays and Tables. We will dig more into what these are later, but let’s first write a quick snippet of code for creating each:

import pyarrow as pa

# Create a array from a list of values
animal = pa.array(["sheep", "cows", "horses", "foxes"], type=pa.string())
count = pa.array([12, 5, 2, 1], type=pa.int8())
year = pa.array([2022, 2022, 2022, 2022], type=pa.int16())

# Create a table from the arrays
table = pa.Table.from_arrays([animal, count, year], names=['animal', 'count', 'year'])
print(table)

So in this example, you can see we constructed 3 arrays of values: animal, count, and year. We can combine these arrays to form the columns of a table. The results of running this code look like so:

animal: string
count: int8
year: int16
----
animal: [["sheep","cows","horses","foxes"]]
count: [[12,5,2,1]]
year: [[2022,2022,2022,2022]]

So now that we have a table to work with, let’s see what we can do with it. The first primary feature of Arrow is to provide facilities for saving and restoring your tabular data (most commonly into the Parquet format, which will feature heavily in future blogs).

Let’s save and load our newly created table:

import pyarrow as pa
import pyarrow.parquet as pq

# Create a array from a list of values
animal = pa.array(["sheep", "cows", "horses", "foxes"], type=pa.string())
count = pa.array([12, 5, 2, 1], type=pa.int8())
year = pa.array([2022, 2022, 2022, 2022], type=pa.int16())

# Create a table from the arrays
table = pa.Table.from_arrays([animal, count, year], names=['animal', 'count', 'year'])

# Save the table to a Parquet file
pq.write_table(table, 'example.parquet')

# Load the table from the Parquet file
table2 = pq.read_table('example.parquet')
print(table2)

Lastly, to finish the basics, let’s try out a compute function (value_counts). We can apply compute functions to arrays and tables, which then allows us to apply transformations to a dataset. We will cover these in greater detail in the next section but let’s start with a simple example:

import pyarrow as pa
import pyarrow.compute as pc

# Create a array from a list of values
animal = pa.array(["sheep", "cows", "horses", "foxes", "sheep"], type=pa.string())
count = pa.array([12, 5, 2, 1, 10], type=pa.int8())
year = pa.array([2022, 2022, 2022, 2022, 2021], type=pa.int16())

# Create a table from the arrays
table = pa.Table.from_arrays([animal, count, year], names=['animal', 'count', 'year'])

count_y = pc.value_counts(table['animal'])
print(count_y)

As you can see, call the library pyarrow.compute as pc and use the built-in count function. This allows us to count the number of values within a given array or table. We chose to count up the number of animals, which produces the following output:

-- child 0 type: string
  [
    "sheep",
    "cows",
    "horses",
    "foxes"
  ]
-- child 1 type: int64
  [
    2,
    1,
    1,
    1
  ]

A practical example

So I decided to skip listing all the datatypes and processors to you and thought I would show you a more realistic example of using Apache Arrow with InfluxDB’s 3.0.

So the plan:

Query InfluxDB 3.0 using the new python client library.
Then we will use a new function to save the table as a series of partitioned Parquet files to disk.
Lastly a second script will reload the partitions and perform a series of basic aggregations on our Arrow Table structure.

Let’s take a look at the code:
create_parquet.py

token = ''
host = 'eu-central-1-1.aws.cloud2.influxdata.com'    
org = 'Jay-IOx'
db = 'factory'

client = InfluxDBClient3.InfluxDBClient3(token=token,
                         host=host,
                         org=org,
                         database=db) 


query = "SELECT vibration FROM machine_data WHERE time >= now() - 1h GROUP BY machineID"
table = client.query(query=query, language="influxql")
print(table)

print("Saving to parquet files...")


# partitioning of your data in smaller chunks
ds.write_dataset(table, "machine_data", format="parquet",
                 partitioning=ds.partitioning(
                    pa.schema([table.schema.field("iox::measurement")])
                ))

So this function you will be unfamiliar;

Write_dataset(… partitioning=ds.partitioning(…)) : This modified method partitions our table into Parquet files based upon the values within our ‘iox::measurement’ column. This will look like a tree of directories. This method helps to separate large datasets into more manageable assets.

Let’s now take a look at the second script, which works with our saved Parquet files:

import pyarrow.dataset as ds

# Loading back the partitioned dataset will detect the chunks                
machine_data = ds.dataset("machine_data", format="parquet", partitioning=["iox::measurement"])
print(machine_data.files)

# Convert to a table
machine_data = machine_data.to_table()
print(machine_data)

# Grouped Aggregation example
aggregation = machine_data.group_by("machineID").aggregate([("vibration", "mean"),("vibration", "max"),("vibration", "min") ]).to_pandas()             
print(aggregation)

In this script we deploy several new functions you might be familiar with if you work with Pandas or other query engines: group_by and aggregate. We use these functions to group our data points based on the measurement and provide a mathematical aggregate to each group (mean, max, min). This generates a new Arrow table based on the aggregations. We then convert the table back to a data frame for readability.

Conclusion

I hope this blog empowers you to start digging deeper into Apache Arrow and helps you to understand why we decided to invest in the future of Apache Arrow and its child products. I also hope it gives you the foundations to start exploring how you can build your own analytics applications from this framework. InfluxDB’s new storage engine emphasizes its commitment to the greater ecosystem. For instance, allowing the exportation of Parquet files gives us the opportunity to analyze our data in platforms such as Rapid Miner and other analytical platforms.

My call to action for you is to check out the code here and discover some of the other processor functionality Apache Arrow offers. A lot of the content coming up will be around Apache Parquet, so if there are any products/platforms that use Parquet that you would like us to talk about let us know. Come join us on Slack and the forums. Share your thoughts — I look forward to seeing you there!