A common use case that is implemented with Aerospike involves storing time series data that arrives at a very high throughput rate. The ability to view that data is real-time is also a desired component of this use case. A demo using price data for equity (stock) prices has been created which will demonstrate both of those capabilities.
First we want to populate a baseline of historical opening and closing stock price data for some symbols. To populate this data, a Java program loaded source data for a few stock symbols, IBM, WMT, XOM, BRK, and AMZN from freely available Yahoo Finance export files. That source data only includes one record per day with information regarding opening, closing, high, and low prices. The program generates data for each second within the price ranges provided in the file. All the generated tick price data for the stock symbol for a single day is stored in a single Sorted List which is then written to the Aerospike database. A separate List entry is appended to another record which encapsulates all closing prices over a single year for that same stock symbol.
The code is as follows:
The next Java program is utilized to endlessly generate an infinite number of real-time stock random ticker values selected stock symbols. As its base, it relies on a CSV file that provides the stock symbol, low/high range of values and the maximum percentage change per second. For each of those symbols, a random stock price will be generated for the current time (second resolution). That price will be added to the Sorted List of ticker prices for that record in the Aerospike database.
Note that the “operation” API call is utilized to add the entry to a ticker price List for that symbol and date on the server WITHOUT requiring the entire list to be downloaded/edited/updated at the client. The ability to execute operations on the database server makes the storage/manipulation of large lists practical; having to download a large record to the client over the network, make a minor change and then load back to the server simply cannot SCALE to large data volumes.
After generating the values for each symbol, the application will sleep for a second.
Last, but not least, we’d like to display the real-time stock ticker data graphically. For this, we’ve integrated Grafana with Aerospike. We’ve modified a publicly available sample for building a Grafana datasource into one for use with Aerospike for purposes of this demo. Query requests are parsed and then fulfilled by retrieving the desired records from the Aerospike database, which is being populate in read-time by the Java program described above. If the requested data is for a range of less than 3 days, then the detailed tick data with a granularity of seconds (one database record symbol per date) is retrieved. If the range is greater, then the data is retrieved with the granularity of days (one database record per symbol per year). Note that retrieval of the multiple Aerospike database records required to fulfill the client request is accomplished via a single API call (get_many). As the database is distributed among multiple servers, the request is fanned out in parallel to the servers that have the requested data.
A sample of the results of demo project in a Grafana window is visible at the top of this blog post.
I hope that this demo code is useful for the reader and provides some useful ideas as to how to employ Aerospike towards capturing/viewing “real-time” time series data, such as for stock ticker (this demo) or IIoT applications. Aerospike’s high throughput, low latency, low TCO is optimal for these types of applications.
Top comments (0)