DEV Community: Dan Voyce

Converting CSV to ORC/Parquet fast without a cluster!

Dan Voyce — Mon, 16 Dec 2019 02:58:14 +0000

How to convert from CSV to a Columnar format without using a cluster?

Converting from a flat format such as CSV or JSON to a columnar storage (such as ORC or Parquet is a task that any data engineer will have to do probably multiple times each week).

CSV's are row based (think Excel - you have Column names at the top and then the values in each row going down).

Columnar formats are different (See ORC Example below):

And the process of converting between a flat, row based format to a columnar format is computationally expensive that usually requires the creation of a cluster to manage this effectively in a reasonable amount of time.

Ways of converting CSV to ORC

I am a great believer in simplicity when it comes to writing files like this. In our previous experiments we realised that although using things like Hive can greatly simplify the process and allow it to be more automated. Systems like Spark can actually manage the process faster and cheaper if you keep things simple.

Hive / Presto

In a nutshell, you create a new table with STORED AS ORC on the definition. You then:

INSERT into orc_table SELECT * from csv_table

This then rewrites your CSV data into a new ORC table on the fly.
This requires a multi-node cluster to ensure things run quickly.

Dataframes

Spark runs on dataframes. You essentially load files into a dataframe and then output that dataframe as a different type of file.

csv = df.read_csv('/path/to/csv/files')
csv.to_orc('/path/for/final/orc/files')

I mean it doesn't really get any simpler than that for a 1-1 conversion. However this still requires a cluster to work and its speed is very much based on the number of nodes you point at it / how you manage node memory.

Enter the GPU

GPU processing used to be the domain of Game designers and some scientist types exclusively. However with the recent introduction of frameworks such as rapids.ai (https://rapids.ai) things have got much easier for the average Joe.

I recently set up my QNAP NAS as a data-science lab (you can read about the setup of it and Rapids.Ai here: https://www.linkedin.com/pulse/setting-up-rapidsai-qnap-nas-dan-voyce).

This included (naturally) an obscene amount of storage, some fast NVME hard-drives and most importantly a GPU (I went with a cheap 1060 GTX with 6GB memory as max speed isn't required for just a PoC).

Using Rapids.ai & cuDF to convert formats using the GPU.

As I have said - my setup is not particularly complex or the most up-to-date.
Mark Litwintschik is a fantastic Big Data consultant who transparently publishes comparisons of various methods of CSV conversion on his blog: https://tech.marksblogg.com/faster-csv-to-orc-conversions.html

In his comparisons he launches a 21 node m3.xlarge cluster on AWS to complete his conversions. According to the AWS cost calculator this costs $430 USD per month if used for a maximum of 2 hours per day (Marks longest conversion)

In comparison a single EC2 instance with a 8GB GPU runs at $54 USD per month for 2 hours usage per day (This is an Linux on i3en.xlarge which is probably overkill for the computing but includes 2500GB of NVME Storage)

There are other factors here that I haven't tested yet such as S3 latency and speed over Hard Drives but in my experience they are roughly on par.

By adding local NVME storage to the instance it means you can process this locally, however for truly huge datasets it is sometimes preferable to just process from S3 directly and output back to S3 - YMMV.

The Dataset

We were asked to produce a sample dataset for Las Vegas for a client. Generally unless clients are super tech savvy they request files in CSV format, unfortunately for us this is not the most efficient way of querying things so I figured a real-world dataset would be a good test of this.

The dataset comes in at just over 2 Billion Rows and has 41 fields. The total size of this data is 397GB uncompressed or around 127GB gzip compressed.

So this is a little larger than the dataset that Mark has done his benchmarks on but should be a good comparison.

A note on partitions:

I am using an older consumer graphics card that only has 6GB of memory.
Whatever Dask creates, the partitions still need to fit into the available GPU memory, obviously having 3000 files in this case is not ideal and whilst I could probably tune it to be less if having this number of files is a deal breaker for you then I would suggest maybe looking at a GPU with higher memory or attaching several GPU's to get a larger memory block. It should still work out much cheaper than a cluster!

First test on "Spinning Rust"

My NAS has 8 x 4TB Seagate Ironwolf SATA drives setup in RAID-10, this usually gets me around 140MB/s read speed.

On the Jupyter notebook that is provided with rapids.ai I ran the following:
The output is included

%%time
import dask_cudf as dc

ddf = dc.read_csv('/data/Data Files/Vegas/datafiles/csv/*.csv.gz', compression='gzip')

CPU times: user 1.82 s, sys: 870 ms, total: 2.69 s
Wall time 6.99 s

%%time
ddf = ddf.repartition(npartitions=3000)

CPU times: user 60.2 ms, sys: 159 µs, total: 60.4 ms
Wall time: 57.6 ms

%%time
ddf.to_orc('/data/Data Files/Vegas/datafiles/orc/')

CPU times: user 1h 4min 4s, sys: 30min 19s, total 1h 34min 23s
Wall time: 41min 57s

42 Minutes - wow!

These tests are already pretty mind blowing. The fact that my consumer NAS with its out of date consumer graphics card can beat the majority of the benchmarks for converting CSV data that Mark provided (that are run on huge clusters) is amazing. I have no doubt that with NVME drives this will be significantly faster.

First test on NVMe drive

I added a 1TB NVMe drive (I bought 2 but it seems like my QNAP doesnt want to recognise anything in NVMe Slot 1 (RMA incoming)), to my NAS and provisioned it as a volume. I copied the dataset to the newly provisioned drives (mounted at /ssd) and re-ran the test:

%%time
import dask_cudf as dc

ddf = dc.read_csv('/ssd/csv/*.csv.gz', compression='gzip')

CPU times: user 1.87 s, sys: 726 ms, total: 2.59 s
Wall time: 2.13 s

%%time
ddf = ddf.repartition(npartitions=3000)

CPU times: user 76 ms, sys: 22 ms, total: 98 ms
Wall time: 97.2 ms

%%time
ddf.to_orc('/ssd/orc/')

CPU times: user 1h 16min 7s, sys: 18min 50s, total: 1h 34min 57s
Wall time: 32min 15s

32 Minutes

This is obviously faster but a bit disappointing, I was expecting around the 20 minute benchmark, however this is likely because I am reading and writing to / from the same drive. Unfortunately with my NAS not recognising the other NVMe drive I can't really do a proper test until it is fixed.

CONCLUSIONS

Using a GPU is basically like using a mini-cluster, but rather instead of spreading the workload out onto multiple machines and therefore multiple CPU cores, it does it all within its own CUDA cores.

The fact that my ASUS GTX 1060 6GB card can do a basic conversion from Row based to Column based formats in less time than a 21 node, large instance cluster should be eye opening at the least.

I will also be testing out our spatial joins on GPU in another article to follow!

Daniel Voyce - CTO

Daniel leads the technology strategy, architecture and development at LOCALLY.
He calls Melbourne, Australia home and is an accomplished technical leader with extensive experience in defining and developing digital roadmaps, highly technical systems and frameworks, and software platforms.

Spatial Big Data Systems - a retrospective

Dan Voyce — Thu, 05 Dec 2019 04:52:30 +0000

This retrospective was presented as a talk at Cakefest 2019 in Tokyo Japan

Background: Our specialty is human movement data, we collect anonymized telemetry over millions of devices each day. This consists of some device information (OS, Brand, Model, Wireless capabilities etc). Also consists of Location data (Latitude, Longitude, Altitude etc).

By understanding human movement and intersecting this data with other datasets we can help businesses understand their customers – we can provide insights to allow a business to accurately predict customer intent / requirements.

We maintain a database of Millions of US POI’s, both shape data and ‘point’ data. By spatially joining the human movement data with this POI data we can get context as to the locations people visit and use this to determine intent.

Big Data: Due to the size of these data sets and the complexity in spatial processing, a normal RDBMS simply won't cut it or allow us to process this data within the 24 hour window we require - let alone allow us to create an interface that allows a human being to design their own audiences on-the-fly and receive instant estimates of the audience size.

The previous article I wrote discusses some of the challenges we faced early on in our development of this platform (https://www.linkedin.com/pulse/what-functional-big-data-dan-voyce), This article describes our methodology of testing various solutions before and after that - to bring us to the final solution we are currently using to process over 200B rows each day in under an hour.

The problem:

"To create a system where a customer can design their own audience by choosing and combining customer intents"

I won't go heavily into the analysis of this problem in this article but we eventually landed upon a system that would generate a "query" (not necessarily SQL) that we can run against our data warehouse to produce the audience.

The tricky part of this - was "How do we get an estimate displayed to the customer of the audience size"?

As the end user can design an audience from any combination of 8 filters (each containing 100's - 1000's of options) pre-caching the counts on each processing wasn't really feasible - especially since we were also providing the ability to filter between specific dates.

The solution:

There are a few well known methods for producing fast estimation of large datasets which I won't go into in this article but in both cases they came with a set of problems we needed to overcome, in the end it all came down to the choice of the underlying data warehouse / query engine we decided to use. This is what this article will focus on.

Why count(distinct()) is king - but definitely not equitable.

As developers most of us understand what count(distinct()) does, what many people don't understand (or think about) is HOW it does what it does.

Counting distinct entities in a huge dataset is actually a hard problem, it is slightly easier if the data is sorted but re-sorting data on each insert becomes expensive depending on the underlying platform used.

Generally a count distinct performs a distinct sort and then counts the items in each "bucket" of grouped values.

If you need a 100% accurate count then this is unfortunately pretty much the only way you can get it on a randomised dataset, there are tricks you can do in how things are structured in the platform to make things more efficient (partitioning, clustering / bucketing for example) but it essentially still has to perform the same operations.

Thankfully we are not in the position where we immediately require a 100% accurate answer - an estimate with a reasonable error margin is just fine for what we need. This opens up a few more options.

Rejected methods / platforms

These were methods we started to test out early on, we had a number of requirements that some were not able to provide (for example Geospatial processing is a pre-requisite we had as many of the counts we would have to provide would be based on Geospatial predicates).

Final method / platform choices

Which ever method we chose it would still need to be underpinned by a solid and future proof data warehouse, in the end we decided upon 2 to power different parts of the platform (to eventually be consolidated into a single data warehouse later in the process).

HyperLogLog++
For many years this was the gold standard in cardinality estimation, for static datasets it is still the recommended method as it is easily stored, very low maintenance and can merge counts from multiple "columns" to get an overall count. Unfortunately for us our queries were not static, we would have had to create a HLL sketch for every single combination of columns which wasn't feasible.

VerdictDB / Keebo

Early on during the process we started working with the guys at VerdictDB. VerdictDB is a fantastic product that uses probability / statistical theory to create estimates of cardinality on large datasets. With the support of the team at Verdict we were able to create a bridge to incorporate this into each of our platforms.

Presto

Presto is a query engine designed to provide an "SQL on Anything" type of interface for data. As all of our data was stored in Google Cloud Storage it was a natural choice. It is blazingly fast even with a small number of nodes and completely customisable. You can see why companies such as Teradata and Starburst data have adopted it to power their platforms.

There was a fair bit of optimisation required around the ORC format we chose to use with Presto but once this was completed we could power our Fast counts with a minimal 3 node cluster running over the scrambles and a 10 node cluster to perform the pre-processing required.

As long as good housekeeping was maintained on the data, the fast counts would usually return within around 5-7 seconds.

BigQuery

BigQuery is a Google product based on Dremel, it uses a massively parallel architecture to plan queries across a huge number of nodes all packaged as a fully managed and on-demand service to the user.

BigQuery BI is an in-memory caching service provided which cuts down on the 3s overhead to queries issued to BigQuery. By caching the scrambles and linking our infrastructure directly into this, we were able to ensure that the query would return in around 2 seconds on average.

Conclusion

With the use of VerdictDB both Presto and BigQuery provided the speed required to allow a human interface to our Data Warehouse, BigQuery out performed Presto in a number of areas when BigQuery BI was thrown into the equation, and although this is still in beta offering only 10GB (should be enough to cache a 1% scramble of 1TB of data), it has huge potential in offering a cost-effective and fast interface to Big Data.

If you want to avoid vendor lock-in then Presto is a fantastic choice, throwing more nodes at the situation would easily get the speed down to the BigQuery equivalent. There are also several other options that exist that could be used as the query interface once the scrambles are built on Presto!

Daniel Voyce - CTO

Solr and CakePHP Integration - Part II

Dan Voyce — Thu, 01 Aug 2019 06:15:31 +0000

This is the second in a series of posts about setting up CakePHP with Solr, if you missed part 1 you can catch it here: https://dev.to/locally/cakephp-and-solr-integration-417e

In this post, I will share my experience with the Solarium PHP Solr client library and how to perform some basic actions in Solr such as adding, removing and searching documents, which was essential for building the CakePHP Solr Behavior for work with the full-text search in Solr and keep the documents always updated.

We use Solr at LOCALLY to provide efficient Geofence and Geospatial processing for our Consumer Engagement platform LOCALLY Engage

Establishing a connection with Solr

Using the Solarium PHP Solr client library in CakePHP, this task is easy, you just need a few lines to make a connection with Solr.

First of all, installing the library:

composer require solarium/solarium

Then, creating the connection:

use Solarium\Client;
protected $client;
public function init()
    {
        $config = [
            'endpoint' => [
                'localhost' => [
                    'host' => "SOLR-HOST",
                    'port' => "SOLR-PORT",
                    'path' => "/solr/SOLR-CORE/",
                ],
            ],
        ];
        $this->client = new Client($config);
    }

Performing a search

Basically, to perform a search in Solr, you just need to use a few commands, there are many options you can set in your query, below I’m showing an example of how to use some of them using the library:

// create a client instance
$client = new Solarium\Client($config);

// get a select query instance
$query = $client->createQuery($client::QUERY_SELECT);

// set start and rows param (comparable to SQL limit) using fluent interface
$query->setStart(2)->setRows(20);

//It passes a term or "*:*" will search everything in solr
$query->setQuery($searchTerm);

// set fields to fetch (this overrides the default setting 'all fields')
$query->setFields(['id','name']);

// this executes the query and returns the result
$resultset = $client->execute($query);

If we create a function to perform a search and pass an array of options as a parameter, it should look like this:

public function search(array $options)
  {
        $query = $this->client->createSelect([
            'start' => isset($options['start']) ? $options['start'] : $this->start,
            'rows' => isset($options['limit']) ? $options['limit'] : $this->rows
        ]);
        $query->getEDisMax();
        $query->setQuery(!empty($options['match_field']) ? $options['match_field'] : '*:*');
        $query->setFields('*');
        $resultset = $this->client->select($query);
        return $resultset->getData()['response'];
}

Extended DisMax (eDismax) Query Parser

This is an interesting option, the eDisMax is a better version of DisMax query parser, this allows you to write queries such as AND, OR, NOT, -, and +. Also respects all the “magic fields” names such as val and query. Find out more about it here.

Handling Solr Response

To handle a response of a search in solr, you can:

Interact Solr response object

// this executes the query and returns the result
$resultset = $client->execute($query);

// display the total number of documents found by solr
echo 'NumFound: '.$resultset->getNumFound();

// show documents using the resultset iterator
foreach ($resultset as $document) {

    echo '<hr/><table>';

    // the documents are also iterable, to get all fields
    foreach ($document as $field => $value) {
        // this converts multivalue fields to a comma-separated string
        if (is_array($value)) {
            $value = implode(', ', $value);
        }

        echo '<tr><th>' . $field . '</th><td>' . $value . '</td></tr>';
    }

    echo '</table>';
}

Return only the response:


              ...
$query = $this->client->createSelect();
              ...
$resultset = $this->client->select($query);
return $resultset->getData()['response'];
Debugging Solr Response

// create a client instance
$client = new Solarium\Client($config);

// get a select query instance
$query = $client->createSelect();
$query->setQuery('*');

// add debug settings
$debug = $query->getDebug();
$debug->setExplainOther('id:MA*');

// this executes the query and returns the result
$resultset = $client->select($query);
$debugResult = $resultset->getDebug();

// display the debug results
echo '<h1>Debug data</h1>';
echo 'Querystring: ' . $debugResult->getQueryString() . '<br/>';
echo 'Parsed query: ' . $debugResult->getParsedQuery() . '<br/>';
echo 'Query parser: ' . $debugResult->getQueryParser() . '<br/>';
echo 'Other query: ' . $debugResult->getOtherQuery() . '<br/>';

echo '<h2>Explain data</h2>';
foreach ($debugResult->getExplain() as $key => $explanation) {
    echo '<h3>Document key: ' . $key . '</h3>';
    echo 'Value: ' . $explanation->getValue() . '<br/>';
    echo 'Match: ' . (($explanation->getMatch() == true) ? 'true' : 'false')  . '<br/>';
    echo 'Description: ' . $explanation->getDescription() . '<br/>';
    echo '<h4>Details</h4>';
    foreach ($explanation as $detail) {
        echo 'Value: ' . $detail->getValue() . '<br/>';
        echo 'Match: ' . (($detail->getMatch() == true) ? 'true' : 'false')  . '<br/>';
        echo 'Description: ' . $detail->getDescription() . '<br/>';
        echo '<hr/>';
    }
}

echo '<h2>ExplainOther data</h2>';
foreach ($debugResult->getExplainOther() as $key => $explanation) {
    echo '<h3>Document key: ' . $key . '</h3>';
    echo 'Value: ' . $explanation->getValue() . '<br/>';
    echo 'Match: ' . (($explanation->getMatch() == true) ? 'true' : 'false')  . '<br/>';
    echo 'Description: ' . $explanation->getDescription() . '<br/>';
    echo '<h4>Details</h4>';
    foreach ($explanation as $detail) {
        echo 'Value: ' . $detail->getValue() . '<br/>';
        echo 'Match: ' . (($detail->getMatch() == true) ? 'true' : 'false')  . '<br/>';
        echo 'Description: ' . $detail->getDescription() . '<br/>';
        echo '<hr/>';
    }
}

echo '<h2>Timings (in ms)</h2>';
echo 'Total time: ' . $debugResult->getTiming()->getTime() . '<br/>';
echo '<h3>Phases</h3>';
foreach ($debugResult->getTiming()->getPhases() as $phaseName => $phaseData) {
    echo '<h4>' . $phaseName . '</h4>';
    foreach ($phaseData as $class => $time) {
        echo $class . ': ' . $time . '<br/>';
    }
    echo '<hr/>';
}

Adding Documents to Solr

In order to keep the indexes up to date, you can add or update documents to Solr when a record is created or updated in your database, for example. Let’s create a function for it.

public function saveSearchIndex($data) 
 {
        // get an update query instance
        $update = $this->client->createUpdate();

        // create a new document for the data,
        // you can do validations before this line
        $document = $update->createDocument();
        //In my case, I am passing an array $data with the keys for my indexes. 
        foreach ($data as $key => $doc) {
            $document->{$key} = $doc;
        }
        // add the document and a commit command to the update query
        $update->addDocument($document);
        $update->addCommit();

        // this executes the query and returns the result
        return $this->client->update($update);
}

Deleting Documents

In a very similar way, you can delete documents in Solr. There are two options for it, delete by id or delete by query. If you are using a database view with multiple tables as Solr source, as I did, delete by query is an option, as my id’s can “repeat” in different tables.

The function below, the conditions to delete an index will be something like in an “AND” condition, so I will delete indexes with the id $id and with a result_type $result_type:

public function deleteSearchIndex($id, $result_type) 
{
      $update = $this->client->createUpdate();
      $update->addDeleteQuery('id:'.$id);
      $update->addDeleteQuery('result_type:'.$result_type);
      $update->addCommit();
      return $this->client->update($update);
}

Same function deleting by id:

public function deleteSearchIndex($id) 
{
      $update = $this->client->createUpdate();
      $update->addDeleteById($id);
      $update->addCommit();
      return $this->client->update($update);
}

That’s it! It is pretty simple work with Solr, once you’ve worked out its fundamentals!

In this blog we’ve seen how to:

Create a connection with Solr using Solarium PHP Solr client library,
Perform a search setting query, limits, fields and using eDisMax Query Parser.
Debug a query response
Handle a response
Add and delete documents in Solr.

In the next blog, I will write about the CakePHP behavior to work with Solr and how to display results to the user.

Lorena Santana - Platform Developer

System analyst/Web development in the last +8 years with different technologies and methodologies. Experience with ERP's, CMS and frameworks such as CakePHP, Bootstrap and Wordpress. Full stack developer, self-taught, expertise in database modeling and maintenence, also in requirements gathering. Experience with agile methodologies and work well under pressure. Experience with interns.

Fixing Mouse Wheel on Chrome in Ubuntu 18.04 Virtualbox / VMWare

Dan Voyce — Tue, 23 Jul 2019 03:02:41 +0000

This has been driving me insane for at least a year now. Like many developers I run a Virtual Machine that holds my day to day development environment (For the rare days I get to actually code!).
One thing that has been a constant annoyance is going from the lovely smooth scrolling of Windows to a horrible messy, jagged, disjointed scrolling experience in Chrome and other apps under Ubuntu 18.04.

The problem (it turns out) was simple - libinput does not like trying to handle the mouse wheel at the same time as the mouse moving.

The solution seemed simple on the surface: Replace libinput with evdev and all should be solved.

Unfortunately Ubuntu 18.04 comes with its own Hardware Enablement layer, this replaces many of the standard components with version locked and layered versions that Canonical provide.

disclaimer: this worked on my system - you might break yours!

The first step is to remove this layer and instead install the standard versions of xorg:

sudo apt remove xserver-xorg-core-hwe-18.04 xserver-xorg-input-all-hwe-18.04 linux-generic-hwe-18.04 xserver-xorg-video-all-hwe-18.04

Now install the standard version of xorg and the evdev driver:

sudo apt install xserver-xorg-core xserver-xorg-input-evdev

Finally you need to edit the xorg config for libinput and change the driver to evdev:

sudo vi /usr/share/X11/xorg.conf.d/40-libinput.conf

and change the following line:

Section "InputClass"
        Identifier "libinput pointer catchall"
        MatchIsPointer "on"
        MatchDevicePath "/dev/input/event*"
        Driver "libinput"

to:

Section "InputClass"
        Identifier "libinput pointer catchall"
        MatchIsPointer "on"
        MatchDevicePath "/dev/input/event*"
        Driver "evdev"

This alone sorts out the "scrolling while moving" problem instantly resulting in a less jagged experience, however for some serious clout you might also want to install imwheel (https://wiki.archlinux.org/index.php/IMWheel is a good guide to getting it going)

My Config for this is:

None,      Up,   Button4, 2
None,      Down, Button5, 2
Control_L, Up,   Control_L|Button4
Control_L, Down, Control_L|Button5
Shift_L,   Up,   Shift_L|Button4
Shift_L,   Down, Shift_L|Button5

"^Navigator$"
None,      Up,   Button4, 2
None,      Down, Button5, 2
Control_L, Up,   Control_L|Button4
Control_L, Down, Control_L|Button5
Shift_L,   Up,   Shift_L|Button4
Shift_L,   Down, Shift_L|Button5

"^chromium$"
None,      Up,   Button4, 2
None,      Down, Button5, 2
Control_L, Up,   Control_L|Button4
Control_L, Down, Control_L|Button5
Shift_L,   Up,   Shift_L|Button4
Shift_L,   Down, Shift_L|Button5

".*gimp*"
Control_L, Up,   Control_L|Button4
Control_L, Down, Control_L|Button5
Shift_L,   Up,   Shift_L|Button4
Shift_L,   Down, Shift_L|Button5
Alt_L,     Up,   Alt_L|Button4
Alt_L,     Down, Alt_L|Button5
Control_L|Alt_L,     Up,   Control_L|Alt_L|Button4
Control_L|Alt_L,     Down, Control_L|Alt_L|Button5

".*nautilus*"
Control_L, Up,   Control_L|Button4
Control_L, Down, Control_L|Button5

I really hope this helps someone as this has been a massive annoyance for me.

CakePHP and Solr Integration

Dan Voyce — Wed, 17 Jul 2019 07:05:21 +0000

Recently, I had to develop a full text search with Apache Solr for the LOCALLY CMS in CakePHP 3 and what an adventure that was! The reason being is that it is important that users are able to search for all information relating to Locations and impressions in our Location Intelligence and Consumer Engagement Platform (LOCALLY Engage)
If Solr is a complete new world to you, as it was to me when I started developing this, here is a quick definition about what Solr (It is pronounced “Solar”) is, provided by Solr project:

Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites.

Solr Integration with CakePHP

The diagram above, illustrates perfectly what I had to do in this task. So, in order to integrate Solr with other server applications for a search, such as LOCALLY CMS, basically we have to:

Define a schema
Deploy Solr
“Feed” Solr with the documents that the users will search
Create a link between Solr and CakePHP via Solarium PHP Solr client library
Implement the search in the CMS

I have a lot to share about all I’ve been working to achieve this task, so I will divide this in a few blog posts. In today's post, I will talk about Solr Core Configuration for the version 6.6, when Solr is configured to work with multi cores.

Full Text Search: Single core or multiple cores?
When you have a relational database, maybe your first choice would be having multiple cores, one for each searchable table of our database, which makes sense. However, we had some things to consider before making this choice:

Solr isn’t relational, all the searchable words are stored in an "inverse index" and this is what makes it so fast;
Multiple cores means multiple configuration, indexation and maintenance;
If you use multiple cores, at some stage you might need to “join” your data.

Our decision at LOCALLY was to have only one core “Search” configured. So, all the searchable data provided by our database tables was imported as documents in Solr using Data Import Request Handler.

Single core configuration

Solr is configured using XML files. For creating a single core in Solr, some files must be configured previously, basically: solrconfig.xml, schema.xml and data-config.xml. I won’t get in the details about solrconfig.xml once this file should work fine with Solr default configuration and no changes were needed for this task.

data-config.xml

This file is where we specify what is the source of our documents, as I mentioned before this was made using the Data Import Request Handler, getting the indexes from MySQL tables. All the configuration must be between tags.

Specifying the dataSource of the documents, a simple JDBC connection with MySQL.

<dataSource type="JdbcDataSource" 
               driver="com.mysql.jdbc.Driver" 
               url="jdbc:mysql://HOST/DATABASE" 
               user="USER" 
               password="PASSWORD" />

For the entity and fields configuration, I used a database View instead of specify multiple tables and all its fields, because we’ve decided to have a single core. Plus you don’t want nobody searching in fields with sensitive information such as password, credit card and tokens, right?! So, also for safety reasons, a view was my choice.

Between the tags we specify another tag where we specify the table name (a view in our case) in the attribute name, primary key in the attribute pk, in the attribute query, a SQL query can be informed.

Inside the tags , we specify the fields of our query that will be visible in Solr after indexing the documents. The attributes column and name are essential.

Here is an example of this configuration.

<document>
        <entity name="MyView"
                pk="id"
                query="SELECT * FROM MyView">
            <!-- Fields that will be returned by the query -->
            <field column="id" name="id"/>
            <field column="name" name="name"/>
            <field column="created" name="created"/>
            <field column="modified" name="modified"/>
            ...
        </entity>
    </document>

And a complete example of data-config.xml.

<dataConfig>
   <dataSource type="JdbcDataSource" 
               driver="com.mysql.jdbc.Driver" 
               url="jdbc:mysql://HOST/DATABASE" 
               user="USER" 
               password="PASSWORD" />
   <document>
      <entity name="MyView" pk="id" query="SELECT * FROM MyView">
         <!-- Fields that will be returned by the query -->
         <field column="id" name="id" />
         <field column="name" name="name" />
         <field column="created" name="created" />
         <field column="modified" name="modified" />
         ...
      </entity>
   </document>
</dataConfig>

schema.xml

This is another file with default configuration, but there are specific some points I’d to highlight and they are essential to have a full text search configured.

This file starts this way:

<?xml version="1.0" encoding="UTF-8"?>
<schema name="example-data-driven-schema" version="1.6">
  <uniqueKey>id</uniqueKey>
  ...
  </schema>

First important point in this document, the tag id must be specified. This is the same field specified in <entity name="MyView" pk="id"></entity> and the same of <field column="id" name="id" />.

The other fields of our entity (query) are informed in this format:

    <!-- Query Fields -->
    <field name="id"  type="int" indexed="true" stored="true"  />
    <field type="string" indexed="true" stored="true" name="name" />
    <field type="date"  indexed="true" stored="true"  name="created" />
    <field type="date"  indexed="true" stored="true"  name="modified" />

For allowing the full text search, that means, the ability to search in any field of our documents, a special field can be informed, in our case the field name text will do the trick, this field is of text_general type, it’s multi valued which allows this field receive multiple values in different formats and types and not stored. After this, we can use a copyField, defined by the tag , this is equivalent to “say” to Solr:

For any index found ( source="*" ) , copy the result to the field ( dest="text") and return the result.

<field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>
<copyField source="*" dest="_text_"/>

Creating the core in Solr

With the files configured, we need to create the following folders in the Solr Server, where “Search” is the name of our core:

/path/to/solr/data/Search

/path/to/solr/data/Search/conf

/path/to/solr/data/Search/data

In /path/to/solr/data/Search/conf, we will place our 3 files: solrconfig.xml, schema.xml and data-config.xml and any other config files such as currency.xml, elevate.xml, etc.

Now, in the Solr Server, in the Core Admin, create a new core called “Search”:

After that, select the core created in the Core selector and navigate to DataImport and execute a full import, for this, select the command “full-import” and click on the button “Execute” and after that “Refresh Status”. Alternatively, you can also check the “Auto-Refresh Status” field and then won’t be necessary click on “Refresh Status”. The response should be similar to this:

Performing a simple query:

That’s all for this post!

In the nexts blog posts I will keep talking about these points:

Solarium PHP Solr client library: Connection, Search options, returning results.
CakePHP Solr Behavior: Performing a search, adding, removing documents.
Displaying Solr search results in your front-end.

Lorena Santana - Platform Developer

PySpark and Parquet - Analysis

Dan Voyce — Tue, 16 Jul 2019 08:28:26 +0000

Pandas is known to be a Data Scientist’s tool box, because of the amazing portfolio of the functionalities provided by the Pandas library. But Pandas performs really bad with Big Data and Data which you cannot hold in memory. That is where Spark comes into picture. Spark is a Cluster computing system build for large scale data processing.

Why use Spark?

Spark is a framework that supports both batch processing (eg. Hadoop and MapReduce), and real time processing (eg. Apache Storm).

Lot of high level API’s available for Spark. You can code in Scala, Java, Python and R.

Abstractions and Concepts

DAG (Direct Acyclic Graph) - Directed graph which controls the flow of execution.
SparkContext - Orchestration within the Spark cluster.
Resilient Distributed Dataset - Immutable, any activity (like a filter or a map) on the RDD created a new RDD.
Transformations - Activities on RDD’s
Actions - Lazy loading, processing as and when needed based on the DAG. Collecting data, get counts are all actions.

What is PySpark?

Pandas, gives you the best tools for Data Analysis, and Spark gives you the capability to work with Big Data. PySpark gives you the best of both worlds. You can work with Big Data, without having to really know the underlying mechanisms of cluster computing. PySpark gives you Pandas like syntax for working with data frames.

For example, reading a CSV in Pandas:

df = pd.read_csv('sample.csv')

#Whereas in PySpark, its very similar syntax as shown below. 

df = spark.read.csv('sample.csv')

Although there are couple of differences in the syntax between both the languages, the learning curve is quite less between the two and you can focus more on building the applications.

Working with Parquet

Parquet is a high performance columnar format database.
To read more about the benefits of storing the data in columnar storage, please visit: https://blog.openbridge.com/how-to-be-a-hero-with-powerful-parquet-google-and-amazon-f2ae0f35ee04.

To read a parquet file from s3, we can use the following command:

df = spark.read.parquet("s3://path/to/parquet/file.parquet")

#Converting compressed CSV files to parquet, with snappy compression, 
#takes approximately 40 minutes for each day’s worth of POI_EVENTS data. 
#This can be done easily with PySpark using the following code. 

for i in range(1, 31):
    print("Processsing Day %02d" % i)
    csv_path = "s3://test-data/2018/05/%02d/*" % i
    p_path = "s3://locally-sanjay/2018/05/%02d/" % i
    df = sqlContext.read.option("header", "false").csv(csv_path)
    df.write.parquet(p_path, mode='overwrite')

Downsides of using PySpark

The main downside of using PySpark is that Visualisation not supported yet, and you need to convert the data frame to Pandas to do visualisation, it is not recommended because converting a PySpark dataframe to a pandas dataframe loads all the data into memory. However, you can use the “sample” method to convert parts of the PySpark dataframe to Pandas and then visualise it.

Best practices using PySpark

pyspark.sql.functions library provide built in functions for most of the transformation work. However you can write your own Python UDF’s for transformation, but its not recommended. The built-ins work inside of the JVM which Spark uses, and running Python UDF’s will make the JVM spin up a python instance and pip all the data to that instance and pip all the output back from the instance to the JVM, which will affect the performance significantly:
Do not iterate row by row in Dataframes.
Do not convert the entire PySpark dataframe to Pandas, always sample and convert to Pandas.

Technology Stack

The following technology stack was used in the testing of the products at LOCALLY:

Amazon Spark cluster with 1 Master and 2 slave nodes (standard EC2 instances)
s3 buckets for storing parquet files.
Zeppelin notebook to run the scripts.
The performance and cost on the Google Cloud Platform needs to be tested.

Ultimately we went with ORC as our main storage format at LOCALLY, but depending on your specific use-case Parquet is also a solid choice. ORC is the main driving force behind LOCALLY's Location Intelligence Platform

Sanjay Kumar - Data Engineer

Graduate student in Data Science, skills in Spark, Presto, BigQuery and Big Data.

Creating a proof of concept for Spatial Joins

Dan Voyce — Tue, 16 Jul 2019 08:15:48 +0000

So as part of our geospatial strategy we have at LOCALLY we have a huge database of POI’s, as part of this we have started to merge this with our library of geofences allowing us to get more granular control and perform things such as a dwell time calculations to help provide more value to our data.

Creating a test dataset:

Now the data we usually use is pretty huge, my test database alone is over 150GB and we don't have time to wait around all day while calculations happen on our local machines so the trick is to create a small and individually verifiable dataset to create a proof of concept for the larger whole.

We have a dataset for the District Of Columbia in the US:

Now - even with this super restricted data set it will still take a large amount of time to perform calculations in QGIS with the huge number of vertices of that surrounding polygon, so to make things simple I use the Geoprocessing tools in QGIS to “simplify” this polygon:

(You can also do this by hand).

Now we have a simplified polygon we can overlay our POI database to get an idea of how much data we are going to be working with:

Yep as expected this is a pretty huge amount, we are actually only interested in a very tiny subset of these (which I will describe later) but this needs to be an iterative process in order to not end up with a 24 hour join on the final dataset.

So first lets discard anything outside of the simplified polygon we have created, we can do this using the “Clip” functionality in QGIS:

Much better. So now this brings us onto how we manage the Geofencing aspect. These pink circles represent every single point POI within the DC Metro (ish) area, however points are just that - single points. We can tell whether a device gets near to a point but we cant really signal intent from that action. This is where we need Geofences. So we have our database of building Geofences we have created for items that interest us

So in this case lets take our Geofences from near the Washington Capitol Building (ignore the non-matching Geofences - the underlying map is different but I wanted the labels):

Each of these buildings here represent some foot traffic locations that we can use as our proof of concept.

Not every POI is aligned with a Geofence of a building, there are 2 ways we can handle this:

Create an inverse dataset that creates a new layer for any point NOT inside a Geofence and then create a small radius around these as a self contained Geofence.
Use a nearest-neighbour function search to approximate their locations.

Actually we will need to do both in a fashion depending on the data but that is outside of the scope of this. For now lets just focus on the items that ARE inside the Geofences. We can see that on the left is the Washington Marriot, the red dot to the right of that is the POI point that shows this.

From this now we can do a join within QGIS to create a new dataset that has the attributes joined:

This will now create a new table in my local test database that contains all of the information of the POI with a join to the Polygon ID it is contained it, with the create separate features for each located features this will also mean that there will be several items per polygon, internally we use proprietary methods to identify separate boundaries for each of these locations thereby splitting up the single polygon into multiple polygons containing individual POI’s.

Daniel Voyce - CTO

Displaying Geofences and Polygons on Google Maps

Dan Voyce — Tue, 16 Jul 2019 08:02:21 +0000

Well, the last few weeks I had been working in the redesign of LOCALLY Engage which will be launched in the next few days! This feature was quite challenging and required me to use things I haven’t used for a long time! One of the things I had to do was plotting a map to display the Beacons and Geofences of a certain customer.

The scope was pretty simple in our case as Geofences were always polygons or circles, while the beacons were represented by markers.

As the proposal of this dashboard was make everything dynamic which means that all the data had to be pulled with AJAX requests. (We decided against moving to React at this time because the legacy system we inherited was already built using CakePHP's templating language) The map info also needed to pulled via AJAX and then displayed in the client side.

After building all the functions in the back end, my decision wouldn’t had been different than use Maps JavaScript API which is a very simple way to achieve this!

Let’s do some coding!

The Server response
Imagining that you have a JSON response like that for Beacons:

{  
   "beacons":[  
      {  
         "lat":-37.829996,
         "lng":144.967707,
         "address":"65 Coventry Street, Southbank, Victoria 3006, Australia"
      },
      {  
         "lat":-37.829857,
         "lng":144.967851,
         "address":"1021 / 65 Coventry street, Southbank, Victoria 3006, Australia"
      }
   ]
}

And a JSON response like that for Geofences:

{ 
  "geofences": [{
                "type": "POLYGON",
                "object": "POLYGON((-76.94644698125 38.74057911875,-76.33609541875 38.74057911875,-76.33609541875 39.35093068125,-76.94644698125 38.74057911875))",
                "description": "A nice Polygon!",
                "lat": 38.94403,
                "lng": -76.539546,
                "coordinates": [
                    {
                        "lat": 38.74057911875,
                        "lng": -76.94644698125
                    },
                    {
                        "lat": 38.74057911875,
                        "lng": -76.33609541875
                    },
                    {
                        "lat": 39.35093068125,
                        "lng": -76.33609541875
                    },
                    {
                        "lat": 38.74057911875,
                        "lng": -76.94644698125
                    }
                ]
            },
             {
                "type": "RADIUS",
                "object": "POINT(-83.4885590000 42.5722669000)",
                "description": "An amazing circle!",
                "lat": 42.5722669,
                "lng": -83.488559,
                "radius": 0.010840289
            }]
}

Creating a Google Map

Let’s create a very basic map to display our objects:

<!DOCTYPE html>
<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no">
    <meta charset="utf-8">
    <title>Beacons and Geofences Map</title>
  </head>
  <body>
    <div id="map"></div>
    <script>
       function initMap() {
        map = new google.maps.Map(document.getElementById('map'), {
          center: {lat: -34.397, lng: 150.644},
          zoom: 5,
          mapTypeId: google.maps.MapTypeId.ROADMAP
        });
      }
    </script>
    <script async defer
    src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap">
    </script>
  </body>
</html>

Displaying Beacons and Geofences

Now, displaying the Beacons, looping the response object and:

var markers = [];
//Creating the icon
var icon = new google.maps.MarkerImage('/path/to/icon/image.png', new google.maps.Size(32, 32));

//Let's display the Beacon address when clicked on the arrow
var markerContent = '<p>' + beacons.address + '</p>';

var infoWindowMap = new google.maps.InfoWindow({
    content: markerContent
});

//Now the marker, our Beacon
var marker = new google.maps.Marker({
    position: {
        lat: beacon.lat,
        lng: beacon.lng
    },
    draggable: false,
    icon: icon
});
// When we click on the marker, show the window with the address
marker.addListener('click', function() {
    infoWindowMap.open(map, marker);
});

//Closing the info Window with when the mouse is out of the beacon
google.maps.event.addListener(marker, 'mouseout', function() {
    infoWindowMap.close();
});

//Setting the beacon in the map
marker.setMap(map);
//Let's store these beacons in an array, so we can cluster them! 
markers.push(marker);
//Clustering the beacons
var options = {imagePath: '/path/to/cluster/image'};
var cluster = new MarkerClusterer(map, markers, options);
To display the geofences, it is as simple as it can be. Looping the geofences object, we need to know what type of object we’ll create, as the API has different options and functions for polygons and circles:

//Configuration for objects type: POLYGON
if (geofence.type == 'POLYGON') {
    var paths = geofence.coordinates;
    var polygon = new google.maps.Polygon({
        paths: paths,
        strokeColor: '#FF0000',
        strokeOpacity: 0.8,
        stokeWeight: 2,
        fillColor: '#FF0000',
        fillOpacity: 0.35
    });

    //Set the polygon in the map
    polygon.setMap(map);

    //Plot the bounds
    var bounds = new google.maps.LatLngBounds();
    for (var i = 0; i < paths.length; i++) {
        bounds.extend(paths[i]);
    }
    map.fitBounds(bounds);
} else {
    //It is a circle!
    var circle = new google.maps.Circle({
        strokeColor: '#FF0000',
        strokeOpacity: 0.8,
        strokeWeight: 2,
        fillColor: '#FF0000',
        fillOpacity: 0.35,
        map: map,
        center: {
            lat: geofence.lat,
            lng: geofence.lng
        },
        radius: geofence.radius * 1609.344,
    });
    //Set the circle in the map
    circle.setMap(map);
    //Plot the bounds
    var bounds = new google.maps.LatLngBounds();
    bounds.union(mapType.getBounds());
    map.fitBounds(bounds);
}

Final Result, Map done!

The map should look as cool as that!

The result is a very nice map combining Geofences and Beacons and this info is handy for many things! The Google Maps JavaScript API really makes the task easy and trustful, with a few code lines, show where your geofences are and what beacons you have around!

Lorena Santana - Platform Developer

Starting our Blockchain Programming Journey

Dan Voyce — Tue, 16 Jul 2019 07:56:09 +0000

We started this journey at the beginning of June 2018. We were a small, talented and very passionate team of IT specialist. We evolved and grew up since them but we are still very compact. We were about to engage into the crypto world and start building a completely new product called LOCALLY Reward.

LOCALLY Reward is an engagement platform build on the top of the NEM Blockchain. It rewards people with NEM cryptocurrency based on the data they share on our systems. We have on the behind scenes a couple of layers of Machine Learning, A.I and Data processing algorithms to help us to build quality data.

We are the mobile team and our main goal on this project is to build a framework for iOS and Android, that allows people to share data and monitor their rewards.

The NEM Account

When you use LOCALLY Reward for the first time, we create a NEM account for you so you can carry on and withdraw your funds. A NEM account is like a bank account on the crypto world, it has an address and a public and private key. The address and public key are used as your identifier to receive and send funds on the blockchain. The private key is like your password to access your account.

NEM has a public API called NIS and at the time we started this project, we thought most of the features related to account management would be handled through API calls and network services. That was not the case and we ended up having to implement all features related to NEM accounts by ourselves (more about that in a second).

The NEM account lives on the heart of LOCALLY Reward, it's a vital piece of infrastructure and it was by far one of the biggest we faced on this project. The main reason behind that is because to create a NEM account we had to also deal with a couple of crypto libraries, algorithms, hashes and data structures that had to be implemented in a very specific order to be in conformance with the NEM specifications.

In addition to that, by the time we started this project, there was a lack of documentation and solid implementations you could follow. Fortunately, NEM has a very active and supportive community and they came up to the rescue when the hard time hit. We wouldn't be able to overcome this issue without them but even with all the support we had, it still cost us a lot of time coordinating in different time zones, communicating, etc. We only reached the bottom of this problem when we had a chance to put our hands on a decent piece of code implemented in C# (Thanks Kailin).

We are porting this implementation to the Mobile Platforms (Swift and Kotlin). We started with Swift because we thought it would be more challenging due to the language volatility and the lack of reference we had at that time.

The NEM account generation is all based on the concept of KeyPairs (Private and Public Keys) where the private key is the pivot of the other elements of creation.

Here is a quick hint of the private key generation:

var privateKeyBytes = [UInt8](repeating: 0, count: keySize)
createPrivateKey(&privateKeyBytes)

As you can see the private key is generated by a 32 bytes array shuffled randomly and converted to hexadecimal.

In Swift we use the private key to generate the public one and then we create a KeyPair.

static func generatePublicKey(fromPrivateKey privateKey: String) -> String {

    var publicKeyBytes = ['UInt8'](repeating: 0, count: keySize)
    var privateKeyBytesEndian = privateKey.toEndianArray(privateKey.bytes.count)
    createPublicKey(&publicKeyBytes, &privateKeyBytesEndian)

    return Data(bytes: publicKeyBytes).toHexString()
}

With the public key and a network selected, we then generate the account address using Sha3 and RipeMd160.

static func address(withPublicKey publicKey: String, network: Network = .mainnet) -> String {

        // First Step
        let sha3 = publicKey.toByteArray().sha3(.keccak256)

        // Second Step
        let sha3data = NSData(bytes: sha3, length: sha3.count)
        let RIPEMD160 = RIPEMD.digest(sha3data) as NSData

        // Third Step
        let networkRIPEMD160Bytes = [network.rawValue] + (RIPEMD160 as Data).bytes

        // Fourth Step
        let networkRIPEMD160BytesSha256 = networkRIPEMD160Bytes.sha3(.keccak256)

        let checksumArray = [networkRIPEMD160BytesSha256[0],
                             networkRIPEMD160BytesSha256[1],
                             networkRIPEMD160BytesSha256[2],
                             networkRIPEMD160BytesSha256[3]]

        let checksumBuffer = networkRIPEMD160Bytes + checksumArray
        let checksumData = Data(bytes: checksumBuffer, count: checksumBuffer.count)

        return Base32Encode(checksumData)
}

With a public, private key and address in hands we can now create a NEM Account.

In Swift we do that in a single line of code:

Account(network: network)

The NEM Wallet

The NEM wallet is basically a place where we store NEM accounts. We use a public account, a selected network and password to generate these digital wallets using crypto algorithms such as AES256 and Keccak. The great thing about this wallet is that it’s fully compatible with any other NEM application. For instance, once you have an account on LOCALLY Reward you can export your wallet and download and access your account details in any other desktop client (Mac, Linux and Windows) [https://nem.io/downloads/].

Here a snippet of the wallet generation:

static func generate(withAccount account: PublicAccount, password: String, network: Int = Network.mainnet.intValue) -> Wallet? {
        guard let encryptedKey = Crypto.encryptPrivateKey(privateKey: account.privateKey, withPassword: password) else { return nil }
        return Wallet(address: account.address, encrypted: encryptedKey.encrypted, iv: encryptedKey.encryptedIV, network: network)
}

The idea here is to create an encrypted private key that will be stored securely on the digital wallet. Let’s have a look inside the encryptPrivateKey method.

At first, we give 20 rounds of derived Sha3 to the password, then we convert the derived Sha3 to hexadecimals and send it with the password to an AES256Encrypt.

static func encryptPrivateKey(privateKey: String, withPassword password: String) -> EncryptedKey? {

    guard privateKey.isHexadecimal else { return nil }

    let derivedSha3 = derivedPassSha(string: password, count: 20)

    return AES256Encrypt(inputString: privateKey, hashKey: derivedSha3.toHexString())
}

The AES256Encrypt receives the private key, the hash generated with the password and a random IV (Initialization Vector).

static func AES256Encrypt(inputString: String, hashKey: String, iv: [UInt8] = AES.randomIV(16)) -> EncryptedKey? {

    let inputStringBytes = inputString.toByteArray()

    let inputStringData = Data(bytes: inputStringBytes, count: inputStringBytes.count)

    guard let encryptedData = inputStringData.aesEncrypt(key: hashKey.toByteArray(), iv: iv) else { return nil }

    return EncryptedKey(encrypted: encryptedData.toHexString(), encryptedIV: iv.toHexString())
}

The algorithm generates a byte array with the private key and another one with the hash that was passed. Then, it sends both the random IV to another AES encryption algorithm to create a byte array. When the byte array returns we generate two hexadecimal strings, one with the returned byte array and another one with the random IV.

Conclusion

In this chapter, we covered some of the challenges we faced in building NEM accounts and digital wallets on the iOS platform. In the next chapter, we will talk about signed transactions (mosaics, signed transactions, messages, etc) and how we built a network layer capable of communicate with the NEM blockchain.

Eduardo Dias - Lead Mobile Platform Developer

I have been working on the software development industry for almost two decades. With a solid understanding of all software development life cycle, I worked in a wide range of projects delivering results that provide great user experience and reach beyond clients expectations.

LOCALLY Reward (LocationCore) & NEM

Dan Voyce — Tue, 16 Jul 2019 07:44:38 +0000

This is the first of some blogs we wrote discussing the development of LOCALLY Reward about our experiences with developing on NEM.

If you don’t know, NEM is a blockchain that has its native currency of XEM, it is designed in such a way that users can interface with it through the use of APIs (mostly anyway - more on that later…), this means that rather than obscure and often untested languages (Solidity for example) you can use tools and environments you are comfortable with.

NEM is a little different than something like Ethereum for example, from my experience NEM is designed as a development platform, it doesn’t currently work with specific smart contracts, instead leaving the functionality and infrastructure up to the developers, this is great to work at the speed your business runs because you can use something like NEM to migrate parts of your business to the blockchain without too much trouble.

So what exactly has LOCALLY Reward been developing?

In short - ALL THE THINGS. It has been a great couple of months, personally I love when a team is in that space of just pure creation, and we have a great team building this product.

The product however isn’t only the tangible application that LOCALLY Reward will be initially released as. We have hired Blockchain Directors, Data Scientists, Platform developers, more app developers and QA engineers. LOCALLY Reward is part of a much bigger product and all pieces need to work together.

So for each of these updates I will focus on some of the items I have experience with developing and leave the mobile jargon to articles written by my other developers.

Trusted NIS servers, Mobile applications and Private Keys

The biggest hurdle we had to overcome recently was surrounding the method of payment of rewards for LOCALLY Reward. We decided from the start that the LOCALLY Reward app would generate a wallet that was compatible with NEM’s NanoWallet, this was both a strategic decision and a short-cut for us. For this LOCALLY Reward Beta App we decided that we would be pushing it out to users both technical and non-technical so the decision to generate a working wallet with NanoWallet was preferable to having the app juggle private keys and thus avoiding the complicated process of using that key to setup a working wallet on NanoWallet. It also means that users will have immediate access to their funds and all mosaics, mosaics can only be transferred by having enough XEM to pay for the transaction fee, so this also solved this issue.

The added benefit of this was that because we are moving the onus from the device to initiate blockchain transactions, we could now treat this account as a “receive only” actor within our system meaning that we can use NIS to sign and broadcast these transactions rather than attempting to sign this transaction locally and running into the 'fee' issue.

(NB: NEM2 / Catapult will have a great feature whereby the Requester can choose who pays the fees - our end goal will be to use this functionality when Catapult public chain is ready!)

This again introduces complexity in other ways, because the main LOCALLY Reward account would now be initiating the transaction to transfer mosaics to the user, the app needed to talk to something to let us know how much to pay the user and when to do so! So far we had managed to avoid any centralisation of the infrastructure or data flow but due to the fact that a mobile app cannot run a local NIS node there isn't much choice in this case - so from a technical standpoint how do we handle this without having to potentially spin up a cluster of servers that can scale horizontally to handle future influx?

Enter Lambda

For those of you not aware, serverless programming is the next big thing (apparently?), personally I haven’t fallen in love with it but it offers some great solutions to specific use cases, in this case it was “How can we produce an infinitely scalable API without spending a fortune on servers?”

The Lambda Function

NB: I am not a Javascript programmer, however I have been doing a lot more Node since this article was written and I am aware how horrible the callbacks are below...

The function below basically allows us to build the transaction and reward the user based on the amount and types of data they have sent to us. This will be run at a predetermined interval (undecided exactly when as of yet).

exports.handler = function (event, context, callback) {

    // Sanity check the data
    if (sanityCheck(event)) {

        var nem = require("nem-sdk").default;
        var crypto = require('crypto');

        var endpoint = nem.model.objects.create("endpoint")('http://ournode', nem.model.nodes.defaultPort);
        var common = nem.model.objects.create("common")("", "privatekeyfromenv");
        var mosaicDefinitionMetaDataPair = nem.model.objects.get("mosaicDefinitionMetaDataPair");
        var transferTransaction = nem.model.objects.create("transferTransaction")(event.address, 0, "Test " + Date.now());
        var mosaicAttachment = nem.model.objects.create("mosaicAttachment")("LOCALLY Reward", "lcc", event.amount * 100);

        transferTransaction.mosaics.push(mosaicAttachment);

        // Need mosaic definition of LOCALLY Reward:lcc to calculate fees, so we get it from network.
        nem.com.requests.namespace.mosaicDefinitions(endpoint, mosaicAttachment.mosaicId.namespaceId).then(function (res) {

            var neededDefinition = nem.utils.helpers.searchMosaicDefinitionArray(res.data, ["lcc"]);
            var fullMosaicName = nem.utils.format.mosaicIdToName(mosaicAttachment.mosaicId);

            if (undefined === neededDefinition[fullMosaicName]) return console.error("Mosaic not found !");

            mosaicDefinitionMetaDataPair[fullMosaicName] = {};
            mosaicDefinitionMetaDataPair[fullMosaicName].mosaicDefinition = neededDefinition[fullMosaicName];

            // The difficulty with fee calculation is that it needs the current supply, many libraries out there are incorrectly coded to use
            // the initial supply
            nem.com.requests.mosaic.supply(endpoint, nem.utils.format.mosaicIdToName(mosaicAttachment.mosaicId)).then(function (res) {

                mosaicDefinitionMetaDataPair[nem.utils.format.mosaicIdToName(mosaicAttachment.mosaicId)].supply = res.supply;

                nem.com.requests.mosaic.supply(endpoint, fullMosaicName).then(function (res) {

                    mosaicDefinitionMetaDataPair[fullMosaicName].supply = res.supply;

                    var transactionEntity = nem.model.transactions.prepare("mosaicTransferTransaction")(common, transferTransaction, mosaicDefinitionMetaDataPair, nem.model.network.data.testnet.id);

                    // Transaction is broadcast here
                    nem.model.transactions.send(common, transactionEntity, endpoint).then(function (res) {
                        response = {
                            statusCode: 200,
                            body: JSON.stringify(res)
                        };
                        callback(null, response);
                    });

                }, function (err) {
                    console.error(err);
                });
            }, function (err) {
                console.error(err);
            });
        }, function (err) {
            console.error(err);
        });
    }    
};

Access to the Lambda function will be handled via Amazon Cognito user pools.

How are users rewarded and what data is accepted?

LOCALLY Reward rewards users at the point of collection, in this case the device will cache the data ready to be sent to LOCALLY Reward, the app will do this at a pre-determined interval (We are thinking for the beta to have it every day but possibly once a week when it is live).

Once the data is ready to be sent several things happen:

The device has maintained a rolling hash of the data that has been collected (from the device, Facebook etc), this hash is ‘finalised’ at the cut off point, this is so the user can compare the hashes on the blockchain transaction with their previous chunk of sent data to ensure they have been paid.

The transaction is initiated with this hash as the message

The user can see and verify that their data has been paid for by comparing the hashes.

Daniel Voyce - CTO