DEV Community: Gianluca Fabrizi

1BRC in PHP FFI + Rust

Gianluca Fabrizi — Fri, 14 Feb 2025 09:13:00 +0000

We have tried multi-threading in PHP to speed up execution time; the results are good, but far from perfect. Is there another way we can improve PHP's performance?

In the previous post, we gave an overview of 1BRC, tried to push the limits of PHP when discussing performance optimization, and ran our best PHP script on an EC2 instance.

The results were not bad, but not noteworthy either: 17.0636 seconds (the fastest Java code took 1.535 seconds).

So what are we supposed to do? Call it a day and get on with our lives? No, obviously not!
We could "cheat" our way to a better score, by abusing one of Python's winning strategies: letting external libraries do the heavy lifting job!

Foreign Function Interface

One of the ways to optimize an interpreted language is by moving the slow operations in an external module, usually written in a low-level language.
In PHP you can write system-wide modules and enable them in php.ini; this is useful for generic functions or for code that is not specific to one application.
Since version 7.4 PHP introduced a new feature: Foreign Function Interface (FFI).
FFI is a method for calling external libraries in your PHP coding, without changing global PHP configuration.
This method is more flexible than dealing with modules, but configuring it could be a bit daunting at first.

Let's try to wrap a Rust solution of 1BRC in a PHP script (yes, ok, we are definitely cheating).

The Rust solution

To keep things simple we need a Rust solution that:

it's fast
it's written in a clear way
it's composed of a few files

There's no need to explain point 1; points 2 and 3 are needed because we are going to modify the code to make it work as a module.
I love Rust, but I'm not a Rust programmer, so the simpler the code the better.

I choose the solution written by Flavio Bizzarri https://github.com/newfla/1brc_rust

Compiling Rust module

First of all, we clone the repository, then we edit the Cargo.toml file to add some options:

[profile.release]
lto = true
strip = true
panic = "abort"
debug = false
opt-level = 3
codegen-units = 1

[lib]
crate-type = ["cdylib", "lib"]
bench = false

In the [profile.release] section, we enabled additional performance optimizations (debug, opt-level, codegen-units); we added the [lib] section, where we specify that we want to compile the source as a cdylib library (shared libraries that can be linked into external programs).

main.rs file is used just to call adv::process; we remove this file and add a run() method in lib.rs:

#[no_mangle]
pub extern "C" fn run(filename: *const c_char) -> *mut c_char {
    let c_str = unsafe {
        assert!(!filename.is_null());
        CStr::from_ptr(filename)
    };
    let path = c_str.to_str().unwrap().to_string();

    adv::process(path)
}

#[no_mangle] disables the mangle (in short: it keeps the function's name in the exported library) and marks this function as "to export".

We are cheating, but in a responsible way 😅: from PHP code, we pass the weather data filename to the Rust module. Then the Rust module returns the station's aggregated data to be displayed.
PHP is a loosely-typed language, while Rust is a strongly-typed language, so moving data between the two can be a bit of a challenge-in-the-challenge. We need libc crate and ffi::CStr from std.

The code needed to convert from PHP String to Rust string slice has been taken from "The Rust FFI Omnibus"; using it's words:

Getting a Rust string slice (&str) requires a few steps:

We need to make sure that the C pointer is not NULL as Rust references are not allowed to be NULL.

Use std::ffi::CStr to wrap the pointer. CStr will compute the string's length based on the terminating NULL. This requires an unsafe block as we will be dereferencing a raw pointer, which the Rust compiler cannot verify meets all the safety guarantees so the programmer must do it instead.

Ensure the C string is valid UTF-8 and convert it to a Rust string slice.

Use the string slice.

In adv.rs we use this code:

let json_string = CString::new(serde_json::to_string(&cities).unwrap()).unwrap();

json_string.into_raw()

to return a JSON string to the PHP script.

That's it for Rust; we can compile the library with:

cargo build --release

The PHP script

On the PHP side first of all we need a class to manage the input and output of th Rust module. Let's create a file called libonebrc.php:

<?php
final class LibOneBrc {
    private static $ffi = null;

    public function __construct() {
        if (is_null(self::$ffi)) {
            self::$ffi = FFI::cdef("char* run(const char* str);", "rust/libonebrc.so");
        }
    }

    public function run($filename) {
       $resultPtr = self::$ffi->run($filename);
       return FFI::string($resultPtr);
    }
}

The constructor's code uses FFI::cdef() to import the Rust function from the rust/libonebrc.so file.
Here we have to declare the extern function's signature using C code, so the Rust c_char parameters, become char*.

NOTE: it's also possible to use a .h header file to specify the function(s) that PHP needs to know about; since we only need one simple function, it is easier to declare it inline in PHP code.

The run() method invokes the run method of the Rust module (self::$ffi->run($filename)). We called both this wrapper method and the Rust function with the same name (run()); this is only a coincidence (...or lack of fantasy); it's not mandatory.
FFI::string converts the pointer to a String usable in PHP.

We also need an index.php file to instantiate this LibOneBrc class and to print the results:

<?php

require_once "libonebrc.php";
$libOneBrc = new LibOneBrc();

$filename = "./rust/measurements.txt";

$result = json_decode($libOneBrc->run($filename));

echo "{" . PHP_EOL;
$isFirstRow = true;
foreach ($result as $key => $value) {
    if (!$isFirstRow) {
        echo "," . PHP_EOL;
    } else {
        $isFirstRow = false;
    }

    echo "\t" . $value->city . '=' . $value->max / 10 . '/' . $value->min / 10 . '/' . round($value->sum / $value->count / 10, 1);
}
echo PHP_EOL . "}" . PHP_EOL;

Nothing interesting here: we call our run() method, passing it the measurements filename.
The JSON string from Rust contains temperatures as integers, so we need to divide them by 10 and calculate the average temperature for each station.

The benchmark

Let's run this code on the EC2 instance. The configuration is the same as last time: an m6a.8xlarge with 32 vCPUs and 128GB of memory. For the hard disk, I opted for a 200GB io1 volume (to reach 10,000 IOPS).

We run it with:

perf stat -o 1B-ffi.log -r 10 -d php app/index.php

and these are the results:

 Performance counter stats for 'php app/index.php' (10 runs):

          58802.93 msec task-clock                       #   29.718 CPUs utilized            ( +-  0.26% )
              4736      context-switches                 #   80.191 /sec                     ( +-  3.80% )
                57      cpu-migrations                   #    0.965 /sec                     ( +- 13.37% )
             52703      page-faults                      #  892.378 /sec                     ( +-  1.33% )
   <not supported>      cycles                                                      
   <not supported>      instructions                                                
   <not supported>      branches                                                    
   <not supported>      branch-misses                                               
   <not supported>      L1-dcache-loads                                             
   <not supported>      L1-dcache-load-misses                                       
   <not supported>      LLC-loads                                                   
   <not supported>      LLC-load-misses                                             

            1.9787 +- 0.0197 seconds time elapsed  ( +-  1.00% )

1.9787 seconds! 🥳 🎉

This is a surprising result, considering the overhead of calling an external module and the fact that we are still making some calculations on the PHP side of the app.

Conclusions

After this 2-parts-journey we can affirm that:

PHP is slow, but the performance improves significantly when using threads
Performance tuning is a game of trade-offs: you can improve the speed of a task by saturating all the CPU cores, but your system will become unresponsive. In PHP this is a problem if your application needs to accept more than one connection at a time
For heavy tasks, you can delegate to optimized external libraries

The full code is available on Github:
https://github.com/gfabrizi/1brc-php-ffi

I hope you enjoyed the post!

1 Billion Rows Challenge in PHP

Gianluca Fabrizi — Fri, 31 Jan 2025 09:09:00 +0000

You've probably heard of the 1BRC (1 Billion Rows Challenge).

In a nutshell, it's a challenge (originally aimed at Java programmers) to write code that can solve a data calculation and aggregation problem on a file with 1 billion rows in the fastest way possible.

While we can all agree that it is not indicative of the quality of the language or the typical usage scenarios, it is an activity that leads to a better understanding of the programming language being used and to the discovery of more powerful techniques.

The challenge was very successful and many have tried to transfer the concept to other programming languages.

Today, we will try to tackle the challenge in the language that, at first glance, seems to have the least to say: PHP.

Let's start by reading the task specifications:

The file with the data (measurements.txt) contains one billion rows. Each row represents the temperature recorded in a weather station and is composed of two fields: the station name and the detected temperature, separated by the ; character.

Example:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2

The task is to read all the lines and calculate the minimum, average and maximum temperatures for each weather station, rounded to one decimal place.

Finally, you have to display this data on the screen in alphabetical order in the format <station name>:<minimum temperature>/<average temperature>/<maximum temperature>.

Example:

{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3, Abéché=-10.0/29.4/69.0, Accra=-10.1/26.4/66.4, Addis Ababa=-23.7/16.0/67.0, Adelaide=-27.8/17.3/58.5, ...}

The winning script took 1,535 seconds to finish; the top 5 finished under 2 seconds.

Getting to this point in PHP seems daunting, but let's see how far we can go!

DISCLAIMER: I will not be using advanced debugging and profiling tools to determine where to optimize during testing, that's not the purpose of this article.

Initial Specifications

We need to find a way to measure the performance of a PHP script. This is where the first problems come in:

How can we measure the execution time?
How much data should be passed to the script to ensure reliable results?
Where should we run the script? Should we do it locally or elsewhere (a dedicated server, a VM, etc...)?

Measuring the execution time

All Linux installations should have the command time; by placing it before a command, the operating system can return the execution time of the command passed.
This would seem to be the ideal solution, but there is a problem: it is not very precise, especially if we make it profile only one execution of the PHP script.
A better approach is to use the command perf and pass it the option -r, followed by the number of times you want to run the command to profile.
Example: perf -r 10 my_command

How much data should be passed to the script

Running the PHP script on a billion rows could take a long time; we can use a small set of 1 million rows to start doing some tests. Then we can gradually increase the number of rows until we reach 1 billion.

Where to run the script

Here, just like before, we can take a step-by-step approach. First we can run the benchmark locally on our computer, to see how the different versions compare.
There's one important thing to remember: close all non-essential programs and make sure that the tests are always run under the same conditions.
The version of PHP locally installed is 8.3.14.

Then, when we have an optimized enough PHP script, we can move on to a dedicated server or a virtual machine.
We will need a dedicated server for a short time; the best option would be to use a VM or a cloud instance.

Writing the code

First attempt

Let's start by writing a solution in PHP in the simplest way we can think of:

<?php

$fp = fopen('data/measurements.txt', 'r');

$stations = [];
while (($line = fgets($fp, 4096)) !== false) {
    [$station, $temp] = explode(';', $line);

    if (!isset($stations[$station])) {
        $stations[$station] = [];
    }

    $stations[$station][] = $temp;
}
fclose($fp);

$results = [];
foreach ($stations as $key => $value) {
    $results[$key] = [];
    $results[$key][0] = min($value);
    $results[$key][1] = max($value);
    $results[$key][2] = array_sum($value);
    $results[$key][3] = count($value);
}

ksort($results);

echo '{', PHP_EOL;
foreach ($results as $name => &$temps) {
    echo "\t", $name, '=', $temps[0], '/', number_format($temps[2]/$temps[3], 1), '/', $temps[1], ',', PHP_EOL;
}
echo '}', PHP_EOL;

We open the file data/measurements.txt for reading, populate an array with this data, and finally calculate the minimum, maximum and average temperature for each weather station.
We sort the results alphabetically and print them on the screen.

We have already created the file measurements.txt with 1 million lines using the semi-official tool create_measurements.py:

python3 create_measurements.py 1_000_000

We can then launch the first PHP script using the command:

perf stat -o results/calculate_1.log -r 10 -d php src/calculate_1.php

The -o option specifies where to save the log with the execution results.
To have reliable data we run the script 10 times (with the -r option).

It took 1.066 seconds on the laptop I use for writing (Intel N5100 CPU with 8GB of RAM).
We don't have enough information to understand whether it is a lot or a little.

Second attempt: `trim()`

From the screen output we see some strange line breaks that shouldn't be there. Let's try to apply a trim() on the temperatures:

$stations[$station][] = trim($temp);

We rerun the script and it takes 1.048 seconds. The improvement is minimal, but the screen output no longer shows those extra line breaks.

Third attempt: `(float)`

The “solution” of using trim() does not seem to be the best; a quick analysis of the code shows that PHP stores the temperatures in the array as strings and not as floats. Let's try casting to float instead:

$stations[$station][] = (float) $temp;

The execution time is 0.62 seconds, a big improvement over the previous attempt!

Perhaps we can also increase the number of rows from 1 million to 10 million to better appreciate the timing variations.
Let's try running the same script on 10 million rows:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in src/calculate_3.php on line 13

PHP's current memory limit is 128MB; the script tries to create an array with all the values of the file measurements.txt, exhausting the 128MB.
Now we could raise that limit, or rewrite the code that takes care of aggregating and processing the data.
Let's proceed with the rewriting of the code avoiding putting the entire input file in memory.

Fourth attempt: `while`

Let's try to read the data and process it in a single while loop:

$results = [];
while (($line = fgets($fp, 4096)) !== false) {
    [$station, $temp] = explode(';', $line);
    $temp = (float) $temp;

    if (!isset($results[$station])) {
        $results[$station] = [$temp, $temp, $temp, 1];
    } else {
        $results[$station][0] = $results[$station][0] < $temp ? $results[$station][0] : $temp;
        $results[$station][1] = $results[$station][1] > $temp ? $results[$station][1] : $temp;
        $results[$station][2] += $temp;
        $results[$station][3]++;
    }
}

This way we don't saturate the memory with the contents of the measurements.txt file.

Execution time on 10 million rows: 6.14 seconds
This time doesn't tell us anything about a possible improvement/deterioration of performance; let's try with the 1 million row dataset.

Execution time on 1 million rows: 0.76 seconds
The time got worse (as expected); we traded the lower RAM usage for a higher execution time.

Fifth attempt: `min()/max()`

This time we try to use the native PHP functions min() and max() to save the minimum and maximum value of each station:

$results[$station][0] = min($results[$station][0], $temp);
$results[$station][1] = max($results[$station][1], $temp);

Execution time on 10 million rows: 6.01 seconds

It's a small improvement, but an improvement nonetheless.

Sixth attempt: `if`

Perhaps we can simplify those two lines of code even further, using two simple if statements:

if ($temp < $results[$station][0]) {
    $results[$station][0] = $temp;
}

if ($temp > $results[$station][1]) {
    $results[$station][1] = $temp;
}

Execution time on 10 million rows: 4.90 seconds

Here we are, another important step in the right direction.

Seventh attempt: `!isset()`

The situation inside the while loop is this:

if (!isset($results[$station])) {
    // ...
} else {
    // ...
}

We are used to thinking of that !isset() as if it were a single statement; in reality, however, there are two: there is the not (!) and the isset().
Let's try to invert the two branches of the if:

if (isset($results[$station])) {
    if ($temp < $results[$station][0]) {
        $results[$station][0] = $temp;
    }

    if ($temp > $results[$station][1]) {
        $results[$station][1] = $temp;
    }

    $results[$station][2] += $temp;
    $results[$station][3]++;
} else {
    $results[$station] = [$temp, $temp, $temp, 1];
}

Execution time on 10 million rows: 4.90 seconds

It's pretty much the same time as the previous attempt...

Eighth attempt: pointer `&`

Another attempt we can make inside the if is to use a pointer to the array element that contains the weather station instead of calling $results[$station][0] every time:

$resultStation = &$results[$station];

Execution time on 10 million rows: 4.65 seconds

There is an improvement in timing, we keep this change.

Ninth attempt: `fgets()`

Let's focus for a moment on the while that takes care of looping the entire measurements.txt file:

while (($line = fgets($fp, 4096)) !== false) {

Why is there a 4096 as the second parameter of the fgets() function?
I saw it in the example code on the PHP documentation page, so I just mindlessly copied it into the script. However, I later checked the documentation and found that the second parameter "limits" lines that are too long.
Some comments on the PHP documentation page suggest there may be a performance issue when passing this second parameter (which is optional by the way).
Let's try removing it and see what happens:

while (($line = fgets($fp)) !== false) {

Execution time on 10 million rows: 4.39 seconds

...this confirms the fact that reading the documentation carefully is always useful

Tenth attempt: `strtok()`

The only point in the code that raises some doubts is the explode() used to split the weather station's name from the temperature.

Let's try replacing it with strtok():

$station = strtok($line, ';');
$temp = (float) strtok(';');

Execution time on 10 million rows: 3.82 seconds

With this latest version of the script, it seems that we have reached the end of the optimizations, there are no further points to optimize.

Test on dedicated hardware

Now that we have our most performant script, we can go ahead and run it on a dedicated machine.

Test on EC2 with 1 billion rows

Let's try to run this script on hardware similar to the one used in the official Java challenge:

CPU: AMD EPYC 7502P 32 cores / 64 threads @ 2.5 GHz
Memory: 128 GB ECC DDR4 RAM
2x SAMSUNG MZQL2960HCJR-00A07, 1 TB, Software RAID-1

The EC2 instance that most closely resembles it (and is cheaper) is the m6a.8xlarge with its 32 vCPUs and 128GB of memory. For the hard disk I opted for a 200GB io1 volume (to reach 10,000 IOPS).

I tried to launch the last script on the EC2 instance; this time I ran the script only 2 times.
The result was this:

 Performance counter stats for 'php src/calculate_10.php' (2 runs):

         261924.05 msec task-clock                       #    1.007 CPUs utilized            ( +-  0.72% )
               813      context-switches                 #    3.127 /sec                     ( +- 28.17% )
                 2      cpu-migrations                   #    0.008 /sec                   
              1486      page-faults                      #    5.715 /sec                   
   <not supported>      cycles                                                      
   <not supported>      instructions                                                
   <not supported>      branches                                                    
   <not supported>      branch-misses                                               
   <not supported>      L1-dcache-loads                                             
   <not supported>      L1-dcache-load-misses                                       
   <not supported>      LLC-loads                                                   
   <not supported>      LLC-load-misses                                             

            260.06 +- 1.89 seconds time elapsed  ( +-  0.73% )

260.06 seconds, or 4 minutes and 20 seconds... a truly disastrous result.

In a previous attempt (the fourth) we had to change the way the data was aggregated, due to a PHP out-of-memory error; we had seen a significant performance degradation due to this change.
Since the EC2 instance has a lot of RAM available, let's try to resume the script from the third attempt by applying the changes made from the fifth attempt onwards and raising the memory_limit in the php.ini file.

The results of this test:

 Performance counter stats for 'php src/calculate_in_memory.php' (2 runs):

         259240.23 msec task-clock                       #    0.992 CPUs utilized            ( +-  0.81% )
              1348      context-switches                 #    5.158 /sec                     ( +- 14.73% )
                 5      cpu-migrations                   #    0.019 /sec                     ( +- 10.00% )
            120810      page-faults                      #  462.253 /sec                     ( +-  0.26% )
   <not supported>      cycles                                                      
   <not supported>      instructions                                                
   <not supported>      branches                                                    
   <not supported>      branch-misses                                               
   <not supported>      L1-dcache-loads                                             
   <not supported>      L1-dcache-load-misses                                       
   <not supported>      LLC-loads                                                   
   <not supported>      LLC-load-misses                                             

            261.38 +- 2.10 seconds time elapsed  ( +-  0.80% )

261.38 seconds, one second slower than the previous version.
Probably the difference is irrelevant on such high-performance hardware.

Multi-thread PHP

The only thing left to try is to write a PHP script that takes advantage of the interpreter's multithread capabilities.
This task turned out to be quite complex for two reasons:

The PHP interpreter must be compiled with the ZTS (Zend Thread Safe) option to launch parallel executions; few Linux distributions provide an interpreter with this feature turned on;
The PHP script needs to be completely rewritten to take advantage of thread concurrency;

For the first point, the only possibility is to compile PHP from sources with the ZTS option active, and then install the PECL parallel module.
On Debian, this is possible using the commands:

# Install build tools and libraries needed
sudo apt-get -y install build-essential autoconf libtool bison re2c pkg-config git libxml2-dev libssl-dev

# Clone and build a stripped-down version of PHP with ZTS support
git clone https://github.com/php/php-src.git --branch=PHP-8.4.3 --depth=1
cd php-src/
./buildconf
./configure --prefix=/opt/php8.4-zts --with-config-file-path=/opt/php8.4-zts/etc/php --disable-all --disable-ipv6 --disable-cgi --disable-phpdbg --enable-zts --enable-xml --with-libxml --with-pear --with-openssl
make -j32
./sapi/cli/php -v
sudo make install

# Install `parallel` module from PECL
sudo /opt/php8.4-zts/bin/pecl channel-update pecl.php.net
sudo /opt/php8.4-zts/bin/pecl install parallel
sudo mkdir -p /opt/php8.4-zts/etc/php/conf.d
echo 'extension=parallel.so' | sudo tee -a /opt/php8.4-zts/etc/php/php.ini
echo 'memory_limit=-1' | sudo tee -a /opt/php8.4-zts/etc/php/php.ini

# Verify module installation
/opt/php8.4-zts/bin/php -i | grep parallel

As you can see, we downloaded and compiled the 8.4.3 version of PHP.

The ZTS version of PHP can be run with the command /opt/php8.4-zts/bin/php.

For the second point (rewriting the PHP script for multithreading), you can take inspiration from the PHP documentation and solutions to the 1BRC challenge available on the Internet.
The one I took heavily from is this one.
The overhead from multithread management mostly comes from having to cycle the measurements.txt file first to split it into chunks that match the number of cores on the machine where the script is running. Each thread will process one of these chunks that, combined, will lead to the final result.

The source code is fully available in the GitHub repository.

The results on the EC2 instance are:

 Performance counter stats for '/opt/php8.4-zts/bin/php src/zts/zts_calculate_1.php 32' (10 runs):

         521063.71 msec task-clock                       #   30.537 CPUs utilized            ( +-  0.17% )
              4416      context-switches                 #    8.521 /sec                     ( +- 12.71% )
                25      cpu-migrations                   #    0.048 /sec                     ( +- 27.73% )
             41060      page-faults                      #   79.228 /sec                     ( +-  0.21% )
   <not supported>      cycles                                                      
   <not supported>      instructions                                                
   <not supported>      branches                                                    
   <not supported>      branch-misses                                               
   <not supported>      L1-dcache-loads                                             
   <not supported>      L1-dcache-load-misses                                       
   <not supported>      LLC-loads                                                   
   <not supported>      LLC-load-misses                                             

           17.0636 +- 0.0181 seconds time elapsed  ( +-  0.11% )

17.0636 seconds... an impressive result considering the previous 260 seconds!

Conclusions

For now, we'll pause here; we're left with a time of 17.06 seconds for the execution on 1 billion rows.

In the next article, we'll see another way to face this challenge with PHP.
I'll leave you with the summary of the test results on the EC2 instance. The first 9 scripts were run on 1 and 10 million rows dataset. Script 10, the one with all the data in RAM (calculate_in_memory.php), and the ZTS script were run on a 1 billion rows dataset.

NOTE: The 1M and 1M lines tests were run with PHP 8.4.2; the 1B lines tests were run with PHP 8.4.3.

The full code is available on Github:
https://github.com/gfabrizi/1brc-php

Getting started with Cypher and RedisGraph / Part III

Gianluca Fabrizi — Mon, 29 May 2023 08:16:00 +0000

Welcome to the last part of the "Getting started with Cypher and RedisGraph" series.
In the first post we looked at what a Graph DB is and why should you use one; the second post explained the basics of querying RedisGraph.

Now we will build a very small example to show everything we learned in the previous posts.

First of all, start redis Insight with docker compose up -d.

At this point you need python3 installed on your local machine; if you don't, check online the instructions for you OS.
You also need click and redis python extensions; you can install them with:

pip3 install click redis

Now that you have everything, you can clone the demo repository

git clone https://github.com/gfabrizi/redisgraph-demo.git

enter the cloned directory and launch the data import:

python3 bulk_insert.py Routes -n City.csv -r ROUTE.csv -h 127.0.0.1 -p 6379

you should see a message like this:

b'City'  [####################################]  100%
19 nodes created with label 'b'City''
b'ROUTE'  [####################################]  100%
81 relations created for type 'b'ROUTE''
Construction of graph 'Routes' complete: 19 nodes created, 81 relations created in 0.006071 seconds

What just happened?

we used a script (bulk_insert.py) from the awesome repository redis-dataset:
https://github.com/redis-developer/redis-datasets
The script is used to import nodes (-n parameter) and relationships (-r parameter). You can specify more than one nodes and relationships file.
We imported nodes representing cities (City.csv) and relations (ROUTE.csv); the import script uses the csv filenames to give name to nodes and relations.
So now we have our graph "Routes" filled with 19 cities and 1786 routes.

It's time to open RedisInsight UI in the browser and start querying!

Visualize data

Go to "Workbench" tab in RedisInsight and launch the generic query to show all nodes and relations:

GRAPH.QUERY Routes "MATCH (a) RETURN a"

you should see a graph like this (remember to check the "All relationships" flag:

Not really helpful, right? 😔
Well, informations are there, now you have to query for what you want to know.

A little background:
Years ago a company asked me to resolve a test for an interview as backend developer. I had 3-4 days to solve this: "Imagine you have 20 goods depots, connected through cargo truck routes (each depot is connected with a maximum of 3 nearby depots). Write a script that - given deposit A and deposit B - finds the best route between them".
I struggled for days with this test, trying to learn and implement Dijkstra algorithm... with no success at all! 🥲
Well, if I knew about GraphDBs at that time, I would have passed that test! 🔝

For example, if you want to know the shortest routes between Memphis and Phoenix, you could use this query:

GRAPH.QUERY Routes "MATCH (source:City {name: 'Memphis'}), (destination:City {name: 'Phoenix'}) with source, destination MATCH p=allShortestPaths((source)-[:ROUTE*]->(destination)) RETURN nodes(p) as cities"

we wrote a first MATCH query, followeb by a second one; to use the matching nodes of the first MATCH in the second one, we need to specify them with the with keyword. This keyword is useful when you have to use one (or more) previously matched node in a following MATCH.
allShortestPaths is a function of MATCH that allows us to find every shortest paths between two nodes; if we choose to see results as text (and not the defaul graph) we could see that Redis found 2 paths:

if we wanted only one shortest path, we could use the shortestPath function:

GRAPH.QUERY Routes "MATCH (source:City {name: 'Memphis'}), (destination:City {name: 'Phoenix'}) RETURN shortestPath((source)-[:ROUTE*]->(destination))"

if you look at the textual output of the results, you'll notice that this time RedisGraph returned not only the nodes, but also the relationships between nodes.
If you only want the nodes involved in the shortestPath, change the query like this:

GRAPH.QUERY Routes "MATCH (source:City {name: 'Memphis'}), (destination:City {name: 'Phoenix'}) RETURN nodes(shortestPath((source)-[:ROUTE*]->(destination)))"

Cool, right?

Conclusion

This series showed the basic of RedisGraph and the Cypher language.
There is much more to explore and learn; if we look just at the "path algorithms", the official documentation offers many functions to use:
https://redis.io/docs/stack/graph/path_algorithm/#find-all-paths-from-a-if-the-trip-is-limited-to-10-kilometers

I hope you have followed this series and enjoyed it as much as I did!😊🎉

Getting started with Cypher and RedisGraph / Part II

Gianluca Fabrizi — Fri, 19 May 2023 08:58:00 +0000

In the first post of this series we looked at what a Graph DB is and why should you use one.
Today we'll learn what is Cypher start querying the graph!

What is Cypher

Cypher it's a declarative query language, developed internally by Neo4j since 2011.
There is an ongoing standardization process (OpenCypher) to make Cypher an open standard.
Cypher let's you easily retrieve data from the graph. It's one of the simplest query language to learn; its syntax is constructed in a way that helps visualize relationship between nodes

RedisInsight

We will be running queries in RedisInsight. If you've read the previous post of this series, you should know how to run a docker based installation of Redis.
After launching docker compose up -d, point your browser to http://localhost:8001 , RedisInsight should welcome you. Accept the EULA and you will be redirected to RedisInsight dashboard.
Next click on the "Workbench" tab (the third icon in the side menu on the left).
The upper right pane of this new screen is where you'll write all the queries.

Adding nodes and edges

Now that both Redis and RedisInsight are up & running, let's create a new graph and add one node to it:

GRAPH.QUERY Social "CREATE (:Person {name: 'Laura Phillips', age: 32})"

Every query is prepended by GRAPH.QUERY and the graph name.
Now click the green arrow to the right or press Ctrl + Enter.

The output should be similar to this:

The first time we create e node in a non-existent graph, RedisGraph will create the graph for us.
So we've just created a new graph called "Social" and added one node of type "Person".
name and age are properties that we add to the node. RedisGrap automatically add another property to each node: the unique id.

We can add more nodes in bulk:

GRAPH.QUERY Social "CREATE (:Person {name: 'Diana Hendrickson', age: 31}), (:Person {name: 'Susan Hendrickson', age: 29}), (:Person {name: 'Peter Steinmetz', age: 30}), (:Person {name: 'Louise Rosol', age: 29}), (:Person {name: 'Bryce Fett', age: 30})"

Now you should have 6 nodes of type Person in the graph. Let's check them out!

Basic querying

GRAPH.QUERY Social "MATCH (p:Person) RETURN p"

Let's analyze this query: MATCH is the keyword that describes the relationship between queried entities. It's used to 'match' something in the graph, based on some parameters. In this case we are asking for Person, and we alias our results with a p (the alias name is not important, you can choose any letter or word).
Nodes are always specified in round brackets.
The RETURN keyword returns every p found.
So this query returns every Person it finds in the graph Social.

The resulting graph should be something like this:

The previous query could also be written as:

GRAPH.QUERY Social "MATCH (a) RETURN a"

This is the generic way to view all nodes in the graph, you'll see this a lot of times. Here we are using an alias a without asking for a specific node type.

Ok, so let's say we want to retrieve all Person(s) who are exactly 30 years old:

GRAPH.QUERY Social "MATCH (p:Person {age: 30}) RETURN p"

this query introduces another feature: match for node's attribute. Here we are matching only nodes of type Person and with an attribute age equal to 30.
You can query for multiple attributes, by separating attribute: value couples with comma between the curly brackets.

Sometimes visualizing a graph of this kind isn't much helpful, a textual result might be more useful.
So click on the </> icon just above the last graph and choose Text; the output of the last query will be shown as text:

Connecting nodes and creating edges

Now we add one relation between two nodes:

GRAPH.QUERY Social "MATCH (a:Person {name: 'Diana Hendrickson'}), (b:Person {name: 'Susan Hendrickson'}) CREATE (a)-[:KNOWS {relation: 'sister'}]->(b)"

The first part of the query is a MATCH to find 2 Person ('Diana Hendrickson' and 'Susan Hendrickson'); we alias them with a and b.

Then we create the relation:

CREATE (a)-[:KNOWS {relation: 'sister'}]->(b)

relationships are always written in square brackets. We are binding node a and node b with a relation of type KNOWS. The KNOWS relation has an attribute relation with value sister.

Redis is informing us that it has created one relationship with one properties.

NOTE: The correct way to add a relation between existent nodes is by matching them first; if we try to directly create the relationship, like so:
CREATE (a:Person {name: 'Diana Hendrickson'})-[:KNOWS {relation: 'sister'}]->(b:Person {name: 'Susan Hendrickson'})
RedisGraph would create 2 new nodes and add the relation between them

The generic query for relationship has this form:

(NodeA)-[:Relationship]->(NodeB)

take one minute to appreciate how eloquent and visually clear the syntax of Cypher can be😯👏.

Ok, now we can add some more relationships in the graph:

GRAPH.QUERY Social "MATCH (a:Person {name: 'Susan Hendrickson'}), (b:Person {name: 'Peter Steinmetz'}) CREATE (a)-[:KNOWS {relation: 'married'}]->(b)"
GRAPH.QUERY Social "MATCH (a:Person {name: 'Susan Hendrickson'}), (b:Person {name: 'Louise Rosol'}) CREATE (a)-[:KNOWS {relation: 'friend'}]->(b)"
GRAPH.QUERY Social "MATCH (a:Person {name: 'Peter Steinmetz'}), (b:Person {name: 'Bryce Fett'}) CREATE (a)-[:KNOWS {relation: 'friend'}]->(b)"
GRAPH.QUERY Social "MATCH (a:Person {name: 'Laura Phillips'}), (b:Person {name: 'Louise Rosol'}) CREATE (a)-[:KNOWS {relation: 'coworker'}]->(b)"
GRAPH.QUERY Social "MATCH (a:Person {name: 'Laura Phillips'}), (b:Person {name: 'Diana Hendrickson'}) CREATE (a)-[:KNOWS {relation: 'coworker'}]->(b)"
GRAPH.QUERY Social "MATCH (a:Person {name: 'Louise Rosol'}), (b:Person {name: 'Diana Hendrickson'}) CREATE (a)-[:KNOWS {relation: 'coworker'}]->(b)"

Let's see what we just did:

GRAPH.QUERY Social "MATCH (p:Person) RETURN p"

the graph is exactly the same as before... why?🤔

The reason is that RedisInshight hides relationships by default; if you enable the "All relationship" slide in the upper right side of the graph, RedisInsight will also show you all the relationships between nodes:

Now try to create a connection between an existent node and a non-existent node:

GRAPH.QUERY Social "MATCH (p:Person {name: 'Laura Phillips'}) CREATE (p)-[:KNOWS {relation: 'married'}]->(:Person {name: 'William Stultz', age:33})"

RedisGraph creates for us the unknown node (Person 'William Stultz') and add the relationship between the known node and the new one.

Querying for relationships

Now that we've created some "connections" between nodes, we can query the graph for relation between nodes.
Let's say we want to show every married Person:

GRAPH.QUERY Social "MATCH (p:Person)-[:KNOWS {relation:'married'}]->(o:Person) RETURN p,o"

Or we need the list of every Person that knows 'Susan Hendrickson':

GRAPH.QUERY Social "MATCH (p:Person)-[:KNOWS]-(:Person {name: 'Susan Hendrickson'}) RETURN p"

here we made some subtle but important changes to the query: first of all, we don't need the Person 'Susan Hendrickson' so we haven't added an alias to it.
Second: in RedisGraph every relation has a direction (starting from Node A ending on Node B). There's no point in giving the :KNOWS relation a direction in the last query (two friends are friends to each other, it's a bi-directional relationship).
So we query for the relation without specifying the direction; the syntax (a)-[:KNOWS]->(b) became (a)-[:KNOWS]-(b).

What if instead of the list of acquaintances of 'Susan Hendrickson' we want just the count?
We can use the COUNT aggregation:

GRAPH.QUERY Social "MATCH (p:Person)-[:KNOWS]-(:Person {name: 'Susan Hendrickson'}) RETURN count(p)"

There is no graph to see here, so RedisGraph only shows the textual output of the COUNT:

Delete nodes and edges

The last query we'll look at today is the DELETE:

GRAPH.QUERY Social "MATCH (p:Person {name: 'William Stultz'}) DELETE p"

again, we first MATCH the node, then we DELETE it.

As you may remember, we created a relationship between this node and another Person ('Laura Phillips'). When we delete a node, all its relationships are deleted too. This may seems obvious, but it's something to keep in mind.

RECAP

In this post we learned the basics of Cypher and how to query RedisGraph. The topic is vast, we haveve covered a small part of it just to get you started.

In the next (and last) post, I'll show you a more practical example of RedisGraph in action.

Getting started with Cypher and RedisGraph / Part I

Gianluca Fabrizi — Mon, 15 May 2023 08:03:00 +0000

I keep a personal to-do list where i annotate technologies and frameworks i'd like to learn. Aside from the perpetual "learn Rust", there's been a "learn graph db" for months now.
Now it's time to start exploring it.
What better way to understand how much I've learned than writing a serie of posts about it?

In this first post I try to explain what is a Graph DB, why I choose RedisGraph and how to use it locally. In the next post we will see how to query the db, how to edit and add relationships. In the third (and last) post we'll be looking at some examples.

Without further ado, let's briefly see what a Graph database is.

What is a Graph database?

In "classic" relational and NoSQL databases, data are structured in a tabular way (you may think of it as a spreadsheet); this makes querying from relations between data harder (usually you have to join two or more tables).
Graph databases are structured following a graph data model.
In "graph data model" relationships between nodes are the priority and the way you lay out your data is more expressive.
Usually Graph databases come with a visualization tool that lets you see how your data is connected, thus helping you getting useful insight from your data.
The rigid schema of a relational database make it hard seek for data relationships, whereas a graph db can make it easy.

Why use a Graph database?

So graph DBs allow you to better structure and visualize your data. But why you may need one?

Top use cases are: Real-time recommendation engines, Knowledge graphs and Fraud detection
But you can also think of a social media network where people are connected with different kind of relations.

Due to their flexibility, Graph DBs can easily adapt to your data model, even if it changes over time.
One of the approach that is become more and more relevant is to use Graph databases in addition to traditional data store.
For instance you can use a MySQL db to store your users, products and sales. Than you can have a Graph db to map your users habits and create your custom recommendation engine.
You always have to keep in mind that each db type has it's own preferred use: graph db wins easily when you have to traverse relationships between entities, while a SQL db is more efficient in aggregating and grouping results.

Why RedisGraph?

Nowadays there are multiple Graph DBs: Neo4j is the first that comes in mind. Then there are AWS Neptune, ArangoDB, TypeDB, etc...

Neo4j has some tricky licensing and they seem to push hard on the "Enterprise" plan: with the community edition you can't have more than one database per installation and you can't create a cluster.
Amazon Neptune is a "pay as you go solution", not suitable for learning. It uses Apache TinkerPop Gremlin and W3C’s SPARQL as query language. It's way too easy to be locked-in.
ArangoDB has a unique multi-model approach (nodes are essentially documents stored in collections). It uses AQL, a proprietary query language.
TypeDB uses it's own query language (TypeQL); you can be locked-in pretty quick.
RedisGraph is a pluggable module for Redis, developed by RedisLabs. While being younger than it's direct competitor Neo4j (and still not as feature rich), it's simpler and faster. New features are being included in each version.

Graph Elements

A graph database consists of:
Nodes: nodes represents entities in your domain.
Edges: edges are the connections between nodes. An edge represents a relationship between a start node and an end node, thus giving the relationship a "direction".
Properties: properties are key-value pairs connected to nodes or edges. With properties you can store extra informations about an entity or a relation
Labels: labels help you categorize elements (nodes and edges)

An easy way to think about graphs is as analogous to the relationship between nouns and verbs. Nodes, or the nouns, are things such as people, places, and items. Relationships, or the verbs, are how they’re connected. People know each other and items are sent to places. The signal in those relationships is powerful. (source)

Running RedisGraph locally

To get started you can go on https://redis.com and sign up for a free Redis cloud tier. The free subscription includes 1 dedicated database, 30MB of RAM and RedisGraph module, so you can use it to start learning RedisGraph.

The most interesting way to start exploring RedisGraph though is by running a local docker container.
Here's a simple docker-compose.yml file:

version: "3"
services:
    redis-stack:
        image: redis/redis-stack:latest
        container_name: redis-stack
        volumes:
            - ./local-data/:/data
        expose:
            - 6379
            - 8001
        ports:
            - "6379:6379"
            - "8001:8001"
        environment:
            - PATH=./bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

You can launch it with docker compose up -d.
If you open your browser and point it to http://localhost:8001 you will see the main interface of RedisInsight.
RedisInsight is the web UI of Redis.
When you finished, you can stop the container with docker compose stop

In the next post

We've covered the basics of what a Graph database is and how it can be used.
In the next post we'll see how to create a database and use Cypher to query it.

✨ Porting Lambda Functions to AWS SAM

Gianluca Fabrizi — Sun, 23 Apr 2023 21:55:22 +0000

Two weeks ago I attended to JsDay 2023 (from Verona, Italy 🇮🇹).
One talk hit me in particular: "Production-ready lambdas with Node.js" by Luciano Mammino.
He explained some trick, tips and best practices to work with AWS Lambda in a production environment.
One best practice is this:

"Stop creating resources manually on your AWS account, like right now! If you are doing this, please STOP"

Ok, when he said this i felt reaaallly really guilty.

AWS SAM

There are many IAC tools that can be used with AWS. Maybe the logic option (or the one that seems to fit better) is using AWS SAM.
The AWS Serverless Application Model (SAM) is a framework for building serverless applications. It's open-source and all the configuration can be written in YAML files.
It consists of a cli tool to be installed (it's separate from aws-cli); you can use it to deploy your infrastructure, deploy (and sync) your application, for local test of your code and much much more.
I confess that i never used it (really never even heard of it...) so let's learn it by porting a previous project in SAM.

THE PROJECT

The project is my previous "Lambda Inception Architectural Pattern" (quite a mouthful, right? 🤭):
https://dev.to/gfabrizi/lambda-inception-architectural-pattern-f67

In that post I wrote blocks and blocks of code to describe roles and policies of the infrastructure... how naive!
So let's start by looking at the SAM documentation:
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html
After digging the examples and the documentation I started writing my template.yaml

THE CODE

First thing first: the roles and policies.
The simplest role to be ported in SAM is the LambdaInceptionWorker:

LambdaInceptionWorker:
  Type: AWS::IAM::Role
  Properties:
    RoleName: LambdaInceptionWorker
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole

just some boilerplate code were we define a role (Type: AWS::IAM::Role) and attach a AssumeRolePolicyDocument that says that every Lambda functions can assume this role. As we saw in the previous post, the LambdaInceptionWorker role is empty, so there's nothing more to add here.

Next is the LambdaInceptionManager:

LambdaInceptionManager:
  Type: AWS::IAM::Role
  Properties:
    RoleName: LambdaInceptionManager
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Policies:
      - PolicyName: LambdaInceptionPassRole
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - iam:PassRole
              Resource: !Sub arn:aws:iam::${AWS::AccountId}:role/LambdaInceptionWorker
      - PolicyName: LambdaInceptionCreateFunction
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:CreateFunction
              Resource: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*
      - PolicyName: LambdaInceptionDeleteFunction
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:DeleteFunction
              Resource: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*
      - PolicyName: LambdaInvokeFunction
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:InvokeFunction
              Resource: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*

Let's analyze the code section by section:

ManagedPolicyArns:
  - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

with this we are attaching an AWS managed policy to the role (the basic execution role, needed by Lambda Function URL).
Then we started adding inline policies to the role; we see just the first policy:

- PolicyName: LambdaInceptionPassRole
  PolicyDocument:
    Version: '2012-10-17'
    Statement:
      - Effect: Allow
        Action:
          - iam:PassRole
        Resource: !Sub arn:aws:iam::${AWS::AccountId}:role/LambdaInceptionWorker

here we are creating a new inline policy called LambdaInceptionPassRole; this policy allows the iam:PassRole action only to the specified resource.
In the Resource line we specify the LambdaInceptionWorker role by passing it's ARN. We are using a builtin variable to specify the account id (${AWS::AccountId}). The keyword !Sub at the beginning of the line indicates that the string contains a variable to be replaced with it's value. Another useful builtin variable is ${AWS::Region}.
The others 3 policies have the same structure, so we skip them.

Then we create the IAM user that will invoke the Inception manager function from the command line:

LambdaInceptionInvoker:
  Type: AWS::IAM::User
  Properties:
    UserName: lambda-inception-invoker
    Policies:
      - PolicyName: LambdaInceptionInvoke
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - lambda:InvokeFunctionUrl
              Resource: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:lambda-inception

The syntax of this block is the same as the previous, nothing new.

Finally we can write the definition of the Inception Manager function:

LambdaInceptionManagerFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: "lambda-inception"
    CodeUri: manager/
    Handler: manager.handler
    Runtime: nodejs18.x
    Architectures:
      - x86_64
    MemorySize: 512
    Timeout: 30
    Role: !GetAtt LambdaInceptionManager.Arn
    FunctionUrlConfig:
      AuthType: AWS_IAM
    Environment:
      Variables:
        LAMBDA_INCEPTION_WORKER_ROLE: !GetAtt LambdaInceptionWorker.Arn

we are using the type AWS::Serverless::Function to specify that we are defining a Lambda Function.
We gave it a name, specify the path where the code lies, the handler and some more common Lambda configuration.

Then we assign a role to the function. !GetAtt is another keyword that returns an attribute; in this case it returns the ARN of the LambdaInceptionManager role seen previously.

With the 3 last lines we pass a variable to the function handler. We can access this variable from the js code with process.env.LAMBDA_INCEPTION_WORKER_ROLE.

DEPLOY

The sam build command is used to processes the AWS SAM template file, application code, and any applicable language-specific files and dependencies (i.e. npm install).

Then we can launch sam deploy to deploy the infrastructure and code on AWS

FINAL NOTES

It was quite a journey 😅
We saw how we can start using an IAC tool to define and manage a cloud infrastructure.
The infrastructure as defined here is far from perfect, this is just a learn-by-doing exercise.

The updated code can be downloaded from:
https://github.com/gfabrizi/lambda-inception-sam

Leave a comment for questions or issues with the code
Thank for reading! 👋

Lambda Inception Architectural Pattern

Gianluca Fabrizi — Thu, 23 Mar 2023 15:48:48 +0000

The problem

In some cases (some "shadier" than others) you may need to have multiple "clean" ip addresses.
One use case could be the obvious web scraper.
Usually when you make multiple GET requests to a host, you may get a captcha to prove you're a human, your ip may be rate limited or banned.

Ideally you want something that is:

simple to use
doesn't need additional software
could be used with traditional CLI commands like cURL
cheap

You can deal with this problem in several ways: use a vpn, use tor proxy (please, please, please don't do this) or use some kind of throwaway ip.
You could use an AWS EC2 instance to be your proxy; when the ip is banned, you take a snapshot of the instance and create a new one.
...or you can be really creative and use AWS Lambda Functions...

The pattern

NOTE: I never saw such pattern applied to Lambda functions, I checked several times. If you know someone who already did this, please let me know in the comments!

The pattern is nothing new: you have a manager always running and one or more worker doing the heavy-lifting.
If we have to keep going with the scraper example, the "heavy-lifting" here is making the GET requests.

Why can't we use a single Lambda?
Lambda functions are by definition "serverless": they aren't tied to the underlying hardware, you can't make assumption about the hardware.
On AWS the Lambda functions have an "execution context": on first run (cold start) AWS create an execution context for the function.
This context is kept alive for an indefinite period of time (usually between 5 and 7 minutes). In this time your Lambda function will respond in less time (warm start) but keep the same ip address.

So we have a manager (let's call it "lambda-inception").
This manager is a Lambda Function URL (a function with a dedicated url you can call).
So we want to call the manager url, passing a payload consisting of the url to scrape.

The manager will create a new Lambda worker Function (without URL, we don't need to access this function), pass the url to scrape to the function, await for response, destroy the worker and return the response to the client.
We destroy the worker after each call because we want a clean ip every time.

So without further ado this is the configuration on AWS:

Roles

We need 2 roles (LambdaInceptionWorker and LambdaInceptionManager) with some custom policies (replace AWS_REGION and AWS_ACCOUNT_ID with your account information):

LambdaInceptionWorker
(no permission policies) We leave the worker role without policies (neither AWSLambdaBasicExecutionRole) so it doesn't log anything on CloudWatch

LambdaInceptionManager
LambdaInceptionPassRole (Custom created):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::AWS_ACCOUNT_ID:role/LambdaInceptionWorker"
        }
    ]
}

LambdaInceptionCreateFunction (Custom created):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lambda:CreateFunction",
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:*"
        }
    ]
}

LambdaInceptionDeleteFunction (Custom created):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lambda:DeleteFunction",
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:*"
        }
    ]
}

LambdaInvokeFunction (Custom created):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:*"
        }
    ]
}

AWSLambdaBasicExecutionRole (AWS standard policy)

Users

If you want to keep your Functions protected (why wouldn't you?) you need to create an IAM user (call it lambda-inception-invoker) with LambdaInceptionInvoke custom policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "lambda:InvokeFunctionUrl",
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:lambda-inception"
        }
    ]
}

We need programmatic access for this user, so after creating the user go to the "Security Credentials" tab and create an access key.

You may need a lambda-deployer user with AWSLambda_FullAccess permission policy if you want to use the script provided on the github repository to deploy the infrastructure on your account.

The code

The code is available here: https://github.com/gfabrizi/lambda-inception
The code uses a mono-lambda approach to manage the routing.

src/app.mjs and src/res.mjs are responsible for managing the basic routing and create an evelope for the response.

The code for the manager lies in index.js:

const worker = await createWorker();

const lambdaClient = new LambdaClient({region: awsRegion, apiVersion: '2015-03-31'});
const invokeCommand = new InvokeCommand({
    FunctionName: worker.FunctionName,
    Payload: JSON.stringify({
        'url': req.body.url,
        'accept': req.body.accept ?? 'text/html,application/xhtml+xml,application/xml;q=0.9',
        'contentType': req.body.contentType ?? 'application/octet-stream',
        'method': req.body.method ?? 'GET',
    }),
    LogType: LogType.Tail,
});

const { Payload, LogResult } = await lambdaClient.send(invokeCommand);
const result = Buffer.from(Payload).toString();

const deleteCommand = new DeleteFunctionCommand({
    FunctionName: worker.FunctionName
});
await lambdaClient.send(deleteCommand);

res.json(result, 200);

here we create a new worker, invoke the worker (passing the url, the method and a few headers).
Then we await for the response of the worker, destroy the worker and return the scraped html to the client.

src/create-worker.mjs wraps the functionality to create a new worker:

const lambda = new LambdaClient({region: awsRegion, apiVersion: '2015-03-31'});
const functionName = 'worker-' + randomUUID();

const functionCommand = new CreateFunctionCommand({
    Code: {
        ZipFile: fs.readFileSync('worker.zip')
    },
    Architectures: [Architecture.x86_64],
    FunctionName: functionName,
    Handler: 'worker.handler',
    PackageType: PackageType.Zip,
    Role: lambdaInceptionWorkerRole,
    Runtime: 'nodejs18.x',
    Description: 'Crawler Worker ' + functionName
});

return lambda.send(functionCommand);

nothing too fancy here: we specify an archive (worker.zip) to be used for creating the worker.

The worker code (worker/worker.js) is just a call to got.js:

let response = await got(event.url, {
    headers: {
        'user-agent': getUserAgent(),
        'Content-Type': event.contentType,
        'Accept': event.accept,
    },
    method: event.method
});

return {
    statusCode: 200,
    body: JSON.stringify(response.body),
};

We use a simple user-agent rotation, just in case...

Make a request

So if everything is correctly configured (refer also to the README.md file in the repository) you can run ./deploy.sh to deploy the manager on a Lambda Function.

Then you can call your function with a simple cURL:

curl -X POST https://XXXXXXXXXXXXXXXXXXXXXX.lambda-url.AWS_REGION.on.aws/crawl \
-H 'Content-Type: application/json' \
-d '{"url":"https://ifconfig.me/", "accept":"application/json", "method":"GET"}' \
--user XXXXXXXXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
--aws-sigv4 "aws:amz:AWS_REGION:lambda"

(replace all the Xs with yout function url and account details. Also replace AWS_REGION with the region you have used)

Links

The code is available here: https://github.com/gfabrizi/lambda-inception

DEV Community: Gianluca Fabrizi

1BRC in PHP FFI + Rust

Foreign Function Interface

The Rust solution

Compiling Rust module

The PHP script

The benchmark

Conclusions

1 Billion Rows Challenge in PHP

Initial Specifications

Measuring the execution time

How much data should be passed to the script

Where to run the script

Writing the code

First attempt

Second attempt: trim()

Third attempt: (float)

Fourth attempt: while

Fifth attempt: min()/max()

Sixth attempt: if

Seventh attempt: !isset()

Eighth attempt: pointer &

Ninth attempt: fgets()

Tenth attempt: strtok()

Test on dedicated hardware

Test on EC2 with 1 billion rows

Multi-thread PHP

Conclusions

Getting started with Cypher and RedisGraph / Part III

What just happened?

Visualize data

Conclusion

Getting started with Cypher and RedisGraph / Part II

What is Cypher

RedisInsight

Adding nodes and edges

Basic querying

Connecting nodes and creating edges

Querying for relationships

Delete nodes and edges

RECAP

Getting started with Cypher and RedisGraph / Part I

What is a Graph database?

Why use a Graph database?

Why RedisGraph?

Graph Elements

Running RedisGraph locally

In the next post

✨ Porting Lambda Functions to AWS SAM

AWS SAM

THE PROJECT

THE CODE

DEPLOY

FINAL NOTES

Lambda Inception Architectural Pattern

The problem

The pattern

Roles

Users

The code

Make a request

Links

Second attempt: `trim()`

Third attempt: `(float)`

Fourth attempt: `while`

Fifth attempt: `min()/max()`

Sixth attempt: `if`

Seventh attempt: `!isset()`

Eighth attempt: pointer `&`

Ninth attempt: `fgets()`

Tenth attempt: `strtok()`