DEV Community: Arnold Acho

How to forward a text message (SMS) from an android phone into your webhook

Arnold Acho — Sat, 18 Mar 2023 06:50:31 +0000

You can now program your android phone to forward messages received on your phone to your server and trigger powerful automations with tools like zapper and IFTTFT. I created an open source application called httpSMS that helps you to set this up with ease

Step 1: Get your API_KEY

Create an account on the httpSMS web application and copy your API key from the settings page. https://httpsms.com/settings

Step 2: Install the httpSMS android app

Install the httpSMS android app on your phone and sign in using your API KEY which you copied above. This app listens for SMS messages received on your android phone. https://github.com/NdoleStudio/httpsms/releases/latest/download/HttpSms.apk

Step 3: Set up a webhook

Once the appliation has been installed, it will be listening for SMS messages received on the android phone. You can configure the application to sent this SMS message to your server URL using a webhook. You can configure this URL under the settings page in the httpSMS application https://httpsms.com/settings

Conclusion

Congratulations, you have successfully set up SMS forwarding from your Android phone to a webhook! This powerful automation tool can help you streamline your business workflow and save you time and effort.

You can also trigger the httpSMS application to send an SMS a simple API. You can find more information on the documentation page at https://docs.httpsms.com

Key-Value store built with CockroachDB, .Net 5.0, and Entity Framework

Arnold Acho — Sun, 07 Nov 2021 19:46:32 +0000

In this article, we are going to build a simple key-value store using Entity Framework 5.0, .Net 5.0, and CockroachDB Serverless.

Prerequisites

Install NuGet packages

Install the Microsoft.EntityFrameworkCore and Microsoft.EntityFrameworkCore.Design nuget packages version 5.0.10. The Design package helps when we want to generate migrations from our model classes.

Install the entity framework core provider for PostgreSQL called Npgsql.EntityFrameworkCore.PostgreSQL version 5.0.10

dotnet add package Microsoft.EntityFrameworkCore --version 5.0.10
dotnet add package Microsoft.EntityFrameworkCore.Design --version 5.0.10
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL --version 5.0.10

Setup connection to CockroachDB cluster

Download the cockroachDB certificate using the command below on powershell. If you use linux/mac, You can find alternative commands here. Also, replace {cluster-id} with the id of your cockroachDB cluster.

mkdir -p $env:appdata\.postgresql\; Invoke-WebRequest -Uri https://cockroachlabs.cloud/clusters/{cluster-id}/cert -OutFile $env:appdata\.postgresql\root.crt

Add a connection string in the appsettings.json file.

replace {host} with your cluster host e.g free-tier.gcp-us-central1.cockroachlabs.cloud
replace {username} with your database username
repalce {password} with your database password
replace {cluster.database} with your cluster databse. e.g if your cluster name is key-value-store-123 on cockroachDB, and you plan to use the default database, then {cluster.database} = key-value-store-123.defaultdb
replace {cert-location} with the directory of the certificate

"ConnectionStrings": {
    "ApplicationDbContext": "Host={host};Port=26257;SslMode=Require;username={username};password={password};Database={cluster.database};Root Certificate={cert-location};"
},

**ProTip: **You can use dotnet user-secrets to ensure that your credentials are not commited to git.

Setup Models and DBConntext

Create an Entities directory in the root of the api project and add the following entity

    namespace KeyValueStore.Entities
    {
        public class KeyValueEntry
        {
            [Required]
            [Key]
            [StringLength(44)]
            public string Key { get; set; } = default!;

            [Required] [Column(TypeName = "text")]
            public string Value { get; set; } = default!;

            [Required]
            public DateTime CreatedAt { get; set; }
        }
    }

KeyValueEntry entity
Create another directory called Database in the Api prject and add an ApplicationDbContext

    namespace KeyValueStore.Database
    {
        public class ApplicationDbContext : DbContext
        {
            public DbSet<KeyValueEntry> KeyValueEntries { get; set; } = default!;

            public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options) : base(options)
            {
            }

            protected override void OnModelCreating(ModelBuilder modelBuilder)
            {
                modelBuilder.Entity<KeyValueEntry>().ToTable("store");
                modelBuilder.Entity<KeyValueEntry>().HasKey(x => x.Key);
            }
        }
    }

Database Migrations

First, register the ApplicationDbContext with the .net dependency injection. Add the following in the ConfigureServices method in Startup.cs

services.AddDbContext<ApplicationDbContext>(options => options.UseNpgsql(Configuration.GetConnectionString(nameof(ApplicationDbContext))));

Dependency Injection
Run the commands below in powershell to generate migrations and update your database.

    dotnet tool install --global dotnet-ef
    dotnet ef migrations add InitialCreate
    dotnet ef database update

Migrate Database

Create Controller

Create a controller called StoreController this will have the methods to add, update and delete key value entries from the cockroachDB database.

Note

Instead of storing the entire key, we create a sha256 has of the key. This helps reduce the amount of disk space needed to store the key
The value entries needs to be encoded in base64 this is to prevent all types of encoding issues on utf-x strings.
I used FlexLabs.EntityFrameworkCore.Upsert nuget version 5.0.0 in the PutAsync method to carry out an upsert database query.

    namespace KeyValueStore.Controllers
    {
        /// <summary>
        /// Add, update, delete and fetch key-value entries.
        /// </summary>
        [ApiController]
        [Route("v1/store/{key:maxlength(256):minlength(1)}")]
        [Consumes(MediaTypeNames.Text.Plain)]
        [Produces(MediaTypeNames.Text.Plain)]
        public class StoreController : ControllerBase
        {
            private readonly ApplicationDbContext _dbContext;

            public StoreController(ApplicationDbContext dbContext)
            {
                _dbContext = dbContext;
            }

            /// <summary>
            /// Get a specific value by the key
            /// </summary>
            /// <param name="key" example="995c0628-3fb3-11ec-9356-0242ac130003">The max length is 256 characters</param>
            /// <param name="cancellationToken">Used to cancel the request</param>
            /// <response code="200" example="dmFsdWU=">Value retrieved</response>
            /// <response code="404">There is no entry for the key in the store</response>
            /// <response code="500">Internal Server Error</response>
            [HttpGet]
            [ProducesResponseType(typeof(string), StatusCodes.Status200OK)]
            [ProducesResponseType(StatusCodes.Status500InternalServerError)]
            public async Task<IActionResult> GetAsync([FromRoute] string key, CancellationToken cancellationToken)
            {
                var item = await _dbContext.KeyValueEntries
                    .FirstOrDefaultAsync(x => x.Key == Hash(key), cancellationToken);

                if (item == null) return NotFound();

                return new OkObjectResult(item.Value);
            }

            /// <summary>
            /// Upsert a key value entry.
            /// </summary>
            /// <remarks>
            /// The body of the request will be stored as the value for the "key"<br/><br/>
            /// <strong>NOTE:</strong>
            /// - The body must be a <see cref="!:https://en.wikipedia.org/wiki/Base64">base64</see> encoded string. <br/>
            /// - The body must be maximum 5kb long. ie. the length should be less than or equal to 5120 characters 
            /// </remarks>
            /// <param name="key" example="995c0628-3fb3-11ec-9356-0242ac130003">The max length is 256 characters</param>
            /// <param name="body" example="dmFsdWU=">The value encoded in base64. e.g the word "value" encoded in base64 is "dmFsdWU="</param>
            /// <param name="cancellationToken">Used to cancel the request</param>
            /// <response code="204">Value set successfully</response>
            /// <response code="400">Validation Error</response>
            /// <response code="500">Internal Server Error</response>
            [HttpPut]
            [Produces(MediaTypeNames.Application.Json)]
            [ProducesResponseType(typeof(IList<string>), StatusCodes.Status400BadRequest)]
            public async Task<IActionResult> PutAsync(
                [FromRoute] string key,
                [FromBody] string body,
                CancellationToken cancellationToken
            )
            {
                var item = new KeyValueEntry
                {
                    Key = Hash(key),
                    Value = body,
                    CreatedAt = DateTime.UtcNow
                };

                await _dbContext.KeyValueEntries
                    .Upsert(item)
                    .On(x => x.Key)
                    .RunAsync(cancellationToken);

                return NoContent();
            }

            /// <summary>
            /// Delete a key value entry.
            /// </summary>
            /// <param name="key" example="995c0628-3fb3-11ec-9356-0242ac130003">The max length is 256 characters</param>
            /// <param name="cancellationToken">Used to cancel the request</param>
            /// <response code="204">Value deleted successfully</response>
            /// <response code="404">There is no entry for the key in the store</response>
            /// <response code="500">Internal Server Error</response>
            [HttpDelete]
            [ProducesResponseType(StatusCodes.Status500InternalServerError)]
            public async Task<IActionResult> DeleteAsync(
                [FromRoute] string key,
                CancellationToken cancellationToken
            )
            {
                var item = await _dbContext.KeyValueEntries
                    .FirstOrDefaultAsync(x => x.Key == Hash(key), cancellationToken);

                if (item == null) return NotFound();

                _dbContext.KeyValueEntries.Remove(item);

                await _dbContext.SaveChangesAsync(cancellationToken);

                return NoContent();
            }

            private static string Hash(string value)
            {
                using var hash = SHA256.Create();
                var byteArray = hash.ComputeHash(Encoding.UTF32.GetBytes(value));
                return Convert.ToBase64String(byteArray).ToLower();
            }
        }
    }

Conclusion

The entire code for this project can be found on github https://github.com/AchoArnold/key-value-store. I also hosted it on azure and you can play with it live on https://key-value-store.ndolestudio.com. Please no DDOS attacks 🙏

Least Frequently Used (LFU) Cache with Constant Time Complexity

Arnold Acho — Fri, 22 Oct 2021 16:18:56 +0000

In this blog post, I will explain to you how to implement an in-memory least frequently used cache(LFU) in Go. The implementation has constant time complexity O(1) for Get, Set and Removing the least frequently used item from the cache when the cache is full.

The LFU cache algorithm presented here is based on this paper written by some very very smart people (hey, they even work at FAANG).

Data Structure

Source: An O(1) algorithm for implementing the LFU cache eviction scheme

The data structure for this cache is a struct with 3 fields.

type Cache struct {
    cap           int
    frequencyList *list.List
    lookupTable   map[interface{}]*lookupTableNode
}

cap

This represents the capacity of the cache, i.e the maximum number of items the cache can contain. If the cache already has cap items and you attempt to add a new item in the cache, the least frequently used item will be evicted.

frequencyList

This is a doubly linked list which

A doubly linked list of all the cache items which have a been accessed n number of times
n, the frequency weight for items in that node

This list is represented by [head] -> [1] -> [2] -> [5]->[9] in the image above. All the items which have been accessed 1 time will be stored in the node with value [1] so the nodes [x] and [y] have been accessed 1 time while the node [b] has been assessed 5 times since it's under the node [5].

So as not to re-invent the wheel, I used the container/list package for the doubly linked list.

lookupTable

This is a hashmap. i.e basic key => value store. The key in this case, is the key of the item in the cache and the value is a pointer to the item's node in the frequencyList doubly linked list.

Cache Operations

Set

Set is used to an item in the LFU cache with key and value. It returns ErrInvalidCap if the cache is not initialized

func (cache *Cache) Set(key interface{}, value interface{}) (err error) {
    // check if cache has been initialized.
    if cache.Cap() <= 0 {
        return ErrInvalidCap
    }

    // if the key already exists, change the value.
    if _, ok := cache.lookupTable[key]; ok {
        cache.lookupTable[key].value = value
        return
    }

    // create a lookup table node.
    lookupTableNode := &lookupTableNode{value: value}

    // check if the lookupTable has enough space. If it doesn't, pop the least frequently used item from the Cache.
    if cache.IsFull() {
        cache.pop()
    }

    // set the lookup table node.
    cache.lookupTable[key] = lookupTableNode

    // if frequency list is empty or the first item in the list doesn't have weight 1 create a new node with weight 1.
    if cache.frequencyList.Len() == 0 || cache.frequencyList.Front().Value.(*frequencyListNode).weight != minFrequencyWeight {
        freqListNode := &frequencyListNode{weight: minFrequencyWeight, list: list.New()}
        freqListNode.element = cache.frequencyList.PushFront(freqListNode)
    }

    // get the first item in the frequency list node. We're sure the item has the weight 1.
    freqListNode := cache.frequencyList.Front().Value.(*frequencyListNode)

    // set the node parent of the newly set item in the frequency list node Cache.
    freqListNodeListNode := &frequencyListNodeListNode{parent: freqListNode, key: key}

    // set the frequencyListNodeListNode in the lookup table node.
    lookupTableNode.frequencyListNodeListNode = freqListNodeListNode

    // add the newly created frequencyListNodeListNode to the frequencyListNode of weight 1.
    freqListNodeListNode.element = freqListNode.list.PushBack(freqListNodeListNode)

    return err
}

Get

Get returns an item for the LFU cache by a key. It returns ErrCacheMiss if there's a cache miss.

func (cache *Cache) Get(key interface{}) (value interface{}, err error) {
    // check if the key exists if it doesn't return with a Cache miss error.
    node, ok := cache.lookupTable[key]
    if !ok {
        return value, ErrCacheMiss
    }

    freqListNode := node.frequencyListNodeListNode.parent

    // check if the next node's weight is equal to current weight +1
    // if not, create a new node with weight = current weight + 1 ans insert if after the current node
    if freqListNode.element.Next() == nil || (freqListNode.element.Next().Value.(*frequencyListNode).weight != freqListNode.weight+1) {
        newFreqListNode := &frequencyListNode{
            weight:  freqListNode.weight + 1,
            element: nil,
            list:    list.New(),
        }
        newFreqListNode.element = cache.frequencyList.InsertAfter(newFreqListNode, freqListNode.element)
    }

    // gets the list with weight = node weight + 1. This node MUST exist because it was created above
    nextFreqListNode := freqListNode.element.Next().Value.(*frequencyListNode)
    node.frequencyListNodeListNode.parent = nextFreqListNode

    // remove node from current frequency list node
    freqListNode.list.Remove(node.frequencyListNodeListNode.element)

    // remove freq list node from the cache's freq list if the list node has NO item in it.
    if freqListNode.list.Len() == 0 {
        cache.frequencyList.Remove(freqListNode.element)
    }

    // setting the element of the node in it's new list
    node.frequencyListNodeListNode.element = nextFreqListNode.list.PushBack(node.frequencyListNodeListNode)

    return node.value, err
}

pop

The pop method removes the least frequently used item from the LFU Cache.

func (cache *Cache) pop() {
    // The frequency list node MUST exist i.e the cache cap.
    freqListNodeListNode := cache.frequencyList.Front().Value.(*frequencyListNode).list.Front().Value.(*frequencyListNodeListNode)

    // Remove key from lookup table.
    delete(cache.lookupTable, freqListNodeListNode.key)

    // remove node from frequency list node.
    cache.frequencyList.Front().Value.(*frequencyListNode).list.Remove(freqListNodeListNode.element)

    // if frequency list node list is now empty, remove the frequency list node from Cache's frequency list.
    if cache.frequencyList.Front().Value.(*frequencyListNode).list.Len() == 0 {
        cache.frequencyList.Remove(cache.frequencyList.Front())
    }
}

Conclusion

The code for this LFU cache is on github, One improvement I'm thinking of is to make this cache generic so it can work in cases where the key is a string, int or some other comparible struct/pointer. The generics implementation can be found in 2.0-dev branch on github.

One Trick to speed up your unit tests with Laravel

Arnold Acho — Mon, 17 Aug 2020 13:00:28 +0000

TLDR; Don't use the base TestCase which ships with laravel out of the box. If your tests don't require Laravel specific functionality, Using PHPUnit's base TestCase is more than 4 times faster.

Benchmark

For this test, I'm using the latest version of Laravel (7.25.0) on a Macbook Pro with 16GB RAM and a 2.3 GHz Quad-Core Intel Core i5 processor.

Setup

Install Laravel from the official guide here

Create a class called Math in app/Math.php

<?php

declare(strict_types=1);

namespace App;

class Math
{
    function sum(int $first, int $second): int {
        return $first + $second;
    }
}

Add the test file MathTest.php in tests/Unit

<?php

declare(strict_types=1);

namespace Tests\Unit;

use App\Math;
use Tests\TestCase;

class MathTest extends TestCase
{
    /**
     * @dataProvider provideDataForSum
     *
     * @param int $first
     * @param int $second
     * @param int $expectedResult
     */
    function testSum(int $first, int $second, int $expectedResult) {
        $math = new Math();
        $actualResult = $math->sum($first, $second);

        $this->assertEquals($expectedResult, $actualResult);
    }

    public function provideDataForSum(): array {
        return [
            'sum two zeros (0 + 0 = 0)' => [
                0,
                0,
                0
            ],
            'sum negative and positive number (-1 + 3 = 2)' => [
                -1,
                3,
                2
            ],
            'sum two positive numbers (1 + 6 = 7)' => [
                1,
                6,
                7
            ],
            'sum two negative numbers (-4 + -8 = -12)' => [
                -4,
                -8,
                -12
            ]
        ];
    }
}

Run the tests using

./vendor/bin/phpunit tests/Unit/MathTest.php

Results: Time: 138 ms, Memory: 16.00 MB
Change line 8 in MathTest.php with use PHPUnit\Framework\TestCase;

Run the tests again using

./vendor/bin/phpunit tests/Unit/MathTest.php

Results: Time: 33 ms, Memory: 6.00 MB

Conclusion

The test with laravel's base test case is 418% slower than the tests with PHPUnit's base test case. Also, This is just 1 test, Imagine how much time you will save on a project with more than 1000 test.

NOTE: From laravel 6.0, The ExampleTest.php file in tests/Unit now uses PHPUnit\Framework\TestCase; but If you are coming from an old version of laravel you may be using the laravel base test Tests\TestCase; instead.

How to build, run and test go2 files locally

Arnold Acho — Sun, 16 Aug 2020 14:58:41 +0000

I worked recently on an in-memory key, value cache with Go and I used interface{} for the key, value types since I didn't want to restrict their usage to a specific type. Now with Generics in go, I refactored this package such that you could explicitly set the types of the key and value when instantiating the cache. I found it difficult to run go2 code locally hence this blogpost.

Dependencies

Docker

Approach

Luckily, there's a docker image with the go2go tool already installed (levonet/golang:go2go). The idea here is to build/run/test the go2 code inside this docker container.

Build

Write your go2 code in a file ending in .go2 e.g cache.go2. From the directory where the file is located, run the command below to build the it

docker run --rm -v "$PWD":/go/src/myapp -w /go/src/myapp levonet/golang:go2go go tool go2go build

Run

Similarly, you can run your code using the command

docker run --rm -v "$PWD":/go/src/myapp -w /go/src/myapp levonet/golang:go2go go tool go2go run

Test

You could also test using the command

docker run --rm -v "$PWD":/go/src/myapp -w /go/src/myapp levonet/golang:go2go go tool go2go test

Creavats

The go2go command doesn't support go modules. I had to work around this by creating a custom docker file with steps to download my dependencies manually.
I can't get test coverage using the go2go test tool. It can do the tests but since it translates the go code inside the same file as the _test.go file. I couldn't get test coverage for files in this directory.

Workarounds

I solved the go.mod problem by creating a custom Dockerfile which downloads my dependencies and adds them to the go2go build path

FROM levonet/golang:go2go

## "apt-get update" updates package list
## "apt-get install git -y" installs 'git' the "-y" flag is there to answer yes to all prompts
RUN apt-get update && apt-get install git -y

# This command clones the dependencies in my GOPATH
RUN git clone https://github.com/dgrijalva/lfu-go /go/src/github.com/dgrijalva/lfu-go && \
    git clone https://github.com/arschles/assert /go/src/github.com/arschles/assert

# This command copies all my files in the directory /go/src/myapp in the container
COPY ./ /go/src/myapp

## This changes the working directory
WORKDIR /go/src/myapp

Build the docker image using the command below the -t cache is to tag the image as cache

docker build . -t cache

You can now build/run/test the .go2 code using the commands below

# Test
docker run --rm lfucache go tool go2go test

# Build
docker run --rm lfucache go tool go2go build

# Run
docker run --rm lfucache go tool go2go run

This can also be done with a CI/CD tool like travis. I have the following travis file the lfucache package

services:
  - docker

before_install:
  # This builds the docker file containing the tests
  - docker build . -t lfucache

script:
  # This tests the go2 files
  - docker run --rm lfucache go tool go2go test

PHP Test Coverage with Travis CI and Code Climate using Clover

Arnold Acho — Fri, 19 Oct 2018 07:34:36 +0000

This article is a guide on how to set-up a PHP based repository on code climate such that it reports the test coverage. When you push a commit to your repository on GitHub, the Travis CI pipelines will run and generate the test coverage report which will then be sent to the code climate platform.

1. Get your code climate Test Reporter ID

Navigate to "Repo Settings -> Test Coverage" as seen on the image below

2. Configure your .travis.yml file

Grab a sample .travis.yml file for a PHP project from this repository https://git.io/fxu0q

NOTE: Copy the .travis.yml file from Example 2 since it uses Clover for test coverage.

In the global section of the .travis.yml file, replace the CC_TEST_REPORTER_ID variable with the ID which you copied from step 1 above.

Change the php version under the php block to your desired version.

Also, under the before_install section, add the script composer install --no-dev such that your composer dependencies are installed before tests are run.

Finally, Change the last line from

 - if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then ./cc-test-reporter after-build --exit-code $TRAVIS_TEST_RESULT; fi

  - if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then ./cc-test-reporter after-build  -t clover --exit-code $TRAVIS_TEST_RESULT; fi

Your .travis.yml file would look like this after the above changes

env:
  global:
  - CC_TEST_REPORTER_ID={put your reporter id here}
  - GIT_COMMITTED_AT=$(if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then git log -1 --pretty=format:%ct; else git log -1 --skip 1 --pretty=format:%ct; fi)
language: php
php:
  - '7.2'
before_script:
  - composer install --no-dev
  - curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
  - chmod +x ./cc-test-reporter
  - ./cc-test-reporter before-build
script:
  - "phpunit --coverage-text --coverage-clover build/logs/clover.xml"
  - if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then ./cc-test-reporter after-build  -t clover --exit-code $TRAVIS_TEST_RESULT; fi

You can get this .travis.yml file on github.

3. Configure phpunit.xml

It is mandatory to configure a whitelist for telling PHPUnit which source code files to include in the code coverage report. If you don't already have a whitelist filter in your phpunit.xml file, add one as such

<filter>
    <whitelist processUncoveredFilesFromWhitelist="true">
        <directory suffix=".php">./src</directory> 
        <exclude>
           <directory>./vendor</directory>
        </exclude>
    </whitelist>
</filter>

In the example above, ./src is the path to the directory where your source files are found and ./vendor is the path to a directory which you want to ignore.

Also, tell PHPUnit where to log the test coverage output by adding a logging block in the phpunit.xml file

<logging>
    <log type="coverage-clover" target="build/logs/clover.xml"/>
</logging>

My complete PHPUnit file looks like this

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="vendor/autoload.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false">
    <testsuites>
        <testsuite name="unit">
            <directory suffix="Test.php">./tests</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./src</directory>
            <exclude>
                <directory>./vendor</directory>
                <directory>./doc</directory>
                <directory>./SampleApp_RoadPlanner/</directory>
            </exclude>>
        </whitelist>
    </filter>
    <logging>
        <log type="coverage-clover" target="build/logs/clover.xml"/>
    </logging>
</phpunit>

You can see this code on GitHub

That Is All!

Next time when your push a build on Travis CI, it will run your tests and send the coverage report to Code Climate.

Sample Test Coverage Report