DEV Community: Michael Salaverry

FastGRPC - FastAPI with the speed of gRPC

Michael Salaverry — Mon, 30 Mar 2026 19:26:29 +0000

barakplasma / FastGRPC

FastGRPC

Add gRPC to your FastAPI app in one line. No protobuf. No config. ~2× faster than HTTP/JSON.

The Pitch

You already have a FastAPI app. FastGRPC reads it at startup and gives you a fully working gRPC server — same handlers, same Pydantic models, zero new files to write or maintain.

from fastgrpc import FastGRPC

# Your existing FastAPI app, unchanged
app = FastAPI()

@app.get("/items/{item_id}", response_model=Item)
async def get_item(item_id: int) -> Item:
    ...

# One line to add gRPC on port 50051
FastGRPC(app, grpc_port=50051)

That's the entire migration.

Performance

Benchmarked on the same FastAPI app, same handler logic, 30,000 requests over ~1 minute with persistent connections (how production clients work):

Scenario	HTTP/JSON	gRPC	Speedup
Small payload (3 fields)	5.38 ms	1.18 ms	4.6×
Large payload (~50 fields, nested)

…

View on GitHub

Chrome Reading List export

Michael Salaverry — Sun, 18 Jun 2023 11:25:20 +0000

The easiest way to export the chrome reading list is to use takeout.google.com and to export the Chrome details. On my Google workspace account, this wasn't possible.

Instead, I navigated to chrome://sync-internals/ on the sync node browser tab and ran this javascript code in the console:



[
  ...$("#sync-node-tree")
    .shadowRoot.querySelector("#tree-item-autogen-id-17")
    .shadowRoot.querySelectorAll(".leaf"),
].map((row) => row.detail.payload);

(The shadowRoot has its own query selector function separate from the window of the console.)

Then, I right clicked the result to paste it into a text editor and save as a json file

Here's a sample of what you can expect to get:

The following is one particular chrome reading list item



{
    "CLIENT_TAG_HASH": "xxxxbEkAs=",
    "CTIME": "2022-07-25 14:41:44 +03",
    "ID": "sZ:ADqtAZw/jKp+WHaGmzqqSHy/YRNDZvdUo0doOqiQFaIMf0Yo59vheTu0WKfrimEtw+TaNyZJxSpEzC1wpOhS6r5qtfZP1kywYw==",
    "MTIME": "2022-07-25 14:41:44 +03",
    "NAME": "https://developer.mozilla.org/en-US/docs/Web/API/Reporting_API",
    "NON_UNIQUE_NAME": "https://developer.mozilla.org/en-US/docs/Web/API/Reporting_API",
    "ORIGINATOR_CACHE_GUID": "",
    "ORIGINATOR_CLIENT_ITEM_ID": "",
    "PARENT_ID": "",
    "SERVER_DEFINED_UNIQUE_TAG": "",
    "SPECIFICS": {
        "reading_list": {
            "creation_time_us": "1658749304297842",
            "entry_id": "https://developer.mozilla.org/en-US/docs/Web/API/Reporting_API",
            "estimated_read_time_seconds": "0",
            "first_read_time_us": "0",
            "status": "UNREAD",
            "title": "Reporting API - Web APIs | MDN",
            "update_time_us": "1658749304297842",
            "update_title_time_us": "1658749304297842",
            "url": "https://developer.mozilla.org/en-US/docs/Web/API/Reporting_API"
        }
    },
    "metadata": {
        "acked_sequence_number": "2",
        "client_tag_hash": "3ewNy9zDJ7HvgpPTJfkN1DbEkAs=",
        "creation_time": "1658749304297",
        "is_deleted": false,
        "modification_time": "1658749304299",
        "sequence_number": "2",
        "server_id": "Z:ADqtAZw/jKp+WHaGmzqqSHy/YRNDZvdUo0doOqiQFaIMf0Yo59vheTu0WKfrimEtw+TaNyZJxSpEzC1wpOhS6r5qtfZP1kywYw==",
        "server_version": "1658749304544398",
        "specifics_hash": "eb+IzTHvIE0GMJ0az8W4a1nUlis="
    },
    "modelType": "Reading List"
}

Benchmark of inter-thread communication options in a concurrent Python GUI

Michael Salaverry — Wed, 24 May 2023 11:32:39 +0000

In this blog post, we'll discuss a benchmarking process I used while refactoring the software for a microscope. The experiment compares the performance of two methods for sending and receiving images between a tkinter frontend and a python backend controlling the microscope: using the blinker package and a Flask-based web server.

Context

We have a custom built microscope at the startup I work at which has a custom built tkinter GUI for controlling it. The architecture when I got there was that the tkinter gui would communicate with the microscope control module via flask / http calls. The idea was to allow future remote control of the microscope from other programs. Additionally, the microscope images needed to be displayed in the UI in real time for focusing the microscope, and verifying the images looked good before saving them to disk. This was previously happening via a python queue which contained PIL images.

There were issues sharing the queue across the various modules. Each module needed to be aware of queue full/empty logic. Each module needed to get a reference to the queue, which was owned by the flask server module. We needed to display the most recent image even if the queue was full of old images that hadn't been displayed yet. The queue was in memory, and the images were each quite large, so there were concerns around memory / RAM capacity if the queue filled up.

I decided to benchmark an alternative I thought of: using the blinker library as an event emitter architecture rather than queues.

Code

Both version of inter-thread communication will check the received image at the end to mimic the save to disk process. And both share the same scan function.

from PIL import Image

def checker(image: Image.Image):
    assert image.size == (4096, 3000)

def microscope_scan() -> Image.Image:
    image = Image.open('./example_thorlabs.png')
    return image

For the benchmark, I start the flask uWSGI server below with waitress-serve --host 127.0.0.1 --port 5000 flask_server:app

from io import BytesIO

from flask import Flask, send_file
app = Flask(__name__)

from scan import microscope_scan

@app.route("/get_latest_image")
def get_latest_image():
    bytes_io = BytesIO()
    microscope_scan().save(bytes_io, format='PNG')
    bytes_io.seek(0)
    return send_file(bytes_io, mimetype='image/png')

The following is the essence of the previous flask version:

from requests import get
from io import BytesIO
from concurrent.futures import ThreadPoolExecutor as Pool
from os import cpu_count

def flasker():
    with Pool(cpu_count()) as p:
        p.map(flask_receiver, range(repititions))

def flask_receiver(x):
    res = get('http://localhost:5000/get_latest_image', timeout=10)
    if res.ok:
        with Image.open(BytesIO(res.content)) as im:
            checker(im)

This get's the latest image from the backend over http calls and checks the size of the received image.

The following is the blinker version:

from blinker import signal

def send_via_blink(image: Image.Image):
    scanned = signal('scanned')
    scanned.send('microscope', image=image)

def blinker():
    for x in range(96):
        send_via_blink(microscope_scan())

scanned = signal('scanned')

@scanned.connect
def blink_receiver(sender, **kwargs):
    checker(kwargs['image'])

I also added a queue version of the benchmark:

############# Queue #############
import queue

q = queue.Queue()

def queueer():
    for x in range(repititions):
        q.put(microscope_scan())

def queue_receiver():
    while True:
        checker(q.get())

Benchmarking

Finally, I used the timeit module to benchmark the performance of the blinker and flasker functions.

from timeit import timeit
times = [
    ("queueer", timeit("queueer()", setup="from __main__ import queueer", number=10)),
    ("blinker", timeit("blinker()", setup="from __main__ import blinker", number=10)),
    ("flasker", timeit("flasker()", setup="from __main__ import flasker", number=10))
]
print("| Method | Time (seconds) |")
print("| ------ | ------ |")
for time in times:
    print(f"| {time[0]} | {time[1]} |")

Based on the results from the timeit benchmark, we can see a significant difference in the time taken by both methods:

Method	Time (seconds)
queueer	0.1009
blinker	0.0301
flasker	1573.2531
async_flasker	1618.7829

The Blinker method is much faster compared to the Flask method. This is likely because Flask introduces additional overhead due to HTTP requests and responses, while Blinker is a lightweight library designed for in-process communication using signals.

Given the substantial difference in performance, it would be advisable to use the Blinker method for sending and receiving images in the refactored microscope software. This will help to ensure faster processing times and a more responsive system. However, it's important to consider other factors such as ease of implementation, maintainability, and any other specific requirements for your project.

Conclusion

This code demonstrates a benchmark experiment comparing the performance of sending and receiving images using the blinker library and a Flask-based web server. By analyzing the results, we can make informed decisions on which approach to use for the refactored microscope software.

The code in executable form is available at https://gist.github.com/barakplasma/ea7ed66109a14d597c24e2374ed975ad

How to send syslog logs from docker containers to a nearby Synology NAS

Michael Salaverry — Sun, 26 Mar 2023 12:53:04 +0000

Using this docker compose yaml:

version: '3.9'
services:
  log-generator-syslog:
    image: mingrammer/flog
    command: flog -f rfc3164 -n 10
    logging:
      driver: syslog
      options:
        syslog-address: "udp://nas.local:514"
        syslog-format: rfc3164
        env: dev
        labels: tlv,michael-laptop
        tag: "{{.Name}}-{{.ID}}"

We can generate fake logs using https://hub.docker.com/r/mingrammer/flog in the BSD syslog format. These logs are sent via UDP to the NAS in the same network

On the receiving NAS end, open the Log Center, and setup an endpoint for the container to write to.

Then, just run the docker compose file with docker compose up and you should see your fake logs in synology! These logs can be searched and filtered

ChatGPT can draw system architecture diagrams using mermaid

Michael Salaverry — Sun, 19 Mar 2023 13:42:39 +0000

Turns out that ChatGPT can draw system architecture diagrams for it's suggestions using mermaid.js

I asked ChatGPT for some system architecture advice for monitoring some windows and linux on-prem servers my company runs using this prompt:

... (conversation including details on my goals)
draw a system architecture design using mermaid and markdown

Here is the design ChatGPT produced:



graph LR
    A[VPN] --> B[On-Prem Windows]
    A --> C[On-Prem Linux]
    A --> D[Docker Containers]
    B --> E{Prometheus}
    C --> E
    D --> E
    E --> F[Grafana]
    G[User] --> F

Vue 3 Mouse and Keyboard event combo

Michael Salaverry — Wed, 12 Oct 2022 09:05:06 +0000

I had trouble figuring out keyboard events in Vue, so I'm writing this for future reference

Edit in Vue SFC Playground

data manipulation: jq vs Miller

Michael Salaverry — Mon, 24 Jan 2022 14:52:40 +0000

Recently, I was confronted with a 700mb json file which I needed to manipulate and explore in order to find the source of a bug. I needed to find any relationships between an occasionally missing field, and the other properties. Finding a pattern would let me focus my debugging efforts.

My first attempt was actually a simple Node.js script which would fs.readFile and use vanilla JS to map/reduce/summarize the JSON. This was a good place to start, as it was fairly easy to launch the Node REPL and to .save and .load as needed between sessions. However, I didn't like the wait associated with the single threaded parsing of the JSON file. While it was the most flexible option, it felt the most manual.

My usual tool for dealing with JSON,

stedolan / jq

Command-line JSON processor

jq

jq is a lightweight and flexible command-line JSON processor.

Unix: Windows:

If you want to learn to use jq, read the documentation at https://stedolan.github.io/jq. This documentation is generated from the docs/ folder of this repository You can also try it online at jqplay.org.

If you want to hack on jq, feel free, but be warned that its internals are not well-documented at the moment. Bring a hard hat and a shovel. Also, read the wiki: https://github.com/stedolan/jq/wiki, where you will find cookbooks, discussion of advanced topics, internals, release engineering, and more.

Source tarball and built executable releases can be found on the homepage and on the github release page, https://github.com/stedolan/jq/releases

If you're building directly from the latest git, you'll need flex bison (3.0 or newer), libtool, make, automake, and autoconf installed. To get regexp support you'll also need to install Oniguruma or clone it as…

View on GitHub

is a workhouse written in C which has a variety of useful verbs for parsing and summarizing JSONs with one liners.
In this case, jq is fast, but still single threaded.

Instead, I turned to a relatively newer tool

johnkerl / miller

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

What is Miller?

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

What can Miller do for me?

With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically pretty-print, and more.

Miller operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map.
Miller handles a variety of data formats including but not limited to the familiar CSV, TSV, and JSON/JSON Lines (Miller can handle positionally-indexed data too!)

In the above image you…

View on GitHub

which serves a similar purpose to jq. When I tried miller, I noticed that it's count-distinct verb was quite fast. I read the Miller documentation, and discovered on https://miller.readthedocs.io/en/latest/cpu/ that Miller is often multi-threaded as a consequence of it's implementation in Go and use of channels. This can give Miller the edge over jq for certain tasks where concurrency can make a difference.

I plan on diving deeper into Miller, especially since it can handle data in CSV, JSON, or other formats. Also, the then-chaining launching separate goroutines sounds like a big win for more complicated data manipulation flows.

Another tool I tried using is sqlite and it's json extension combined with it's readfile extension. This was inspired by this question / answer I found

Jun 27 '20

A way do this without CSV or a 3rd party tool is to use the JSON1 extension of SQLite combined with the readfile extension that is provided in the sqlite3 CLI tool. As well as overall being a "more direct" solution, this has the advantage of handling JSON NULL values…

Open Full Answer

and ended up being a fast way to run sql on my JSON file. By running an sql statement in the sqlite3 REPL like this

INSERT INTO my_table SELECT 
  json_extract(value, '$.user_agent')
FROM json_each(readfile('my_data.json'));

I was able to create an SQL table based on the JSON keys I was interested in from each line. However, it wasn't a great fit for my data, as many lines are missing common keys, and I didn't want sparse rows. Also, I would have to write my own verbs in SQL, whereas Miller provides many verbs out of the box. However, sqlite3 was quite fast to provide an answer, probably as a result of it's query optimizations.

I was interested in trying

duckdb / duckdb

DuckDB is an in-process SQL OLAP Database Management System

DuckDB

DuckDB is a high-performance analytical database system. It is designed to be fast, reliable and easy to use. DuckDB provides a rich SQL dialect, with support far beyond basic SQL. DuckDB supports arbitrary and nested correlated subqueries, window functions, collations, complex types (arrays, structs), and more. For more information on the goals of DuckDB, please refer to the Why DuckDB page on our website.

Installation

If you want to install and use DuckDB, please see our website for installation and usage instructions.

Data Import

For CSV files and Parquet files, data import is as simple as referencing the file in the FROM clause:

SELECT * FROM 'myfile.csv'
SELECT * FROM 'myfile.parquet';

Refer to our Data Import section for more information.

SQL Reference

The website contains a reference of functions and SQL constructs available in DuckDB.

Development

For development, DuckDB requires CMake…

View on GitHub

for the same reason I tried sqlite3, but ultimately didn't pursue it since I had the json extension was easier than converting the json to csv first, and then importing to duckdb later.

Append a file into a zip file in Python

Michael Salaverry — Fri, 10 Dec 2021 09:27:14 +0000

I was trying to find a way to append a file into a zip file in python, but I could not find an easy way to do it.

When using zipfile built into python, using the 'a' append method doesn't overwrite files the way I expected it to. So this python module will overwrite the existing file when appending a file (which to me is the obvious solution).

There's no lack of StackOverflow posts and answers, but all of those were too specific to the specific post/issue.

So I decided to scratch my own itch, and make a library to handle it for myself and others.

https://pypi.org/project/appendzip/0.0.4/

barakplasma / append-zip

append a file into an existing zip file, overwriting the existing file of the same name if needed

append-zip

Appends a file to a zip file, overwriting the existing file there if necessary

Performance

Not efficient; extracts all the files in the zip, copies over the new file, and compresses a brand new zip replacing the original one. You will need enough disk space to duplicate the zip file.

Caveats

for some reason, windows has a different file length after unzipping (by 10-20 bytes). So beware how this works on windows

Getting started

install (on Mac) $ python3 -m pip install appendzip

from appendzip.appendzip import appendzip
# before appendzip calendar.txt in the zip archive test.zip contains 2021-01-02
# before appendzip calendar.txt outside the zip contains 2022-02-03
appendzip(
            pathlib.Path('test.zip'),
            pathlib.Path('calendar.txt'),
            'calendar.txt'
)
# after appendzip calendar.txt inside the zip contains 2022-02-03
# after, there is still only one file in the zip archive test.zip

View on GitHub

MIT licensed

example of how to use it:

install (on Mac)
$ python3 -m pip install appendzip

from appendzip.appendzip import appendzip
# before appendzip calendar.txt in the zip archive test.zip contains 2021-01-02
# before appendzip calendar.txt outside the zip contains 2022-02-03
appendzip(
            pathlib.Path('test.zip'),
            pathlib.Path('calendar.txt'),
            'calendar.txt'
)
# after appendzip calendar.txt inside the zip contains 2022-02-03
# after, there is still only one file in the zip archive test.zip

Hanukkah lighting in WebXR

Michael Salaverry — Sun, 05 Dec 2021 09:21:18 +0000

To celebrate Hanukkah (חנוכה), I made a WebXR scene where the user can pan, walk around (WASD), and/or enter VR in the Western Wall (הכותל המערבי) using AFrame, an HTML/JS framework for creating WebXR experiences.

aframevr / aframe

🅰️ Web framework for building virtual reality experiences.

A-Frame

A web framework for building virtual reality experiences.

Site — Docs — School — Slack — Blog — Newsletter

Examples

Find more examples on the homepage, A Week of A-Frame, and WebVR Directory.

Features

👓 Virtual Reality Made Simple: A-Frame handles the 3D and WebVR boilerplate required to get running across platforms including mobile, desktop, Vive, and Rift just by dropping in <a-scene>.

❤️ Declarative HTML: HTML is easy to read and copy-and-paste. Since A-Frame can be used from HTML, A-Frame is accessible to everyone: web developers, VR enthusiasts, educators, artists, makers, kids.

🔌 Entity-Component Architecture: A-Frame is a powerful framework on top of three.js, providing a declarative, composable, reusable entity-component structure for three.js. While A-Frame can be used from HTML developers have unlimited access to JavaScript, DOM APIs, three.js, WebVR, and WebGL.

⚡ Performance: A-Frame is a thin framework on…

View on GitHub

To make a WebXR scene, I used the webcomponents proved by AFrame to create a scene with

a 360 image as the sky
a number of cylinders as candles
a number of gltf models for flames and the hanukiah (special thanks to Santiago Shang for the CC licensed beautiful GLTF model)
and some english text (doesn't support hebrew / RTL as far as I can tell)

Here is the live demo (click the ▶️ to see it in your browser, desktop or mobile)

Future plans involve adding interactivity, so that using a Quest 2 controller / pointer lets you light each candle, or to integrate hebcal so that it can automatically light the right number of candles.

Add Notifications to a 3rd party Website

Michael Salaverry — Wed, 17 Nov 2021 09:09:02 +0000

Ever had to monitor a long running process on a webpage like Jenkins? Constantly switching tabs to check if a long running task is finished is a pain, and we can ameliorate it with a small amount of code.

MDN has a great article on the notification API so I'll skip that part in this blog post. (Flavio Copes also wrote a great introduction to the Notification API) What I want to show you is how to use that API in a page.

There's a great tool called Violent Monkey which lets you add your own Javascript userscripts to run on a 3rd party website. Using a userscript, we can trigger the Notification API for almost any page event. (There's also a cool separate tool for adding your own CSS called stylus which is outside the scope of this blog).

The trick is to add an event listener or selector in Javascript for the thing you want to notify on. For example, when Jenkins finishes a build, it changes the header color from blue to green or red. We can use a CSS selector via document.querySelector() that when a classname has been added to a specific DOM node, that our script will trigger a notification.

Check out my example using the selector method:

Notice that it runs in a somewhat recursive loop, with a base case that does not trigger a second alert.

Which pages do you need a notification for?

showdev: Serverless IoT Dashboard

Michael Salaverry — Thu, 23 Sep 2021 13:48:30 +0000

I want to showdev how I built a serverless IoT (internet-of-things) dashboard for a Tasmota / BME680 air quality sensor.

Preview of the real-time dashboard:

Live Demo / Repository:

barakplasma / Browser-MQTT-Dashboard

receives and stores MQTT messages relayed from a Tasmota ESP8266 and displays them on a realtime graph and table dashboard

Browser-MQTT-Dashboard

receives and stores MQTT messages relayed from a Tasmota ESP8266 and displays them on a realtime graph and table dashboard

Access it at

View on GitHub

The hardware:
Prototype:

(I made the case out of corrugated cardboard I had lying around, and sealed it up with a few more sheets of cardboard glued together. I did leave vents so that it can detect the actual air too)

The architecture:

The data is collected using Tasmota on an ESP8266. I soldered a BME680 directly to the ESP8266 pins and mapped the pins accordingly using the web config. I configured Tasmota to send the data to a free public MQTT broker provided by emqx. It would be ok to self host broker, but that wouldn't be as serverless. The Tasmota/ESP8266 is connected to my home wifi, and since it's only sending messages out, no special NAT changes or Port Forwarding was required.

The serverless static site architecture is based on MQTT.js reading from an MQTT broker, and displaying the data using Chart.js with chartjs-plugin-streaming for real-time chart updates.
I decided to use vanilla javascript DOM updates to keep it simple. The static site stores the messages within the browser for historical purposes, and to enable the first load to happen with live data. Static site hosting and CI/CD is done through Github (Pages and Actions).

mqttjs / MQTT.js

The MQTT client for Node.js and the browser

MQTT.js is a client library for the MQTT protocol, written in JavaScript for node.js and the browser.

MQTT.js is an OPEN Open Source Project, see the Contributing section to find out what this means.

Important notes for existing users

v5.0.0 (07/2023)

Removes support for all end of life node versions (v12 and v14), and now supports node v18 and v20.
Completely rewritten in Typescript 🚀.
When creating MqttClient instance new is now required.

v4.0.0 (Released 04/2020) removes support for all end of life node versions, and now supports node v12 and v14. It also adds improvements to debug logging, along with some feature additions.

As a breaking change, by default a error handler is built into the MQTT.js client, so if any errors…

View on GitHub

chartjs / Chart.js

Simple HTML5 Charts using the <canvas> tag

Simple yet flexible JavaScript charting for designers & developers

Documentation

All the links point to the new version 4 of the lib.

In case you are looking for an older version of the docs, you will have to specify the specific version in the url like this: https://www.chartjs.org/docs/2.9.4/

Contributing

Instructions on building and testing Chart.js can be found in the documentation. Before submitting an issue or a pull request, please take a moment to look over the contributing guidelines first. For support, please post questions on Stack Overflow with the chart.js tag.

License

Chart.js is available under the MIT license.

View on GitHub

nagix / chartjs-plugin-streaming

Chart.js plugin for live streaming data

chartjs-plugin-streaming

Chart.js plugin for live streaming data

chartjs-plugin-streaming 2.x requires Chart.js 3.0.0 or later. If you need Chart.js 2.x support, use the following versions.

For Chart.js 2.9.x, 2.8.x or 2.7.x, use version 1.9.0 (tutorials and samples)
For Chart.js 2.6.x, use version 1.2.0

Documentation

Development

You first need to install node dependencies (requires Node.js):

npm install

The following commands will then be available from the repository root:

npm run build      # build dist files
npm run build:dev  # build and watch for changes
npm run lint       # perform code linting
npm run package    # create an archive with dist files
npm run docs       # generate documentation (`dist/docs`)
npm run docs:dev   # generate documentation and watch for changes

License

chartjs-plugin-streaming is available under the MIT license.

View on GitHub

Splunk Alert on Percentage Change in text Field Frequency

Michael Salaverry — Sun, 05 Sep 2021 17:17:07 +0000

Splunk is a tool for querying and summarizing unstructured or structured text. You can use Splunk to search, summarize, and alert on application logs.

You can use Splunk to send alerts via email or Pagerduty when we notice a rise in certain error logs. It's important for us to monitor our error logs so that we know when our users are experiencing issues with our products. Errors are one of the "Golden Signals" for application health as described in the Google Site Reliability Engineering book online (link).

Here's a hypothetical log that we need to alert on:
2021-09-05T16:35:18+00:0 level=ERROR logger=com.enterprise.payments PAY - reason=null payeeId=aceja payStatus=FAILED host=appDeployment-alfzf
Which also has other logs, such as success logs
2021-09-05T13:35:18+00:0 level=INFO logger=com.enterprise.payments PAY - reason=accepted payeeId=baehg payStatus=SUCCESS host=appDeployment-alfzf
Or in between logs
2021-09-05T17:35:18+00:0 level=WARN logger=com.enterprise.payments PAY - reason=decline payeeId=gaeaj payStatus=FAILED host=appDeployment-alfzf

In the past, we setup our alerts to trigger when we get x number of error logs, where x is some number from 1 to 100 errors. This works most days, since usually we have steady traffic/usage.

However, when our application gets high traffic, we can arrive at a triggered alert even when the system is operating "normally". The goal of our alerts and monitoring is to tell us when something unusual, or exceptional is happening; something we need to fix. Getting alerts that we have high traffic is nerve wracking, and wakes us up late at night for no reason. These false positives create pager burnout quickly.

In order to avoid false alarms / false positives, I refactored one of our Splunk alerts from quantity-of-errors based, to percentage-of-requests-which-fail based.

Here's an example of the query before:
index=example (PAY "FAILED" logger=com.enterprise.payments)
which would alert when it finds greater than x results. This has the advantage of being simple, but also the downside of a false positive during high traffic. We don't know how many payments were successful, or just declined cards, not an error in our server flow, but rather in the user's cash flow.

Here's an example of the query after making it alert on percentage of errors:

index=example (PAY logger=com.enterprise.payments) 
| eval failureRate=if(match(payStatus, "FAILED"), 100, 0)
| timechart avg(failureRate) as percentFail 
| where percentFail > 5

By converting payStatus to a number, we can use avg later on if(match(payStatus, "FAILED"), 100, 0). This gives us a nice 0-100% timechart graph.

Additionally, the where percentFail > 5 is critical, since it lets us set the alert to fire when it finds greater than 1 result. Using the where limits the number of results to results when the percentage of failures was high.

Another side benefit is that we get the splunk error percentage directly in the PagerDuty alert, letting us know what to expect once we click out of PagerDuty and into Splunk.

The final query powering our alert can also be tied to other relevant transaction logs by payeeId, avoid alerting if the volume of events is too low, and provide reasons for each log in the results table:

DEV Community: Michael Salaverry

FastGRPC - FastAPI with the speed of gRPC

barakplasma / FastGRPC

FastGRPC

The Pitch

Performance

Chrome Reading List export

Benchmark of inter-thread communication options in a concurrent Python GUI

Context

Code

Benchmarking

Conclusion

How to send syslog logs from docker containers to a nearby Synology NAS

ChatGPT can draw system architecture diagrams using mermaid

Vue 3 Mouse and Keyboard event combo

data manipulation: jq vs Miller

stedolan / jq

Command-line JSON processor

jq

johnkerl / miller

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

What is Miller?

What can Miller do for me?

duckdb / duckdb

DuckDB is an in-process SQL OLAP Database Management System

DuckDB

Installation

Data Import

SQL Reference

Development

Append a file into a zip file in Python

barakplasma / append-zip

append a file into an existing zip file, overwriting the existing file of the same name if needed

append-zip

Performance

Caveats

Getting started

Hanukkah lighting in WebXR

aframevr / aframe

🅰️ Web framework for building virtual reality experiences.

A-Frame

Examples

Features

Add Notifications to a 3rd party Website

showdev: Serverless IoT Dashboard

barakplasma / Browser-MQTT-Dashboard

receives and stores MQTT messages relayed from a Tasmota ESP8266 and displays them on a realtime graph and table dashboard

Browser-MQTT-Dashboard

mqttjs / MQTT.js

The MQTT client for Node.js and the browser

Table of Contents

Important notes for existing users

chartjs / Chart.js

Simple HTML5 Charts using the <canvas> tag

Documentation

Contributing

License

nagix / chartjs-plugin-streaming

Chart.js plugin for live streaming data

chartjs-plugin-streaming

Documentation

Development

License

Splunk Alert on Percentage Change in text Field Frequency