DEV Community: Piotr

How to (not) run Automerge in React-native

Piotr — Sat, 02 Aug 2025 00:00:00 +0000

TL;DR: I don’t know how how to run Automerge in RN, I’ve tried ⚛️ 🏎 react-native-webassembly but it’s not yet possible due to technical reasons.

There is hope. Callstack published new RN library callstackincubator/polygen which allows to run webassembly in RN, but I have yet to try it.

Problem

I wanted to create an app that has one big note, that is shared and synced between my devices. There are two layers to it:

Presentation: A single markdown formatted note, which can be done using https://github.com/Expensify/react-native-live-markdown
Storage and sync: There were no obvious answers here, I didn’t want to relay on cloud providers. I found a nice library: https://automerge.org/ which has a nice selling point:
- Automatic merging: Automerge is a Conflict-Free Replicated Data Type (CRDT
- Network-agnostic: Allows the use of any connection-orientated protocol
- Portable: Imlemented in Javascript-WASM and Rust, with FFI bindings

Choosing the right tool for the job

This WASM part caught my eye since there are libraries that claim to bring WASM to react-native:

⚛️ 🏎 react-native-webassembly - A c++ turbo module that is a wrapper around wasm3 - A fast WebAssembly interpreter and the most universal WASM runtime.

callstackincubator/polygen - from the callstack website: “Polygen instead performs Ahead-of-time compilation of WebAssembly modules into C/C++ code using the wonderful wasm2c tool. After that, Polygen generates additional glue code so that the compiled WebAssembly module can be used from JavaScript code.”

So both projects are essentially wrappers around other libraries. I decided to go with the react-native-webassembly since it offered Android support, while polygen (as I’m writing it) iOS only.

react-native-webassembly

Of course adding this library to the project was not simple npm install react-native-webassembly failed to compile

I got somewhat cryptic error:

error: error in backend: failed to perform tail call elimination on a call site marked musttail

Quick google led me to a simmilar issue on PowerPC, but I’m compiling for android… two searches later I got it: Musttail is causing crash when compiling wasm3 for android armv7-a so I read through the issue and I saw wasm3 being mentioned

The issue is with the configuration of wasm3, I fixed it the same way the @bald-man did, disabling musttail. Here’s the whole patch, feel free to take it:

diff --git a/node_modules/react-native-webassembly/cpp/m3_config_platforms.h b/node_modules/react-native-webassembly/cpp/m3_config_platforms.h
index 50b86ac..8deaf8d 100644
--- a/node_modules/react-native-webassembly/cpp/m3_config_platforms.h
+++ b/node_modules/react-native-webassembly/cpp/m3_config_platforms.h
@@ -82,7 +82,7 @@
 # endif
 # endif

-# if M3_COMPILER_HAS_ATTRIBUTE(musttail)
+# if M3_COMPILER_HAS_ATTRIBUTE(musttail) && !defined( __arm__ )
 # define M3_MUSTTAIL __attribute__ ((musttail))
 # else
 # define M3_MUSTTAIL

Open questions

Why this is an issue. https://github.com/llvm/llvm-project/pull/109943 is closed in llvm, did the fix not propagate? No idea
What are the consequences of this fix, what is the musttail?

wasm

Before I proceed with the steps that I took, I need to explain what is WASM.

WASM - it’s a low level assembly language, that has both binary and text format. It runs the code in a sandboxed environment. That means that by default it does not have access to any Web APIs. Those are given by providing imports when loading WASM module.

Here’s the example WASM module in WAT (text) format:

(module
  (func $i (import "my_namespace" "imported_func") (param i32))
  (func (export "exported_func")
    i32.const 42
    call $i))

You can convert any .wasm binary into such text format using: https://webassembly.github.io/wabt/demo/wasm2wat/

How react-native-webassemly loads WASM

In react-native-webassembly those imports can be provided like this:

const module = await WebAssembly.instantiate<{
  exported_func: () => number;
  // ...
}>(bufferSource, {
  // Define the scope of the import functions.
  my_namespace: {
    imported_func: (value: number) => console.error(value),
  },
});

Those functions are then passed to wasm3::module by the c++ turbomodule somwehere in lines 367-411 but I don’t quite understand how he gets the callback to call the function he wants (I suspect it has to do something with _doSomethingWithFunction). My understanding goes like this:

Imports are passed as callback from the JS side to the RNWebassembly_instantiate function.

reactNativeWebAssembly.RNWebassembly_instantiate({
    iid,
    bufferSource: bufferSourceBase64,
    stackSizeInBytes,
    callback: ({ func, args, module }) => {
        ...
        return maybeFunction(...args.map(parseFloat));
    },
});

Then on the native side the callback is passed to m3_LinkRawFunctionEx:

Function callback = params.getProperty(runtime, "callback").asObject(runtime).asFunction(runtime);

std::shared_ptr<facebook::jsi::Function> fn = std::make_shared<facebook::jsi::Function>(std::move(callback));

/* export initialization */
for (u32 i = 0; i < io_module->numFunctions; ++i) {
    ...
          m3_LinkRawFunctionEx(io_module, M3GetResolvedModuleName(f).data(), functionName->data(), signature.data(), &_doSomethingWithFunction, static_cast<void*>(id->data()));

And the imported function which needs to be called is called using _doSomethingWithFunction:

m3ApiRawFunction(_doSomethingWithFunction)
{
    ...
    resultDict.setProperty(*context.rt, "module", facebook::jsi::String::createFromUtf8(*context.rt, std::string(_ctx->function->import.moduleUtf8)));
    resultDict.setProperty(*context.rt, "func", facebook::jsi::String::createFromUtf8(*context.rt, functionName));
    resultDict.setProperty(*context.rt, "args", ConvertStringArrayToJSIArray(*context.rt, result, length));

    Value callResult = originalFunction.call(*context.rt, resultDict);
    ...
}

Loading automerge in RN

First try

This was my first attempt:

const AutomergeWasm = require('@/assets/automerge.wasm');
type AutomergeModuleType = typeof import("@automerge/automerge/slim");
const module = await WebAssembly.instantiate<AutomergeModuleType>(AutomergeWasm);

Guess what’s wrong? I forgot imports 🫠, just for reference here’s the error that I’ve got:

error unknown value_type

not telling me much…

Second try

Adding the imports was not that straightforward (or at least I did not know a better way). So I’ve just copied the web/automerge_wasm.jsand exported __wbg_get_imports which provides those imports for the web.

function __wbg_get_imports() {
    const imports = {};
    imports.wbg = {};
    imports.wbg.__wbindgen_object_drop_ref = function(arg0) {
        takeObject(arg0);
    };
    ...

https://www.npmjs.com/package/@automerge/automerge-wasm

And… again…

error unknown value_type

Side note: To check what imports are needed I put the .wasm file into https://webassembly.github.io/wabt/demo/wasm2wat/ and looked at the import statements, then a quick grep -r __wbindgen_object_drop_ref quickly found the missing functions.

  (import "wbg" "__wbindgen_string_get" (func $wasm_bindgen::__wbindgen_string_get::hbef6b8ade2155369 (type $t87)))

Debugging time

How to debug this thing? I’ve decided to go straight into modifying the node_modules/react-native-webassembly/cpp/react-native-webassembly.cpp and then recompiling with npx expo run:android, which proved to work nicely.

I added logs and turns out that it didn’t even run the ‘import’ part, it failed on parsing the module:

wasm3::module mod = env.parse_module(buffer, decoded.length());

So I went to google once again and found this issue: https://github.com/wasm3/wasm3/issues/352

Which is a bummer.

Question: What are reference types? And how do I know that the .wasm file has them?

WASM docs states:

Reference types classify first-class references to objects in the runtime store.reftype::=funcref ∣ externrefThe type funcref denotes the infinite union of all references to functions, regardless of their function types. The type externref denotes the infinite union of all references to objects owned by the embedder and that can be passed into WebAssembly under this type. Source: https://webassembly.github.io/reference-types/core/syntax/types.html#syntax-reftype

How to spot a reftype in the text format? Per the WASM spec:

So i went back to the https://webassembly.github.io/wabt/demo/wasm2wat/ loaded wasm and:

Bingo!

 (module $automerge_wasm.wasm
  (type $t0 (func))
  (type $t1 (func (result i32)))
  (type $t2 (func (result externref)))
  (type $t3 (func (param i32)))
  (type $t4 (func (param i32) (result i32)))
  (type $t5 (func (param i32) (result i32 i32)))
  (type $t6 (func (param i32) (result i32 i32 i32)))
  (type $t7 (func (param i32) (result i64)))
  (type $t8 (func (param i32) (result f64)))
  (type $t9 (func (param i32) (result externref)))

Conclusions

When I started writing this I had no idea what the WASM was and how to load it into react-native. Even though I failed catastrophically this was a nice learning experience. I’m sharing it here as a future reference for myself and to turn those X hours of fighting to run this thing into something concrete.

In the future I’ll probably look into some other library like https://github.com/yjs/yjs which also offers CRDTs but this time there are some people which got it to work in react-native https://github.com/yjs/yjs/issues/381 which may be easier and I might actually get some work done 😅

👀 turns out the reference-types are not supported everywhere and at some point you had to explicitly enable them in wasm-bindgen.

Podman on debian How to fix podman no services after restart

Piotr — Sat, 12 Apr 2025 12:00:00 +0000

I noticed that with static build of podman, services didn’t start after I restarted my server, even though they had Restart=Always policy.

This is an issue that can popup when installing podman from a static build installation on debian.

Turns out that this issue is due to missing podman-restart.service that starts all the services after boot.

This file was originally from github repo, but with static podman build it is not present.

Create directory for

mkdir -p ~/.config/systemd/user/

Create file podman-restart.service:

[Unit]
Description=Podman Start All Containers With Restart Policy Set To Always
Documentation=man:podman-start(1)
StartLimitIntervalSec=0
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=true
Environment=LOGGING="--log-level=info"
ExecStart=/usr/local/bin/podman $LOGGING start --all --filter restart-policy=always
ExecStop=/usr/local/bin/podman $LOGGING stop --all --filter restart-policy=always

[Install]
WantedBy=default.target

Reload units:

systemctl --user daemon-reload; journalctl -f

Enable service and timer:

systemctl --user enable podman-restart.service

How to install latest version of podman on debian/ubuntu

Piotr — Sat, 12 Apr 2025 00:00:00 +0000

I like stability of debian. I use it as my host distro for almost every server I have. But debian repositories (stable) have a very old version of podman. The same goes for debian testing, there’s simply no podman 5>= avaliable. Debian backports don’t work either - there’s no backport of podman.

That left me with one other option - static build of podman.

Installation on amd64 host

Copied from https://github.com/mgoltzsche/podman-static?tab=readme-ov-file#binary-installation-on-a-host

Download the statically linked binaries of podman and its dependencies:

curl -fsSL -o podman-linux-amd64.tar.gz https://github.com/mgoltzsche/podman-static/releases/latest/download/podman-linux-amd64.tar.gz

Verify signature:

curl -fsSL -o podman-linux-amd64.tar.gz.asc https://github.com/mgoltzsche/podman-static/releases/latest/download/podman-linux-amd64.tar.gz.asc
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0CCF102C4F95D89E583FF1D4F8B5AF50344BB503
gpg --batch --verify podman-linux-amd64.tar.gz.asc podman-linux-amd64.tar.gz

Install binaries on the host:

tar -xzf podman-linux-amd64.tar.gz
sudo cp -r podman-linux-amd64/usr podman-linux-amd64/etc /

Install additional binaries:

apt install uidmap nsenter iptables slirp4netns

Add UID/GID mapping

Check /etc/subuid and /etc/subgid for mappings:

cat /etc/subuid
cat /etc/subgid

If theres nothing in here add mapping:

sudo sh -c "echo $(id -un):100000:200000 >> /etc/subuid"
sudo sh -c "echo $(id -gn):100000:200000 >> /etc/subgid"

Persist containers after logout

Enable linger:

loginctl enable-linger <USERNAME>

Enable podman socket

systemctl enable --now --user podman.socket

Enable automatic restart of services

systemctl enable podman-restart.service

Source: https://github.com/containers/podman/blob/main/contrib/systemd/system/podman-restart.service.in

Install podman compose

apt install pipx
pipx install podman-compose

Check if everything is working

$ podman run quay.io/podman/hello

!... Hello Podman World ...!

         .--"--.
       / - - \
      / (O) (O) \
   ~~~| -=(,Y,)=- |
    .---. /` \ |~~
 ~/ o o \ ~~~~.----.~~
  | =(X)= |~ / (O (O) \
   ~~~~~~~ ~| =(Y_)=- |
  ~~~~~~~ | U |~~

Project: https://github.com/containers/podman
Website: https://podman.io
Desktop: https://podman-desktop.io
Documents: https://docs.podman.io
YouTube: https://youtube.com/@Podman
X/Twitter: @Podman_io
Mastodon: @Podman_io@fosstodon.org

Podman on debian How to fix no podman-auto-update.service

Piotr — Sat, 12 Apr 2025 00:00:00 +0000

This is an issue that can popup when installing podman from a static build Installation on Debian.

To fix it we have to create a update service and timer. These were originally from github repo, but with static podman build they aren’t.

Create directory for

mkdir -p ~/.config/systemd/user/

Create file podman-auto-update.service:

[Unit]
Description=Podman auto-update service
Documentation=man:podman-auto-update(1)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/podman auto-update
ExecStartPost=/usr/local/bin/podman image prune -f

[Install]
WantedBy=default.target

Create file podman-auto-update.timer:

[Unit]
Description=Podman auto-update timer

[Timer]
OnCalendar=daily
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target

Reload units:

systemctl --user daemon-reload; journalctl -f

Enable service and timer:

systemctl --user enable podman-auto-update.service
systemctl --user enable podman-auto-update.timer

ZFS Snapshots using sanoid and syncoid

Piotr — Sun, 06 Apr 2025 00:00:00 +0000

Proxmox by default backups vms as tar archives/system images, which is not suitable for me, as I want to use zfs snapshots to ensure that I don’t use many system resources. Then I want to replicate those snapshots to a remote system for an offsite backup.

Installation

Follow the instructions from: https://github.com/jimsalterjrs/sanoid/blob/master/INSTALL.md

After you’re done see if the sanoid.timer works. This timer will execute a policy from /etc/sanoid/sanoid.conf (empty by default).

Configuration

Find all the things that you want to snapshot by running:

$ zfs list

NAME USED AVAIL REFER MOUNTPOINT
rpool 45.1G 412G 104K /rpool
rpool/ROOT 2.34G 412G 96K /rpool/ROOT
rpool/ROOT/pve-1 2.34G 412G 2.34G /
rpool/data 28.6G 412G 96K /rpool/data
rpool/data/vm-100-disk-0 128K 412G 64K -
rpool/data/vm-100-disk-1 6.08G 412G 5.96G -
rpool/data/vm-100-disk-2 19.7G 412G 18.7G -
rpool/data/vm-101-disk-0 2.73G 412G 2.70G -
rpool/var-lib-vz 14.1G 412G 14.1G /var/lib/vz

Create file /etc/sanoid/sanoid.conf:

[rpool/data]
        use_template = production
        recursive = yes
        process_children_only = yes

#############################
# templates below this line #
#############################

[template_production]
        frequently = 0
        hourly = 36
        daily = 30
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

This template will backup all datasets in rpool/data but NOT the rpool/data itself. In this case I want to create snapshots of all the vm disks.

Remote replication

Our setup will consist of two machines:

production machine
remote machine, a repote machine that will store our snapshots

Remote machine configuration

Create dataset into which we’ll replicate the data:

zfs create data/production-machine-xxx

Remote replication uses ssh as its transport layer and we don’t want to give the remote machine root access.

We need to create a new user: adduser remote-backup, and give him privileges to write to it:

zfs allow -u remote-backup compression,mountpoint,create,mount,receive,rollback,destroy data/production-machine-xxx

Production machine configuration

SSH Config

Edit ~/.shh/config:

Host remote-machine:
  Hostname <ip addr>
  User remote-backup

Copy ssh key to remote machine:

ssh-copy-id remote-machine

Timer

Create /etc/systemd/system/zfs-sync.timer:

[Unit]
Description=Run syncoid daily

[Timer]
OnBootSec=30min
OnCalendar=*-*-* 21:00:00 Europe/Warsaw
Persistent=true

[Install]
WantedBy=timers.target

Example values that can be used in OnCalendar (credit: Rockibul Islam):

Explaination Systemd timer
Every Minute *-*-* *:*:00
Every 5 minutes *-*-* *:*/5:00
Every 15 minutes *-*-* *:*/15:00
Every 30 minutes *-*-* *:*/30:00
Every 1 hour *-*-* *:00:00
Every other hour *-*-* */2:00:00
Every 12 hour *-*-* */12:00:00
Between certain hours *-*-* 9-17:00:00
Daily *-*-* 00:00:00
Every Night *-*-* 01:00:00
Every Night at 2am *-*-* 02:00:00
Every morning *-*-* 07:00:00
Every midnight *-*-* 00:00:00
Every sunday Sun *-*-* 00:00:00
Every friday at midnight Fri *-*-* 00:00:00
Every weekday Mon...Fri *-*-* 00:00:00
Every weekend Sat,Sun *-*-* 00:00:00
Every 7 days * *-*-* 00:00:00
monthly * *-*-01 00:00:00
Every quarter * *-01,04,07,10-01 00:00:00
Every 6 months * *-01,07-01 00:00:00
Every year * *-01-01 00:00:00

Oneshot service

Create file /etc/systemd/system/zfs-sync.service:

[Unit]
Description=Send zfs snapshots to remote machine
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/syncoid --no-privilege-elevation -r --skip-parent rpool/data remote-machine:data/production-machine-xxx
[Install]
WantedBy=default.target

Enable the timer and service:

systemctl daemon-reload; journalctl -f
systemctl enable zfs-sync.timer
systemctl enable zfs-sync.service

Start zfs-sync.service manually to sync the first snapshot:

systemctl start zfs-sync.service

You can list snapshots on the remote machine by running:

zfs list -t snapshot

Setting obsidian-shellcommands plugin on flatpak

Piotr — Sun, 30 Mar 2025 10:17:23 +0000

Running Obsidian from flatpak comes with its own set of problems. One of them is that Obsidian is ran in sandbox environment - which makes sense when we consider security, but to be frank - it’s annoying when I have to do something non-standard.

I wanted to run this command from obsidian (it updates my git repo):

npx quartz sync --no-pull

but after adding it to there:

I was welcomed with an error that it couldn’t find npx command, which makes sense - I’m using NVM to manage node.

Tip

To make it work I had to make sure that npx was added to Obsidians $PATH variable and obsidian has permissions to this directory.

I used Flatseal to:

Give Obsidian permissions to /home/$USER directory - that’s where bin/ of NVM resides
Add NVM to $PATH:

I also had to update the script to execute from the git root:

cd "$(git rev-parse --show-toplevel)" && npx quartz sync --no-pull

Now running Ctrl + P and typing Update page updates the page directly from obsidian.

Fix file permissions in podman volumes

Piotr — Sun, 30 Mar 2025 00:00:00 +0000

How to fix common permissions issues in podman volumes (both local and bind mounts)?

The obvious way to fix them is to chown, but as a user we cannot execute this command - users are not permitted to change ownership of files for security reasons. Buttt: podman has a command podman unshare (I know, this name is not intuitive…) that lets us pretend to be root the same way as podman rootless containers do (root inside container is a current user, more on that in this article).

We can execute podman unshare chown -R 0:0 ./volume/path to change ownership of the volume to you user (that is id -u on HOST).

Or if you want to change it to the user that was set:

inside container, created by command eg. podman run --user 1000
inside quadlet: User=1000

Run this command:

podman unshare chown -R 1000:1000 ./volume/path

References

Thoughts on hosting my blog

Piotr — Tue, 31 Dec 2024 00:00:00 +0000

My blog https://garden.piotrk.it was built using Quartz and so far I’m loving it. I’ve set it up such that it publishes automatically using Cloudflare Pages because I wanted to get something running fast. One advantage of Cloudflare is that it gives me analytics for free. My end goal is to move out of Cloudflare and self-host.

Updating this site is super easy,:

npx quartz sync --no-pull

and the site is up!

To make this process less painful (I’m lazy and don’t want to open a terminal) I’ve set up Obsidian Shellcommands plugin. You can read abut this process here setting obsidian-shellcommands plugin on flatpak.

Now running Ctrl + P and typing Update page updates the page directly from obsidian.

My debugging toolbox: strace

Piotr — Sat, 03 Aug 2024 08:00:00 +0000

From time to time, my usual debugging tools: good ol printf, logger.log, console.log, gdb, pdb are not enough to solve the problem. This happens more than often when multiple processes/threads are involved or performance is an issue.

One such tool that I found really useful is strace.

Linux Performance Observability Tools. Source: https://www.brendangregg.com/Perf/linux_observability_tools.png

See man strace (great read)!.

TL;DR

Traces all syscalls (aka how your program interacts with os)
It’s like printf but it automatically prints everything being read/written to any file descriptor (file, socket etc.)
Can show execution stack trace after each system call (flag -k)
and follow forks! (with -f)
Automatically decodes pids / file descriptors (flags --decode-pids=comm / --decode-fds=all)
Each line follows: callname(**args) = result ¹
Slows down traced program ², sometimes ~ 100 times

To showcase these features, I’ve created a toy program. I’m going to show you how to discover what’s it doing.

Debugging missing config files

Some programs are not very verbose. This one complains that it cannot open a file, but it doesn’t specify which one.

$ ./demo
Cannot open file.

Running strace ./demo will print all the relevant output. You can filter it by adding -eread,write,open,openat,writev (common syscalls).

$ strace -eread,write,open,openat,writev ./demo
openat(AT_FDCWD, "example.txt", O_RDONLY) = -1 ENOENT (No such file or directory)
write(1, "Cannot open file.\n", 18Cannot open file.

Program tries to open file example.txt but it doesn’t exist!

Debugging running program

Lets run the demo in background:

$ ./demo &
[1] 104794

And connect to it with strace (-p [PID]):

$ strace -eread,write,open,openat,writev -p 104794
strace: Process 104794 attached
writev(3, [{iov_base="2137\n2137\n2137\n2137\n2137\n2137\n21"..., iov_len=8190}, {iov_base="2137\n", iov_len=5}], 2) = 8195

So our program is writing 2137\n to file descriptor 3. What file is it? Use --decode-fds=all! And bam it’s writing to file: demo.txt

$ strace -eread,write,open,openat,writev --decode-fds=all -p 104794
strace: Process 104794 attached
writev(3</home/pk/projects/strace/demo.txt>, [{iov_base="2137\n2137\n2137\n2137\n2137\n2137\n21"..., iov_len=8190}, {iov_base="2137\n", iov_len=5}], 2) = 8195

If our binary has debug symbols you can event print preety stack traces with -k:

strace -eread,write,open,openat,writev -p 104794 -k
strace: Process 104794 attached
writev(3, [{iov_base="2137\n2137\n2137\n2137\n2137\n2137\n21"..., iov_len=8190}, {iov_base="2137\n", iov_len=5}], 2) = 8195
 > /usr/lib/x86_64-linux-gnu/libc.so.6(writev+0x17) [0x11aa57]
...
 > /home/pk/projects/strace/demo(waitDemo()+0x85) [0x138e]
 > /home/pk/projects/strace/demo(main+0x12) [0x149e]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_init_first+0x90) [0x29d90]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x29e40]
 > /home/pk/projects/strace/demo(_start+0x25) [0x1245]

Example traced program

This program was used to showcase neat features of strace.

#include <iostream>
#include <fstream>
#include <unistd.h>
using namespace std;

void waitDemo() {
 ofstream file;
 file.open("./demo.txt");
 if (!file.is_open()) {
 exit(1);
 }
 while(true) {
 file << "2137\n";
 usleep(1000);
 }
}

void openFileDemo() {
 ifstream file;
 file.open ("example.txt");
 if (!file.is_open()) {
 cout << "Cannot open file.\n";
 exit(78);
 }
}

int main () {
 openFileDemo();
 waitDemo();
 return 0;
};

Compile with:

g++ main.cpp -g -o demo

More about strace

strace(1) - Linux man page ↩︎
strace Wow Much Syscall, Gregg Brendan ↩︎

Python Numpy Array Shared Between Processes

Piotr — Fri, 05 Jan 2024 09:03:05 +0000

Python’s multiprocessing.shared_memory module provides a way to share a block of memory between processes. It’s usefull in ml contexts, where an app will runs ML model in separate process and wants share the results with its parent process. Frigate NVR does exacly that, see: object_detection.py.

# Define properties of shared memory
SHARED_MEMORY_NAME = "shm-1"
SHARED_MEMORY_SHAPE = (5, 5)
NP_DATA_TYPE = np.float64

# Calculate shared numpy array size
dsize = np.dtype(NP_DATA_TYPE).itemsize * np.prod(SHARED_MEMORY_SHAPE).astype(int)

# Create allocate shared memory with a given name
shm = shared_memory.SharedMemory(
 create=True,
 size=dsize,
 name=SHARED_MEMORY_NAME,
)

To access it from a different process, create SharedMemory with create=False and the same name.

For example:

class ShmModifier(Process):
 def __init__ (self) -> None:
 # Required as per: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process
 Process. __init__ (self)

 # Get shared memory
 self.out_shm = shared_memory.SharedMemory(name=SHARED_MEMORY_NAME, create=False)

 # Map it to numpy array
 self.out_np = np.ndarray(
 SHARED_MEMORY_SHAPE, dtype=np.float32, buffer=self.out_shm.buf
 )

 def run(self) -> None:
 # Overrides a numpy array (runs in a different process)
 self.out_np[0, 0] = 1
 self.out_shm.close()

Full working example

import numpy as np
from multiprocessing import Process
import multiprocessing.shared_memory as shared_memory

SHARED_MEMORY_NAME = "shm-1"
SHARED_MEMORY_SHAPE = (5, 5)
NP_DATA_TYPE = np.float64

class ShmModifier(Process):
 def __init__ (self) -> None:
 # Required as per: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process
 Process. __init__ (self)

 # Get shared memory
 self.out_shm = shared_memory.SharedMemory(name=SHARED_MEMORY_NAME, create=False)

 # Map it to numpy array
 self.out_np = np.ndarray(
 SHARED_MEMORY_SHAPE, dtype=np.float32, buffer=self.out_shm.buf
 )

 def run(self) -> None:
 # Overrides a numpy array (runs in a different process)
 self.out_np[0, 0] = 1
 self.out_shm.close()

if __name__ == " __main__":
 # Calculate shared numpy array size
 dsize = np.dtype(NP_DATA_TYPE).itemsize * np.prod(SHARED_MEMORY_SHAPE).astype(int)

 # Create allocate shared memory with a given name
 shm = shared_memory.SharedMemory(
 create=True,
 size=dsize,
 name=SHARED_MEMORY_NAME,
 )
 out_np = np.ndarray(SHARED_MEMORY_SHAPE, dtype=np.float32, buffer=shm.buf)

 # Fill arr with zeros
 out_np[:] = np.zeros_like(out_np)

 print("Array before running the process:")
 print(out_np)
 dp = ShmModifier()
 dp.start()
 dp.join()

 print("Array after running the process:")
 print(out_np)
 shm.unlink()

Running this script will result in:

Array before running the process:
[[0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]]
Array after running the process:
[[1. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]]

Sources

Django Vue (Axios) Cors headers + CSFR tokens

Piotr — Mon, 14 Aug 2023 13:44:05 +0000

Basic CORS and CSFR settings for Django Rest Framework (DRF) + Axios:

# Settings.py:
...

CORS_ORIGIN_ALLOW_ALL = False
CORS_ALLOWED_ORIGINS = [
    "http://localhost:5173",
]

CSRF_TRUSTED_ORIGINS  = [
    "http://localhost:5173",
]

CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SAMESITE = "None"
SESSION_COOKIE_SAMESITE = "None"

CORS_ALLOW_CREDENTIALS = True
CORS_ALLOW_HEADERS = [
    "accept",
    "authorization",
    "content-type",
    "user-agent",
    "x-csrftoken",
    "x-requested-with",
    "x-xsrf-token"]
...

In Vuejs:

// axios config:
xsrfCookieName: 'csrftoken',
xsrfHeaderName: 'X-CSRFToken',
withCredentials: true,

Read what these do before using them. I'm not responsible for any security issues that may arise from using these settings.

Bugs I encountered:

Reason: expected ‘true’ in CORS header ‘Access-Control-Allow-Credentials

References

Poco F1 Restarts on Fastboot Amd

Piotr — Sat, 08 Jul 2023 19:15:20 +0000

Poco F1 - Restarts on Fastboot (AMD)

I tried to flash custom recoery for my Poco F1 (codename “beryllium”). I installed ADB drivers and drivers for my phone. Windows device manager said everything was ok so I procedded and entered fastboot mode (Volume down + Power). I entered command:

fastboot devices

And the fastboot logo disappered and was replaced with text in top left corner: “Press any key to shutdown”. Then I remembered that the guide said this:

Also, have a Xiaomi Account with Mobile Number of Sim1 ready for sign-in whenever required prior to Bootloader Unlocking and make sure you have a couple of Mobile Data MBs at least. And an INTEL PC/Laptop. For AMD you may need to do via WSL Ubuntu. If you encounter issues try to power off via the options available and boot straight to the fastboot or recovery depending in whichever stage you are.

Source: [ROM][13][OFFICIAL] PixelOS [AOSP][STABLE][16/02/2023]

I have AMD CPU so I had to do something else. I found a solution on XDA Developers forum.

Solution

Create a new *.bat file and execute it. It should contain:

@echo off
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags\18D1D00D0100" /v "osvc" /t REG_BINARY /d "0000" /f
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags\18D1D00D0100" /v "SkipContainerIdQuery" /t REG_BINARY /d "01000000" /f
reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags\18D1D00D0100" /v "SkipBOSDescriptorQuery" /t REG_BINARY /d "01000000" /f

pause

This will add some registry keys that will allow you to use fastboot on your phone on AMD CPU.

Credit: [FIX] Fastboot not working on Ryzen CPU