DEV Community: Artur Daschevici

Rusty puppets, Websockets and Voyeurism (part II): Driving Chromium in Docker with a Window

Artur Daschevici — Wed, 19 Nov 2025 09:05:00 +0000

TL;DR

You swapped Chrome → Chromium for better arm64 support, strapped on VNC + noVNC to watch the chaos, made Alpine optional to chase size gains, and pimped a Makefile so everything feels like a dead man's switch.

The Backstory

In some of my previous experiments I have used browser automation with Chrome to extract info from pages, usually via a wrapper around CDP. CDP has a pretty neat and quite huge API for driving Chrome and Chromium based browsers. Instead of using an existing wrapper, I built my own in Rust with tungstenite to communicate over WebSocket.

Why?

Ideally what I want is a server style primitive component that is able to speak CDP over a websocket without having to babysit a local browser instance. Normally the way you run most of the chrome automations is by running the process and if you want some transparency into what is happening you flip --headless flag to false, but this is not what you want, the automation is outside the docker container, the container only holds the browser with the remote debugging port exposed. Something else that tripped me up was that there were no Chrome repos with arm64 builds that i could run on my M1 mac, so I switched to Chromium that supports arm64 better.

Yes, "headless" is efficient; no, I don't trust it until I can see it wiggle, that is why an extra feature that felt right was having VNC enabled on one of the container variants.

What?

Components (lego brick style)

Chromium (not Chrome) – better support for arm64 builds, CDP at 9222.
Socat - port forwarding for CDP, since binding to 0.0.0.0 did not work well with Chromium.
Xvfb + lightweight WM (fluxbox) – fake display for the VNC stream.
VNC server (x11vnc) – the eyeballs.
noVNC + websockify – view it in the browser at :6080.
supervisord – herd cats (multiple daemons).
Alpine (optional) – minimal base; trade-offs: fonts, glibc shims, weird edges.
Rust client – talks to http://container:9222/json

Architecture

+------------------ Docker Container --------------------+
|                                                        |
|  [Xvfb] --- [WM] --- [VNC Server] --- [noVNC]          |
|                         ^             (HTTP 8080)      |
|                         |                              |
|                    screen:0                            |
|                                                        |
|  [Chromium --headless=new --remote-debugging-port=9222]|
|                                         (WS:9222)      |
|                                ^                       |
|                                |                       |
|                            [Socat proxy]               |
+--------------------------------------------------------+

 Outside:
   Rust (chromiumoxide/others) --> http://host:<exposed-socat-port>
   Human -> http://host:6080 (noVNC)

How?

I. Image strategy (two by two):

Essentially what I wanted was efficiency at the container level and at the browser level. I applied that in practice by using alpine for smaller image size and running headless for better runtime efficiency. However running blind is kinda hard to introspect so I added two additional images with VNC support and those came with both alpine and debian flavors.

Debian/Ubuntu base: fewer papercuts, bigger image, fastest to "it works."
Alpine base: smallest, but bring your own fonts, codecs, and glibc cuddles.

Pro tip: start Debian for DX, ship Alpine once you tame fonts/codecs.

II. Dockerfile (there is a common base between debian and alpine)

#....
# Default ports
ENV CHROME_PORT=9222
ENV SOCAT_PORT=9224
ENV VNC_PORT=5900
ENV NOVNC_PORT=6080

#....packages common to both debian and alpine
  vim \ # want to edit stuff?
  chromium \ # duh
  socat \ # port forwarding for the 9222 remote debugging port
  curl \ # healthcheck and manual testing
  net-tools \ # netstat useful for listing open ports
  iproute2 \ # ss useful for listing open ports
  ca-certificates \
  procps \


#.... Install VNC and GUI components
  supervisor \
  xvfb \
  x11vnc \
  websockify \
  fluxbox \
  git \

#....

# Install noVNC from source (most reliable method)
RUN cd /opt && \
  git clone --depth 1 https://github.com/novnc/noVNC.git && \
  cd noVNC && \
  ln -s vnc.html index.html

#....

# Create necessary directories with proper permissions
RUN mkdir -p /home/chrome/data && \
  mkdir -p /var/log/supervisor && \
  chown -R chrome:chrome /home/chrome && \
  chown -R chrome:chrome /var/log/supervisor


# Copy startup script
ARG STEALTH=basic
COPY ${STEALTH}.sh /usr/local/bin/start-chrome.sh
RUN chmod +x /usr/local/bin/start-chrome.sh && \
  chown chrome:chrome /usr/local/bin/start-chrome.sh

#....


EXPOSE ${SOCAT_PORT} ${VNC_PORT} ${NOVNC_PORT}

CMD ["supervisord", "-n", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

III. Dockerfile (lightweight Debian to get a more frictionless experience):

FROM debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive

# Install base dependencies
RUN apt-get update && apt-get install -y \
  #....
  fonts-liberation \
  fonts-noto-color-emoji \
  fonts-roboto \
  fonts-noto \
  libasound2 \
  libatk-bridge2.0-0 \
  libatk1.0-0 \
  libatspi2.0-0 \
  libcups2 \
  libdbus-1-3 \
  libdrm2 \
  libgbm1 \
  libgtk-3-0 \
  libnspr4 \
  libnss3 \
  libwayland-client0 \
  libxcomposite1 \
  libxdamage1 \
  libxfixes3 \
  libxkbcommon0 \
  libxrandr2 \
  xdg-utils \
  && rm -rf /var/lib/apt/lists/*

# Install VNC and GUI components
RUN apt-get update && apt-get install -y --no-install-recommends \
  gnupg \
  #....
  python3 \
  python3-numpy \
  && rm -rf /var/lib/apt/lists/*

#....

# Create a non-root user
RUN useradd -m -s /bin/bash chrome

#....

# Copy supervisord config
COPY supervisord.conf.debian /etc/supervisor/conf.d/supervisord.conf

#....

IV. The alpine version, similar but not quite the same, some commands are different and some of the package names and dependencies differ too:

FROM alpine:3.19

#....

# Install base dependencies and Chromium
RUN apk add --no-cache \
  # Core utilities
  bash \ # using some custom scripts to launch the browser
  # ....
  xdpyinfo \ # x11 utilities that are not included by default
  xauth \
  xprop \
  xwininfo \
  # Fonts
  font-liberation \
  font-noto \
  font-noto-emoji \
  font-noto-cjk \
  # Chromium dependencies
  libstdc++ \
  harfbuzz \
  nss \
  freetype \
  ttf-freefont \
  wqy-zenhei \
  # Audio/Video libraries
  alsa-lib \
  at-spi2-core \
  cups-libs \
  dbus-libs \
  libdrm \
  mesa-gbm \
  libxcomposite \
  libxdamage \
  libxfixes \
  libxkbcommon \
  libxrandr \
  wayland-libs-client \
  # X11 libraries
  libx11 \
  libxext \
  libxrender \
  libxtst \
  libxi

# Install VNC and GUI components
RUN apk add --no-cache \
  #....
  python3 \
  py3-numpy \
  py3-pip

#....

# Create a non-root user (Alpine uses adduser instead of useradd)
RUN adduser -D -s /bin/sh chrome

#....

# Copy supervisord config
COPY supervisord.conf.alpine /etc/supervisor/conf.d/supervisord.conf

#....

V. In order to manage the multiple containers k8s would be overkill so I used docker-compose instead. I like it for simple experiments when I have to do quick and incremental iteration for testing my containers. There is less of a need for cleaning up things after and less of a chance to make mistakes with docker run commands.

name: chrome-cdp

x-chrome-common: &chrome-common
  image: chrome-cdp:${DISTRO:-debian}-${MODE:-headless}-stealth-${STEALTH:-basic}
  container_name: chrome-${DISTRO:-debian}-${MODE:-headless}-stealth-${STEALTH:-basic}
  networks: [cdpnet]
  shm_size: "${HEADLESS_SHM_SIZE}"
  restart: unless-stopped
  healthcheck:
    test: ["CMD", "curl", "-fsS", "http://0.0.0.0:${SOCAT_PORT}/json/version"]
    interval: 10s
    timeout: 3s
    retries: 5
    start_period: 15s

services:
  chrome-headless:
    <<: *chrome-common
    build:
      context: ./headless
      dockerfile: Dockerfile.${DISTRO:-debian}
      args:
        STEALTH: ${STEALTH:-basic}
    ports:
      - "${CDP_HOST_PORT}:${SOCAT_PORT}"

  chrome-gui:
    <<: *chrome-common
    build:
      context: ./gui
      dockerfile: Dockerfile.${DISTRO:-debian}
      args:
        STEALTH: ${STEALTH:-basic}
    ports:
      - "${CDP_HOST_PORT}:${SOCAT_PORT}"
      - "${VNC_PORT}:5900"
      - "${NOVNC_PORT}:6080"

networks:
  cdpnet:
    driver: bridge

as you can see the docker-compose file is slightly streamlined by using YAML anchors and aliases to avoid repetition between the two services. But this is not the most interesting DX improvements I added.

VI. The pimped out Makefile. The big deal with this one is that I got everything working almost like an extension of make, where previously I would just be creating targets to wrap docker-compose now I added a bunch of more advanced params to allow parametrization of each target instead of doing different names for each variant.

SHELL := /bin/bash
ENV_FILE ?= .env
STEALTH ?= basic
MODE ?= headless
DISTRO ?= debian

# Container name (adjust to match your docker-compose service name)
CONTAINER_NAME ?= chromium
COMPOSE_FILE ?= docker-compose.yml

.PHONY: health list-containers ps stats stats-all stats-live top \
   ports ports-all ports-detailed logs logs-chrome logs-chrome-live \
   shell rebuild up down chrome-version chrome-tabs chrome-health verify-chrome-flags \
   restart-all stop-all wsurl

# ============================================
# Configuration
# ============================================
CHROME_IMAGE_PREFIX := chrome-cdp
CHROME_IMAGES := \
 $(CHROME_IMAGE_PREFIX):debian-headless-stealth-basic \
 $(CHROME_IMAGE_PREFIX):debian-headless-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):debian-gui-stealth-basic \
 $(CHROME_IMAGE_PREFIX):debian-gui-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):alpine-headless-stealth-basic \
 $(CHROME_IMAGE_PREFIX):alpine-headless-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):alpine-gui-stealth-basic \
 $(CHROME_IMAGE_PREFIX):alpine-gui-stealth-advanced

# Build docker filter arguments
DOCKER_FILTERS := $(foreach img,$(CHROME_IMAGES),--filter "ancestor=$(img)")

# Command to get container
GET_CONTAINER = docker ps $(DOCKER_FILTERS) -q | head -1

# Command to get all containers
GET_ALL_CONTAINERS = docker ps $(DOCKER_FILTERS) -q

# ============================================
# Helper Functions
# ============================================
# Check if container exists and set CONTAINER variable
define require_container
 $(eval CONTAINER := $(shell $(GET_CONTAINER)))
 @if [ -z "$(CONTAINER)" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) container running"; \
  echo "Available images: $(CHROME_IMAGES)"; \
  exit 1; \
 fi
 @echo "✓ Using container: $(CONTAINER)"
endef

# ============================================
# Targets
# ============================================
help:
 @echo "Chrome Container Management"
 @echo ""
 @echo "Available targets:"
 @echo "  make list-containers    - List all running chrome containers"
 @echo "  make verify-chrome-flags - Verify stealth flags in Chrome"
 @echo "  make stats              - Show container stats"
 @echo "  make stats-all          - Show stats for all chrome containers"
 @echo "  make logs               - Show container logs"
 # @echo "  make exec CMD=<cmd>     - Execute command in container"
 @echo "  make shell              - Open shell in container"

list-containers:
 @echo "Running chrome-cdp containers:"
 @docker ps $(DOCKER_FILTERS) --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

up:
 @echo "Building and starting chromium container in $(MODE) mode with $(DISTRO) (stealth: $(STEALTH))"
 DISTRO=$(DISTRO) MODE=$(MODE) STEALTH=$(STEALTH) docker compose --env-file $(ENV_FILE) up chrome-$(MODE) -d --build

down:
 docker compose --env-file $(ENV_FILE) down

logs:
 $(call require_container)
 @docker logs -f $(CONTAINER)

shell:
 $(call require_container)
 @docker exec -it $(CONTAINER) bash

rebuild:
 DISTRO=$(DISTRO) MODE=$(MODE) STEALTH=$(STEALTH) docker compose --env-file $(ENV_FILE) build --no-cache chrome-$(MODE)

ps:
 docker compose --env-file $(ENV_FILE) ps

health:
 @echo "Headless:" && curl -fsS http://127.0.0.1:$$(grep ^CDP_HOST_PORT $(ENV_FILE) | cut -d= -f2)/json/version | jq -r .Browser || true
 @echo "GUI:" && curl -fsS http://127.0.0.1:$$(grep ^CDP_PORT_GUI $(ENV_FILE) | cut -d= -f2)/json/version | jq -r .Browser || true

# Quick helper to print a page websocketDebuggerUrl (requires jq)
wsurl:
 @curl -s "http://127.0.0.1:$$(grep ^CDP_PORT_HEADLESS $(ENV_FILE) | cut -d= -f2)/json/new?about:blank" | jq -r .webSocketDebuggerUrl
 @curl -s "http://127.0.0.1:$$(grep ^CDP_PORT_GUI $(ENV_FILE) | cut -d= -f2)/json/new?about:blank" | jq -r .webSocketDebuggerUrl


# ============================================
# Chrome-specific commands
# ============================================
chrome-version:
 $(call require_container)
 @docker exec $(CONTAINER) chromium --version

chrome-tabs:
 $(call require_container)
 @CHROME_PORT=$$(docker port $(CONTAINER) | grep -E '9222|9223|9224' | head -1 | awk -F' -> ' '{print $$2}'); \
 if [ -z "$$CHROME_PORT" ]; then \
  echo "❌ Chrome DevTools port not found"; \
  exit 1; \
 fi; \
 echo "Fetching tabs from http://$$CHROME_PORT/json"; \
 curl -s "http://$$CHROME_PORT/json" | jq -r '.[] | "\(.id): \(.title) - \(.url)"'

chrome-health:
 $(call require_container)
 @CHROME_PORT=$$(docker port $(CONTAINER) | grep -E '9222|9223|9224' | head -1 | awk -F' -> ' '{print $$2}'); \
 if [ -z "$$CHROME_PORT" ]; then \
  echo "❌ Chrome DevTools port not found"; \
  exit 1; \
 fi; \
 echo "=== Chrome DevTools Health Check ==="; \
 echo "Endpoint: http://$$CHROME_PORT"; \
 echo ""; \
 if curl -sf "http://$$CHROME_PORT/json/version" >/dev/null 2>&1; then \
  echo "✅ Chrome DevTools is responding"; \
  echo ""; \
  curl -s "http://$$CHROME_PORT/json/version" | jq -r '"Browser: \(.Browser)\nProtocol Version: \(."Protocol-Version")\nUser Agent: \(."User-Agent")"'; \
 else \
  echo "❌ Chrome not responding"; \
  exit 1; \
 fi

verify-chrome-flags:
 $(call require_container)
 @echo "Verifying Chrome flags in container $(CONTAINER)..."
 @docker exec $(CONTAINER) sh -c \
  "cat /proc/\$$(pgrep -o chromium)/cmdline | tr '\0' '\n' | grep -E 'disable-blink-features|user-agent'" \
  && echo "✅ Stealth flags detected" \
  || echo "❌ No stealth flags"

stats:
 $(call require_container)
 @echo "Container stats:"
 @docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}" $(CONTAINER)

stats-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 echo "Stats for all chrome containers:"; \
 docker stats --no-stream --format "table {{.Container}}\t{{.Image}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}" $$CONTAINERS

stats-live:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 docker stats $$CONTAINERS

top:
 @CONTAINER=$$($(GET_CONTAINER))
 @echo "=== Top Processes in Container ==="
 @docker top $$(docker-compose -f $(COMPOSE_FILE) ps -q $$CONTAINER)

# ============================================
# Bulk operations
# ============================================
restart-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No containers to restart"; \
  exit 1; \
 fi; \
 echo "Restarting all chrome containers..."; \
 docker restart $$CONTAINERS

stop-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No containers to stop"; \
  exit 1; \
 fi; \
 echo "Stopping all chrome containers..."; \
 docker stop $$CONTAINERS

# Show port mappings for a single container
ports:
 $(call require_container)
 @echo "=== Port Mappings for Container $(CONTAINER) ==="
 @echo ""
 @docker port $(CONTAINER) | awk -F' -> ' '{print $$1 "\t→\t" $$2}' | column -t -s $$'\t'

# Show port mappings for all chrome containers in a table
ports-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 echo "=== Port Mappings for All Chrome Containers ==="; \
 echo ""; \
 printf "%-15s %-40s %-20s %-20s\n" "CONTAINER ID" "IMAGE" "CONTAINER PORT" "HOST BINDING"; \
 printf "%-15s %-40s %-20s %-20s\n" "---------------" "----------------------------------------" "--------------------" "--------------------"; \
 for container in $$CONTAINERS; do \
  IMAGE=$$(docker inspect --format='{{.Config.Image}}' $$container); \
  SHORT_ID=$$(echo $$container | cut -c1-12); \
  docker port $$container | while IFS= read -r line; do \
   CONTAINER_PORT=$$(echo "$$line" | awk -F' -> ' '{print $$1}'); \
   HOST_BINDING=$$(echo "$$line" | awk -F' -> ' '{print $$2}'); \
   printf "%-15s %-40s %-20s %-20s\n" "$$SHORT_ID" "$$IMAGE" "$$CONTAINER_PORT" "$$HOST_BINDING"; \
  done; \
 done

# Detailed port information with service names
ports-detailed:
 $(call require_container)
 @echo "=== Detailed Port Information for Container $(CONTAINER) ==="; \
 echo ""; \
 IMAGE=$$(docker inspect --format='{{.Config.Image}}' $(CONTAINER)); \
 echo "Container: $(CONTAINER)"; \
 echo "Image: $$IMAGE"; \
 echo ""; \
 printf "%-35s %-20s %-25s %-15s\n" "SERVICE" "CONTAINER PORT" "HOST BINDING" "PROTOCOL"; \
 printf "%-35s %-20s %-25s %-15s\n" "----------------------------------" "--------------------" "-------------------------" "---------------"; \
 docker port $(CONTAINER) | while IFS= read -r line; do \
  CONTAINER_PORT=$$(echo "$$line" | awk -F' -> ' '{print $$1}' | cut -d'/' -f1); \
  PROTOCOL=$$(echo "$$line" | awk -F' -> ' '{print $$1}' | cut -d'/' -f2); \
  HOST_BINDING=$$(echo "$$line" | awk -F' -> ' '{print $$2}'); \
  case $$CONTAINER_PORT in \
   9222|9223|9224) SERVICE="Chrome DevTools Socat Proxy" ;; \
   5900) SERVICE="VNC Server" ;; \
   6080) SERVICE="noVNC Web" ;; \
   *) SERVICE="Unknown" ;; \
  esac; \
  printf "%-35s %-20s %-25s %-15s\n" "$$SERVICE" "$$CONTAINER_PORT" "$$HOST_BINDING" "$$PROTOCOL"; \
 done

# View Chrome startup script logs
logs-chrome:
 $(call require_container)
 @echo "=== Supervisor Chrome Program Logs ==="
 @docker exec $(CONTAINER) cat /var/log/supervisor/chrome.log 2>/dev/null || \
  echo "Chrome log file not found"

# Live tail of Chrome logs
logs-chrome-live:
 $(call require_container)
 @echo "=== Live Chrome Logs (Ctrl+C to exit) ==="
 @docker logs -f $(CONTAINER) 2>&1 | grep --line-buffered "CHROMIUM\|chromium\|Chrome"

Perfomance & size notes

Alpine and Debian size are roughly the same with the current layout, so whatever I am doing wrong, needs a bit more investigation. The list of installed packages can probably be trimmed a bit more, maybe I get some extra savings but I don't expect them to be noticeable at small scale.

Security notes

Haven't added any specific security protocols so don't run this in production without:

Gating with VPN or bind to 127.0.0.1.
Add auth to noVNC/websockify or stick it behind a reverse proxy with auth.
Run as non-root (done above), and keep --no-sandbox only inside containers you control.
Pin package versions for reproducible builds.

WTF moments (and fixes)

Had a bunch a pain with the chromium remote debugging port as it would not bind to 0.0.0.0:2222 on the container so I had to add socat to forward the port. When trying to connect to the remote debugging port of chromium i was getting a connection reset by peer error. Turns out chromium only binds to localhost inside the container unless you use socat to forward the port.
When the browser was not launching inside the container, I thought I messed up, but, as it turns out, any kind of --headless flags should probably be omitted when running with VNC, otherwise the browser will not show up in the VNC session.
this is a hack and probably you can fix it in a better way, but I am lazy, so 2MB of overhead on alpine is me adding bash. I used bash to start chrome with the wrapper script, add the flags all that jazz.
in my first attempt I tried to use chrome, but there were some silly issues with arm64 builds, so I switched to chromium which has better support for arm64.

My conclusion

I will use this setup to automate via CDP. It decouples the browser from the language. I know that this is not ideal as a solo dev but hey it's fun. So I will give it a try.

How I optimized my blog images using Rust

Artur Daschevici — Wed, 12 Feb 2025 10:15:00 +0000

Why?

Ages ago I read a blog post about how images can be optimized and that gives visitors to your site a better experience as the site loads faster yadda, yadda. Why would anyone not want this?

My blog is built with zola so I am a bit of a rust fanboy and want to use rust whenever it makes sense and my novice skills can handle it. Not going to lie, I am not a rust expert and for me ChatGPT is a useful tool as it most often than not points me in the right direction.

But I digress, a few days ago I decided to resurrect my blog, I tried to write a bit more consistently last year and got a streak of a few articles going, was feeling pretty good about it, but then my daughter was born and I was thrown in the gauntlet of figuring things out as a first time dad and the blog was left to rot. Since I am a bit more web marketing savvy, I decided to add some SEO to my blog, maybe I get some more visitors to it and get a sense of how popular it is.

I am trying to be polite with the people that land on my blog and not track them so I don't use cookies. I host my stuff on cloudflare since that gives the best bang for my buck. In other words I want my blog to be performant and free to host.

My blog uses some analytics that are available through cloudflare but they are very respectful of user privacy in that they are GDPR and CCPA compliant. This saves me the hassle of having to add a cookie consent form that disrupts the user navigation experience. I both like and dislike the analytics from cloudflare as the numbers I am seeing are a bit weird as I am only seeing a constant number.

Since I learned a bit more about SEO and about Google Search Console I decided to check my blog's performance and see what I can do to improve it. Submitted my sitemap and ran a performance check and even if performance was at 100/100 I saw that the images were not optimized.

My OCD kicked in and I had to figure out a way to address it, especially since I remembered that I read an article talking about this. I dug into it a bit and noticed he is using ImageMagik, cavif and cwebp to optimize the images, I decided to go a different way, essentially almost reinventing the wheel. I built a rust cli that converts bigger png and jpeg images to webp or cavif.

How?

The step by step process looks like this:

Change the shortcode for images to try and render the optimal image if supported by the browser
write the rust tool that traverses the directory tree and convert images to webp or avif
integrate the tool into the github action pipeline
perform caching on the github workflow to avoid spending too many github minutes on the actual conversion

What's in it for me?

1. Storage Cost Savings

AVIF can be ~70% smaller than PNG while maintaining similar or better quality.
If you're storing images on AWS S3, Google Cloud Storage, DigitalOcean Spaces, or another cloud provider, reducing storage by 70% directly cuts storage costs by the same percentage. Example:
- 100GB of PNGs → ~30GB of AVIF
- If storage costs $0.023 per GB (AWS S3 Standard):
- PNG: $2.30/month
- AVIF: $0.69/month
- Savings: ~$1.61 per 100GB/month (~70%)

2. Egress Bandwidth Cost Savings

Most cloud providers charge for outbound bandwidth (data transferred to users).
Smaller AVIF files mean lower bandwidth usage, leading to significant savings.
AVIF reduces bandwidth usage by ~70% compared to PNG. Example with AWS CloudFront:
- Data transfer cost (to the internet): $0.085 per GB
- If you serve 1TB of PNGs per month:
- PNG: 1TB → $85/month
- AVIF (70% smaller): 0.3TB → $25.50/month
- Savings: ~$59.50 per TB/month (~70%)

3. CDN Caching & Requests

Many e-commerce sites use a CDN (Cloudflare, CloudFront, Fastly, etc.).
Smaller images:
- Improve cache hit ratio (more images fit in CDN cache).
- Reduce origin fetch requests, further lowering egress costs.
- Speed up load times, improving user experience.

Total Cost Savings Estimate

Cost Factor	PNG	AVIF	Savings
Storage (100GB)	$2.30	$0.69	$1.61 (70%)
Egress (1TB/month)	$85.00	$25.50	$59.50 (70%)
Total Savings per TB	$87.11/month

The Playbook

Step 1: Shortcode

I avoided using javascript for this since html already gives a mechanism to render an image with a fallback

<picture>
  <source srcset="{{id}}.avif" {% if alt %}alt="{{alt}}" {% endif %} />
  <source srcset="{{id}}.webp" {% if alt %}alt="{{alt}}" {% endif %} />
  <img src="{{id}}.png" {% if alt %}alt="{{alt}}" {% endif %} />
</picture>

Step 2: Rust tool

The way I structure my posts is that each post lies neatly inside its own folder, along with all the images and any other extra assets that add some sort of value to the content.

So, from the theme I grab all the png images

    let mut input_paths: Vec<Params> = glob("content/**/*.png")?
        .filter_map(Result::ok)
        .map(|path| Params {
            path,
            should_recreate: args.recreate,
            ..Default::default()
        })

The trouble is the theme also contains some images which need to be converted. At the moment the only image is my logo

    let theme_image_paths: Vec<Params> = glob("themes/**/*.jpg")?
        .filter_map(Result::ok)
        .map(|path| Params {
            path,
            should_recreate: args.recreate,
            should_resize: true,
        })
        .collect();

In order to save time, converting images that exist already is a bit redundant so the tool checks if the path exists already and if it does, conversion is skipped. Working with paths is surprisingly straightforward. I was previously quite afraid to write code in rust because I feared the overhead.

The actual code is stupid easy to understand and reason about, even for me

    //  webp file path
    let webp_file_path = parent_dir.join(format!("{}.webp", file_stem).as_str()); // Convert to .webp as an example
    // was it already converted?
    if !webp_file_path.exists() {

A big chunk of the code lies in the conversion code which also gave me the most brain pain.
We converted webp using the webp crate

fn convert_to_webp(img: &DynamicImage, output_path: &str) -> AnyResult<()> {
    let encoder = WebpEncoder::from_image(img).unwrap();
    let webp_data = encoder.encode(75.0); // Quality 75
    let mut file = File::create(output_path)?;
    file.write_all(&webp_data)?;
    println!("Saved WebP to {}", output_path);
    Ok(())
}

And then avif using the avif crate

fn convert_to_avif(img: &DynamicImage, output_path: &str) -> AnyResult<()> {
    let (width, height) = img.dimensions();

    let rgba = img.to_rgba8();
    let encoded_avif = Encoder::new()
        .with_quality(50.0)
        .with_alpha_quality(50.0)
        .with_speed(10)
        .with_alpha_color_mode(AlphaColorMode::UnassociatedClean)
        .with_num_threads(Some(4));
    let avif_pixels = rgba
        .pixels()
        .map(|p| Rgba {
            r: p[0],
            g: p[1],
            b: p[2],
            a: p[3],
        })
        .collect::<Vec<Rgba>>();

    let EncodedImage {
        avif_file,
        color_byte_size,
        alpha_byte_size,
        ..
    } = encoded_avif
        .encode_rgba(Img::new(
            &avif_pixels,
            width.try_into().unwrap(),
            height.try_into().unwrap(),
        ))
        .unwrap();
    let mut file = File::create(output_path)?;
    file.write_all(&avif_file)?;
    println!("Saved AVIF to {}", output_path);
    Ok(())
}

The reason converting to avif was a bit more convoluted was due to the requirement for pixels to be in rgba format. I had to convert the image to rgba and then convert the pixels to Rgba format(thank you libs with different types). This was a bit of a pain but I managed to get it working.

I'm not an image processing expert so the solution was the result of a long conversation with trail and error with ChatGPT, then again this is why I love rust and how strict it is. It forces you to write code in a way that if it runs it most likely is correct.

Step 3: Github action

The action installs and enables rust so that the cli can be used

- name: Install Rust
  run: |
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
    rustup toolchain install nightly
    rustup default nightly
    echo "$HOME/.cargo/bin" >> $GITHUB_PATH

- name: Verify Rust Installation
  run: |
    rustup --version
    rustc --version
    cargo --version

- name: Build CLI tool
  run: |
    cargo build --manifest-path ./helpers/image-optimizer/Cargo.toml --release --verbose

- name: Perform the optimization
  run: ./helpers/image-optimizer/target/release/image-optimizer

One thing I particularly liked was the fact that it is possible to use a relative path to Cargo.toml which means no mucking about with paths.

Step 4: Caching

Now one thing about rust that is a bit of a bummer is that builds take quite some time. I guess that is the price to pay for static memory analysis. I would love to do a deep dive at some point on the optimization of rust build times but that is a story for another time.

The one thing that github tends to hold you accountable for is the number of build minutes you use when a workflow runs, so having rust install itself, download dependencies and then run a build for the tools can quickly add up.

The optimization for build times covers caching cargo dependencies but also caching the built binary.

I cached most things that I was able to but I am getting mixed results when trying to cache apt packages. It simply does not seem to work as intended in the naive approach.

- name: Install OS Dependencies (if needed)
  run: |
    # Create a file listing your required packages, one per line.
    cat > apt-packages.txt << EOF
    nasm
    EOF
    sudo apt-get update
    sudo apt-get install -y --no-install-recommends $(cat apt-packages.txt)

- name: Cache Rust toolchain
  uses: actions/cache@v3
  with:
    path: ~/.rustup
    key: ${{ runner.os }}-rustup-${{ hashFiles('rust-toolchain') }}
    restore-keys: |
      ${{ runner.os }}-rustup-

  ...
- name: Cache Cargo dependencies and target
  uses: actions/cache@v3
  with:
    path: |
      ~/.cargo/registry
      ~/.cargo/git
      ./helpers/image-optimizer/target
    key: ${{ runner.os }}-manual-cargo-${{ hashFiles('**/Cargo.lock') }}

The savings in time by using the caching is quite substantial. The first run of the workflow took 15 minutes, the run that had cached the deps was less than 1 minute. Even with a substantial amount of images this will most likely not be a bottleneck.

Conclusion

Optimizing images can decrease load on the server up to 70%
... and it also improves performance which is beneficial for SEO
Optimizing github workflows can save you a lot of wait time
... and github minutes
While this was interesting to do, I optimized for something that did not move the needle at all, it just made the evaluation in the Google Search Console a bit better.

This is what I have to blame my OCD for. I am happy with the result and I learned quite a few things about rust and image processing. I am also happy that I managed to get the avif conversion working as it is a format that is not yet widely supported but is the most efficient format out there.

Rustify some puppeteer code(part I)

Artur Daschevici — Thu, 27 Jun 2024 15:30:00 +0000

Why?

Rust is pretty amazing but there are a few things that you might be weary about. There are few war stories of companies building their entire stack on rust or and then living happily ever after. Software is an ever evolving organism so in the darwinian sense the more adaptable the better. Enough of that though, not here to advocate any particular language or framework, what I want is to share my experience with writing an equivalent scraper in rust to my previous post where I used golang and chromedp.

The experience using go with chromedp to automate chrome was pretty good, it is not as powerful as what is available in puppeteer so I figured I would have a look at what might be available in the rust landscape.

What?

In rust there are several libraries that deal with browser automation, a few I have had a look at are:

fantocini - A high-level API for programmatically interacting with web pages through WebDriver, but I want chrome devtools protocol instead.
rust-headless-chrome - chrome devtools protocol client library in rust, not as active as the crate I wound up using.
chromiumoxide - this is the one that seem to be the most active in terms of development so it looks like a good choice at time of writing.

TL;DR

As I was reading one of my older posts that focuses on quasi live coding I realized it was boring as hell, and if your attention span is that of a goldfish, like mine is, it would probably make sense to just drop in a link to the repo so that you can download the code and try it out yourself. The repo is a collection of rust prototypes that I have been building for fun and learning, haven't had yet a compelling reason to use rust in production unfortunately 😢.

How?

To my surprise the code was closer in structure to the puppeteer version than it was to the chromedp. The chromedp version uses nested context declarations to manage the browser and page runtimes, the rust version uses a more linear approach. You construct a browser instance and then you can interact with it as a user would. This points at the fact that the chromiumoxide api is higher level.

The way you can set things up to keep your use cases separate is by adding clap to your project and use command line flags to select the use case you want to run.

You will see that I have covered most cases but not everything is transferable from puppeteer or chromedp to the chromiumoxide version. I will not go through the setup of rustup, rust toolchain or cargo as this is a basic and well documented process, all you have to do is search for getting started with rust and you will find a bunch of resources.

Show me the code

1. Laying down the foundation

set up my project root

cargo new rust-crawl-pupp
cd rust-crawl-pupp
cargo install cargo-edit # this is useful for adding and upgrading dependencies

add dependencies via cargo add

[dependencies]
chromiumoxide = { version = "0.5.7", features = ["tokio", "tokio-runtime"] } # this is the main dependency
chromiumoxide_cdp = "0.5.2" # this is the devtools protocol
clap = { version = "4.5.7", features = ["derive", "cargo"] } # this is for command line parsing
futures = "0.3.30" # this is for async programming
tokio = { version = "1.38.0", features = ["full"] } # this is the async runtime
tracing = "0.1.40" # this is for logging
tracing-subscriber = { version = "0.3.18", features = ["registry", "env-filter"] } # this is for logging

add clap command line parsing to the project so that each different use case can be called via a subcommand
define your imports

use clap::{Parser, Subcommand};

define your command structs for parsing the command line arguments, this will allow for each use case to be called with its own subcommand like so cargo run -- first-project, cargo run -- second-project, and so on.

#[derive(Parser)]
#[command(
    name = "OxideCrawler",
    version = "0.1",
    author = "artur",
    about = "An example application using clap"
)]
struct Cli {
    #[command(subcommand)]
    command: Commands,
}
#[derive(Subcommand, Debug)]
enum Commands {
    FirstProject {},
    SecondProject {},
    ...
}

the way you can hook this into the main function is via a match statement that will call the appropriate function based on the subcommand that was passed in.

let args = Cli::parse();
...
match &args.command {
    Commands::FirstProject {} => {
        let user_agent = spoof_user_agent(&mut browser).await?;
        info!(user_agent, "User agent detected");
    }
    ...
}

2. Starting browser and the browser cleanup

use the launch method and its options to start the browser, if the viewport and window size are different, the browser will start in windowed mode, with the page size being smaller.

let (mut browser, mut handler) = Browser::launch(
    BrowserConfig::builder()
        .with_head() // this will start the browser in headless mode
        .no_sandbox() // this will disable the sandbox
        .viewport(None) // this will set the viewport size
        .window_size(1400, 1600) // this will set the window size
        .build()?,
)
.await?;

let handle = tokio::task::spawn(async move {
    loop {
        match handler.next().await {
            Some(h) => match h {
                Ok(_) => continue,
                Err(_) => break,
            },
            None => break,
        }
    }
});

the browser cleanup needs to be done correctly and there are two symptoms that you will see if you missed anything:
- the browser will not close - hangs at the end
- you might get a warning like the following:
```
  2024-06-26T08:40:01.418414Z  WARN chromiumoxide::browser: Browser was not closed manually, it will be killed automatically in the background
```
to correctly clean up your browser instance you will have to call these on the code paths that close the browser
```
browser.close().await?;
browser.wait().await?;
handle.await?;
```

3. Use cases

In the repo each use case lives in its own module most of the time. There are some cases where you might have two living in the same module when they are very closely related, like in Use Case c..

a. Spoof your user agent:

The only way I have found to set your user agent was from the Page module via the set_user_agent method

  let page = browser.new_page("about:blank").await?;
  page.set_user_agent(
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) \
       Chrome/58.0.3029.110 Safari/537.36",
  )
  .await?;
  page.goto("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
      .await?;

b. Grabbing the full content of the page is pretty straightforward

      let page = browser
        .new_page("https://scrapingclub.com/exercise/list_infinite_scroll/")
        .await?;
    let content = page.content().await?;

c. Grabbing elements via css selectors,

  let elements_on_page = page.find_elements(".post").await?;
  let elements = stream::iter(elements_on_page)
      .then(|e| async move {
          let el_text = e.inner_text().await.ok();
          match el_text {
              Some(text) => text,
              None => None,
          }
      })
      .filter_map(|x| async { x })
      .collect::<Vec<_>>()
      .await;

performing relative selection from a specific node and mapping the content to rust types

  ...
  let product_name = e.find_element("h4").await?.inner_text().await?.unwrap();
  let product_price = e.find_element("h5").await?.inner_text().await?.unwrap();
  Ok(Product {
      name: product_name,
      price: product_price,
  })
  ...

d. When the page has infinite scroll you will have to scroll to the bottom of the page to be able to collect all the elements you are interested in. To achieve this you need to inject javascript into the page context and trigger a run of the function. The chromiumoxide api seems to have really decent support for this, I faced much less resistance than I did with chromedp and go.

  let js_script = r#"
      async () => {
        await new Promise((resolve, reject) => {
          var totalHeight = 0;
          var distance = 300; // should be less than or equal to window.innerHeight
          var timer = setInterval(() => {
            var scrollHeight = document.body.scrollHeight;
            window.scrollBy(0, distance);
            totalHeight += distance;

            if (totalHeight >= scrollHeight) {
              clearInterval(timer);
              resolve();
            }
          }, 500);
        });
    }
  "#;
  let page = browser
      .new_page("https://scrapingclub.com/exercise/list_infinite_scroll/")
      .await?;
  page.evaluate_function(js_script).await?;

e. When you need to wait for an element to load, this was not exactly part of the chromiumoxide api so I had to hack it together. Given my limited rust expertise there probably a better way to do this but this is what I managed to come up with. If the async block runs over the timeout then the element_result will be an error, otherwise poll the dom for the element we are looking for.

  use tokio::time::{timeout, Duration};
  ...
  let element_result = timeout(timeout_duration, async {
      loop {
          match page.find_element(selector).await {
              Ok(element) => return Ok(element),
              // Wait for a short interval before checking again
              Err(e) => tokio::time::sleep(Duration::from_millis(100)).await,
          }

      }
  })
  .await;

4. Fixtures to replicate various scenarios

Some websites, actually most websites have some sort of delay for loading different parts of the page, in order to prevent blocking the entire page. To replicate this behavior fixtures can be used to inject nodes into the dom with a delay. For the more edge case scenarios I created fixtures to emulate edge behaviors while not actually having to remember a website that is live and behaves like that.

The HTML is really basic:

  <div id="container">
      <!-- New node will be appended here -->
  </div>

  <script src="script.js"></script>

The script.js file is slightly more, but still fairly straightforward:

  document.addEventListener('DOMContentLoaded', () => {
    // Function to create and append the new node
    function createDelayedNode() {
      // Create a new div element
      const newNode = document.createElement('div');

      // Add some content to the new node
      newNode.textContent = 'This is a new node added after a delay.';

      // Add some styles to the new node
      newNode.style.padding = '10px';
      newNode.style.marginTop = '10px';
      newNode.style.backgroundColor = '#f0f0f0';
      newNode.style.border = '1px solid #ccc';
      newNode.id = 'come-find-me';

      // Append the new node to the container
      const container = document.getElementById('container');
      container.appendChild(newNode);
    }

    // Set a delay (in milliseconds)
    const delay = 3000; // 3000ms = 3 seconds

    // Use setTimeout to create and append the node after the delay
    setTimeout(createDelayedNode, delay);
  });

What it will do is create a new node with some text content and some styles, then append it to the container div after a delay of 3 seconds.

Why to be continued?

What I hate more than to be continued in a TV show where I don't have the next episode available is a blog post that has code that looks reasonable and that it might work, but doesn't. So going by the lesser of two evils principle I decided to make this a two parter which will give me the time to write and test the other use cases in order to make sure everything works as expected.

Conclusions

This is one of the few times I have stuck with rust through the pain and I have to say it was a better experience than I had with go and chromedp
writing the code was slightly faster since there was less boilerplate to write
messing around with wrappers and unwrap() was challenging but probably in time it gets easier
the code in rust looks more like puppeteer than the go version did

In Part II I will cover dealing with bot protection, handling frames, forms and more. Stay tuned!

OpenAI api RAG system with Qdrant

Artur Daschevici — Wed, 19 Jun 2024 12:35:00 +0000

Why?

OpenAI has been making it easier and easier to build out GPT agents that make use of your own data to improve the generated responses of the pretrained models.

Agents give a way to inject knowledge about your specific proprietary data into your pipeline, without actually sharing any private information about it. You can also improve the recency of your data too which makes you less dependent on the model's training cycle.

OpenAI has improved the DX, UX and APIs since version 3.5, and has made it easier to create agents and embed your data into your custom GPTs. They have lowered the barrier to entry which means that virtually anyone can build their own assistants that would be able to respond to queries about their data. This is perfect for people to experiment on building products. IMO this is a very good approach to enable product discovery for the masses.

Most big AI contenders on the market provide you with a toolbox of high level abstractions and low to no code solutions. The weird thing about my approach to learning things is that not having some understanding of the first principles of the tech I'm using makes me feel a bit helpless, this is why I figured trying to build my own RAG system would be a good way to figure out the nuts and bolts.

What?

I wanted to get a project for running my own pipeline with somewhat interchangeable parts. Models can be swapped around so that you can make the most of the latest models either available on Hugginface, OpenAI or wherever.

Because things are moving so fast in model research the top contenders are surpassing each other every day pretty much. A custom pipeline would allow us to quickly iterate and test out new models as they evolve. This allows you to try out new models and just as easily rollback your experiment.

What I wound up building is a Streamlit app that uses qdrant to index and search data extracted from a collection of pdf document. The app is a simple chat interface where you can ask questions about the data and get responses from a mixture of GPT-4 and the indexed data.

How?

1. Setting up the environment

use pyenv to manage python versions

   # update versions
   pyenv update
   # install any python version
   pyenv install 3.12.3 # as of writing this
   # create a virtualenv
   ~/.pyenv/versions/3.12.3/bin/python -m venv .venv
   # and then activate it
   source .venv/bin/activate

2. Install the dependencies

   # install poetry
   pip install poetry
   # install the dependencies
   poetry install

the dependencies section of the pyproject.toml file should look like this:

   ...
   [tool.poetry.dependencies]
    python = "^3.12"
    streamlit = "^1.32.1"
    langchain = "^0.1.12"
    python-dotenv = "^1.0.1"
    qdrant-client = "^1.8.0"
    openai = "^1.13.3"
    huggingface-hub = "^0.21.4"
    pydantic-settings = "^2.2.1"
    pydantic = "^2.6.4"
    pypdf2 = "^3.0.1"
    langchain-community = "^0.0.28"
    langchain-core = "^0.1.31"
    langchain-openai = "^0.0.8"
    instructorembedding = "^1.0.1"
    sentence-transformers = "2.2.2"
   ...

3. Set up the loading of the variables from a config file

a nice way to manage settings is to use pydantic and pydantic-settings

   from pydantic import Field, SecretStr
   from pydantic_settings import BaseSettings, SettingsConfigDict

   class Settings(BaseSettings):
       model_config = SettingsConfigDict(env_file="config.env", env_file_encoding="utf-8")
       hf_access_token: SecretStr = Field(alias="HUGGINGFACEHUB_API_TOKEN")
       openai_api_key: SecretStr = Field(alias="OPENAI_API_KEY")

this way you can load the settings from config.env but variables in the environment override the ones in the file.

a nice extra is that you also get type checking and validation from pydantic including SecretStr types for sensitive data.

4. Set up the UI elements

Streamlit makes it quite easy to strap together a layout for your app. You have a single script that can run via the streamlit binary:

   streamlit run app.py

The gallery has many examples of various integrations and components that you can use to build your app. You have smaller components like inputs and buttons but also more complex UI tables, charts, you even have ChatGPT style templates.

For our chat interface we require very few elements. Generally to create them you only need to use streamlit to initialize the UI.

   import streamlit as st
   ...
   def main():
       st.title("ChatGPT-4 Replica")
       st.write("Ask me anything about the data")
       question = st.text_input("Ask me anything")
       if st.button("Ask"):
           st.write("I'm thinking...")
           response = get_response(question)
           st.write(response)
   ...
   main()

The one thing I find a bit awkward is the fact that if you have elements that need to be conditionally displayed the conditions tend to resemble the javascript pyramid of doom if you have too many conditionals in the same block.

Below is a simple example so you can see what I mean:

   if len(pdf_docs) == 0:
       st.info("Please upload some PDFs to start chatting.")
   else:
       with st.sidebar:
           if st.button("Process"):
               with st.spinner("Processing..."):
                   # get raw content from pdf
                   raw_text = get_text_from_pdf(pdf_docs)
                   text_chunks = get_text_chunks(raw_text)

                   if "vector_store" not in st.session_state:
                       start = time.time()
                       st.session_state.vector_store = get_vector_store(text_chunks)
                       end = time.time()
                       # create vector store for each chunk
                       st.write(f"Time taken to create vector store: {end - start}")

This makes me think that it is probably not designed for complex UIs but rather for quick prototyping and simple interfaces.

5. pdf data extraction

I used the PyPDF2 library to extract the text from the pdfs. The library is quite simple to use and you can extract the text from a pdf file with a few lines of code.

   import PyPDF2

   def get_text_from_pdf(pdf_docs):
       raw_text = ""
       for pdf in pdf_docs:
           pdf_file = pdf["file"]
           pdf_reader = PyPDF2.PdfFileReader(pdf_file)
           for page_num in range(pdf_reader.numPages):
               page = pdf_reader.getPage(page_num)
               raw_text += page.extract_text()
       return raw_text

The extracted text should be chunked into smaller pieces that can be used to create embeddings for the qdrant index.

   def get_text_chunks(raw_text):
       text_chunks = []
       for i in range(0, len(raw_text), 1000):
           text_chunks.append(raw_text[i:i + 1000])
       return text_chunks

6. Setting up the `qdrant` server via `docker`

The best way to set up qdrant is to use docker and to keep track of the environment setup docker-compose is a nice approach. You can set up the qdrant server with a simple docker-compose.yml file like the one below:

   version: '3.9'

   services:
     qdrant:
       image: qdrant/qdrant:latest
       ports:
         - "6333:6333" # Expose Qdrant on port 6333 of the host
       volumes:
         - qdrant_data:/qdrant/data # Persistent storage for Qdrant data
       environment:
         RUST_LOG: "info" # Set logging level to info

   volumes:
     qdrant_data:
       name: qdrant_data

7. Indexing the data

The qdrant client can be used to index the embeddings and perform similarity search on the data. You can pick and choose the best model for embeddings for your data and swap them out if you find a better one.

   def get_vector_store(text_chunks, qdrant_url="http://localhost:6333"):
       embeddings = HuggingFaceInstructEmbeddings(model_name="avsolatorio/GIST-Embedding-v0", model_kwargs={"device": "mps"})
       vector_store = Qdrant.from_documents(
           text_chunks,
           embeddings,
           url=qdrant_url,
           collection_name="pdfs",
           force_recreate=True,
       )
       return vector_store

8. sending the query

In order to send the query to qdrant you again need to embed it to allow to do a similarity search over your collection of documents.

   def get_response(question, qdrant_url="http://localhost:6333"):
        embeddings = HuggingFaceInstructEmbeddings(model_name="avsolatorio/GIST-Embedding-v0", model_kwargs={"device": "mps"})
        query_vector = embeddings.encode(question)
        vector_store = Qdrant(url=qdrant_url, collection_name="pdfs")
        response = vector_store.search(query_vector, top_k=1)
        return response

9. Analysis

You can swap out any of the components in this project with something else. You could use Faiss instead of qdrant, you could use OpenAI models for everything(embeddings/chat completion) or you could use open models.

You can forego the UI and simply use fastapi to create an API to interact with the PDF documents. I hope this gives you some sense of the possibilities that are available to you when building your own RAG system.

Conclusions

you can build your own agent and have it respond to queries about your data quite easily
streamlit is great for prototyping and building out simple interfaces
qdrant is good for performing similarity search on your data
when building RAG systems you need to make use of embedding models to encode your data
embedding models are the most taxing parts of the pipeline
if you have pluggable parts in your pipeline you can swap them out easily to save costs
pydantic and pydantic-settings are great for adding type checking and validation to your python code

Converging project boilerplates with copier

Artur Daschevici — Tue, 11 Jun 2024 10:00:00 +0000

Why?

Technically when you start a new project the best way to approach it is by using the CLI tool of the realm, such as svelte-kit, astro, django-cli etc..., you get the idea. The huge bonus to doing this is that you get the best practices baked in and as new standards are created the CLI gets updated.

So far the frontend has been a lot luckier with the tools as far as project generation goes, every major framework having come out with their own project generation tool, some having more than one possibly due to multiple schools of thought.

There are some backend frameworks that have project generation tools too but so far it seems to be difficult to agree on the structure. The best you can do is find a way to structure it that looks like the majority and makes sense for you. I have been building spiders and crawlers for data ingestion pipelines using python at first and then node and go. Even more recently I have been looking at hacking out some tweaks in some of my neovim plugins(that is lua).

For example neovim plugins have a pretty standard setup, they will have a folder layout something like the following:

scratcher.nvim/
├── README.md
├── lua
│   └── scratcher
│       └── init.lua
└── plugin
    └── scratcher.lua

The standard way of naming things seems to be gravitating towards having some conventions as you can see so setting up a new plugin would be pretty much repetitive and automatable. And it will probably save you some time and willpower in the long run, provided you have some sense of what your final architecture needs to look like.

How?

If you are coming from python like I am then you may already be familiar with cookiecutter. I have been in the situation a few times where it might have made sense to use it, but every time it was a matter of balancing out the timeline and trying to stay away from over engineering.

Lately though the stuff I have been dealing with has been slightly on the more experimental side so churning out something new is something that happens quite often, so it makes more sense to have a prebaked architecture for specific project styles.

From the project templating libraries I was aware of cookiecutter and copier. cookie-cutter uses json for driving the generation while copier uses yaml.

In the end I used copier as I tend to favor yaml because it allows for comments in the config. It makes it easy to plop random pieces of info or even docs in there. Since the config files driving the wizard it can become quite convoluted and also difficult to read as you would normal code so having the ability to document different options is probably a plus I would think.

The library allows you to build a sort of setup wizard where you can set up your desired flow of questions, and you can use the choices supplied to drive what folders will be used in the final project boilerplate. This is pretty nifty as it gives you the ability to customize stuff all the way down to the build process.

Another neat thing is that when you have decent chunks of code that can be shared, so you can just put that in your boilerplate, so it will essentially give you things just the way you like them.

Cherry pick of features

There are a few notable features that I would kick myself if I didn't mention:

you can define choices for your options by using the choices key in your copier.yml file:

project_type:
type: str
help: What type of project are you creating?
default: neovim-plugin
choices:
    - neovim-plugin
    - golang-cli
    - python-cli

you can include different files in the root level copier.yml thus breaking down the wizard in composable parts, for example the CI/CD parts can be shared across projects
```
!include shared-conf/ci-cd.*.yml
```
defaults are very powerful and can also make use of current runtime context which is quite nice. Essentially you can think of it as a way to have your very own project wizard that is tweaked for every one of your needs.

Cool use-cases

you can define a folder/file be created conditionally depending on an option selection eg:

You define your copier.yml like this to give you a choice into the type of project:
```
project_type:
type: str
help: What type of project are you creating?
default: neovim-plugin
choices:
    - neovim-plugin
    - golang-cli
    - python-cli
```
you then create a conditionally rendered folder, the naming follows jinja templating rules, so it might look something like the following
```
{% if project_type == 'neovim-plugin' %}{{project_name}}.nvim{% endif %}
```
It looks a bit strange, it will probably not work on Windows and it might look daunting at first but hopefully you will only need to revisit the hierarchy when you update your project structure template. This is not something I would expect to happen very often.
in more advanced use cases you may need to write some custom python code for transforming/processing of entities in your jinja templates, or template strings.
To hook this in you need to enable the jinja template extensions and add a separate package copier-templates-extensions
```
_jinja_extensions:
- copier_templates_extensions.TemplateExtensionLoader
- extensions/context.py:ContextUpdater
```
This allows you to load specific extensions in your project generator runtime and it can serve different functions. The following snippet illustrates a way you can update the context:
```
from copier_templates_extensions import ContextHook

class ContextUpdater(ContextHook):
    def hook(self, context):
        new_context = {}
        new_context["onboarding"] = "first steps with " + context["project-type"]
        return new_context
```
For example you might decide to create a file called "first steps with .txt" once the project is generated. In template form the file name would be {{ onboarding }}

Conclusions:

you can build hybrid boilerplates for your project and drive generating the folder hierarchy from a single repo
the notation is a bit weird with templated folder names, will not work on Win
the templated naming is also very powerful allowing for conditional creation of folders

Gopherizing some puppeteer code

Artur Daschevici — Thu, 06 Jun 2024 22:00:00 +0000

Why?

As developers we sometimes get a bad case of the shiny new object syndrome. I hate to say it but every time I start hacking on something new, the urge to add something new is quite overwhelming. It is really tough to keep an interest in projects for a long time and it starts to become tedious the deeper you go into the weeds. I suppose this is why people list growth as one of their top motivations.

I consider that anything new is an opportunity for growth, and doing something over and over in a similar manner quickly becomes a tedious. I've been building various types of scrapers since 2011, and it all started because I wanted to automate a workflow and save myself some time. The time spent on automating this was probably more than if I had done this by hand but it was interesting interacting via http from code and crunching the data automatically.

The amount of data on the web is pretty crazy, you have various sources and multiple types of data that can be combined in very interesting ways. Back in those days dropshipping was becoming huge and people were performing arbitrage across Amazon/Ebay/local flea-markets etc.. Tools that were able to perform analytics across these shops were quite trendy, and the market was slightly less crowded, so for me building crawlers seemed like a nice idea to build out a good customer base.

Nowadays due to RAG systems, gathering data automatically, breaking it down and feeding it into embedding models and storing it in vector databases for LLM information enhancement has come back into the spotlight. In between then and now there have been a few changes in the way data is served up for consumption. Off the top of my head:

single page apps have gained huge traction, most everyone turning to building their content in a js bundle, loading everything on the fly as the page loads
websites have become fussy about having their data used by unknown parties, so they have been closing down access and have become very litigious(#TODO: maybe add some cases of court cases Linkedin vs those guys, Financial Times vs OpenAI)
bot detection and prevention - this one is funny since it is like a flywheel, it built 2 lucrative markets overnight - bot services and anti bot protection
TBH, it's difficult to predict where this might be heading, it kind of feels like people have been aiming to move all their datas into data centers but since data is becoming so guarded...will they move back to paper?

Because of SPAs and the wide adoption of js in websites it is much more convenient to use some sort of browser automation to crawl pages and extract the information. This makes it less prone to badgering the servers, and having to reverse engineer the page content loading, so you will probably want to use either a chrome developer tools protocol or webdriver flavored communications protocol with the browser. Back in the day IIRC I have also used the PyQt bindings for acessing the Qt browser component but nowadays its mostly straight-up browsers.

These days my goto is puppeteer. It's a weird tool that can be easily be used to scrape data from pages. The reason I say it is weird is mainly due to the deceiving nature of the internals, essentially using two js engines that communicate via the cdp protocol that is a a very dense beast and does not play nice with complex objects.

Recently it has become more appealing to me to use strongly typed languages. This is probably because I have started to narrow down my experiments to very small code samples that illustrate one thing at time. I would go as far as to call it experiment driven development. Duck typing is fun as you can print pretty much anything you want. I was thinking to use rust but it has a very tough learning curve. Node is pretty nice with mjs but it's confusing sometimes when it crosses over between the two event loops, also while it is good for communicating on cdp it is not really designed for sync code and python is a bit boring for me so I decided to look at go. Since it is a google language I expected it to have decent support for cdp, and the learning curve is slightly gentler than rust.

How?

Looking at the alternatives there are two that stand out chromedp and rod. Rod looks like it is the prodigal son of behave and cucumber some well established BDD frameworks. Personally I am not finding the MustYaddaYadda... very readable and combining it with other custom APIs would probably make it become inconsistent. It has a few nice things in the way it abstracts iframes but I am just unable to go past the higher level API.

In the end I wound up choosing chromedp. It works pretty well for most use cases, there are some places where it doesn't quite cut it and I wish it did, but by now I have come to terms there is no one technology to rule them all, wouldn't it be nice if that existed?

You can install it via go get -u github.com/chromedp/chromedp and then you can start using it in your code. It has quite a few submodules and related projects that you may want to use depending on your concrete use case.
Generally if your use case is only data extraction and you have no tricky actions to deal with(page is bot resistant, some elements are loaded at later times, iframe hell etc...).

import (
    "context"
    "log"
    "time"

    "github.com/chromedp/chromedp"
    "github.com/chromedp/cdproto/cdp"
    // for slightly more advanced use cases
    "github.com/chromedp/cdproto/browser"
    "github.com/chromedp/cdproto/dom"
    "github.com/chromedp/cdproto/storage"
    "github.com/chromedp/cdproto/network"
)

The pleasant surprises

Well, calling them surprises is a bit of a stretch, I have been golang over the years and I have to admit it is a pretty nice ecosystem and language.
chromedp automates chrome or any binary that you are able to communicate with via cdp. The API is somewhat intuitive, haven't found myself diving into the guts of it very often to figure out how stuff works. The good part is that once you extract the data from the nodes you are interested in you can map it to go structs and make use of the go typing system.

For example you can grab a list of elements via selector:

    var productNodes []*cdp.Node
    if err := chromedp.Run(ctx,
        // visit the target page
        chromedp.Navigate("https://scrapingclub.com/exercise/list_infinite_scroll/"),
        chromedp.Evaluate(script, nil),
        chromedp.WaitVisible(".post:nth-child(60)"),
        chromedp.Nodes(`.post`, &productNodes, chromedp.ByQueryAll),
    ); err != nil {
        log.Fatal("Error while trying to grab product items.", err)
    }

then map each element to a struct

    for _, node := range productNodes {
        if err := chromedp.Run(ctx,
            chromedp.Text(`h4`, &name, chromedp.ByQuery, chromedp.FromNode(node)),
            chromedp.Text(`h5`, &price, chromedp.ByQuery, chromedp.FromNode(node)),
        ); err != nil {
            log.Fatal("Error while trying to grab product items.", err)
        }
        products = append(products, Product{name: name, price: price})
    }

Another nice perk is that go is built with concurrency in mind so crunching the extracted data can be a lot more performant than in puppeteer.

Yet another pretty nifty thing I found is that you can deliver a binary that can be compiled for multiple platforms and can be distributed easily. This is a huge plus given that you may not really know who the user of the tool might be in the end.

The ugly parts

The way to communicate with the browser is still through the cdp protocol and sometimes you need to pass objects only objects that can be serialized.

If you need to work with objects that can't be serialized you will need to inject js into the page context and interact with it.

When you have a page that contains iframes it is problematic to trigger events on the elements inside them. You can extract data from it but triggering events gets messy as you need js for that.
An example of how you might extract data from an iframe might look something like this:


    var iframes []*cdp.Node
    if err := chromedp.Run(ctx, chromedp.Nodes(`iframe`, &iframes, chromedp.ByQuery)); err != nil {
        fmt.Println(err)
    }

    if err := chromedp.Run(ctx, chromedp.Nodes(`iframe`, &iframes, chromedp.ByQuery, chromedp.FromNode(iframes[0]))); err != nil {
        fmt.Println(err)
    }
    var text string
    if err := chromedp.Run(ctx,
        chromedp.Text("#second-nested-iframe", &text, chromedp.ByQuery, chromedp.FromNode(iframes[0])),
    ); err != nil {
        fmt.Println(err)
    }

But in order to trigger events on elements inside the iframe you can't just use the chromedp API, and since chromedp.Evaluate does not take a Node as context you will need to perform all the actions in javascript and that will make the resulting code a bit of a mishmash of go and js.

puppeteer also has some extra packages that can be used like puppeteer-stealth but chromedp does not seem to have an equivalent for that at this time. The rod package has rod stealth but I haven't tried it since the API is not to my liking.

The other slightly dissappointing missing feature is that when running in headless mode all the GPU features are disabled because it is running in a headless-chrome container which does not have a display server. Puppeteer is able to run with GPU features enabled allowing it to pass the webgl fingerprinting tests.

Conclusion

in some ways puppeteer is still better than chromedp, working with iframes falls short
rod is a nice alternative but its API looks like it was designed for testing, reminds me of cucumber
chromedp is a nice alternative to puppeteer if you are looking to build a binary that can be distributed easily
it is a bit more performant than puppeteer due to the concurrency model in go

Keep your estimates boring

Artur Daschevici — Wed, 29 May 2024 09:12:00 +0000

Why?

Most everyone I know has got shiny object syndrome. We all want to work on the latest and greatest. I myself am part of that crowd very much. Whenever I start on a project I will never pin the versions for the libraries. That adds anywhere between 0% and 50% on top of the project timeline. The most notable example is Javascript with its myriad of libraries. You would think everyone is familiar with this meme by now 😅.

In the first article in the series we talked about getting anchors right and trying to stay away from the uniqueness bias when choosing a good anchor. In How Big Things Get Done the authors also bring in a concept that was new to me: the reference class.
The phrase was originally coined in the 1970s by the psychologist Daniel Kahneman and his colleague Amos Tversky and is regularly used in the context of reference class forecasting.
Daniel and Amos refer to two types of views when estimating a project:

inside view which is the view while working on the project with your personal biases
outside view which is the view from the outside, looking at the project as a whole and comparing it to similar projects

Enough with the theory, in practice, what we actually want to achieve is to have our estimates work and be as accurate as possible.

How?

Now that we have the lingo down, let's get into the nitty-gritty. We want to figure out how to calculate the estimates, you got that right calculate. The calculations are based on being able to cut down your project from being a special and unique snowflake to a project that is similar to others. It's a combination of statistical and historical analysis of other projects as similar as possible to yours.

Going on a tangent here, wouldn't it be cool if we could have a database of software project estimates with numerical data, situational requirements and conditions, and perhaps how long the project took in the end? 🤔

You want to reduce your project to something as generic as possible then look for data about other projects like it. As a software developer you may be tempted to think that it is special and unique, but finding the commonalities will help you get a better estimate.
We could make use of both inside view and outside view:

outside view: see how long similar projects took(take the median) - I am referring to the reduced version where you cut out any product differentiators
inside view: see how long the differentiators will take (this you can break down further as well into common tasks and unique tasks)

... do you see where I am going with this? It's all turtles all the way down 🐢. Now you can already see how things can be broken down further into smaller pieces and how similarities can make estimating easier.

The numbers show a 30% increase in accuracy when using reference class forecasting, that is the outside view, with 50% not being uncommon.

The aspect that is different from plain anchor-based estimates is that you choose an anchor that is based on the reference class which makes it closer to the objective reality.

Into the future with AI

Last couple of years I have been working in the field of AI and I have been an avid reader of various papers and consumed a decent amount of tutorials and courses. Deep learning is an amazingly powerful tool that is able to draw conclusions based on the importance of a particular feature of the project and classify it.

If we had the data about projects we could train a multi-class classifier to predict the time it would take to complete a project(S/M/L/XL). This could be a great Trello plugin for example.

Linear regression can be another simpler approach to do a numeric estimate of the project timeline. Now, this feels like we are taking all the joy out of the agile SDLC, but remember this is only supposed to be used as a data focused approach, from the outside view i.e. looking objectively at the data, so no hard feelings to be had 😉.

Thinking a bit further we could have an LLM + RAG system that looks at the database of projects we have broken down, does a similarity search and gives us some kind of standard estimates.

The data would probably be a huge challenge for this one. You would have to get data from various sources and have it clean, usable and can be used to train the models.

Conclusion

Stay boring: don't get caught up in the thinking your project is special and unique
Use the reference class: look at similar projects and see how long they took
Use both views: inside view and outside view to get a better estimate
Use data: if you have it, use it to your advantage
Ask for help: if you are not sure, ask someone who has done it before, outside perspective can add a layer of objectivity

Why I still struggle with estimates

Artur Daschevici — Sun, 26 May 2024 09:20:00 +0000

Why?

Most of my articles so far have been slightly heavy on the more technical side, this is what is looking like my forté. I find writing code and solving different problems really interesting.
OFC solving problems, especially while hacking comes with a fair degree of uncertainty and frustration...when things go wrong or simply everything turns into a big ol' rabbit hole.

I am not 100% certain whether the frustration comes from not being able to finish projects or from the perceived inability to stick to a self-imposed (albeit artificial) deadline.

So this time you will be reading some of my musings about estimations and some interesting aspects that can be found in a recent book I read that put some things into perspective for me.

How?

I have been trying for years to get close to delivering something on time. The bigger the project the more likely it is to veer off track.
The way you can get to near perfectly predictable timelines is if all the pieces in your project can be estimated with razor sharp precision and the way they will all fit together in the finished product.
Every part needs to be standard and should fit together like off the shelf components, not only that the different components need to fit together well at the end.
The book gives some statistics across projects of how often they are on time, on quality and budget, the number is dismally low across the projects that were analyzed in the book - only 0.5%.
There are some giant scale projects that have succeeded like the Empire State Building and the Guggenheim in Bilbao. The approach that seems to work well for removing the guess work is building the parts before the whole and using tried and tested solutions.

Where does it go wrong?

In software engineering mistakes are not as irrecoverable as building a physical structure wrong, of course as the impact of software systems grows the line is more and more blurred.
You can image there may come a time when the resilience of software will become as important as that of our homes...or maybe it already is and we just are unable to wrap our heads around it.
Most project examples described are delayed because of several reasons.

the initial assumption are vague and contain hidden complexity while sounding very simple and clear
the finished product looks slick and polished, catches the eye, but the architectural feasibility is not assessed beforehand
optimistic estimates
doing something that has never been done before
the human factor
inexperience
unknown unknowns

My A-HA moment

I had not been able to formalize this, and had not come across a term that resonated with me until I read a case study in the "How big things get done" by Bent Flyvbjerg and Dan Gardner.
The story refers to a newspaper columnist that engages in writing a biography. His estimate was based on his experience of writing particularly long articles. This prior experience bias is called an "anchor".
In the story the writer estimates the biography of ~17 chapters at 9 months to a year, using as an anchor estimate the fact that one long article takes 3 weeks to research and write. Needless to say this estimate was off by a factor of 7. In the end it took 7 years.
The story does have a happy end in his case however this is an outlier among the various case studies.
It turns out that anchors are a very common pattern we use for estimating how long something will take.

Typically we try to find similarities within our prior experience. The catch with software projects is that technology evolves so quickly and company/team culture is so unique that it makes anchors very much a guesstimation rule of thumb rather than a rigorous framework you might use.
There are the extrinsic aspects that change and of course there are the intrinsic goals such as writing better code, designing better architectures and products, faster development whatever motivates you.
Now, given all these things that evolve over time, what would you think the probability of your experience with something in the past would equate to your estimates being accurate for a similar project two years in the future in a different company?

One thing that works is breaking the project down into decent sized components and experiment on building the components, and make the project about putting things together.
You want to work with pretty large components, yet small enough that the experiments churn out fast. It's a balancing act but in the end solving a puzzle with 1000 pieces is much harder than one with 5.
I think Agile came out of this need for faster iteration and predictability...but in the end the probability of an estimate to be accurate to the minute is very low.

Conclusions

projects are a sum of experiments
projects are an aggregate of the experiences of the participants
boring technology is easier to estimate
avoiding employee churn helps with estimating projects

Screenshots optimization on OpenAI tokens

Artur Daschevici — Fri, 17 May 2024 06:30:28 +0000

Prologue

In my previous post I have taken the approach of extracting data from the html of the page, and in order to keep token usage low decided to also clean it. Here I will be looking at optimizing the token usage further.

Funnily enough taking screenshots of the element we are looking for, the number of tokens is drastically reduced.

Why?

When working with OpenAI APIs things are not exactly free. They are cheap but depending on how often you run certain operations it can add up quite quickly. Fortunately the API has some metrics for measuring usage for the different operations so that you don't get any surprises.

How?

So what you can do is pick a website at random, for the purpose of the exercise we could use homegate. It's a real estate listing site, so you might use it if you wanted to find a place to rent in Zürich.

Let's use puppeteer as per usual to load up the search results page that is listed above.

import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";


const urls = [
  "https://www.homegate.ch/rent/real-estate/zip-8002/matching-list?ac=2.5",
];

puppeteer.use(StealthPlugin());

async function grabSelectorScreenshot() {
  // usual browser startup:
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
  );
  for (const url of urls) {
    await page.goto(url, { waitUntil: "networkidle0" });
    // the following code will scroll to the bottom of the page
 }
}

Once we have loaded the listing search results you want to select the list element and take a screenshot of it.

To grab a picture of a single element on the page what you want to do is the following:

// you need to grab the right selector for your usecase
    const element = await page.$(".ResultListPage_resultListPage_iq_V2");
    const designatedPathPng = `./screenshots/${hashed}-list-ss.png`;
    await element.screenshot({ path: designatedPathPng, type: "png" });

This looks fairly straightforward and works fine, however if we have a look at the screenshot you will notice that some parts have not rendered.

The gotcha 😕

So...it seems there is a small conundrum with the randomly chosen website, it lazy renders parts as they come into view, which means we have to scroll down and get the bottom part visible before we can grab the snapshot. Just to be clear with this approach we will rely solely on the visual aspect of the page so there is no data available if the elements are not rendered.

To do this we will attempt to scroll down all the way to the bottom of the page and grab the search results:

    await page.evaluate(async () => {
      await new Promise((resolve, reject) => {
        var totalHeight = 0;
        var distance = 300; // should be less than or equal to window.innerHeight
        var timer = setInterval(() => {
          var scrollHeight = document.body.scrollHeight;
          window.scrollBy(0, distance);
          totalHeight += distance;

          if (totalHeight >= scrollHeight) {
            clearInterval(timer);
            resolve();
          }
        }, 500);
      });
    });

This tiny piece of code scrolls down to the bottom of the page and you can even delay it so it doesn't stress the host. Just to be clear you should always be ethical in your data collection and try not to annoy the site owners too much. It gets particularly dicey when going into the realm of commercial applications as then your legal bases need to be rock solid.

Finishing it up with a bit of AI

After we have scrolled all the way down to the bottom of the page we can come back to the snapshot of the element. It should now return the list of listings.

You can save the screenshot and send it out to OpenAI and grab the info from the image.

async function run(propertyInfoImage, imageType) {

  const response = await openai.chat.completions.create({
    model: "gpt-4-turbo",
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: `Can you extract the property prices out of this
            image and send me the results? Can you send me the output as JSON?`,
          },
          {
            type: "image_url",
            image_url: {
              // imageType is png/jpeg
              url: `data:image/${imageType};base64,${propertyInfoImage}`,
            },
          },
        ],
      },
    ],
  });
  // spinner.succeed('Received response from OpenAI');
  console.log(response.choices[0]);
  console.log(response.usage);
}

There are a couple of interesting things about the images:

different type of image files might yield better results or better token usage metrics
the number of tokens used is considerably slower than in the case of text, which is pretty interesting eg:

{ prompt_tokens: 799, completion_tokens: 289, total_tokens: 1088 }

Conclusion

it's very surprising that using images has a lower token usage than using text by several orders of magnitude
chunking might be more challenging since you need to rely on visual marker detection which feels a bit more subjective
it is pretty interesting what you can do with visuals, especially when it comes to extracting data. You might find it interesting interpreting charts automatically or other types of information encoded into images and extracting it automatically.

Hacking out an AI spider with Node

Artur Daschevici — Sat, 11 May 2024 09:06:38 +0000

Prologue

Python is great for AI software, but I prefer NodeJS for crawling with a headless browser, IMO python has some good libraries for parsing the content but puppeter is slightly more powerful when looking at headless browsers.
That said I wanted to do an experiment and try to pull some data from a page without going through the tedious process of checking every element of interest and discarding the fluff.
The nice thing about puppeteer is also that you don't need to reverse engineer the requests going into the page, figure out cookies and whatnot.

Why?

Web/data scraping is pretty cool, it gives you the power to combine different data sources and devise interesting ways to draw conclusions about it. It is even more interesting these days with ChatGPT, you can simply dump some data and ask it what it can do with it, or if it can extract usable info from it. That is in fact quite nice.

What?

We're going to be looking at a listing site for rentals in Italy, picked something at random for no real reason, mostly to see if it works.

Let's plan this out:

grab the content
identify the listings wrapper and grab the innerHTML of it
pass in the HTML to gpt-4-turbo via the API and construct a dialogue with it to extract the data we are looking for

How?

We're going to install a few dependencies:

# we like speed so while npm is nice, pnpm is faster
npm i -g pnpm
pnpm i openai puppeteer-extra dotenv

Then figure out the main listing element, that at the time of writing was something easy like .in-realEstateResults.

Some quick docs fumbling and you get a working script for loading the page and grabbing the innerHTML of the listing element like so:

  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
  );
  const element = await page.$(".in-realEstateResults");
  const innerHtml = await page.evaluate(
    (element) => element.innerHTML,
      element,
  );

The openai library api is quite straightforward and they offer quite a few examples too. The library provides some responses and also some metrics about the usage you had for your specific query.

  const response = await openai.chat.completions.create({
    model: "gpt-4-turbo",
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: `Can you extract the property prices out of this
            html and send me the results? The text is in italian so
            you should translate that. Can you send me the output as JSON?`,
          },
          {
            type: "text",
            text: propertyInfoHtml,
          },
        ],
      },
    ],
  });
  console.log(response.choices[0]);
  console.log(response.usage);

Another extremely cool thing is that you can simply tell the model to extract data as json and what you will have done is convert html to a json api. Basically you have a magicbox that you can tell what you want to extract from some structure. You simply have a piece of flexible code that can extract data from various html pages with different selectors, so no more hassle figuring that stuff out. And the output is quite well structured and would require minimal after-processing.

[\n' +
      '    {\n' +
      '        "price": "€ 115.000",\n' +
      '        "description": "Bilocale via Felice Casorati 33, Borgo Venezia, Verona",\n' +
      '        "rooms": "2 locali",\n' +
      '        "size": "60 m²",\n' +
      '        "bathroom": "1 bagno",\n' +
      '        "floor": "Piano 1",\n' +
      '        "elevator": "Ascensore",\n' +
      '        "balcony": "Balcone"\n' +
      '    },\n' +
      '    {\n' +
      '        "price": "€ 120.000",\n' +
      '        "description": "Trilocale via Arnaldo Da Brescia, 27, Porto San Pancrazio, Verona",\n' +
      '        "rooms": "3 locali",\n' +
      '        "size": "77 m²",\n' +
      '        "bathroom": "1 bagno",\n' +
      '        "floor": "Piano R",\n' +
      '        "elevator": "No Ascensore",\n' +
      '        "cellar": "Cantina"\n' +
      '    },\n
...

How much though?

This processing though can quickly get out of hand even with the low pricing, so you want to cache results. You also need to pay attention to the context window of the model you are using. The last line with response.usage tells you how many tokens you have used in the query.

 { prompt_tokens: 30337, completion_tokens: 972, total_tokens: 31309 }

This works no problem with gpt-4, but smaller models that you can host locally for example might have an issue with this. There are pages out there that can be quite large also.

Trimming the fat

If you think about it a big part of html is non structure defininig. What that means is that it is either display information(css) or interaction capabilities(events and such). The interesting part is that it does not have any true value for the data which is what we are after.
What if we removed all of that? Crazy idea right?

pnpm i jsdom

So let's add an intermediate step before sending the html out to OpenAI:

    const dom = new JSDOM(innerHtml);
    spinner.text = "Cleaning up HTML";
    const document = dom.window.document;
    const elements = document.querySelectorAll("*");
    elements.forEach((element) => {
      Array.from(element.attributes).forEach((attribute) => {
        element.removeAttribute(attribute.name);
      });
    });
    const cleanedHtml = dom.serialize();

And presto, the metrics of the trimmed down version:

{ prompt_tokens: 7790, completion_tokens: 586, total_tokens: 8376 }

Pretty handy...less than one third.

Conclusions

the gpt-4-turbo apis are definitely worth exploring, 15 minutes exploration is not doing it justice
you can tweak the content and let go of everything that is not important for your query
caching might work in some cases depending on what data you want to pull
slimming down the context actually seems to make quite a difference

Adding search to static websites

Artur Daschevici — Sat, 04 May 2024 22:37:59 +0000

In the beginning

In a previous post I touched a bit on how lately building static websites has piqued my interest. I have been contemplating starting a blog and fell into my usual routine of digging myself into a rabbit hole. It's kinda annoying the amount of yak-shaving I volunteer my brain for 🤣.

One of the things I am finding limiting is that due to the nature of static sites you kinda need to flip the script a little bit, data comes in at build time normally. This in turn makes it a bit weird to think about site search.

Normally search would be based on some kind of database, search index and a server side algorithm and filter API.

Why?

In this particular instance I think the answer is self explanatory, the search feature is not really a luxury anymore. It allows for better discoverability, and even though the outreach is from various external posts via backlinks, without search the blog would look rudimentary.

How?

While you usually wouldn't fetch data for pages in a static website, the best performing search in terms of relevance would require server side implementation. One example of this that is also workable on static websites is Algolia. It's pretty good and it's great that it also supports static websites.

Algolia is paid so you might want to take that into consideration if you are thinking about, especially you are not some kind of social media influencer.

I am not that so my first instinct is looking into free options. Luckily there are free options that are fully implemented on the client side.

Architecturally they implement a dumbed down version of the server side alternative. As mentioned before I started digging into rust and zola and came across this really amazing blog post about implementing a static search using WASM and rust.

Essentially what the dumbed down version is, it's an index implementation that allows for a fully embedded search experience, allowing it to work in a static website. It is not designed for millions of pages and performance tends to degrade as you reach larger numbers.

When getting into performance territory you might want to work on the performance of the index, there are multiple options, you could implement fuse filters or Bloom Filters or XOR Filters like the ones suggested in the blog post. If you want to go for a further performance bump, server side is your best bet.

Implementation quirks

I chose fuse.js for one of my projects. For smaller scale it seems to perform just fine. I haven't reached any performance bottleneck yet.

When It comes to implementing search you want to aim for keyword density, but also facets. This is where the pure static solution may feel a bit naive.

So, we are talking static websites, which means that in order to do a faceted search you need to filter content based on meta embedded in the pages before you pass it in to the fuse.js search "engine".

You can do the filtering in JS, or do an exact match search on a single field using fuse.js.

import Fuse from 'fuse.js';

const options = {
  includeScore: true,
  keys: [
    {
      name: "author"
    }
  ],
  useExtendedSearch: true
}

const books = [
  { title: 'The Great Gatsby', author: 'F. Scott Fitzgerald' },
  { title: 'The Da Vinci Code', author: 'Dan Brown' },
  { title: 'The Catcher in the Rye', author: 'J.D. Salinger' }
]

const fuse = new Fuse(books, options);

async function main(engine) {
  let results = engine.search('="Dan Brown"')
  console.log(results)
}

main(fuse)

Then once you have the facet filtered results you can pass it on to the regular fuzzy search that would give you the static search.

async function main(engine) {
  let results = engine.search({
    $and: [{ $path: ['author'], $val: 'cott'}, {$path: 'title', $val: 'The'} ],
  })
  console.log(results)
}

const fuse = new Fuse(books, options);

main(fuse)

In a server side implementation this would be done in a single operation usually.

Conclusions

you can do some of the search functionality without a backend server - think static
while it may not scale, do you really have that as a hard requirement? Paul Graham's do things that don't scale
this approach is good for prototyping because it allows you to check the traction of the search

Building static websites

Artur Daschevici — Tue, 30 Apr 2024 06:19:18 +0000

Static websites

For anyone who is not yet familiar, static websites are websites that you can deploy, host and serve as prebundled and the website itself is a collection of static pages that are served as HTML with the site navigation map constructed at build time.

There are some elements that you can include that emulate dynamic content(e.g. content that would normally be fetched from a database), but that is still only updated after the site, or parts of it are rebuilt.

This approach has seen a proliferation of platforms that offer this as a service(Netlify, Vercel, Cloudflare etc.) and also a proliferation of frameworks with different strengths and weaknesses(list of frameworks supported cloudflare).

Why?

One might opt for a static website rather than a dynamic one is that content might not be changing very often and that makes the need for interacting with a backend application server on every request redundant.

That said you can update the content on a cron based approach, every minute, every twelve hours or a mix of on demand and cron, and have the content refresh.

Architecture and pitfalls

The idea when working on a static website is that 90% of the website content is created when the site is built. It is not necessarily static, but it changes at a much slower rate than say a real time application would. For example a Google maps style app for tracking your current route would not be a good candidate for a static website, a parcel tracking website could be static no problem, provided the updates are infrequent.

One nice aspect of static websites is that it makes it more secure, since the access to the core is gated, for the most part, and you can focus on the real dynamic component endpoints and tighten security on those, in effect decreasing the attack surface by quite a bit.

In the end it is a numbers game, infrequent could mean that you only update the site a few times each day. The reason why this is a numbers game is that if you set up your flow optimally the hosting costs can be extremely low if not free which can be a great perk if you are unsure about product market fit or general vision.

A standard flow in static websites is that data comes in from some third party data source that can trigger builds on change(think headless CMS with webhooks - e.g.Contentful). Each change in the CMS should trigger a rebuild of the website.

Most CI/CD platforms offer some kind of free tier for builds and most times the free tier is measured in minutes, so faster builds means more builds, fresher content.

There is another big caveat you should be aware of, that is, the number of inbound requests. In the past couple of years there have been some war stories where people have been billed absurd amounts of money because they were subject to DDoS attacks.

So I would advise to use some kind of DDoS protection on your site as the inbound traffic comes at a cost too.

Case studies

Case study 1: Netlify, Gatsby and build times

The first time I started building static websites is when I discovered Gatsby. I built several projects using Gatsby and hosted it on Netlify free tier. It felt like a really robust architecture and I loved that it was free.

I was working with React so using this model and using Gatsby made a lot of sense. I built all my React workshops as static websites, particularly since it has a free tier and Gatsby has many templates to choose from.

At one point though I realized there is a scaling problem with my build minutes. I knew that golang has considerably faster builds and in my case the easy fix is swapping over to Hugo.

Case study 2: Hugo

Moving over to hugo cut my build times to a third of the time. It was pretty amazing and given the number of builds I was running each month it made the free minute allowance suffice.

Hugo also has a nice template library and one of them was quite fitting for my use case. I think the people at AWSWorkshops are using it for doing their workshops.

Case study 3: Zola

Since then I got more and more into Rust, but I haven't found many static website builders with nearly as much traction as any of the others I have mentioned. I think it has to do with the size of the community. At one point I was doing a rust codehunt and I asked about it and was left under the impression that zola is somewhat deprecated. Of course I was unable to find many resources to either confirm or reject this, so if anyone out there has some insights I would be happy to reframe my thinking.

It comes with a list of templates too, some look quite nice.

One thing worth highlighting with zola is that rust build times are not as fast as Hugo. Go is optimized for very fast builds and it also has fewer compile time checks so it only makes sense that it is faster to build.

Case study 4: Astro

I started looking into other options recently and stumbled upon Astro. It's OSS and it has very good integration with Vercel, but can be deployed on any of the platforms for static hosting. (updated courtesy of Iain Simmons)

Astro is quite nice since it is very hybrid friendly allowing you to integrate any framework in your flow and with a base of Astro templating. It's an elegant way of approaching micro forntends with a static twist to it.

Not everything is easy or even possible in Astro, but you can use a mixture of Svelte, React, Angular and Vue in a single website, if you wanted to.

Astro also has a pretty extensive gallery of themes that you could use.

Conclusions

building static websites is fun for me, I get to build cheap apps that have clear rules for optimization
build performance is key
you get a lower attack surface out of the box
if you are going to build a blog, static might be a really good way to go
beware of scaling with number of visitors

DEV Community: Artur Daschevici

Rusty puppets, Websockets and Voyeurism (part II): Driving Chromium in Docker with a Window

TL;DR

The Backstory

Why?

What?

Architecture

How?

Perfomance & size notes

Security notes

WTF moments (and fixes)

My conclusion

How I optimized my blog images using Rust

Why?

How?

What's in it for me?

1. Storage Cost Savings

2. Egress Bandwidth Cost Savings

3. CDN Caching & Requests

Total Cost Savings Estimate

The Playbook

Step 1: Shortcode

Step 2: Rust tool

Step 3: Github action

Step 4: Caching

Conclusion

Rustify some puppeteer code(part I)

Why?

What?

TL;DR

How?

Show me the code

1. Laying down the foundation

2. Starting browser and the browser cleanup

3. Use cases

4. Fixtures to replicate various scenarios

Why to be continued?

Conclusions

In Part II I will cover dealing with bot protection, handling frames, forms and more. Stay tuned!

OpenAI api RAG system with Qdrant

Why?

What?

How?

1. Setting up the environment

2. Install the dependencies

3. Set up the loading of the variables from a config file

4. Set up the UI elements

5. pdf data extraction

6. Setting up the qdrant server via docker

7. Indexing the data

8. sending the query

9. Analysis

Conclusions

Converging project boilerplates with copier

Why?

How?

Cherry pick of features

Cool use-cases

Conclusions:

Gopherizing some puppeteer code

Why?

How?

The pleasant surprises

The ugly parts

Conclusion

Keep your estimates boring

Why?

How?

Into the future with AI

Conclusion

Why I still struggle with estimates

Why?

How?

Where does it go wrong?

My A-HA moment

Conclusions

Screenshots optimization on OpenAI tokens

Prologue

Why?

How?

6. Setting up the `qdrant` server via `docker`