DEV Community: John Fls

Building a VPN status tracker for Russia: what one month of constant blocks taught me

John Fls — Tue, 19 May 2026 18:44:13 +0000

A month ago I launched a small site that tries to answer one question: which VPN actually works on my mobile carrier right now.
People vote, the votes go into a public table, and that's basically the whole product.

The reason I built it is dumb. My wife uses MegaFon, I use MTS. We bought the same VPN. It works for her, it works for me half the
time, and the company's support told me to restart my phone. After a few weeks of this, I realized that the question "is this VPN
any good" has stopped meaning anything in 2026. The right question is "is this VPN good on my specific carrier in my specific city
this week", and nobody answers that.

So I built a place where people answer it for each other. Live votes, no five-star reviews, just three options: works, partially
works, broken. With the carrier name attached.

The site is one month old. The first month was very educational and I want to share what I learned, because none of it is in any
tutorial.

You will move VDS providers at least once

The first hosting I used was a generic budget VPS. The IP was clean as far as my domain was concerned, but the same provider hosted
a long list of sites that were already in the national banned-domains registry. Pirate streaming, fake forex, the usual.

This is not directly your problem. Until it is. Russian network filtering sometimes escalates from per-domain to per-IP blocking
when one of the banned neighbors does something clever (encrypted handshakes, redirect tricks). When that happens, everyone on the
same subnet goes down at once. Your perfectly legal site is collateral damage.

I migrated to a different provider before I got swept up in this, but I almost waited too long. The lesson is: when you pick hosting
for a project that lives in a region with infrastructure-level censorship, check who your neighbors are. There are free tools that
list other domains on a given IP. Spend ten minutes there before you spend a year.

Search engines have weirder rules than you think

I had two domains over the project's lifetime. The first one got hit by a domain-level block (registry entry, not my fault). I set
up the second one, put proper 301 redirects on every path from the old to the new, and assumed Yandex would figure it out.

Yandex did not figure it out. For about three weeks the old domain kept showing up in search results, sending people through the
redirect chain, and the new domain was indexed but underranked. Then one Tuesday morning the algorithm collectively decided the old
domain was dead, dropped it from results entirely, and didn't yet move the rankings to the new one.

My traffic went from 450 visits per day to 48 in twenty-four hours.

The fix is to file an explicit "site move" form in the search console. Even when your redirects are textbook correct. The algorithm
won't process it automatically fast enough to save you from the gap.

I'm three days into the recovery from this. Things are coming back, but slowly. If you're running anything with even one domain
change in its history, go check that your search console knows about it explicitly.

Server Actions and proxies fight each other in silence

This was the technical part that took me by surprise. Modern web frameworks (won't name names, you can guess) ship CSRF protection
that compares the request's Origin header against the X-Forwarded-Host value. If they don't match, every form submission gets
rejected with a generic error.

If you have any kind of reverse proxy chain (and most production sites do, even if you didn't set it up that way), the inner proxy
can quietly overwrite the headers in a way that makes them no longer match. The browser sees a generic 500 error. The user thinks
your site is broken. You spend hours debugging the form before realizing the framework is refusing to even run your code.

The fix involves a couple of lines in your proxy configuration. The bug-hunt is twelve hours.

The general lesson, which I keep relearning: your proxy is lying to you about what it sends downstream. The first thing you build
for any project that goes behind a reverse proxy should be a way to inspect actual headers as the backend sees them. I should have
had this from day one.

What this month felt like, honestly

The biggest gap between expectation and reality was the share of time spent on infrastructure versus product. I expected to spend my
month writing features. I spent it migrating hosts, debugging mirror merges, fixing CSRF mismatches, and reading documentation for
filtering systems that aren't documented in any official place.

The site itself is fine. It works. People use it (around 460 registered users in the bot now, growing). I'm happy with it. But the
unglamorous work of keeping it accessible turned out to be the actual product, and I want anyone considering similar projects to
know that going in.

For anyone curious what the thing looks like, it's at vpnstatus.site. It's in Russian because that's where
the audience is. The technical and operational lessons translate.

What I would tell my one-month-ago self:

Pick hosting by who your neighbors are, not by the price per gigabyte.
Tell your search console about every domain change immediately, in writing, through the actual form. Don't wait for the algorithm to be smart.
Build header inspection on day one if you have any proxy in front of your backend.
Expect that the unglamorous infrastructure work is the work. Plan time for it.

The product was the easy part.

Rufler: One YAML File to Rule Your Autonomous AI Swarm

John Fls — Wed, 15 Apr 2026 16:55:11 +0000

The Problem: Too Many Commands, Not Enough Orchestration

If you've worked with ruflo (the TypeScript AI agent orchestration CLI from the claude-flow family), you know it's powerful. It gives you swarm topologies, hive-mind coordination, shared memory, autopilot loops, and the ability to spawn Claude Code as an autonomous agent.

But actually using it on a project looks something like this:

ruflo init --force
ruflo daemon start
ruflo memory init --backend=hybrid
ruflo swarm init --topology=hierarchical --max-agents=8 --strategy=specialized
ruflo hive-mind init --topology=hierarchical --consensus=raft --max-agents=8
ruflo autopilot config --max-iterations=100 --timeout=180
ruflo autopilot enable
ruflo hive-mind spawn --count=4 --role=specialist --claude \
  --non-interactive --dangerously-skip-permissions=true \
  --objective="$(cat my_assembled_prompt.txt)"

That's 8 commands just to get started. And the objective prompt? You have to assemble it by hand -- project context, task description, every agent's role and prompt, autonomy instructions, memory checkpointing rules.

Do this across multiple projects, add multi-task workflows, background runs, log management, and graceful teardown -- and you're writing a lot of shell scripts.

Enter Rufler

Rufler is a Python wrapper that replaces all of the above with:

rufler run

One command. One YAML file. The entire ruflo pipeline runs automatically.

Here's what a rufler_flow.yml looks like:

project:
  name: my-api
  description: Go microservice with REST API and PostgreSQL.

task:
  main: |
    Build a Go HTTP server on :8080 with CRUD endpoints,
    PostgreSQL storage, and integration tests.
  autonomous: true

agents:
  - name: architect
    type: system-architect
    role: specialist
    seniority: lead
    prompt: Design the API schema and package layout.

  - name: coder
    type: coder
    role: worker
    seniority: senior
    prompt: Implement the server following the architect's design.
    depends_on: [architect]

  - name: reviewer
    type: reviewer
    role: specialist
    seniority: senior
    prompt: Review all code for quality and security.
    depends_on: [coder]

Rufler reads this, composes a structured objective prompt (agents sorted by seniority, with GATE/HANDOFF blocks for depends_on), and runs the full ruflo pipeline.

What Rufler Adds Over Raw Ruflo

Raw ruflo	With rufler
8+ shell commands to boot	`rufler run` (one command)
Assemble objective prompts by hand	Auto-composed from YAML
No multi-task support	Sequential or parallel task groups
No task decomposition	AI-powered decomposition (`task.decompose: true`)
Mixed ANSI/JSON/text logs	Clean NDJSON -- every line is parseable JSON
No way to see live progress	`rufler follow` -- 4-panel TUI dashboard
No run history	Docker-style registry: `rufler ps`, `rufler tasks`, `rufler tokens`
No resume after interruption	Automatic resume from last completed task
Manual MCP server setup	Declare MCP servers in YAML

In short: ruflo is the engine; rufler is the ignition, dashboard, and cruise control.

Features Worth Highlighting

Docker-Style Run Lifecycle

Every rufler run gets an 8-char hex id. You can list runs with rufler ps, inspect tasks with rufler tasks, check tokens with rufler tokens, and stop gracefully with rufler stop. Works from any directory.

rufler run -d          # detach (like docker run -d)
rufler ps              # list running
rufler follow a1b2     # live TUI for run a1b2
rufler stop a1b2       # graceful shutdown

Task Tracking and Resume

Each task gets a sub-id (a1b2c3d4.01) with a tracked lifecycle: queued -> running -> exited / failed / stopped. Token usage is counted per task.

When you stop a run mid-way and restart, rufler automatically resumes from the first incomplete task:

rufler run -d        # tasks 1-2 complete, stopped on task 3
rufler stop
rufler run -d        # resumes from task 3
# => "resuming: skipping 2 completed tasks, starting from task_3"

AI-decomposed task files are reused too -- no wasted claude calls on re-decomposition. Use --new to force a clean start.

Live TUI Dashboard

rufler follow renders a 4-panel Rich dashboard that tails all per-task logs:

+-----------------------------------------------------------------+
| rufler follow  [running]  claude-opus-4  tokens: 94.6K  00:04:23|
+-----------------------------------------------------------------+
| Tasks  2/4            | Session                                 |
| > task_1     done     | model: claude-opus-4-6                  |
| > task_2     running  | tokens: in=98  out=829                  |
|   task_3     queued   | turns: 14                               |
|   task_4     queued   | last tool: Write                        |
+-----------------------+-----------------------------------------+
| Conversation (task_2)                                           |
| 14:23  think  Planning the API routes...                        |
| 14:24  tool   Write(src/api/routes.go)                          |
| 14:24  result File created successfully                         |
+-----------------------------------------------------------------+
| Log                                                             |
| 14:22  OK    task_end task_1                                    |
| 14:23  INFO  session init (claude-opus-4-6)                     |
+-----------------------------------------------------------------+

You see the AI's thinking, text output, tool calls, and results in real time.

MCP Server Management

Declare MCP servers directly in the YAML -- rufler registers them with Claude Code via claude mcp add:

mcp:
  servers:
    - name: my-db
      command: npx
      args: ["-y", "@anthropic/mcp-postgres"]
      env:
        DATABASE_URL: "postgresql://localhost/mydb"

    - name: sentry
      transport: http
      url: "https://mcp.sentry.dev/mcp"

Supports stdio, HTTP, and SSE transports with env vars and headers.

Agent Dependencies (Soft DAG)

Declare depends_on and rufler injects coordination prompts:

agents:
  - name: architect
    ...
  - name: coder
    depends_on: [architect]  # waits for architect's approval

The coder agent won't start until the architect writes an approval to shared memory. Cycles and self-dependencies are rejected at config load time.

Getting Started

# Install
git clone https://github.com/lib4u/rufler.git
cd rufler && pip install -e .

# Verify dependencies
rufler check

# Create a sample flow
rufler init

# Edit and run
$EDITOR rufler_flow.yml
rufler run --dry-run    # inspect the plan
rufler run -d           # launch detached
rufler follow           # watch it work

Prerequisites: Python 3.9+, Node.js 20+, Claude Code CLI (claude), and ruflo (resolved automatically via npx).

When to Use Rufler vs Raw Ruflo

Use rufler when:

You want a single config file checked into your repo
You're running multi-task workflows (sequential or parallel)
You need task-level tracking, resume, and token accounting
You want a live dashboard instead of raw log tailing
You're managing MCP servers across projects

Use ruflo directly when:

You need fine-grained control over individual ruflo commands
You're building custom orchestration scripts
You need features rufler doesn't wrap yet

Rufler doesn't replace ruflo -- it's a layer on top. Every ruflo command still works alongside rufler.

Repository: github.com/lib4u/rufler
Built on: ruflo by @ruvnet + Claude Code CLI by Anthropic
License: MIT

Golang with Colly: Use Random Fake User-Agents When Scraping

John Fls — Fri, 10 Jan 2025 16:13:28 +0000

One of the primary reasons for getting blocked while performing web scraping is the use of improper or default user-agents.

Fortunately, adding random fake user-agents to your Go Colly scrapers is straightforward and simple.

What Are Fake User-Agents?

User-agents are strings used by websites to identify the client making the request, providing information about the application, operating system (e.g., Windows, macOS, Linux), and browser (e.g., Chrome, Firefox, Safari) being used. These strings are sent to servers as part of the HTTP request headers.

For instance, here’s an example of a user-agent when accessing a website using Chrome on an Android device:

'User-Agent': 'Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36'

When scraping a website, it’s crucial to set a user-agent for each request. If you don’t, the website can detect non-human traffic and block your scraper.

By default, Go Colly uses this user-agent for its requests:

"User-Agent": "colly - https://github.com/gocolly/colly",

This default user-agent reveals that Colly is being used, making your scraper easy to detect and block. To avoid this, customizing the user-agent is essential.

That is why we need to manage the user-agents Go Colly sends with our requests.

How To Set A Fake User-Agent In Go Colly

Implementing a fake user-agent with Go Colly is a breeze. You can modify the request headers by setting a custom user-agent in the OnRequest() callback, ensuring each outgoing request uses a different or randomized string.


package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
)

func main() {
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", "1 Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148")
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

From here our scraper will use this user-agent for every request.

However, if you are scraping at scale then using the same user-agent for every request isn't best practice as it makes it easier for the website to detect you as a scraper.

To solve this problem we will need to configure our Go Colly scraper to use a random user-agent with every request.

How To Rotate Through Random User-Agents

For using the generated user agent, it is enough to use special packages, in the following examples we will use github.com/lib4u/fake-useragent

package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
    uaFake "github.com/lib4u/fake-useragent"
)

func main() {

    // Init user-agent faker
    ua, err := uaFake.New()
    if err != nil {
        fmt.Println(err)
    }
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", ua.Filter().GetRandom())
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

Now thanks to the fact that we added just a couple of lines of code, we can get the random user-agent.

However, using a simple random user-agent is not always enough; below we will look at options where we will use specific fake user agents.

The library github.com/lib4u/fake-useragent offers thousands of fake user agents, from the real world database.

// Get random user-agent in string
fmt.Println(ua.GetRandom())  // Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1

// Get user-agent string from a specific browser
fmt.Println(ua.Filter().Chrome().Get())
// Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36

fmt.Println(ua.Filter().Firefox().Get())
//Mozilla/5.0 (Android 14; Mobile; rv:133.0) Gecko/133.0 Firefox/133.0

fmt.Println(ua.Filter().Safari().Get())
//Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1

Now I’ll show you using the Go Colly library as an example.

package main

import (
    "bytes"
    "log"
    "github.com/gocolly/colly"
    uaFake "github.com/lib4u/fake-useragent"
)

func main() {

    // Init user-agent faker
    ua, err := uaFake.New()
    if err != nil {
        fmt.Println(err)
    }
    // Instantiate default collector
    c := colly.NewCollector(colly.AllowURLRevisit())

    // Set Fake User Agent
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("User-Agent", ua.Filter().Chrome().Platform(uaFake.Desktop).Get())
    })

    // Print the Response
    c.OnResponse(func(r *colly.Response) {
        log.Printf("%s\n", bytes.Replace(r.Body, []byte("\n"), nil, -1))
    })

    // Fetch httpbin.org/headers five times
    for i := 0; i < 5; i++ {
        c.Visit("http://httpbin.org/headers")
    }
}

This way we set up the generation of random user-agents for the desktop version of the browser Google Chrome.

For now every time we visit, the website will identify us as a random desktop user, thus a simple substitution of the user agent can help you when parsing websites, but do not forget about the use of proxies and other system headers.

https://github.com/lib4u/fake-useragent
https://github.com/gocolly/colly

Grequest is inspired by the Request library for Python for GO

John Fls — Mon, 06 Jan 2025 20:49:48 +0000

Simple and lightweight golang package for http requests. based on powerful net/http

Grequest is inspired by the Request library for Python and Guzzle in PHP, the goal is to make a simple and convenient library for making http requests in go

The library has a flexible API with methods that return a pointer to the library structure, which allows you to declaratively describe a request using a chain of methods.

Installation

go get github.com/lib4u/grequest

Usage

Simple get request anf get string response

req := app.Get("https://jsonplaceholder.typicode.com/todos/1").Do()
    response := req.Body().GetStrings()
    fmt.Println(response)

Get a response and writing it to a json structure

type AutoGenerated struct {
    UserID    int    `json:"userId"`
    ID        int    `json:"id"`
    Title     string `json:"title"`
    Completed bool   `json:"completed"`
}
...
   var myJsonResponse AutoGenerated
    req := app.Get("https://jsonplaceholder.typicode.com/todos/1").Do()
    err := req.Body().GetWithJsonStruct(&myJsonResponse)
    if err != nil {
        fmt.Println("do ..")
    }
    fmt.Println(myJsonResponse.Title)

Simple post request with json payload body and get status of response

data := LoginRequest{
        Username: "example",
        Password: "12345",
    }
    req := app.Post("https://example.site/login").Body().SetJson(data).Do()
    fmt.Println(req.Status().GetCode())

Simple save file from response

// file will saved as ../files/image.png
app.Get("https://example.com/image.png").Do().Body().SaveFile()
//OR
app.Get("https://example.com/image.png").Do().Body().Path("/user/files").SaveFile()
//OR 
app.Get("https://example.com/image.png").Do().Body().ToFile("path/savedimage.png")

Sending form with files and text fields and set header

req := app.Post("https://example.site/form/")
    req.Header().Set("Client", "number_1")
    form := req.FormData().WithMultipart()
    form.AddField("first_name", "John")
    form.AddField("last_name", "Doe")
    form.AddFile("photo", "my_photo.png")
    form.AddFile("documents", "example.txt")
    form.AddFile("documents", "example2.txt")
    form.Push()
    req.Do()
    .....

Request with authentication

//With basic auth
req := app.Post("https://example.site/secret)
    req.Header().Set("Client", "number_1")
    req.Auth().SetBasic("user", "password")
    req.Do()
    ...
    //Sets bearer token
    req := app.Post("https://example.site/secret)
    req.Header().Set("Client", "number_1")
    req.Auth().SetBearer("myToken")
    req.Do()

Work with cookie

//Save cookie to file 
//By default this saved in  cookies/example.site/cookies.json
req := app.Post("https://example.site/cookies")
    req.Cookie().Save()
    ...
    // Load saved cookies form cookies/example.site/cookies.json
    reqWithCookie := app.Post("https://example.site/cookies")
    reqWithCookie.Cookie().Load()
    reqWithCookie.Do()
    ...