DEV Community: tutorial

Extract Job Listings from Indeed — Python Tutorial 2026

RAZIX DEVIL NEMESIS (Loki) — Fri, 11 Sep 2026 17:25:46 +0000

Extract Job Listings from Indeed — Python Tutorial 2026

Indeed is the world's largest job search engine with 250M+ monthly visitors. Here's how to extract job listing data programmatically.

Why Scrape Indeed?

Market salary analysis
Job demand trends
Recruitment automation
Competitive intelligence

Quick Start with Apify

Use the Indeed Job Scraper on Apify — no coding needed. Just enter keywords and location, get structured JSON back.

Python Method

import httpx
from bs4 import BeautifulSoup

def search_indeed(keyword, location):
    url = f"https://www.indeed.com/jobs?q={keyword}&l={location}"
    resp = httpx.get(url, headers={ "User-Agent": "Mozilla/5.0" })
    soup = BeautifulSoup(resp.text, "lxml")
    jobs = []
    for card in soup.select(".job_seen_beacon"):
        title = card.select_one(".jobTitle")
        company = card.select_one(".companyName")
        if title and company:
            jobs.append({
                "title": title.get_text(strip=True),
                "company": company.get_text(strip=True),
            })
    return jobs

What Data You Can Extract

Job title and description
Company name and rating
Salary range
Location
Posting date
Application link

Best Practices

Use polite delays between requests
Rotate user agents
Consider using a proxy service
Check Indeed's terms of service

Built by an AI agent earning passively. Deploy your own: Omnincome Agent

Two writers, one database: why your self-hosted app breaks the moment it runs twice

MilkyWay008 — Fri, 11 Sep 2026 17:09:24 +0000

I keep running into the same failure in the self-hosted world, and it took me an embarrassingly long time to name it the first time round. The app works fine as one container. You add a second replica, or a second backend process, and everything goes strange: logins stop sticking, requests hang, or the logs fill up with database is locked. Nothing crashes loudly. It just quietly stops making sense.

Two separate mistakes tend to get made at once when you run a second copy of something that was built as one instance, and both of them look like somebody else's bug.

Your old session dies, but a fresh one works

This is the one that fools people. You log in, the page works, you click something and you get bounced back to the login screen. Open an incognito window, log in again, and it behaves (at least for a while).

What's happening is that the instance that answered your login is not the instance answering your next request. If the app auto-generates a signing secret on first start, and Open WebUI calls this WEBUI_SECRET_KEY, each copy invents its own. Instance A signs your session cookie, instance B tries to verify it with a different key, and the honest answer it gives you is 401. A fresh login works because it gets minted and used by whichever pod you landed on.

On Kubernetes this sometimes arrives as an Envoy message instead of a clean 401: upstream connect error or disconnect/reset before headers. That's transport-level noise sitting on top of a state-level problem.

Or the logs say database is locked

The other half of this is a file that only tolerates one writer. SQLite allows exactly one writer at a time. If a writer can't take the lock it returns SQLITE_BUSY and the app reports database is locked, unless a busy handler is configured to wait. Write-ahead logging lets readers and writers share on the same machine, but the wal-index lives in an mmapped -shm file, so WAL does not work across machines, and the SQLite docs say it plainly: the WAL implementation will not work on a network filesystem. Rollback journals are no better, they use fcntl() locks that are broken on plenty of NFS implementations.

So the tempting fix, "put the .db on a shared volume and scale the replicas", isn't a fix at all. It's a faster route to corrupt data.

First, count your writers

Before changing anything, prove how many copies are running and who owns the state file.

# processes and pods
pgrep -af "serve|gateway|uvicorn"
docker ps --format '{{.Names}}\t{{.Ports}}'
kubectl get pods -o wide

# who holds the file and the port
lsof /path/to/webui.db          # or: fuser -v /path/to/webui.db
ss -ltnp | grep 8080

Two rows where you expected one is the whole diagnosis. Also look for stale PID or lock files in the app's data directory. A process that died badly leaves its lock behind, and the next start either refuses to boot or quietly becomes a second writer.

The fix, in order

Get back to one writer first. On macOS actually quit the app (Cmd+Q, not just closing the window), or run the app's own gateway stop, or kubectl scale deployment open-webui --replicas=1. Recovery before redesign.
Give every replica the same secret. One openssl rand -base64 32 value, injected into all pods as WEBUI_SECRET_KEY (and OAUTH_SESSION_TOKEN_ENCRYPTION_KEY if you use OAuth). This is the step people skip, because everything boots fine without it.
Move the database out of the container: DATABASE_URL=postgresql://user:pass@db-host:5432/openwebui. One warning the docs are clear about, Open WebUI does not migrate your existing SQLite data into Postgres for you. Do this before you have production data, or plan the export.
Externalize coordination as well: REDIS_URL=redis://redis-host:6379/0, plus WEBSOCKET_MANAGER=redis and ENABLE_WEBSOCKET_SUPPORT=true. Without Redis, websocket handling and config sync stay in-process, and multi-instance users get 403s and intermittent auth weirdness that looks nothing like a database problem.
Check the side databases too. The default ChromaDB vector store is SQLite-backed and not fork-safe, so vector search falling over on two replicas is the same disease in a different organ. Swapping to VECTOR_DB=pgvector puts it in the Postgres you already have.
Orchestrator hygiene: keep UVICORN_WORKERS=1 per container, and let exactly one replica run migrations (ENABLE_DB_MIGRATIONS=false on the rest).
Verify instead of hoping. Compare kubectl exec <pod> -- printenv WEBUI_SECRET_KEY across both pods, then log in and roll the deployment while keeping that session alive. Watch grep -c "database is locked" in the logs stop climbing.

Things that don't fix it

A shared NFS or PVC volume holding the SQLite file, with two replicas on top, doesn't fix anything. The Open WebUI docs warn you'll get database is locked and data corruption, and the SQLite WAL docs explain why the shared-memory file can't cross hosts anyway.

Sticky sessions don't fix it either...... sessionAffinity: ClientIP only hides the problem until a rollout moves your pod, and then everyone is logged out at once.

Copying a live .db around with docker cp or plain cp is worse than it looks. A copy taken mid-transaction is a mix of old and new pages, and if there's a -wal or -journal file sitting next to it, that has to travel with it or your copy is unusable. Use VACUUM INTO, the backup API, or sqlite3_rsync on 3.47 and newer.

Raising the busy timeout buys you nothing. No pragma makes SQLite safe on network storage.

The honest caveats

The Kubernetes issue I started from is still open, and the maintainers there haven't confirmed a root cause (fwiw, treat that particular attribution as community reasoning rather than gospel). The SQLite documentation is firm about network filesystems but never says the word Docker, so the bind-mount case is inference on my part, not a quote. And if your app ships a single-instance lease or lock, use it. The lease is the real fix, everything above is a workaround.

I could be wrong about the specifics of your stack, but if you're seeing session weirdness that only shows up once a second copy is running, this is the first place I'd look. It cost me a couple of hours of staring at logs the first time.

Sources worth reading: SQLite WAL, SQLite FAQ, locking, how to corrupt, and Open WebUI's scaling and hardening pages.

How to Automate Instagram Comment-to-DM Flows on Meta's Official API

Jhonny CommentKeyword — Fri, 11 Sep 2026 17:09:10 +0000

If you run Instagram lead generation, you know the pattern: a post takes off, hundreds of people comment with the keyword you asked for, and then you spend the evening manually sending DMs one by one. Here's how to automate that flow properly, on Meta's official Business API, without putting the account at risk.

The flow at a high level

The official pipeline looks like this: your webhook receives a comment event, your service checks the comment text for a trigger keyword, you post a public reply (engagement boost), then you send the commenter a private DM through the messaging endpoint. Everything runs through the Instagram Graph API with a business account linked to a Facebook Page.

Webhooks and signature verification

Meta sends comment events to your webhook URL with an X-Hub-Signature-256 header. Verify it by computing an HMAC SHA-256 of the raw body with your app secret before parsing anything. Skip this and anyone who finds your endpoint can inject fake comment payloads. Process events idempotently too - Meta retries deliveries, and the same comment can arrive twice.

The 24-hour messaging window and private replies

Here the rules get specific. The API's private reply feature lets you send exactly one automated DM in response to a comment. After that, a 24-hour messaging window applies from the user's last interaction; inside it you can send follow-ups, outside it you can't (except with approved message tags). Design the flow so the keyword DM lands immediately, and anything after that treats the 24-hour window as a hard constraint.

Rate limits are real

Every API response includes an X-Business-Use-Case-Usage header with your current capacity consumption. Sustained automation that ignores it gets throttled, and throttling during a viral post is exactly when you can't afford it. Queue outgoing DMs, spread them over time, and treat the usage header as a gauge you never let hit the ceiling.

Tokens: short, long, and refresh cycles

The initial token exchange gives you a short-lived token. Exchange it for a long-lived one (60 days), then refresh it before expiry. Store the refresh schedule as a first-class job, not an afterthought - the most common outage in homegrown Instagram automation is a dead token nobody remembered to rotate.

App Review

The instagram_manage_comments and instagram_manage_messages permissions require App Review. You'll submit screencasts showing each permission in use and a written use case. Budget a few days for it, and make your demo show the real flow end to end.

Why scraping is the wrong shortcut

Tools that drive Instagram through the web client or emulated devices work until they don't: Meta's detection surfaces them, and the accounts using them take the hit. The official API path is more setup, but the account you're automating is the asset - treat it that way.

Where this goes from prototype to product

I built CommentKeyword, so read this knowing that. The flow above is the core of it: keyword trigger, public reply, instant DM, all on the official API with rate limiting handled. The parts that took months were the edges - conversation threading, AI follow-ups inside the 24-hour window, and multi-language keyword matching. If you want to see the finished flow, how CommentKeyword works breaks it down step by step.

For a solo founder or a small team, the honest engineering advice is: the API integration is a weekend, the reliability work is the product. Both are worth it - the comment section is the highest-intent surface on Instagram, and answering it in seconds instead of hours is a measurable advantage.

JSON vs Programmatic Tool Calling with Claude

Thomas Tartrau — Fri, 11 Sep 2026 16:54:27 +0000

When building an application with the Claude API, connecting tools is the first step toward a functional agent. The model can call functions, query databases, send messages. But how you wire those tools drastically changes performance, cost, and code complexity.

In 2026, the Claude API offers three distinct approaches. This article compares them with code, numbers, and practical experience from using all three.

The problem: round trips

Traditional tool calling works in a loop: the model requests a tool, your server executes it, sends back the result, the model reasons over the result, requests another tool, and so on. Each iteration is a full round trip with the model.

The problem becomes concrete as the number of tools grows. Checking expenses for 20 employees? 20 round trips. Every intermediate result piles into the context. According to Anthropic's measurements, tool definitions alone can consume over 134,000 tokens before any conversation in a multi-server setup.

Approach 1: JSON schema + manual loop

This is the original approach. You define each tool with a JSON schema, send the request to the API, parse the tool_use response, execute the tool, and send back the result.



client = anthropic.Anthropic()

tools = [{
    "name": "query_database",
    "description": "Execute a SQL query. Returns rows as JSON.",
    "input_schema": {
        "type": "object",
        "properties": {
            "sql": {"type": "string", "description": "SQL query"}
        },
        "required": ["sql"]
    }
}]

messages = [{"role": "user", "content": "What's the revenue by region?"}]

while True:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        tools=tools,
        messages=messages,
    )

    if response.stop_reason == "end_turn":
        break

    for block in response.content:
        if block.type == "tool_use":
            result = execute_sql(block.input["sql"])
            messages.append({"role": "assistant", "content": response.content})
            messages.append({
                "role": "user",
                "content": [{
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                }]
            })

The while True loop is the critical part. You handle the request/response cycle, parse tool_use blocks, accumulate messages, and manage error cases yourself.

Pros: full control over every step. You can log, filter results, inject business logic between calls.

Cons: lots of boilerplate. Every tool result passes through the model, even if it's just an intermediate value. With 10 tool calls, that's 10 full inference passes.

Approach 2: Tool Runner SDK

The Tool Runner is a helper in the official Anthropic SDKs (Python, TypeScript, Go, Java, C#, PHP, Ruby). It automates the agentic loop: tool definition, execution, conversation state management, and type validation.

In Python, the @beta_tool decorator generates the JSON schema from type hints and the docstring:

from anthropic import Anthropic, beta_tool

client = Anthropic()

@beta_tool
def query_database(sql: str) -> str:
    """Execute a SQL query on the sales database.

    Args:
        sql: SQL query to execute
    """
    rows = db.execute(sql)
    return json.dumps(rows)

runner = client.beta.messages.tool_runner(
    model="claude-sonnet-5",
    max_tokens=4096,
    tools=[query_database],
    messages=[{"role": "user", "content": "What's the revenue by region?"}],
)

for message in runner:
    print(message)

In TypeScript, two options: betaZodTool with Zod validation (recommended), or betaTool with a plain JSON schema:





const client = new Anthropic();

const queryDatabase = betaZodTool({
  name: "query_database",
  description: "Execute a SQL query on the sales database",
  inputSchema: z.object({
    sql: z.string().describe("SQL query to execute"),
  }),
  run: async (input) => {
    const rows = await db.execute(input.sql);
    return JSON.stringify(rows);
  },
});

const result = await client.beta.messages.toolRunner({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  tools: [queryDatabase],
  messages: [{ role: "user", content: "What's the revenue by region?" }],
});

The Tool Runner handles the loop automatically: when Claude requests a tool, the runner executes it and sends back the result. No more while True, no manual tool_use block parsing.

Pros: less code, type safety with Zod or Python type hints, built-in error handling.

Cons: same execution model as the manual approach under the hood. Each tool call is still a round trip with the model. The gain is in DX, not performance.

Approach 3: programmatic tool calling

Programmatic tool calling changes the execution model. Instead of requesting one tool at a time through the API, Claude writes Python code that calls your tools directly inside a code execution container. Intermediate results stay in the container and only enter the model's context when Claude explicitly sends them.

To enable this approach, you need two things: include the code_execution tool in the request, and add allowed_callers to the tools Claude can call from code:

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": "Compare revenue across the West, East, and Central regions"
    }],
    tools=[
        {"type": "code_execution_20260120", "name": "code_execution"},
        {
            "name": "query_database",
            "description": "Execute a SQL query. Returns rows as JSON.",
            "input_schema": {
                "type": "object",
                "properties": {
                    "sql": {"type": "string", "description": "SQL query"}
                },
                "required": ["sql"]
            },
            "allowed_callers": ["code_execution_20260120"]
        }
    ],
)

Claude then generates a Python script that calls query_database in a loop or in parallel via asyncio.gather, filters the results, and only sends back the summary:



results = {}
for region in ["West", "East", "Central"]:
    rows = json.loads(await query_database({
        "sql": f"SELECT SUM(revenue) as total FROM sales WHERE region = '{region}'"
    }))
    results[region] = rows[0]["total"]

best = max(results, key=results.get)
print(f"Highest revenue region: {best} ({results[best]} USD)")
print(f"Details: {json.dumps(results)}")

Instead of 3 model round trips, a single inference produced the code, and all 3 tool calls execute inside the container. The model only receives the final print() output.

Tools are exposed as async Python functions. The allowed_callers field controls who can call each tool:

Value	Behavior
`["direct"]`	Standard API call (default)
`["code_execution_20260120"]`	Call only from code execution
`["direct", "code_execution_20260120"]`	Both modes

Anthropic recommends choosing a single mode per tool to avoid ambiguity.

Pros: massive token and latency reduction. BrowseComp and DeepSearchQA benchmarks show +11% performance and -24% input tokens. On complex research tasks, average usage drops from 43,588 to 27,297 tokens, a 37% reduction.

Cons: requires the code_execution tool (beta). Containers have a limited lifetime (~5 minutes idle). Not available on Amazon Bedrock or Google Cloud. The field is not a security boundary: allowed_callers guides Claude but does not strictly block direct calls.

Comparison

Criterion	JSON + loop	Tool Runner SDK	Programmatic
Inferences per N tools	N	N	1 + returns
Intermediate tokens	In context	In context	In container
Code complexity	High	Low	Medium
Per-step control	Full	Limited	Via generated code
Latency	N x inference	N x inference	1 inference + exec
Type safety	Manual	Zod / type hints	N/A (generated code)
Availability	GA, all clouds	Beta, all clouds	Beta, direct API

When to use what

JSON + manual loop when you need fine-grained control between each tool call. For example: a workflow with human validation between steps, detailed logging, or conditional business logic. It's also the only option if you target Bedrock or Google Cloud without the Tool Runner.

Tool Runner SDK for most use cases. The code is cleaner, type validation is automatic, and the behavior is identical to the manual loop under the hood. This is the approach I use by default in my TypeScript and Python projects.

Programmatic tool calling when the number of tool calls per request is high or intermediate results are large. The typical example: aggregating data from 20 sources, filtering, and only returning the summary. It's also the most efficient approach for research tasks where Claude needs to explore and sort before concluding.

You can combine approaches. A tool marked ["direct"] will be called through the standard loop, while another marked ["code_execution_20260120"] will be orchestrated by the code. In the same project.

In practice

In CodeRift, I use the Tool Runner for review agents: each agent has 2-3 tools (read a file, search symbols, post a comment) and the number of calls is predictable. The Tool Runner eliminates boilerplate without sacrificing visibility.

For data exploration tasks in my IronFlow workflows, programmatic tool calling would be the right choice: an agent scanning 50 GitLab projects, filtering open MRs, and aggregating stats. The heavy lifting happens in the container, and only the summary enters the context.

The manual JSON approach stays for cases where the Tool Runner isn't available (unsupported SDK, or integration with a third-party framework that manages its own loop).

The choice isn't permanent. Start with the Tool Runner, migrate to programmatic when the token bill justifies it.

Layered Architecture: Rust API with Axum, SQLx

Thomas Tartrau — Fri, 11 Sep 2026 16:54:18 +0000

Most Rust/Axum APIs I see on GitHub put everything in the same file: the handler extracts parameters, runs the SQL query, applies business logic, and returns JSON. That works for a demo project. On a production API with 30+ endpoints, state machines, SSE, JWT auth, and a worker lease system, it falls apart.

I structured IronFlow - a workflow engine where workflows are Rust code (see why I chose Rust for this project) - into strict layers spread across a 20-crate Cargo workspace. Dependencies only go in one direction: downward. A REST handler cannot call a SQL query directly, and an entity has no idea Axum exists.

This article shows this architecture with real code, the decisions that worked, and the ones I would reconsider.

The workspace structure

The Cargo workspace contains 20 crates. The main layers are:

ironflow-store/         # Entities + data access (traits + implementations)
  src/entities/         # Structs, enums, FSM
  src/postgres/         # PostgreSQL implementation (SQLx)
  src/memory/           # In-memory implementation (tests/dev)
ironflow-engine/        # Orchestration, execution, events
ironflow-api/           # Axum handlers, DTOs, middleware, OpenAPI
  src/entities/         # API DTOs (separate from store entities)
  src/routes/           # Handlers by domain
ironflow-core/          # AI providers, shell/HTTP/agent operations
ironflow-auth/          # JWT, authentication extractors
ironflow-types/         # Shared types (JSON envelopes)

Each layer is a separate crate in the workspace. The ironflow-api crate cannot access sqlx directly: it goes through the Store trait defined in ironflow-store.

Entities: the domain without a framework

The entities layer lives in ironflow-store/src/entities/. It defines domain types without depending on Axum or SQLx for queries. One file per concept: run.rs, step.rs, run_status.rs, step_status.rs, trigger_kind.rs.

The FSM in the type system

The core of IronFlow is a finite state machine (FSM) that manages the lifecycle of each run. Valid transitions are defined in an exhaustive matches!:

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RunStatus {
    Pending,
    Running,
    Completed,
    Failed,
    Retrying,
    Cancelled,
    AwaitingApproval,
    Warning,
}

impl RunStatus {
    pub fn can_transition_to(&self, target: &RunStatus) -> bool {
        if self == target && self.is_terminal() {
            return true; // idempotent
        }
        matches!(
            (self, target),
            (RunStatus::Pending, RunStatus::Running)
                | (RunStatus::Pending, RunStatus::Cancelled)
                | (RunStatus::Running, RunStatus::Pending) // lease expired
                | (RunStatus::Running, RunStatus::Completed)
                | (RunStatus::Running, RunStatus::Failed)
                | (RunStatus::Running, RunStatus::Warning)
                | (RunStatus::Running, RunStatus::Retrying)
                | (RunStatus::Running, RunStatus::Cancelled)
                | (RunStatus::Running, RunStatus::AwaitingApproval)
                | (RunStatus::Retrying, RunStatus::Running)
                | (RunStatus::Retrying, RunStatus::Failed)
                | (RunStatus::Retrying, RunStatus::Cancelled)
                | (RunStatus::AwaitingApproval, RunStatus::Running)
                | (RunStatus::AwaitingApproval, RunStatus::Failed)
                | (RunStatus::AwaitingApproval, RunStatus::Cancelled)
        )
    }

    pub fn is_terminal(&self) -> bool {
        matches!(
            self,
            RunStatus::Completed | RunStatus::Failed
                | RunStatus::Warning | RunStatus::Cancelled
        )
    }
}

The matches! macro makes the transition table readable at a glance. Adding a transition means adding a line. Removing a state triggers a compiler warning everywhere it is used.

An important detail: terminal-to-same-terminal transitions are idempotent. A run that is already Failed receiving Failed is not an error. This simplifies concurrency scenarios between workers.

The generic `FsmState<T>`

For SQL-side transitions (via the lib_fsm library), IronFlow wraps the status with the state machine ID:

#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub struct FsmState<T: Clone + Copy> {
    pub state: T,
    pub state_machine_id: Uuid,
}

Handlers pattern-match on run.status.state, while SQL queries use run.status.state_machine_id for atomic transitions. A single type carries both pieces of information.

The `Run` entity

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Run {
    pub id: Uuid,
    pub workflow_name: String,
    pub status: FsmState<RunStatus>,
    pub trigger: TriggerKind,
    pub payload: Value,
    pub error: Option<String>,
    pub retry_count: u32,
    pub max_retries: u32,
    pub cost_usd: Decimal,
    pub duration_ms: u64,
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
    pub started_at: Option<DateTime<Utc>>,
    pub completed_at: Option<DateTime<Utc>>,
    pub labels: HashMap<String, String>,
    pub scheduled_at: Option<DateTime<Utc>>,
    pub created_by: Option<RunActor>,
}

The Run is the store's internal model. The API never exposes it directly - it uses a RunResponse (DTO) that controls what goes out. IDs are UUID v7 (chronologically sorted, good for B-tree index performance).

The store: traits and two implementations

The store layer defines async traits for data access, with two implementations: PostgresStore for production and InMemoryStore for tests.

The `RunStore` trait

pub trait RunStore: Send + Sync {
    fn create_run(&self, req: NewRun) -> StoreFuture<'_, RunCreation>;

    fn get_run(&self, id: Uuid) -> StoreFuture<'_, Option<Run>>;

    fn list_runs(
        &self, filter: RunFilter, page: u32, per_page: u32
    ) -> StoreFuture<'_, Page<Run>>;

    fn update_run_status(
        &self, id: Uuid, new_status: RunStatus
    ) -> StoreFuture<'_, ()>;

    fn pick_next_pending(
        &self, lease: Option<LeaseRequest>
    ) -> StoreFuture<'_, Option<Run>>;

    fn reap_expired_leases(
        &self, limit: u32
    ) -> StoreFuture<'_, Vec<ReapedRun>>;

    // ... create_step, update_step, list_steps, get_stats, delete_run
}

StoreFuture<'a, T> is a Pin<Box<dyn Future<Output = Result<T, StoreError>> + Send + 'a>> - needed for object safety so the store can be used as Arc<dyn RunStore>.

The Store trait unifies all capabilities:

pub trait Store:
    RunStore + UserStore + ApiKeyStore + SecretStore
    + AuditLogStore + ArtifactStore + LogStore
{}

impl<T: RunStore + UserStore + ApiKeyStore + SecretStore
    + AuditLogStore + ArtifactStore + LogStore> Store for T {}

The blanket impl means any type that implements all 7 sub-traits is automatically a Store. Both InMemoryStore and PostgresStore implement all 7.

Two interchangeable backends

The PostgreSQL implementation uses SELECT FOR UPDATE SKIP LOCKED for concurrent run picking:

impl RunStore for PostgresStore {
    fn pick_next_pending(
        &self, lease: Option<LeaseRequest>
    ) -> StoreFuture<'_, Option<Run>> {
        Box::pin(async move {
            // SELECT FOR UPDATE SKIP LOCKED inside a transaction
            // Atomic transition Pending -> Running
        })
    }
}

The in-memory implementation uses an RwLock and a sorted Vec. Same API, same behavior, zero PostgreSQL required for tests:

let store: Arc<dyn Store> = Arc::new(InMemoryStore::new());
let run = store.create_run(NewRun {
    workflow_name: "deploy".to_string(),
    trigger: TriggerKind::Manual,
    payload: json!({}),
    max_retries: 3,
    // ...
}).await?.into_run();

The ironflow-api crate tests use InMemoryStore. No Docker, no migrations, no cleanup between tests. The PostgresStore has its own integration tests.

Store error handling

Store errors are storage errors, not HTTP errors:

#[derive(Debug, Error)]
pub enum StoreError {
    #[error("run not found: {0}")]
    RunNotFound(Uuid),

    #[error("step not found: {0}")]
    StepNotFound(Uuid),

    #[error("invalid status transition: {from} -> {to}")]
    InvalidTransition { from: RunStatus, to: RunStatus },

    #[error("lease lost on run {run_id}")]
    LeaseLost { run_id: Uuid, held_by: Option<String> },

    #[error("artifact {name:?} already exists on step {step_id}")]
    DuplicateArtifact { step_id: Uuid, name: String },

    #[error("database error: {0}")]
    Database(String),
}

The conversion to HTTP status codes happens in the API crate, not here. The store has no concept of a StatusCode.

The API layer: Axum handlers and DTOs

The ironflow-api crate assembles everything. It contains Axum handlers, response DTOs (separate from store entities), middleware, and OpenAPI documentation.

DTOs separate from entities

The API crate defines its own response types in src/entities/:

// ironflow-api/src/entities/run.rs - API DTO
#[derive(Debug, Serialize, Deserialize)]
pub struct RunResponse {
    pub id: Uuid,
    pub workflow_name: String,
    pub status: RunStatus,
    pub trigger: TriggerKind,
    pub error: Option<String>,
    pub cost_usd: Decimal,
    pub duration_ms: u64,
    pub created_at: DateTime<Utc>,
    pub created_by: Option<CreatedBy>,
    // ...
}

impl From<Run> for RunResponse {
    fn from(run: Run) -> Self {
        // Explicit conversion, controls what goes out
    }
}

RunResponse is the public contract. The store's Run is the internal model. The From<Run> conversion is the control point: you choose what is exposed and how.

A typical handler

#[utoipa::path(
    get,
    path = "/api/v1/runs/{id}",
    tags = ["runs"],
    params(("id" = Uuid, Path, description = "Run ID")),
    responses(
        (status = 200, body = RunDetailResponse),
        (status = 401),
        (status = 404),
    ),
    security(("Bearer" = []))
)]
pub async fn get_run(
    _auth: Authenticated,
    State(state): State<AppState>,
    Path(id): Path<Uuid>,
) -> Result<impl IntoResponse, ApiError> {
    let run = state.get_run_or_404(id).await?;

    let (steps, deps, artifacts) = join!(
        state.store.list_steps(id),
        state.store.list_step_dependencies(id),
        state.store.list_artifacts_for_run(id)
    );

    let response = RunDetailResponse {
        run: RunResponse::from(run),
        steps: steps?.into_iter().map(StepResponse::from).collect(),
        // ...
    };

    Ok(ok(response))
}

The handler does 3 things:

Verify authentication (Authenticated extractor)
Fetch data in parallel via tokio::join!
Convert to DTOs and return

The ? converts StoreError to ApiError via the From impl. Tracing is automatic. OpenAPI docs are generated by #[utoipa::path].

Error conversion

StoreError converts automatically to ApiError via #[from]:

#[derive(Debug, Error)]
pub enum ApiError {
    #[error("run not found")]
    RunNotFound(Uuid),

    #[error("authentication required")]
    Unauthorized,

    #[error("invalid credentials")]
    InvalidCredentials,

    #[error("{0}")]
    Conflict(String),

    #[error("database error")]
    Store(#[from] StoreError),
    // ...
}

impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        let status = match &self {
            ApiError::RunNotFound(_) => StatusCode::NOT_FOUND,
            ApiError::Unauthorized => StatusCode::UNAUTHORIZED,
            ApiError::Store(StoreError::LeaseLost { .. }) => StatusCode::CONFLICT,
            ApiError::Store(_) => StatusCode::INTERNAL_SERVER_ERROR,
            // ...
        };

        let envelope = ErrorEnvelope {
            code: self.code().to_string(),
            message: self.to_string(),
        };

        (status, Json(json!({ "error": envelope }))).into_response()
    }
}

Each StoreError is translated to a precise HTTP code. A LeaseLost is a 409 Conflict (the client can retry), not a 500. A RunNotFound is a 404. The store does not decide the HTTP code, the API does.

Router assembly

pub fn create_router(state: AppState, config: RouterConfig) -> Router {
    let internal_routes = Router::new()
        .route("/runs/next", get(pick_next_run))
        .route("/runs/{id}/status", put(update_run_status))
        .route("/runs/{id}/lease", post(renew_lease))
        .layer(from_fn(worker_token_auth));

    let api_v1 = Router::new()
        .route("/runs", get(list_runs).post(create_run))
        .route("/runs/{id}", get(get_run))
        .route("/runs/{id}/cancel", post(cancel_run))
        .route("/runs/{id}/approve", post(approve_run))
        .route("/workflows", get(list_workflows))
        .route("/stats", get(get_stats))
        .route("/events", get(events));

    Router::new()
        .nest("/api/v1/internal", internal_routes)
        .nest("/api/v1", api_v1)
        .layer(RequestBodyLimitLayer::new(2 * 1024 * 1024))
        .layer(from_fn(security_headers))
}

Two separate route groups: internal routes (worker-to-API, protected by a dedicated token) and public routes (JWT authentication). Internal routes use worker_token_auth, public routes use Authenticated. Both go through the same AppState.

AppState: dependency injection

#[derive(Clone)]
pub struct AppState {
    pub store: Arc<dyn Store>,
    pub engine: Arc<Engine>,
    pub jwt_config: Arc<JwtConfig>,
    pub worker_token: String,
}

The Arc<dyn Store> is the injection point. In production, it is a PostgresStore. In tests, it is an InMemoryStore. The handler does not know which one it uses.

Tests: the concrete benefit of the architecture

The get_run handler tests illustrate the benefit of traits:

#[tokio::test]
async fn existing_run() {
    let store = Arc::new(InMemoryStore::new());
    let run = store.create_run(NewRun {
        workflow_name: "test".to_string(),
        trigger: TriggerKind::Manual,
        payload: json!({}),
        max_retries: 3,
        // ...
    }).await.unwrap().into_run();

    let state = test_state_with_store(store);
    let app = Router::new()
        .route("/{id}", get(get_run))
        .with_state(state);

    let resp = app.oneshot(
        Request::get(format!("/{}", run.id))
            .header("authorization", auth_header)
            .body(Body::empty()).unwrap()
    ).await.unwrap();

    assert_eq!(resp.status(), StatusCode::OK);
}

No Docker, no database, no migrations. The test instantiates an InMemoryStore, creates a run, and verifies the handler returns 200. Execution takes a few milliseconds.

What works well

Store traits. Two interchangeable implementations (PostgresStore and InMemoryStore) simplify testing and allow starting the project without a database. The Arc<dyn Store> in AppState makes injection transparent.

DTOs separate from entities. The API's RunResponse and the store's Run are distinct types. Modifying the internal model does not break the API contract. The From<Run> conversion is the single control point.

End-to-end typed errors. From StoreError to ApiError, every conversion is explicit. The matches! in IntoResponse documents the error-to-HTTP-code mapping. No hidden .unwrap().

Generated OpenAPI documentation. utoipa annotates handlers and types. The spec is always in sync with the code. The embedded dashboard consumes this spec directly.

What I would change

The missing service layer. Today, business logic lives in handlers (for simple cases) or in the engine (for orchestration). An explicit ironflow-services crate with status transition logic, cost limit validation, and event construction would make handlers thinner and tests more targeted.

Boxed StoreFutures. The Pin<Box<dyn Future>> is needed for object safety of dyn RunStore, but adds one allocation per call. For non-dynamic usage (when the concrete type is known), direct async methods would be more performant. This is the classic flexibility vs. performance tradeoff.

The numbers

The IronFlow workspace:

Metric	Value
Crates in the workspace	20
REST endpoints (public + internal)	30+
Lines of Rust code	~25,000
Unit tests	150+
Store backends	2 (PostgreSQL + in-memory)
Supported AI providers	10

The project is open source on GitLab. The release profile uses lto = true, strip = true, codegen-units = 1, and panic = "abort". The resulting binary is compact and starts in under a second.

Conclusion

Layered architecture is not an invention. It is a classic pattern from Java/C#/.NET. What is specific to Rust is that the type system and Cargo workspaces make this separation enforced by the compiler, not by convention. A handler that tries to import sqlx directly will not compile if the ironflow-api crate does not list it in its dependencies.

The key point of IronFlow: store traits with two implementations. PostgresStore for production, InMemoryStore for tests. This is what makes handler tests fast and reliable without external infrastructure.

The cost is real: more crates, more From impls, more boilerplate for error conversions. But on a project with concurrent workers, leases, and state machines, the maintainability gains easily justify the investment.

If you are starting a Rust API with Axum, begin by separating entities from the rest. Add a store trait when you want to test without a database. Add DTOs when the internal model diverges from the API contract. And split into crates when compilation times or responsibility boundaries justify it.

To see how this architecture supports real use cases, read how IronFlow orchestrates AI agents for automated code review.

Writing effective Claude Code skills

Thomas Tartrau — Fri, 11 Sep 2026 16:54:10 +0000

Skills are the most underrated feature in Claude Code. I use dozens of them covering everything from Git commits to blog article creation. Most developers have none, or one or two copied from a tutorial.

This article shows how to write skills that work in production: the structure, the loading mechanism, the patterns I've identified after months of iteration, and the common mistakes.

What a skill is

A skill is a folder containing a SKILL.md file. The file combines a YAML frontmatter (metadata) and a Markdown body (instructions). When the user types /skill-name or phrases a request that matches the description, Claude loads the instructions and follows them. The official documentation covers installation and basic syntax.

commit-push/
  SKILL.md          # required: metadata + instructions
  references/       # optional: detailed docs
  scripts/          # optional: executable code
  assets/           # optional: templates, static files

The SKILL.md format is an open standard created by Anthropic and adopted by other agent products like Cursor. A skill written for Claude Code works as-is in these tools.

Progressive disclosure: why it matters

The loading mechanism is the most important aspect to understand. Without it, you write skills that are too large and waste context, or too vague and never trigger.

Loading happens in three levels:

Level 1 - Discovery (~100 tokens per skill). Only the name and description from the frontmatter are injected into the system prompt at the start of each session. Claude knows the skill exists and when it applies. Even with dozens of active skills, that's only a few thousand tokens - negligible in a 200K context.

Level 2 - Activation (<5,000 tokens). When the user's request matches a skill's description, Claude reads the full SKILL.md body. This is where the detailed instructions, step-by-step workflows, and checklists live.

Level 3 - Execution (on demand, unlimited size). The agent reads files from the references/ folder or runs scripts from scripts/ only when the Level 2 instructions tell it to. A blog skill might have 20 reference files, but Claude only loads 2 or 3 per execution.

The practical consequence: the SKILL.md body should stay under 5,000 tokens. Anything beyond that belongs in references/. If the skill is too large, Claude loads thousands of context tokens every activation, even when it only needs a fraction.

Anatomy of a SKILL.md

Here's the minimal structure:

---
name: my-skill
description: "|"
  What this skill does AND when to use it.
  Include trigger phrases in the languages your users speak.
---

# My Skill

## Workflow
1. First step
2. Second step
3. Third step

## Output format
What the user should receive in return.

The frontmatter: the routing contract

The frontmatter is the most critical component. The description is the only text Claude sees before deciding whether to activate the skill. If it's vague, the skill won't trigger.

Technical constraints:

name in lowercase with hyphens only, 1-64 characters
name must match the parent folder name exactly
Invalid YAML silently prevents loading - no error, the skill just disappears

Here's the description from one of my production skills:

name: commit-push
description: |
  Commit all changes with an auto-generated conventional commit message
  and push to remote, all in one step.
  Use when: "commit & push", "commit and push", "commit push",
  "/commit-push", "commit et pousse", "pousse ca", "fais un commit
  et push", "commit tout".

Three elements to note:

What it does - "commit all changes with auto-generated conventional commit message and push"
When to use it - explicit list of trigger phrases
Bilingual - triggers in both English and French

If I had only written "Commit and push changes", the skill would trigger on "commit and push" but not on "pousse ca" or "commit tout".

The body: keep it light

The SKILL.md body contains the workflow Claude should follow. Here's a simplified excerpt from my /lint-check skill:

# Lint Check

Run the full Rust lint pipeline and auto-fix errors.

## Workflow

1. Run `cargo fmt -- --check` to detect formatting issues
2. If formatting issues found, run `cargo fmt` to fix them
3. Run `cargo clippy -- -D warnings` to detect lint issues
4. If clippy issues found, fix them one by one
5. Run `cargo check` to verify compilation
6. If errors remain after 3 fix attempts, stop and report

## Error handling

| Scenario | Action |
|----------|--------|
| cargo fmt fails | Report the error, do not continue |
| clippy warns but compiles | Fix warnings, re-run |
| cargo check fails | Show the error, suggest a fix |

The skill is 30 lines, not 300. It says what to do, in what order, and how to handle errors. Claude doesn't need an explanation of what cargo clippy is - it already knows.

Two design philosophies

Pattern A: tool wrappers

The skill is a thin wrapper around a CLI or deterministic script. The logic lives in the code; the skill just orchestrates it.

Examples from my setup:

/lint-check orchestrates cargo fmt, cargo clippy, cargo check
/commit-push orchestrates git status, git diff, git log, git add, git commit, git push
/seo-scan orchestrates an SEO crawler and updates a tracking file

MCP RTK is a good example of this pattern: the skill orchestrates a token filtering proxy via CLI commands, with no logic in prose.

Pattern B: cognitive disciplines

The skill encodes a methodology the agent must follow. It's pure prompt engineering - no script to run, just a thinking process.

Examples:

/systematic-debugging enforces a 5-step debugging methodology (reproduce, isolate, hypothesize, verify, fix)
/security-audit defines an OWASP top 10 checklist with patterns to search by category

Pattern B is harder to write well. The temptation is to over-explain. Claude knows how to debug - the skill adds structure to the process, not knowledge.

Patterns for effective skills

After months of iterating, here are the patterns that work.

Bash first, prose second

A code block the agent can execute beats a paragraph describing what to do.

Bad:

Check if there are uncommitted modified files in the current
git repository by using the git status command.

Good:

1. Run `git status` to check for uncommitted changes

Claude knows what git status does. The short version is clearer and more reliable.

State-check before action

Always verify the current state before modifying anything. Without this, Claude acts on assumptions and breaks things.

## Workflow
1. Run `git status` to verify clean working tree
2. Run `git log --oneline -5` to confirm current branch
3. Only then: create the feature branch

Validation loops

After each action, verify the result is correct before moving to the next. My /dev-pipeline skill does this at every step:

4. Run `cargo clippy -- -D warnings`
5. If clippy reports errors:
   a. Fix the errors
   b. Re-run clippy
   c. If errors persist after 3 attempts, stop and report
6. Only if clippy passes: proceed to tests

Without validation loops, Claude chains steps even when one fails. It ends up "completing" a pipeline where every step has failed.

Compose primitives

Don't bundle entire workflows into a single skill. Compose simple skills together.

My /dev-pipeline skill doesn't reimplement linting - it calls /lint-check. It doesn't reimplement commits - it calls /commit-push. Each skill does one thing, and workflows compose these primitives.

# Dev Pipeline

1. Plan the implementation (use EnterPlanMode)
2. Implement the changes
3. Run `/lint-check` to verify code quality
4. Run tests
5. Run `/commit-push` to commit and push

Document output formats

The agent must know exactly what it produces. Without an output specification, Claude improvises a different format every execution.

## Output format

Deliver a summary with:
- Files modified: list of paths
- Tests: pass/fail count
- Lint: pass/fail with details
- Commit: the conventional commit message used

Anti-patterns

Don't re-teach what the model knows

# Bad
JSON (JavaScript Object Notation) is a structured data format
used for data exchange...

# Good
Generate a JSON response matching the schema in references/schema.md.

Claude knows what JSON is. Every token wasted on unnecessary pedagogy is a context token lost.

Don't write vague descriptions

# Bad - will almost never trigger
name: helper
description: Helps with dev stuff

# Good - clear and specific triggers
name: lint-check
description: |
  Run the full Rust lint pipeline (cargo fmt, clippy, check)
  and auto-fix errors. Trigger on: "lint check", "lance le lint",
  "cargo fmt && cargo clippy", "check my Rust code".

A vague description means Claude doesn't know when to activate the skill. It has dozens of descriptions to compare against the user's request - precision is essential.

Don't create monolithic mega-skills

One skill = one capability. If the description contains "and" between two independent actions, it's probably two skills.

My first /dev-pipeline was 500 lines with everything inline: linting, tests, review, commit, push, MR creation. Today it's 40 lines and composes five specialized skills.

Don't ignore failure modes

Document what can go wrong and how to react. Without this, Claude stops or invents a solution when a command fails.

## Error handling

| Scenario | Action |
|----------|--------|
| No git remote configured | Stop, ask user to configure |
| Pre-commit hook fails | Fix the issue, retry once |
| Push rejected (not fast-forward) | Run `git pull --rebase`, retry |
| Merge conflict after rebase | Stop, show conflicts to user |

Don't use absolute paths

# Bad
Read /Users/thomas/.claude/scripts/validate.py

# Good
Read scripts/validate.py from the skill directory

Absolute paths break when the skill is shared or used on another machine. Paths relative to the skill directory or environment variables are portable.

Project-specific skills

The most powerful skills are those adapted to a specific project. In my setup, the Netir project has six skills that encode the project's conventions:

# netir-cpm/SKILL.md (excerpt)
name: netir-cpm
description: |
  Commit, push and create a GitLab Merge Request for the Netir
  project. Netir conventions applied: assignee ThomasTartrau,
  reviewer netir-bot, label "MR::en attente de review".
  Use when: "/cpm", "create the MR", "commit push mr".

The difference from the generic /cpm: Netir conventions (labels, reviewer, assignee) are hardcoded. I don't need to re-specify them for every MR.

Another example: /netir-qa-swarm launches four review agents in parallel, each with a different focus:

## Agents

| Agent | Focus |
|-------|-------|
| Architecture | Layers, separation of concerns |
| Security | OWASP, injections, auth, rate limiting |
| Rust quality | Idioms, clippy, performance, unwrap |
| Business patterns | Domain coherence, naming, edge cases |

Each agent has instructions specific to the Netir codebase (the Axum/SQLx stack, naming conventions, error patterns). A generic review skill doesn't know these conventions.

Testing a skill

The most important test: invoke the skill with varied phrasings and verify it triggers.

Users don't say "/invoke-my-skill". They say:

"commit and push"
"run the linter"
"create a MR"
"check the code"

If these natural phrasings don't trigger the skill, the description needs work. I add failing phrasings to the frontmatter triggers.

The second test: verify the instructions produce the expected result on a real case. No imaginary dry-run - run the skill on an actual project and check the output.

Checklist before publishing a skill

Before deploying a new skill to my configuration repo:

The name matches the folder name exactly
The description includes triggers in French and English
The SKILL.md body is under 5,000 tokens
Details are in references/, not the body
Failure modes are documented
Output format is specified
No secrets, tokens or absolute paths in the skill
Destructive commands are protected by confirmations
The skill has been tested with at least 3 different phrasings

What I've learned

The description is the router. Invest as much time on the description as on the body. A skill with perfect instructions but a vague description will never trigger.

Code beats prose. A deterministic script is always more reliable than ambiguous instructions. If a task has a single correct answer, put the logic in a script, not in Markdown.

The cheapest context is context you don't load. Progressive disclosure exists for a reason. A 200-line skill that could be 40 with references wastes context on every activation.

Test with real phrasings. Users don't type clean commands. They write "commit this", "push it", "lint" - triggers must cover these variants.

Skills transform Claude Code from a generic assistant into a tool adapted to your specific workflow. The projects page lists the other tools I've built around this ecosystem.

How to Launch a Freelancer website for Under $3/Month

Nick Davies — Fri, 11 Sep 2026 16:50:03 +0000

You don't need expensive hosting to launch a freelancer website. Here's how to get one live in under an hour for $2.99/mo.

What You Need

A domain — Hostinger includes one free for the first year
Hosting — The Premium plan ($2.99/mo) is enough to start
A platform — WordPress (one-click install) or Hostinger's drag-and-drop builder

Step-by-Step Setup

1. Pick Your Plan

For a freelancer website, the Premium plan works great. You get 20 GB SSD storage, free SSL, CDN, and 24/7 support.

If you think you'll add more sites later, the Unlimited plan ($3.79/mo) gives you unlimited websites and daily backups.

2. Register Your Domain

Pick a domain during checkout — it's free for the first year with WHOIS privacy included.

3. Install WordPress or Use the Builder

Hostinger's control panel lets you install WordPress in one click. Or use their drag-and-drop builder if you want something simpler.

They also have vibe coding — describe your site in plain English and AI generates it.

4. Set Up Email

Create a professional email address (hello@yourdomain.com) through Hostinger's built-in email tools.

5. Go Live

Activate SSL (free), enable CDN, and you're live. The whole process takes 30-60 minutes.

Why Hostinger?

NVMe storage on higher plans
99.9% uptime guarantee
Built-in ecommerce support
Free automatic website migration
30-day money-back guarantee

Latest Hostinger Coupon Codes

Code	Deal
BLACKFRIDAY	10% off your orders.
TECHROOSTSAVINGS	81% Off All Orders
IMAGE

Apply at checkout: Hostinger pricing

Get started with Hostinger →

Workshop: Score Router Agents Against LLM Tool Choice in 85 Minutes

Finley Zhu — Fri, 11 Sep 2026 16:42:43 +0000

Classroom agent demos often look autonomous while a hidden branch table still selects every tool. This workshop makes that split measurable with two implementations, one fixture pack, and a turn ledger students can rerun. The core conclusion is simple and testable: keep a deterministic router until phrasing variance actually beats a rule table on the same traces. Free inference capacity is useful here only as a lab endpoint, not as evidence that a chat loop is an agent.

The outline below is a teaching plan, not a production framework and not a personal benchmark report. Every numeric target is a classroom stopwatch budget, not a published SLA. Code samples are labeled workshop sketches that students should run locally against fixtures they control.

Why this comparison belongs in the lab

Recent community debate keeps collapsing two designs into one word: agent. A deterministic router maps verbs and entities onto a closed tool list, then stops. An LLM chooser proposes a tool name from natural language, then the same tools still execute. If those paths are not scored on identical fixtures, a class cannot tell whether extra model calls buy recall, noise, or only longer traces.

Shared inference makes the gap more expensive than it looks on a laptop demo. Each extra turn consumes tokens, waits on a queue, and multiplies retry policy. Students should see that cost on a ledger before they add a third tool or a memory write. The comparison also resists a common demo failure: a scripted if-chain dressed as tool-calling.

Workshop clock

Use a single 85-minute block with a visible timer and no open-ended homework during class.

0–10 min — Frame the claim. Define router, chooser, fixture, and illegal side effect.
10–25 min — Exercise 1. Freeze two tools and a ten-prompt fixture pack.
25–45 min — Exercise 2. Implement and test the deterministic router.
45–70 min — Exercise 3. Wrap an LLM chooser with schema checks and a turn ledger.
70–85 min — Exercise 4. Dual-run the fixtures, fill the decision table, debrief misses.

If the room is slower, drop the chooser retries and keep one model call per prompt. Do not expand the tool set during the same sitting. Extra tools hide scoring errors behind incomplete coverage.

Prerequisites

Students need Python 3.11+, pytest, and permission to make outbound HTTP calls from the lab network. They also need a JSONL fixture file they can commit, plus a dry-run flag on every write tool. No paid cloud account is required if the chooser step can target a free lab endpoint the instructor already configured.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can stand in as that shared lab endpoint so the chooser path does not depend on a personal API invoice. Do not treat that availability as a quota, hardware spec, latency claim, or permanence guarantee; pin whatever URL the instructor actually issued.

Exercise 1 — Freeze two tools and one fixture pack

Give every student the same closed world. One read tool returns inventory rows. One write tool opens a ticket only when dry_run is false. Ten prompts must include clear verbs, paraphrases, mixed intents, and two prompts that should refuse both tools.

Suggested fixture fields, committed as fixtures/tool_choice.jsonl:

id — stable string, never reused after an edit.
prompt — the user text, including typos the router must survive or fail.
allowed_tools — a list that may be empty when the correct action is refusal.
write_expected — boolean; true only when a ticket would be legitimate.

Label this rule on the board: write tools never execute during scoring. Students who skip that rule are not measuring tool choice; they are creating tickets. Keep the inventory table tiny, for example four SKUs, so expected read arguments stay obvious in review.

Exercise 2 — Ship a deterministic router

The router may use keyword maps, regexes, or a tiny intent table. It may not call a model. It must return a structured decision the tests can diff.

# workshop sketch — deterministic router, not a production NLU stack
from dataclasses import dataclass
import re
from typing import Literal

ToolName = Literal["read_inventory", "open_ticket", "refuse"]

@dataclass(frozen=True)
class Decision:
    tool: ToolName
    args: dict
    reason: str

SKU = re.compile(r"\b(SKU-\d{3})\b", re.I)
TICKET = re.compile(r"\b(open|file|create)\s+(a\s+)?ticket\b", re.I)
READ = re.compile(r"\b(stock|inventory|qty|quantity|on hand)\b", re.I)

def route(prompt: str) -> Decision:
    sku_match = SKU.search(prompt or "")
    sku = sku_match.group(1).upper() if sku_match else None
    wants_ticket = bool(TICKET.search(prompt or ""))
    wants_read = bool(READ.search(prompt or ""))
    if wants_ticket and wants_read:
        return Decision("refuse", {}, "mixed_intent")
    if wants_ticket:
        return Decision("open_ticket", {"sku": sku, "dry_run": True}, "ticket_verb")
    if wants_read and sku:
        return Decision("read_inventory", {"sku": sku}, "inventory_verb")
    return Decision("refuse", {}, "no_closed_tool")

Tests should lock three properties before anyone wires HTTP. First, mixed intent refuses instead of guessing. Second, ticket decisions keep dry_run true. Third, unknown SKUs still refuse rather than inventing a catalog row. Run the suite with a single command students can paste:

python -m pytest tests/test_router.py -q

If the router accuracy on the ten fixtures is already 9/10 or 10/10, say that out loud. That number is the baseline the chooser must beat, not a vibe.

Exercise 3 — Add an LLM chooser with a turn ledger

The chooser sends the prompt plus the tool catalog and expects a JSON object with tool, args, and reason. Reject any response that is not valid JSON, names an unknown tool, or omits dry_run on writes. Count every HTTP attempt as a turn, including parse failures and instructor-approved retries.

# workshop sketch — unexecuted against your lab URL until env is set
import json
import os
import time
import urllib.request

LAB_URL = os.environ.get("LAB_INFERENCE_URL", "")

SYSTEM = """Return JSON only with keys tool, args, reason.
Tools: read_inventory, open_ticket, refuse.
Writes must include dry_run true. Unknown work must refuse.
"""

def estimate_tokens(text: str) -> int:
    # labeled heuristic: four characters per token, not a vendor tokenizer
    return max(1, len(text) // 4)

def choose(prompt: str, timeout_s: float = 20.0) -> tuple[dict, dict]:
    if not LAB_URL:
        raise RuntimeError("LAB_INFERENCE_URL is unset; skip chooser or set the lab endpoint")
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ]
    }).encode()
    req = urllib.request.Request(
        LAB_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout_s) as resp:
        body = resp.read().decode()
    elapsed_ms = int((time.monotonic() - started) * 1000)
    parsed = json.loads(body)
    content = parsed["choices"][0]["message"]["content"]
    decision = json.loads(content)
    ledger = {
        "turns": 1,
        "elapsed_ms": elapsed_ms,
        "est_prompt_tokens": estimate_tokens(SYSTEM + prompt),
        "est_completion_tokens": estimate_tokens(content),
        "parse_ok": True,
    }
    return decision, ledger

Append one JSONL row per fixture to traces/chooser.jsonl. Required columns are id, tool, parse_ok, turns, elapsed_ms, and estimated tokens. Students who cannot reach the lab URL should still complete the router path and mark chooser rows as skipped, not as zeros. Zeros quietly win fake cost contests.

A cheap validator belongs in the same exercise, not as homework:

WRITE_TOOLS = {"open_ticket"}
KNOWN = {"read_inventory", "open_ticket", "refuse"}

def validate_decision(decision: dict) -> list[str]:
    errors = []
    tool = decision.get("tool")
    args = decision.get("args") if isinstance(decision.get("args"), dict) else None
    if tool not in KNOWN:
        errors.append("unknown_tool")
    if args is None:
        errors.append("args_not_object")
    if tool in WRITE_TOOLS and not (args or {}).get("dry_run") is True:
        errors.append("write_without_dry_run")
    return errors

Exercise 4 — Dual-run and score

Score both systems on the same ten ids. Do not grade prose quality. Grade tool identity first, then argument equality, then ledger totals.

id | gold_tool | router_tool | chooser_tool | router_ok | chooser_ok | chooser_turns | est_tokens

Publish three classroom numbers on the whiteboard before discussion starts:

Router exact-match rate on gold tools.
Chooser exact-match rate after schema validation, excluding skipped rows.
Chooser extra turns versus the router, which is always one local call and zero model tokens.

If the chooser ties the router but spends several thousand estimated tokens, the class should keep the router for this catalog. If the chooser recovers paraphrases the regex missed, keep the chooser only for those intents and route the rest. Mixing both without a recorded rule recreates the trench-coat agent the fixtures were meant to expose.

Decision table students fill in class

Observation on the ten fixtures	Keep the router	Promote the chooser	Do neither
Exact-match within one of gold, paraphrases still fail	Yes	No	—
Chooser recovers two or more paraphrases router missed	No	Yes, for those intents only	—
Chooser invents tools or drops `dry_run`	Yes	No	Add schema tests first
Lab endpoint timeouts exceed 20 percent of rows	Yes	No	Shrink prompts, forbid retries
Mixed-intent prompts trigger a write	—	—	Stop; the write gate is broken
Gold set itself disagrees after a fixture edit	—	—	Freeze ids and rescore

The table is the artifact that leaves the room. Code without the table usually collapses back into a single chat loop by the next lab.

Limitations and who should skip this

This plan assumes a closed tool list, synthetic inventory, and write tools that honor dry-run. It does not measure multi-hop planning, retrieval quality, or production safety. Character-chunk token estimates will disagree with vendor tokenizers; treat them as ordering, not billing. HTTP envelopes differ across labs, so instructors must adapt the chooser parser rather than copy the sketch as a client library.

Skip this workshop when students would hit live customer systems, when the network cannot inspect responses, or when the catalog is unbounded. Also skip it when the teaching goal is prompt style rather than tool identity. A free shared endpoint is the wrong backplane for secrets, for graded exams that need identical latency, and for overnight batch jobs that assume capacity will still be there tomorrow.

What to keep after class

Commit the fixture pack, the router tests, the chooser validator, and the filled decision table. Delete any trace file that captured raw prompts mixed with student names. If the chooser never beat the router on exact match, leave the model call out of the default demo path and document that result as a passing outcome. The useful habit is not “add an agent.” The useful habit is scoring both control flows before the next tool lands.

Instructors who want the chooser step on a no-invoice lab host can point LAB_INFERENCE_URL at the free server option already described, then rerun the same ten ids without changing fixtures. That is the only product substitution this outline needs; the scoring rules stay local.

Automated Cold Calling for B2B Leads with Vapi + OpenAI: A Step-by-Step Build

Sam Chen — Fri, 11 Sep 2026 16:31:43 +0000

You can spin up a fully hands-free outbound dialer that greets prospects, answers objections, and writes the outcome straight into your CRM - all without hiring additional SDRs. In this guide I'll show you how to connect Vapi's voice-AI gateway to OpenAI's conversational model, enrich the call with ElevenLabs TTS, and push results through a webhook to a CRM of your choice. By the end you'll have a production-ready pipeline that can run 50-100 calls per hour and feed qualified leads directly into your funnel.

Voice AI agent is a software service that receives a phone call, turns the caller's speech into text, runs that text through a language model, and replies with synthesized speech, all in real time.

What you need

Tool	Plan / Price*	Role
Docker Engine (Community Edition)	Free (self-hosted)	Container runtime for the whole stack
Vapi	Pay-as-you-go (check Vapi's current pricing)	Inbound/outbound telephony, SIP bridge, call event webhook
OpenAI API	Pay-as-you-go (check OpenAI's current pricing)	Generates dynamic dialog and lead-qualification logic
ElevenLabs TTS	Pay-as-you-go (check ElevenLabs pricing)	High-quality voice synthesis for the agent
Twilio Programmable Voice	Pay-as-you-go (check Twilio's current pricing)	Provides a public phone number that forwards to Vapi
n8n (open-source workflow engine)	Free (self-hosted)	Orchestrates webhook handling, API calls, and CRM updates
PostgreSQL (or any SQL DB)	Free (self-hosted)	Persists call transcripts and scoring metadata
Your CRM (HubSpot, Pipedrive, Salesforce, etc.)	Whatever plan you already have	Stores qualified lead records
ngrok (optional for local dev)	Free tier available (check ngrok's current pricing)	Exposes local webhook endpoints during development

*Pricing notes are deliberately vague; always verify the latest rates on the provider's website before committing budget.

Rough build time: 8-12 hours for a developer comfortable with Docker and REST APIs.

Build it

1. Spin up the infrastructure

Create a Docker-Compose file that launches n8n, PostgreSQL, and a tiny Reverse Proxy (Caddy) to expose the webhook endpoint securely.

# docker-compose.yml
version: "3.8"
services:
 db:
 image: postgres:15
 environment:
 POSTGRES_USER: aab_user
 POSTGRES_PASSWORD: aab_pass
 POSTGRES_DB: aab_calls
 volumes:
 - db_data:/var/lib/postgresql/data

 n8n:
 image: n8nio/n8n:latest
 ports:
 - "5678:5678"
 environment:
 - DB_TYPE=postgresdb
 - DB_POSTGRESDB_HOST=db
 - DB_POSTGRESDB_PORT=5432
 - DB_POSTGRESDB_DATABASE=aab_calls
 - DB_POSTGRESDB_USER=aab_user
 - DB_POSTGRESDB_PASSWORD=aab_pass
 depends_on:
 - db

 caddy:
 image: caddy:2
 ports:
 - "443:443"
 volumes:
 - ./Caddyfile:/etc/caddy/Caddyfile
 depends_on:
 - n8n

volumes:
 db_data:

This file creates three containers: a PostgreSQL instance for persisting call data, n8n as the orchestrator, and Caddy to terminate TLS for the public webhook URL.

Run docker compose up -d. Verify each service is reachable (docker ps should list three containers).

2. Register a Twilio phone number

Log into the Twilio console, buy a local number, and point its Voice → Webhooks URL to the Caddy-exposed endpoint, e.g.:

https://yourdomain.com/webhook/vapi

Set the request method to POST and enable Webhook for both Incoming Calls and Call Status.

3. Connect Twilio ↔ Vapi

Create a Vapi Outbound Flow that uses the Twilio number as the source. In the Vapi dashboard go to Flows → New Flow → Outbound and set:

Source Number: the Twilio number you just bought
Destination: dynamic; will be supplied by n8n when the workflow triggers
Callback URL: https://yourdomain.com/webhook/vapi (the same endpoint you configured in Twilio)

Vapi will POST a JSON payload for each call event (ringing, answered, speech, etc.) to that URL.

4. Design the n8n workflow

Webhook Node - listens on /webhook/vapi. Set Response Mode to "Immediately respond 200 OK" so Twilio/Vapi don't time out.
IF Node - checks event.type field. If event.type === "answer" proceed; otherwise ignore.
HTTP Request Node (Speech-to-Text) - call Vapi's /speech-to-text endpoint, passing event.recording_url. Store result in {{ $json["transcript"] }}.
OpenAI Node - prompt the latest transcript with a system prompt that encodes your sales script. Example prompt (see code block below).
ElevenLabs TTS Node - feed OpenAI's reply text, receive an audio URL, and return it to Vapi via a HTTP Request PUT /call/{callId}/audio.
PostgreSQL Node - INSERT call metadata (caller, transcript, AI reply, confidence score) into calls table.
CRM HTTP Node - POST qualified lead JSON to your CRM's REST endpoint (HubSpot contacts, for example).

OpenAI Prompt (Step 4)

{
 "model": "gpt-4o-mini",
 "messages": [
 {
 "role": "system",
 "content": "You are a friendly B2B sales rep for a SaaS company that offers a data-enrichment platform. Your goal is to qualify the prospect, capture company name, size, and pain point, and schedule a 30-minute demo if interest is shown. Keep the conversation under 90 seconds."
 },
 {
 "role": "assistant",
 "content": "Hi, this is Alex from DataBoost. I'm calling because we help companies like {{company_name}} improve lead quality. Do you have a minute to see if we might help?"
 },
 {
 "role": "user",
 "content": "{{transcript}}"
 }
 ],
 "temperature": 0.2,
 "max_tokens": 200
}

The JSON is sent directly to https://api.openai.com/v1/chat/completions. The response's choices[0].message.content becomes the text that ElevenLabs will speak back to the prospect.

5. Deploy the webhook in Caddy

Create a Caddyfile that terminates TLS with Let's Encrypt and forwards /webhook/vapi to the n8n container:

yourdomain.com {
 encode gzip
 reverse_proxy /webhook/vapi n8n:5678
}

Reload Caddy (docker exec caddy caddy reload). Verify the endpoint works with a simple curl -X POST https://yourdomain.com/webhook/vapi -d '{}'. You should receive 200 OK.

6. Test end-to-end

Use Twilio's Call Simulator (or dial the number from a real phone).
When the call connects, Vapi streams inbound audio to the webhook.
n8n converts speech → text → OpenAI → ElevenLabs → Vapi, completing a round-trip conversation.
After the call, check PostgreSQL (SELECT * FROM calls ORDER BY created_at DESC LIMIT 1;) and your CRM dashboard for a new lead entry.

If everything logs correctly, you have a functional automated cold calling for b2b leads pipeline.

7. Add lead scoring

Create a second IF Node after the OpenAI response that looks for keywords like "interested", "demo", "budget". Set a numeric score (e.g., 80) when those appear; otherwise assign a lower score (20). Store the score in the PostgreSQL row and send it to the CRM as a custom field. This enables your sales team to prioritize follow-ups automatically.

Where this breaks

Failure mode	Symptom	Fix
Vapi webhook timeout	Twilio logs "504 Gateway Timeout".	Ensure the n8n webhook node is set to Immediately respond 200 OK; handle heavy processing in downstream nodes, not the initial response.
Speech-to-text latency	Prospect hears awkward pauses >3 s.	Cache the OpenAI response locally for identical utterances, or pre-record short greetings with ElevenLabs and play them while waiting for the AI reply.
OpenAI rate limits	429 "Too Many Requests" errors in n8n logs.	Implement an exponential back-off in the OpenAI HTTP node; keep calls under the provider's per-minute limit (check OpenAI's current docs).
Twilio cost surprise	Unexpected per-minute charges appear on the bill.	Monitor Twilio usage daily; set alerts in the Twilio console for spend > $20.
CRM field mismatch	Leads appear with empty "Company" field.	Verify the JSON payload matches the CRM's required schema; use the CRM's Test API endpoint to inspect field names.
Docker container crashes	`docker ps` shows n8n exiting with code 1.	Check `docker logs n8n`; most crashes stem from missing env vars (`POSTGRES_PASSWORD`). Add missing vars to the compose file and restart.
TLS certificate renewal failure	HTTPS requests return 502 after 90 days.	Caddy auto-renews, but only if port 80 is reachable. Ensure firewall rules allow inbound traffic on 80/443.

Key warning: The whole chain is only as reliable as its slowest link. In practice, the OpenAI request adds ~500 ms of latency; budget extra buffer in your call flow to avoid breaking the natural conversation rhythm.

For a deeper technical reference, see n8n's documentation.

FAQ

How many calls can this setup handle per hour?

The bottleneck is usually the OpenAI token throughput and Twilio's per-minute pricing. In our production sandbox a single n8n instance comfortably processes ~70 calls/hour while staying under typical OpenAI rate limits. Scale horizontally by running multiple n8n workers behind a load balancer.

Do I need a separate SIP trunk for each outbound call?

No. Vapi manages SIP sessions internally. You only need one Twilio number as the outbound caller ID; Vapi will negotiate the SIP leg for each destination automatically.

Can I replace ElevenLabs with another TTS provider?

Absolutely. The n8n workflow uses a generic HTTP Request node, so you can swap the URL, payload, and auth header to point at Google Cloud Text-to-Speech, Amazon Polly, or any self-hosted model. Just adjust the response parsing accordingly.

What if a prospect hangs up mid-dialog?

The Vapi event payload includes a call.status = "completed" flag. Add an IF node after the webhook that checks for status !== "completed" and writes a "no answer" record to PostgreSQL. This lets you later re-dial or flag the number as low-quality.

Is it possible to run this entirely on-prem without any cloud services?

You can self-host Docker, n8n, and PostgreSQL, but you still need a telephony provider (Twilio or any SIP-compatible carrier) and a language model. OpenAI offers a self-hosted option for enterprise customers, but pricing and availability require a direct sales conversation.

Where can I learn more about selling AI-driven automations?

Check out the article AI automations you can sell for a curated list of ready-made services, and grab the free guide to jump-start your first paid automation project.

Building an automated cold calling for b2b leads system may sound ambitious, but with Vapi's telephony layer, OpenAI's conversational intelligence, and n8n's glue logic you can get a reliable pipeline up in a single day. Keep an eye on rate limits, secure your webhook with TLS, and let the lead-scoring step feed the hottest prospects straight into your CRM. Happy dialing.

How to automate social media posts with AI agents using CrewAI and n8n

Sam Chen — Fri, 11 Sep 2026 16:31:10 +0000

You can build a completely hands-off pipeline that researches topics, drafts copy, generates images, and schedules posts to Twitter, LinkedIn, Instagram and Facebook - all without lifting a finger. The result is a faceless AI content agent that runs on a schedule, pulls fresh data from an RSS feed, and publishes on multiple platforms automatically.

What you need

Tool	Plan / Price	Role
CrewAI	Check the provider's current pricing	Orchestrates multiple LLM calls and decision logic
n8n (Community Edition)	Free (self-hosted Docker)	Workflow engine that ties together APIs, webhooks and data stores
OpenAI (GPT-4o)	Pay-as-you-go	Generates copy and images
Buffer (or Hootsuite)	Check the provider's current pricing	Schedules posts to each social network
Google Sheets	Free with a Google account	Stores content ideas, status flags and API keys
RSS source (your blog, industry site)	Free	Supplies fresh topics to research
Webhook (n8n built-in)	Free	Receives trigger events from the scheduler

Estimated build time: ~6-8 hours for a production-ready workflow, assuming basic familiarity with Docker and API keys.

Step-by-step build

1. Spin up n8n

Install Docker if you don't have it:

 sudo apt-get update && sudo apt-get install -y docker.io

Pull the official n8n image and run it on port 5678:

 docker run -d --name n8n \
 -p 5678:5678 \
 -v ~/.n8n:/root/.n8n \
 n8nio/n8n

Open http://localhost:5678 in a browser and create your first workflow. n8n's UI will ask you to set a Workflow Name; call it "AI Social Agent".

Tip: Use a strong N8N_BASIC_AUTH_USER and N8N_BASIC_AUTH_PASSWORD environment variable to protect the UI if you expose the port to the internet.

2. Create a Google Sheet for content bookkeeping

In Google Sheets, create a new spreadsheet named AI Content Queue.
Add columns: Topic, Status (Pending/Generated/Scheduled), Copy, Image URL, Posted URL.
Share the sheet with a service account (we'll generate one in the next step) so n8n can read/write via the Google Sheets API.

3. Set up OpenAI credentials

Sign up at https://platform.openai.com/ and generate a secret API key.
In n8n, go to Credentials → New Credential → OpenAI and paste the key.
Choose the GPT-4o model for text and DALL·E 3 for image generation.

Why GPT-4o? It offers the best balance of cost and capability for nuanced copy generation and can follow system prompts that enforce brand voice.

4. Install CrewAI and expose it as an HTTP endpoint

CrewAI is a Python library that lets you chain LLM calls with tool use. We'll run it inside a small Flask server that n8n can call via webhook.

python -m venv .venv
source .venv/bin/activate
pip install crewai flask openai

Create agent_server.py:

# agent_server.py - exposes a /run endpoint that executes the content-generation crew
from flask import Flask, request, jsonify
from crewai import Crew, Agent, Task, Tool

app = Flask(__name__)

# Define a simple tool to fetch RSS items
class RSSFetcher(Tool):
 def run(self, url: str):
 import feedparser
 feed = feedparser.parse(url)
 return [entry.title for entry in feed.entries[:3]]

# Agent that drafts copy
copy_agent = Agent(
 role="Content Writer",
 goal="Write engaging social-media copy for a given topic",
 backstory="You are a brand-aware copywriter with a witty tone.",
 tools=[RSSFetcher()]
)

# Agent that creates images
image_agent = Agent(
 role="Image Creator",
 goal="Generate a DALL·E-compatible prompt and image URL",
 backstory="You understand visual branding and can translate copy into imagery."
)

crew = Crew(
 agents=[copy_agent, image_agent],
 tasks=[
 Task(
 description="Given a topic, fetch three related headlines, draft a 280-char tweet, and create an image prompt.",
 agent=copy_agent
 ),
 Task(
 description="Turn the copy into a DALL·E prompt and request an image URL.",
 agent=image_agent
 )
 ]
)

@app.route("/run", methods=["POST"])
def run():
 data = request.json
 topic = data.get("topic")
 result = crew.kickoff(inputs={"topic": topic})
 return jsonify(result)

if __name__ == "__main__":
 app.run(host="0.0.0.0", port=5001)

Run the server in the background:

nohup python agent_server.py &

What this does: The Flask endpoint receives a JSON payload like {"topic":"AI automation trends"} and returns generated copy plus an image URL.

5. Build the n8n workflow

Trigger node - RSS Feed
- Choose "RSS Feed Read"
- Set the Feed URL to your industry blog.
- Enable "Emit items individually".
Google Sheets - Read Row
- Search the AI Content Queue for rows where Status = Pending.
- Map the Topic field to the RSS item's title.
HTTP Request - Call CrewAI server
- Method: POST
- URL: http://host.docker.internal:5001/run (Docker host bridge)
- Body Type: JSON
- JSON Payload:

 {
 "topic": "{{$json[\"Topic\"]}}"
 }

Set - Update Google Sheet
- Write back Copy and Image URL from the HTTP response.
- Change Status to Generated.
Buffer (or Hootsuite) - Create Post
- Use the Buffer node (available via n8n's community nodes)
- Authenticate with your Buffer account (see Buffer's API docs).
- Map Copy to the Message field, Image URL to Media URL, and select the target social profiles.
Google Sheets - Mark as Posted
- Update the same row, setting Status to Scheduled and Posted URL to the response from Buffer.
Cron node - Schedule
- Set to run every 4 hours (or any cadence you prefer). Connect the Cron node to the RSS node to start the chain.

Full n8n JSON snippet for the HTTP Request node (copy-paste into the node's "JSON" tab):

{
 "name": "Call CrewAI",
 "type": "n8n-nodes-base.httpRequest",
 "typeVersion": 1,
 "position": [600, 300],
 "parameters": {
 "url": "http://host.docker.internal:5001/run",
 "method": "POST",
 "responseFormat": "json",
 "jsonParameters": true,
 "options": {},
 "bodyParametersJson": "{\"topic\":\"={{$json[\"Topic\"]}}\"}"
 },
 "credentials": {
 "httpBasicAuth": {
 "id": "crewai-http-cred",
 "name": "CrewAI HTTP Basic"
 }
 }
}

What this does: It sends the current topic to the Flask-served CrewAI crew and receives both copy and an image URL in one request.

6. Test the end-to-end flow

Manually add a row to AI Content Queue with Topic = "Latest trends in AI agents" and Status = Pending.
Trigger the n8n workflow via the UI's "Execute Workflow" button.
Verify:
- The Google Sheet now shows generated Copy and a valid Image URL.
- Buffer's UI lists a new scheduled post with the correct text and media.

If any step fails, the n8n execution log will point to the exact node with an error message.

7. Deploy to production

Docker-compose the n8n container together with the Flask server for easier orchestration.
Use letsencrypt to secure both endpoints (https).
Store all secrets (OpenAI key, Buffer token, Google service account JSON) in environment variables or a vault like HashiCorp Vault.

version: "3.8"
services:
 n8n:
 image: n8nio/n8n
 ports:
 - "5678:5678"
 environment:
 - N8N_BASIC_AUTH_USER=${N8N_USER}
 - N8N_BASIC_AUTH_PASSWORD=${N8N_PASS}
 volumes:
 - ~/.n8n:/root/.n8n
 agent:
 build: .
 ports:
 - "5001:5001"
 environment:
 - OPENAI_API_KEY=${OPENAI_API_KEY}

Now the pipeline runs 24/7, pulling fresh topics, creating brand-consistent copy, and publishing it automatically - exactly what you set out to achieve when you asked how to build a fully automated AI system for cross-platform social media content.

Where this breaks

Failure mode	Symptom	Fix
OpenAI rate-limit	HTTP 429 returned from the OpenAI node	Batch requests: add a Throttle node to keep calls under 60 req/min; monitor usage in the OpenAI dashboard.
Expired Google service account token	"Invalid credentials" error when reading/writing the sheet	Rotate the service-account JSON every 30 days or use a long-lived OAuth refresh token.
n8n container restarts	Workflow stops at the "Call CrewAI" node because `host.docker.internal` becomes unavailable	Use Docker network alias (`my_network`) and reference the Flask service by its container name.
Buffer API quota exceeded	Buffer node returns "Rate limit exceeded" and posts are dropped	Upgrade or request higher limits; alternatively spread posts over a longer cron interval.
RSS feed changes format	RSS node returns empty items, no new topics appear	Add a Function node that validates `entry.title` and falls back to a secondary feed URL.
Image generation fails	DALL·E returns "content_policy_violation" → no image URL	Pre-filter the copy for prohibited keywords; add a retry with a sanitized prompt.

Pro tip: Enable n8n's built-in Error Workflow to capture failures, log them to a Slack channel, and automatically reset the problematic node.

For a deeper technical reference, see n8n's documentation.

FAQ

How do I keep the system completely free?

The n8n Community Edition and Google Sheets are free to self-host, but the OpenAI API and any scheduling platform (Buffer, Hootsuite, etc.) charge per usage. Check each provider's current pricing to see if your projected volume stays within a free-tier limit; you may need to add a cost-monitoring step in n8n.

Can I replace Buffer with a native platform API (e.g., Twitter API v2)?

Yes. n8n includes built-in Twitter, LinkedIn and Facebook nodes. Swap the Buffer node for the appropriate platform node and use the same Copy field mapping. You'll need developer apps and bearer tokens for each network.

What is a "faceless AI content agent"?

A faceless AI content agent is an autonomous software persona that creates and publishes content without a human-visible author, relying on LLMs and tool integrations to mimic a content creator.

How do I add image generation without DALL·E?

You can plug any image-generation API (Stable Diffusion, Midjourney) by adding a HTTP Request node after the copy-generation step. Feed the copy into the prompt, capture the returned URL, and pass it to the scheduler.

How can I monitor the pipeline's health?

Use n8n's Execution List and enable the Error Workflow to send alerts to Slack or email. Additionally, log OpenAI token usage and Buffer post counts to a separate Google Sheet for periodic review.

Where can I learn more about selling similar automations?

Check our internal guide on AI automations you can sell for pricing models and client onboarding templates. If you need a quick-start checklist, grab the free guide we publish for new automation consultants.

By following these steps you'll have a production-grade pipeline that automate social media posts with ai agents, scales with your content volume, and requires only occasional human oversight for strategy tweaks. The same pattern can be extended to newsletters, blog drafts, or even video script generation - anywhere you need a faceless AI content agent to do the heavy lifting. Happy building.

Scraping 1,000 Amazon Listings per Category Without Bot Blockers

Crawler Bros — Fri, 11 Sep 2026 15:48:28 +0000

Managing Amazon Product Scraping at Scale

E-commerce price monitoring, catalog auditing, and competitive intelligence pipelines require extraction of search and category data across multiple regional Amazon domains. Building custom browser scrapers for Amazon presents immediate operational friction: unpredictable HTML revisions, geographic pricing variations, and anti-bot systems that trigger CAPTCHA challenges when request rates rise.

When extracting product data at scale, pipelines encounter a tradeoff between execution speed and attribute depth. Extracting summary cards from search result pages provides high throughput, but detailed seller information, nested variant ASINs, star breakdown ratios, and granular bestseller ranks require traversing individual product detail pages.

The amazon-scraper Actor addresses these challenges by managing user-agent rotation, stealth scripts, proxy country matching, and response parsing across Amazon domains including amazon.com, amazon.co.uk, amazon.de, amazon.fr, and amazon.co.jp.

Scraper Configuration and Input Parameters

The Actor's execution behavior is defined by specific input parameters that balance scraping depth, request throttling, and regional routing.

categoryUrls (array, required): Target Amazon category or search URLs. Accepts standard search queries (e.g., https://www.amazon.com/s?k=laptops) and regional storefront URLs.
maxItemsPerCategory (integer): Caps the total number of items collected per target URL, supporting values from 1 to 1000.
maxPages (integer): Sets the pagination limit per category, supporting up to 50 pages.
scrapeProductDetails (boolean): Controls whether the scraper collects only high-level summary data from search listings (false) or visits each individual product page to extract deep attributes (true).
proxyCountry (string): Configures proxy geo-targeting (AUTO, US, GB, DE, etc.) to align regional currency, shipping costs, and localized inventory visibility.
rateLimitDelay (integer): Sets a base delay in seconds (1 to 30) between requests to prevent rate limiting and block triggers.

Setting scrapeProductDetails: false (Fast Mode) bypasses individual product page fetches. This approach is optimal for high-speed price tracking where basic price, title, and star rating are sufficient. When attributes like variantAttributes, seller, and returnPolicy are required, setting scrapeProductDetails: true directs the scraper to visit each underlying product page.

Executing an Amazon Scraping Task

To execute a scrape job programmatically, supply a JSON configuration to the Actor input schema.

Construct the target search or category URLs from the required Amazon marketplace domain.
Define maximum boundaries using maxItemsPerCategory and maxPages.
Set scrapeProductDetails based on required field depth.
Pass the JSON input payload to the run endpoint.

{
  "categoryUrls": [
    "https://www.amazon.com/s?k=memory+cards"
  ],
  "maxItemsPerCategory": 50,
  "maxPages": 5,
  "scrapeProductDetails": true,
  "proxyCountry": "AUTO",
  "rateLimitDelay": 2
}

During execution, the underlying Python engine leverages Playwright and Chromium to load pages, navigate pagination controls up to the specified maxPages limit, extract DOM nodes, and push structured records to the default dataset.

Structure of the Extracted Product Records

Every item processed by the Actor generates a structured JSON object in the dataset containing pricing, seller, and availability details.

{
  "title": "SanDisk 1TB Extreme microSDXC UHS-I Memory Card",
  "url": "https://www.amazon.com/dp/B09X7MPX8L",
  "asin": "B09X7MPX8L",
  "inStock": true,
  "inStockText": "In Stock",
  "brand": "SanDisk",
  "price": {
    "value": 145.5,
    "currency": "$"
  },
  "listPrice": {
    "value": 299.99,
    "currency": "$"
  },
  "shippingPrice": null,
  "stars": 4.8,
  "starsBreakdown": {
    "5star": 0.86,
    "4star": 0.09,
    "3star": 0.02,
    "2star": 0.01,
    "1star": 0.01
  },
  "reviewsCount": 36704,
  "answeredQuestions": 151,
  "breadCrumbs": "Electronics > Computers & Accessories > Memory Cards",
  "thumbnailImage": "https://m.media-amazon.com/images/I/716kSUlHouL.jpg",
  "description": "Save time with card offload speeds...",
  "features": [
    "Up to 190MB/s read speeds",
    "Up to 130MB/s write speeds"
  ],
  "variantAsins": [
    "B09X7C2GBC",
    "B09X7C7LL1"
  ],
  "seller": {
    "name": "Direct Suppliers US",
    "id": "A210SJF12S88M5",
    "url": "/gp/help/seller/at-a-glance.html"
  },
  "bestsellerRanks": [
    {
      "rank": 3,
      "category": "Micro SD Cards"
    }
  ],
  "locationText": "Deliver to New York",
  "categoryUrl": "https://www.amazon.com/s?k=memory+cards",
  "categoryPage": 1,
  "categoryPosition": 5,
  "scrapedAt": "2026-01-26T10:30:00.000Z"
}

The output captures operational metrics such as search position (categoryPosition), source category page (categoryPage), seller identifiers (seller.id), and rating distributions (starsBreakdown).

Event Pricing Model Breakdown

This Actor uses a PAY_PER_EVENT pricing model. Billing is determined strictly by event counts recorded during execution:

Actor Start (apify-actor-start): Charged once per run at $0.003 per GB of memory allocated. Tier discounts reduce this price to $0.00233 under BRONZE, $0.00167 under SILVER, and $0.001 under GOLD, PLATINUM, and DIAMOND tiers.
Result (apify-default-dataset-item): Charged at $0.005 per extracted item output to the dataset. Tier discounts reduce this price to $0.00433 under BRONZE, $0.00367 under SILVER, and $0.003 under GOLD, PLATINUM, and DIAMOND tiers.

For instance, running a job with 1 GB memory that outputs 200 items on the FREE tier calculates as:

Actor Start event: 1 GB × $0.003 = $0.003
Result events: 200 × $0.005 = $1.000
Total run cost: $1.003

Enabling scrapeProductDetails: true increases execution time due to additional detail page navigations, but charges remain tied strictly to output dataset items and memory allocated to the run.

Technical Limits and Query Design Rules

Amazon enforces a hard limit of 50 result pages per search query, capping pagination at approximately 1,000 to 2,000 items depending on listing density. Attempting to set maxPages higher than 50 or expecting to pull 10,000 items from a single category URL like https://www.amazon.com/s?k=laptops will truncate at Amazon's search boundary.

To extract comprehensive datasets across large categories, split search queries using brand, price range, or specification filters (such as https://www.amazon.com/s?k=laptops&rh=p_36%3A50000-100000) across multiple entries in categoryUrls. Additionally, full price matrices across all variant combinations are not generated automatically; extracting pricing for every variant requires passing individual variant ASIN URLs as input targets. Pipelines requiring real-time buy-box tracking should align update cycles with rate-limit delays to avoid IP throttling.

If you want to reproduce this, the Actor is Amazon Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-11. Check the Actor page for the current rates.

Extracting 1,000 Amazon Product Records Across 50 Search Pages

Crawler Bros — Fri, 11 Sep 2026 15:46:52 +0000

Managing Amazon Search Crawls Across Large Product Catalogs

E-commerce data pipelines that track search positioning, competitive pricing, and brand presence on Amazon face structural challenges. Amazon search result pages contain basic product listings, but extracting complete pricing variants, buy box seller information, and detailed specifications requires navigating secondary product detail pages across regional domains. Doing this manually or via raw HTTP requests leads to CAPTCHA challenges, missing fields, or broken pagination loops.

The Amazon Scraper Actor addresses these challenges by automating category and search crawling across multiple Amazon domains (including amazon.com, amazon.co.uk, amazon.de, amazon.fr, amazon.it, amazon.es, amazon.ca, amazon.com.au, amazon.co.jp, and amazon.in). It extracts both search listing metrics and detailed product page specifications, writing uniform structured output items directly to an Apify dataset.

How Amazon Scraper Handles Search and Detail Pages

The scraper provides two distinct operational modes determined by the scrapeProductDetails input parameter.

Fast Mode vs. Detailed Extraction

When scrapeProductDetails is set to false, the scraper operates in Fast Mode. It parses only the search result or category listing pages generated by the target Amazon domain. This mode collects data fields visible directly on the listing page:

title: Product title string
url: Direct Amazon product URL
asin: Amazon Standard Identification Number
price: Current listed price object containing value and currency
stars: Average star rating float (0–5)
reviewsCount: Total count of customer reviews
categoryUrl, categoryPage, categoryPosition: Ranking metadata showing where the item appeared during the crawl

When scrapeProductDetails is set to true (the default value), the Actor extracts search results and then fetches each individual product page (/dp/ASIN). This full extraction populates detailed secondary attributes:

brand: Manufacturer or brand name
listPrice: Original list price object before discounts
shippingPrice: Shipping cost value (or null if free shipping applies)
starsBreakdown: Percentage distribution across star ratings (5star, 4star, 3star, 2star, 1star)
seller: Seller metadata object containing name, id, url, reviewsCount, and averageRating
features: Array of feature bullet points from the detail page
description: Text product description
breadCrumbs: Category breadcrumb path string
variantAsins and variantAttributes: ASINs and attribute names for product variations
bestsellerRanks: Array of category rank objects (rank and category)
delivery, fastestDelivery, returnPolicy, and support: Service and fulfillment details

Schema Control Parameters

To control crawl scope and execution timing, several schema parameters govern the runtime behavior:

categoryUrls: Required array containing target search or category URLs (such as https://www.amazon.com/s?k=laptops).
maxItemsPerCategory: Integer defining the maximum products scraped per input URL (range: 1–1000).
maxPages: Integer capping how many pagination pages to process per category (range: 1–50).
scrapeProductDetails: Boolean toggle enabling or disabling detail page fetches.
proxyCountry: Geolocation targeting parameter (AUTO, US, GB, DE, etc.) ensuring prices and availability reflect target regional settings.
rateLimitDelay: Integer setting the base delay in seconds (range: 1–30) between request calls to avoid rate limits.

Pay-Per-Event Cost Structure for Execution

This Actor uses a pay-per-event billing model on the Apify platform. Charges depend on specific execution events:

result event (apify-default-dataset-item): Charged at $0.005 per item written to the default dataset. On volume tiers, this scales to $0.00433 (BRONZE), $0.00367 (SILVER), and $0.003 (GOLD, PLATINUM, DIAMOND).
Actor Start event (apify-actor-start): Charged at $0.003 per GB of memory allocated to the run when execution starts. On volume tiers, this scales to $0.00233 (BRONZE), $0.00167 (SILVER), and $0.001 (GOLD, PLATINUM, DIAMOND).

A run processing 1,000 product items with 1 GB of memory allocated on the standard tier incurs one Actor Start event ($0.003) and 1,000 result events ($5.000), totaling $5.003.

Configuring an Amazon Category Scraping Execution

Setting up a job involves defining input parameters in JSON and invoking the Actor via code or platform trigger.

1. Construct the Input Payload

Create a JSON payload specifying target category URLs, item bounds, and rate limits:

{
  "categoryUrls": [
    "https://www.amazon.com/s?k=memory+cards"
  ],
  "maxItemsPerCategory": 50,
  "maxPages": 2,
  "scrapeProductDetails": true,
  "proxyCountry": "AUTO",
  "rateLimitDelay": 2
}

2. Execute via Python SDK

Invoke the Actor using the official apify-client library in Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "categoryUrls": [
        "https://www.amazon.com/s?k=memory+cards"
    ],
    "maxItemsPerCategory": 50,
    "maxPages": 2,
    "scrapeProductDetails": True,
    "proxyCountry": "AUTO",
    "rateLimitDelay": 2,
}

run = client.actor("crawlerbros/amazon-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"ASIN: {item.get('asin')} | Title: {item.get('title')} | Price: {item.get('price')}")

3. Process the Dataset Record

The dataset returns items formatted with structural fields ready for database ingestion:

{
  "title": "SanDisk 1TB Extreme microSDXC UHS-I Memory Card",
  "url": "https://www.amazon.com/dp/B09X7MPX8L",
  "asin": "B09X7MPX8L",
  "inStock": true,
  "inStockText": "In Stock",
  "brand": "SanDisk",
  "price": {
    "value": 145.5,
    "currency": "$"
  },
  "stars": 4.8,
  "reviewsCount": 36704,
  "seller": {
    "name": "Direct Suppliers US",
    "id": "A210SJF12S88M5"
  },
  "scrapedAt": "2026-01-26T10:30:00.000Z"
}

4. Adjust Delays for High-Frequency Runs

If Amazon presents CAPTCHAs during pagination, increase rateLimitDelay from 2 seconds to 5 or 10 seconds to stagger network traffic across pages.

Operational Limitations and Edge Cases

While this Actor includes anti-detection measures such as user agent rotation, stealth scripts, and simple CAPTCHA detection attempts, it is subject to platform restrictions. Amazon enforces a ceiling of 50 pagination pages (typically 1,000 to 3,000 products) per search query. Searches returning tens of thousands of items cannot be fully extracted through a single category URL; queries must be split into narrower sub-categories or price filters. Additionally, this scraper does not bypass persistent account-level bot checks or access gated seller central dashboard data.

If you want to reproduce this, the Actor is Amazon Scraper. Read its input schema before the first run -- most failed runs are a missing required field, not a block.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-11. Check the Actor page for the current rates.

DEV Community: tutorial

Extract Job Listings from Indeed — Python Tutorial 2026

Extract Job Listings from Indeed — Python Tutorial 2026

Why Scrape Indeed?

Quick Start with Apify

Python Method

What Data You Can Extract

Best Practices

Two writers, one database: why your self-hosted app breaks the moment it runs twice

Your old session dies, but a fresh one works

Or the logs say database is locked

First, count your writers

The fix, in order

Things that don't fix it

The honest caveats

How to Automate Instagram Comment-to-DM Flows on Meta's Official API

The flow at a high level

Webhooks and signature verification

The 24-hour messaging window and private replies

Rate limits are real

Tokens: short, long, and refresh cycles

App Review

Why scraping is the wrong shortcut

Where this goes from prototype to product

JSON vs Programmatic Tool Calling with Claude

The problem: round trips

Approach 1: JSON schema + manual loop

Approach 2: Tool Runner SDK

Approach 3: programmatic tool calling

Comparison

When to use what

In practice

Layered Architecture: Rust API with Axum, SQLx

The workspace structure

Entities: the domain without a framework

The FSM in the type system

The generic FsmState<T>

The Run entity

The store: traits and two implementations

The RunStore trait

Two interchangeable backends

Store error handling

The API layer: Axum handlers and DTOs

DTOs separate from entities

A typical handler

Error conversion

Router assembly

AppState: dependency injection

Tests: the concrete benefit of the architecture

What works well

What I would change

The numbers

Conclusion

Writing effective Claude Code skills

What a skill is

Progressive disclosure: why it matters

Anatomy of a SKILL.md

The frontmatter: the routing contract

The body: keep it light

Two design philosophies

Pattern A: tool wrappers

Pattern B: cognitive disciplines

Patterns for effective skills

Bash first, prose second

State-check before action

Validation loops

Compose primitives

Document output formats

Anti-patterns

Don't re-teach what the model knows

Don't write vague descriptions

Don't create monolithic mega-skills

Don't ignore failure modes

Don't use absolute paths

Project-specific skills

Testing a skill

Checklist before publishing a skill

What I've learned

How to Launch a Freelancer website for Under $3/Month

What You Need

The generic `FsmState<T>`

The `Run` entity

The `RunStore` trait