DEV Community: api

A note said the run scraped 0 rows. The platform's own record said 6.

Devil Scrapes — Tue, 22 Sep 2026 09:26:38 +0000

Quick answer

A note in this repo declared the FINRA BrokerCheck Scraper unpublishable: its own shipped prefill, run verbatim, "SUCCEEDS with 0 rows." That's a serious defect if true — a customer running the exact default input would get a clean, billed, empty run. When we went back to fix it, we pulled the actual platform record for the cited run instead of trusting the note. GET /v2/actor-runs/dTWAgCsoPf6DgFkP0 came back status: SUCCEEDED, statusMessage: "Processed 2 queries, skipped 0 due to transport failures; emitted 6 row(s)", chargedEventCounts: {"result-emitted": 6}. The dataset had itemCount: 6, and the six items were real BrokerCheck records for CRD 5998211 and "john smith." The Actor was fine. The note was wrong, and it sat unchallenged for weeks.

How does a wrong finding survive that long?

Because a written verdict is easy to trust and expensive to re-check, and once one agent files "0 rows" against a run ID, every downstream decision treats that as settled fact rather than a claim worth re-opening. The original note presumably reflected something real at the moment it was written — maybe a genuine platform blip, maybe a mis-cited run ID — but nothing forced a second look before the Actor got shelved on the strength of it. A cached verdict is not the same thing as the primary record, and the two only diverge when someone actually goes back to the source.

Re-checking took two independent paths, and both disagreed with the note. First, the platform's own run record for the exact ID the finding cited — status, charged events, and dataset item count, pulled straight from the API, not summarized secondhand. Second, a fresh local apify run against the live FINRA endpoint with the unmodified pre-fix code and the identical prefill: 6 rows back in about 9 seconds. Same build, same input, same result both times, and neither matched "0 rows."

So what was actually wrong with it?

Something real, just not that. proxyConfiguration shipped as a bare {"useApifyProxy": true} — no group named — which on this account silently resolves to the datacenter pack rather than the residential one the build brief called for. The platform's own run logs don't echo proxyConfiguration back, so this couldn't even be confirmed from a log read; it had to be traced from the input schema itself. That's now fixed: apifyProxyGroups: ["BUYPROXIES94952"] and apifyProxyCountry: "US" are pinned explicitly in the default, the prefill, and the QA fixture, with a regression test guarding against a silent revert to the bare form.

We also added a guard the original finding's concern deserved even though its specific claim didn't hold up: a whole-run zero-rows check. If every query in a run gets a real HTTP answer but the run as a whole produces zero rows, it now fails loud rather than reporting a clean SUCCEEDED — a deliberate narrowing from this codebase's usual "empty search still succeeds" default, because a batch of CRD numbers or names that all miss is far more likely to be a customer typo than a real "no such broker exists."

A finding that says "this run returned 0 rows" is a claim about that run, not a fact about the Actor — and the fastest way to know which one you're looking at is to pull the platform's own record for the run ID, not the summary of it.

What the Actor gives you

One merged row per broker or firm hit — search by name or by CRD number, mixed freely in the same list, with purely numeric entries resolved as direct CRD lookups and everything else searched. Each row carries registration scope, disclosure flag, current and previous employments, registered states and SROs, exam counts, and — when fetchFullDetail is on — the full disciplinary disclosure history merged onto the same row instead of left on a separate detail page you'd otherwise have to fetch yourself.

Honest limitations 🚧

This is a lookup-and-merge tool against FINRA's own public API, not a discovery engine — you supply names or CRD numbers, it doesn't enumerate brokers on its own. A name search that matches nothing for a given query still succeeds for that query (only a whole-run zero-match batch fails loud).

FAQ

Do I need a FINRA account or API key?
No — this uses FINRA BrokerCheck's public, keyless JSON API, the same data backing brokercheck.finra.org.

Can I mix broker names and CRD numbers in one run?
Yes. Purely numeric entries are fetched directly by CRD; everything else is name-searched.

What happens if one name in my batch doesn't match anything?
That query succeeds with zero rows and is named in the run's status message. Only a batch where every single query comes back empty fails loud.

Does the disclosure history come from a separate lookup?
Not one you have to make yourself — with fetchFullDetail on, disclosures, employments, and registration counts are merged from the CRD detail endpoint onto the same row as the search hit.

Pricing

$0.20 per run plus $0.0028 per broker/firm row — $3.00 per 1,000 results. A run that matches nothing costs only the start fee.

→ FINRA BrokerCheck Scraper on Apify

Built by Devil Scrapes. We rotate fingerprints, retry with backoff, and — as this one shows — we re-check the primary record before we trust a verdict about our own Actor.

Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows

Devil Scrapes — Tue, 22 Sep 2026 09:26:07 +0000

Quick answer

Our new TCGPlayer Card Prices scraper passed cloud QA, delivered rows, and would have overcharged every customer by 22%. The deep run that caught it took four minutes and one line of analysis: count distinct IDs, not rows.

Searching charizard at the Actor's own default depth returned 50 rows carrying 39 distinct product IDs. Eleven products were billed twice. Nothing failed, nothing logged an error, and the row count looked exactly right.

Why did pagination produce duplicates at all?

Because TCGPlayer's search endpoint is relevance-sorted, and relevance is not a stable ordering.

The API pages with a plain from/size offset — ask for from=0&size=10, then from=10&size=10, and so on. That contract only holds if the underlying result list stays fixed between requests. On a relevance-ranked search it doesn't: scores shift slightly between calls, items move across the page boundary, and a product that sat at position 10 on your first request sits at position 11 on your second — so you fetch it twice and never see whatever got pushed to 9.

This is the quiet failure mode of offset pagination generally. It is not specific to TCGPlayer, and it is not a bug on their side — it is what happens when you page through a ranked list with an offset instead of a cursor.

The fix is unglamorous: track the IDs you have already emitted, skip repeats, and keep paging until you have the number of distinct items the caller asked for.

before:  50 rows, 39 distinct productIds
after:  100 rows, 100 distinct productIds   (2 queries x 50)

Why "it returned rows" is the wrong thing to check

Because every cheap signal agrees with you when you are wrong.

Our QA harness clamps depth to keep smoke tests cheap — it rewrote maxResultsQuery: 50 down to 3 and said so, in an explicit warning: "the run therefore proves page one only." Three rows come off one page. One page cannot exhibit page-overlap. So the duplicate bug was, by construction, invisible to the test that passed.

That is the general shape worth taking away: a test that clamps the parameter under test proves the thing you were not worried about. The warning was right there in the output and it was advisory — which is a polite way of saying nobody had to act on it.

We also found the same class of problem in a second Actor the same morning: 200 rows carrying 127 distinct filing IDs, because it emitted one row per filing-debtor pair while its documentation promised one row per filing. Same tell, entirely different cause — one was a pagination defect, the other a contract mismatch between the code and its own README. Counting distinct IDs surfaced both.

So the audit is now a script rather than a habit, and it checks two things on a real full-depth run:

Distinct identities vs. row count. Duplicates mean either a pagination defect or an undocumented row granularity. Both are worth knowing before a customer finds out.
Fields that are empty in every single row. A column that never populates is a promise the product does not keep. We caught one of those the same day: an advertised supplier-website field that was null on 150 of 150 rows.

Under pay-per-event pricing this is not just a data-quality question. Every duplicate row is a billed event. A 22% duplicate rate is a 22% overcharge, and it would have been entirely invisible on our side — the dashboards would have shown a healthy Actor with good volume.

What the Actor gives you

One row per card product matching your search terms, across every game on TCGPlayer — Magic, Pokémon, Yu-Gi-Oh and the rest. Each row carries product_id, product_name, product_line, set_name, set_code, rarity, market_price, lowest_price, lowest_price_with_shipping, total_listings, foil_only and a product_url. Optionally, per-seller listings for the current offers on a product.

Multiple search queries in one run, deduplicated by product ID across pages — which, as above, is the entire point.

Honest limitations 🚧

Search-based: you get the products a query matches, not a full set catalogue dump. Per-seller listings are off by default because they multiply run time and row count — turn them on deliberately. Prices are what TCGPlayer's marketplace reports at fetch time, so for a volatile card, a row is a timestamp and not a standing quote. And a query that genuinely matches nothing succeeds with zero rows, which is the correct answer rather than a fault.

FAQ

Do I need a TCGPlayer API key?
No. No key, no login, no seller account.

What does it cost?
$3.20 per 1,000 products — a $0.20 start fee plus $0.003 per product row. You pay for distinct products, not for our pagination mistakes.

Can I track a card's price over time?
Yes. Run the same queries on a schedule and diff on product_id; market_price and lowest_price are the fields that move.

Which games are covered?
Every product line TCGPlayer carries — the product_line field tells you which one each row came from.

Add EU VAT validation to Google Sheets

Alexander Nitrovich — Tue, 22 Sep 2026 09:25:13 +0000

Introduction

Integrating EU VAT validation into Google Sheets can dramatically streamline your business processes. Validating these numbers ensures compliance with EU tax regulations and can be seamlessly integrated into your existing workflows using API solutions. In this article, we'll explore how to efficiently set up EU VAT validation in Google Sheets with a step-by-step guide using Google Apps Script, leveraging a developer-centric API to automate and enhance your operations.

Why Validate EU VAT Numbers?

The EU VAT system mandates that businesses validate VAT numbers to ensure compliance with regulations, avoid hefty fines, and streamline cross-border trade. Manual validation is fraught with challenges such as human error and time inefficiency. Automated API-based validation not only saves time but enhances accuracy and reliability, relieving businesses from manual compliance headaches.

Overview of Our API-Driven VAT Validation Solution

Our API offers comprehensive VAT validation services, providing fast, accurate, and easily integrated solutions. With capabilities like checking VAT number validity, retrieving company details, and compliance status, integrating this API opens up possibilities for businesses dealing with invoicing and ecommerce transactions across Europe.

Benefits include:

Speed: Validate VAT numbers in milliseconds.
Accuracy: Reliable results backed by EU databases.
Ease of integration: API-first design for smooth incorporation into existing systems.

Real-world use cases encompass ecommerce platforms ensuring seller validity, enabling invoice processors to verify company VAT details swiftly.

Setting Up Your Google Sheets Environment

To start integrating VAT validation, access Google Apps Script by navigating to Extensions > Apps Script from your Google Sheets. Prepare your spreadsheet by designating cells for VAT numbers and alongside them, spaces for validation responses. Ensure you grant necessary permissions for web access and script actions.

Step-by-Step Guide: Integrating VAT Validation into Google Sheets

Follow these steps to integrate VAT validation:

Write the Google Apps Script: Copy the provided script that calls the EuroValidate API.
Make the API Call: Use the UrlFetchApp to call the API endpoint with required headers containing your API key.
Parse the Response: Capture and handle the JSON response to extract relevant VAT details.
Error Handling: Implement logging for unsuccessful requests or exceptions.

Here’s an example of the script:

function validateVatNumber(vatNumber) {
  var apiKey = "YOUR_API_KEY";
  var endpoint = "https://api.eurovalidate.com/v1/vat/" + encodeURIComponent(vatNumber);

  var options = {
    "method": "GET",
    "headers": {
      "Authorization": "Bearer " + apiKey
    },
    "muteHttpExceptions": true
  };

  try {
    var response = UrlFetchApp.fetch(endpoint, options);
    var responseCode = response.getResponseCode();

    if (responseCode === 200) {
      var result = JSON.parse(response.getContentText());
      return result;
    } else {
      Logger.log("Error: " + responseCode);
      return {error: "API error: " + responseCode};
    }
  } catch (e) {
    Logger.log("Exception: " + e);
    return {error: e.toString()};
  }
}

function onOpen() {
  var ui = SpreadsheetApp.getUi();
  ui.createMenu("VAT Tools")
    .addItem("Validate VAT", "menuValidateVat")
    .addToUi();
}

function menuValidateVat() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
  var vatNumber = sheet.getActiveCell().getValue();
  var result = validateVatNumber(vatNumber);

  if (result.error) {
    SpreadsheetApp.getActiveSpreadsheet().toast("Error validating VAT: " + result.error);
  } else {
    sheet.getActiveCell().offset(0, 1).setValue(result.status ? "Valid VAT number" : "Invalid VAT number");
    SpreadsheetApp.getActiveSpreadsheet().toast("VAT validation complete");
  }
}

Code Walkthrough & Examples

The script configures the API key and endpoint, makes a GET request to validate the VAT number, and parses JSON responses. Customize the request by changing the API key or endpoint URL as needed. Common issues include incorrect API keys or network restrictions, addressed by checking setup and permissions.

Example API Response:

Valid VAT:

vat_number: "NL820646660B01"
status: "valid"
company_name: "Test Company BV"

Invalid VAT:

vat_number: "FR40303265045"
status: "invalid"

Best Practices & Tips for Integration

Secure your API keys by storing them in environment variables or script properties rather than hardcoding. Manage rate limits by queuing requests judiciously. Stay updated with any API changes by checking EuroValidate API documentation.

Conclusion

Adding automated EU VAT validation to your Google Sheets workflow is straightforward with API integration, yielding immediate compliance benefits and operational efficiency. As you prepare to implement this solution, remember the power of automation in enhancing accuracy and saving resources.

Ready to supercharge your spreadsheet workflows with automated EU VAT validation? Sign up for a free API trial today and see how easily you can integrate powerful VAT checks into Google Sheets!

I Turned 40 Product Photos Into Video With a Photo to Video AI Generator (and One Template)

Hamimelon2026 — Tue, 22 Sep 2026 09:24:58 +0000

A friend runs a small online store and handed me 40 product photos with one request: "can these move?" Doing that one photo at a time, writing a fresh prompt each time, would have taken the whole weekend. So I built a small template system before I opened any photo to video ai generator at all.

This post covers the template that made 40 photos manageable, where I did the actual generating, and a script that stitches the results into one reel.

The problem with animating photos one at a time

The first five photos went fine. By photo six, my prompts had drifted: different lighting language, different camera moves, no consistent mood across a single product line. A viewer would notice the catalog didn't feel like one catalog.

The fix wasn't a smarter prompt. It was one prompt template and a spreadsheet.

A photo to video ai generator is really just the image to video AI case applied to a whole catalog: one photo in, one clip out, repeated forty times. That's the quickest way to animate a photo at the scale a small store actually needs, whether it's an AI product video generator for a shop or a portfolio reel for a photographer. The generator handles one photo fine by default. Forty in one voice is a planning problem, not a prompting problem.

Build the prompt list before you generate anything

I keep one CSV with a filename, a product name, and a mood per photo, and fill the rest from a template:

# build_prompts.py - fill a prompt template from a CSV of photos
import csv

TEMPLATE = (
    "Slow {move} on {product}, {mood} lighting, shallow depth of field, "
    "cinematic color grade, no camera shake"
)
MOVES = {"hero": "push-in", "detail": "orbit", "lifestyle": "pan right"}

with open("photos.csv") as src, open("prompts.csv", "w", newline="") as out:
    reader = csv.DictReader(src)
    writer = csv.writer(out)
    writer.writerow(["filename", "prompt"])
    for row in reader:
        move = MOVES.get(row["shot_type"], "push-in")
        prompt = TEMPLATE.format(move=move, product=row["product"], mood=row["mood"])
        writer.writerow([row["filename"], prompt])

print("wrote prompts.csv")

photos.csv needs only filename,product,mood,shot_type per row. Run the script once and every photo gets a prompt in the same voice, ready to paste in one at a time or hand to a teammate.

A few things kept the catalog feeling like one catalog instead of forty unrelated clips:

One mood per product line, not per photo. I set mood at the product level and let shot_type vary the move.
A short, fixed vocabulary for moves. Three move words, reused everywhere, read as a style. Ten different move words read as inconsistency.
The template stays boring on purpose. Cinematic color grade and no camera shake are in every prompt because I never want to debate them per photo.

Where I generated: photo to video AI in one workspace

For the generating itself, I used VOKOO, a multi-model AI creation platform built around video. Its tagline is "Create more. Switch less," and for 40 photos with one voice, that consistency mattered more than any single clip. I dropped in a photo, pasted its prompt from prompts.csv, and had a clip to review before I finished my coffee.

Photo in, motion out

The AI video generator turns a photo plus a prompt into a video. Make a video before the idea gets cold. Each row in prompts.csv became one generation.

Fix a weak photo before you animate it

Some of the 40 were low-res phone shots. The AI photo editor and image upscaler let me edit, refine, and make small or blurry images crisp and usable without leaving the flow.

Switch models when one style doesn't fit

The AI agent lets me try different models without rebuilding my workflow. The hero shots needed a different touch than the lifestyle shots, so I split them across two models without changing anything else.

Check the cost before scaling to 40

I can pick quality and generation specs per stage and see the estimated credit cost before I submit. I ran three test photos at draft quality first, then committed to all 40 at the spec that looked right. One place to generate, edit, enhance, and animate.

Turn the batch into one reel

Forty separate clips aren't a catalog video. This stitches them into one reel with a short crossfade between each pair, reading the order straight from your CSV:

#!/usr/bin/env bash
# make_reel.sh - concat clips in prompts.csv order with a crossfade
# usage: ./make_reel.sh clips/ 0.5   (clip folder, fade duration in seconds)
DIR="$1"; FADE="${2:-0.5}"
mapfile -t FILES < <(tail -n +2 prompts.csv | cut -d',' -f1 | sed 's/\.[a-zA-Z0-9]*$/.mp4/' | sed "s|^|$DIR/|")

CHAIN="[0:v]"
FILTER=""
INPUTS=""
OFFSET=0
for i in "${!FILES[@]}"; do
  INPUTS+=" -i ${FILES[$i]}"
  if [ "$i" -gt 0 ]; then
    DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "${FILES[$((i-1))]}")
    OFFSET=$(echo "$OFFSET + $DUR - $FADE" | bc)
    NEXT="v$i"
    FILTER+="${CHAIN}[$i:v]xfade=transition=fade:duration=$FADE:offset=$OFFSET[$NEXT];"
    CHAIN="[$NEXT]"
  fi
done

eval ffmpeg -y $INPUTS -filter_complex "\"${FILTER%;}\"" -map "${CHAIN}" -an reel.mp4
echo "wrote reel.mp4"

Every filename in prompts.csv needs a matching .mp4 in the clips folder. The script builds an xfade chain in order and writes one reel.mp4.

Keeping 40 generations affordable

The draft-then-commit habit scales linearly: test a few photos at a low spec, confirm the look, then run the rest at the spec you'll actually use. The estimated cost shown before submit is what made testing at scale comfortable instead of anxious.

If you'd like an LLM to write the mood column in photos.csv from a product description, RouteAI provides a cost-effective, OpenAI-compatible API gateway with multiple models, so setup stays simple.

Try this next

A photo to video ai generator handles one photo well by default. Handling forty in the same voice takes a template, not a better prompt written forty times. VOKOO did the generating; the template kept it consistent. Stop managing tools. Start making things.

Here's a short test you can run this weekend:

Fill photos.csv with five real photos and run build_prompts.py.
Generate all five and drop the clips in clips/.
Run make_reel.sh clips/ 0.5 and watch the result.
Check the estimated cost before you scale to the rest of your catalog.

If you want an easy AI video generator that keeps simple AI video creation simple and still leaves room to explore, try VOKOO at https://vokoo.ai.

Two new x402 APIs for AI agents: http-cache + css-audit (2026-09-22)

HAL GOBVAN — Tue, 22 Sep 2026 09:13:51 +0000

What got shipped (cycle 64)

Two more paid x402 endpoints on the URL Metadata API — bringing the catalog to 62 paid routes.

/api/http-cache ($0.0005 USDC)

Probes /favicon.ico, /robots.txt, and /sitemap.xml and audits every cache-related header the server sends back:

Cache-Control decomposition: max-age, s-maxage, public, private, no-store, no-cache, immutable, stale-while-revalidate, stale-if-error.
Validation headers: ETag, Last-Modified.
Vary parser — flags the legacy User-Agent antipattern that kills edge caching.
Surrogate-Control + CDN-specific fingerprints (CF-Cache-Status, X-Cache, X-Served-By, X-Vercel-Cache, X-Amz-Cf-Id, X-Cloud-Trace-Context).
Age header for proxy freshness check.

Returns cache_validator_score 0-100 with A-F grade. Live: stripe.com=10/F (no Cache-Control on static), github.com=40/F (ETag + Last-Modified present, no max-age).

/api/css-audit ($0.0005 USDC)

Audits the full CSS surface of a page so an AI agent knows what it costs to render:

External <link rel=stylesheet> count + inline <style> block sizes + total inline CSS bytes.
Per-sheet HEAD probe: Content-Type, Cache-Control quality (max-age >=86400 = good for versioned CSS), filename critical-CSS hint.
8 @media coverage probes: prefers-color-scheme (dark/light), prefers-reduced-motion, prefers-reduced-transparency, forced-colors, monochrome, any-hover, pointer:fine.
7-framework fingerprint (Tailwind / Bootstrap / Bulma / Foundation / Materialize / Tachyons / Open Props) via class-name string-count.
CSS-in-JS heuristic (any inline block >= 2KB).
Render-blocking detection vs <link rel=preload as=style> coverage.

Returns css_audit_score 0-100 with A-F grade. Live: stripe.com=55/D (preload-everything, no dark-mode, Tailwind-class heavy), github.com=15/F (28 render-blocking stylesheets, no preload).

Why these two specifically

Both are gaps that surface constantly when an AI agent builds a scraping pipeline:

Cache audit: agents that don't know whether a URL is cacheable end up either over-fetching (wasting bandwidth + hitting rate limits) or under-fetching (serving stale data). A single X-PAYMENT call resolves the question.
CSS audit: agents that need to estimate render cost, detect framework assumptions, or decide whether to inline-vs-stream CSS need the surface mapped. The framework fingerprint also helps when deciding which CSS-utility libraries to assume on a target site.

Try it

Both routes are live at https://epson-rpm-america-satisfy.trycloudflare.com — send a USDC payment of $0.0005 (500 atomic units) to 0xCa0a6c6Aa7A8F0D5893636CF166Ea2b44fb6500c on Base mainnet, then GET:

/api/http-cache?domain=stripe.com (or ?url=<URL>)
/api/css-audit?domain=github.com (or ?url=<URL>)

Full 62-route catalog: GET /.well-known/x402 or https://epson-rpm-america-satisfy.trycloudflare.com/llms.txt.

How RCS Messaging Can Work With CRM Software: Architecture, Data Flows, & Automation

Software Solutions — Tue, 22 Sep 2026 08:53:16 +0000

While traditional SMS has served as the default transactional notification channel for years, it operates as a disconnected, unverified plain-text pipe. Integrating Rich Communication Services (RCS) directly with Customer Relationship Management (CRM) platforms—such as Salesforce, HubSpot, Zoho, or custom internal backends—bridges the gap between core customer data and native mobile chat interfaces.

Instead of broadcasting generic 160-character texts, an RCS-CRM integration allows applications to pull real-time CRM attributes (first names, deal stages, account history) to render branded, interactive cards with embedded actions, while passing user interactions back into the CRM via webhooks.

Here is an architectural and technical breakdown of how RCS messaging works alongside CRM software, the data models required, and how to build event-driven messaging workflows.

Why Integrate RCS With a CRM?

Integrating an RCS API directly with your CRM solves critical communication bottlenecks that plague standard SMS:

Elimination of Data Silos: Isolated messaging tools create fragmented conversation histories. An integrated pipeline logs outbound cards, inbound user replies, read receipts, and button postbacks directly to the contact's CRM activity timeline.
Context-Aware Personalization: Accessing CRM fields dynamically allows you to inject tailored media (e.g., custom proposal PDFs, specific product photos left in a cart, or assigned account manager details) rather than generic text.
Closed-Loop Action Tracking: Standard SMS relies on raw external URLs with third-party web analytics. RCS postback actions allow the CRM to capture immediate, structured button taps inside the conversation thread.

Technical Workflow: How RCS and CRM Integration Works

The architectural interaction between a CRM database, an RCS API, and a mobile recipient follows a bi-directional event loop:

┌────────────────────────────────────────┐
│                  CRM                   │ ◄── (Record Change / Lifecycle Event)
└───────────────────┬────────────────────┘
│
▼
┌────────────────────────────────────────┐
│          Customer / Lead Data          │ ──► Extracts Context (Name, Order ID, Stage)
└───────────────────┬────────────────────┘
│
▼
┌────────────────────────────────────────┐
│           Business Workflow            │ ──► Evaluates Triggers & Compiles Payload
└───────────────────┬────────────────────┘
│
▼  [ HTTPS POST API Request ]
┌────────────────────────────────────────┐
│                RCS API                 │ ──► Dispatches Rich Card / Carousel
└───────────────────┬────────────────────┘
│
▼  [ IP Delivery ]
┌────────────────────────────────────────┐
│                Customer                │
└───────────────────┬────────────────────┘
│
▼  [ User Taps Button / Enters Text ]
┌────────────────────────────────────────┐
│            Response / Event            │
└───────────────────┬────────────────────┘
│
▼  [ Asynchronous POST ]
┌────────────────────────────────────────┐
│                Webhook                 │
└───────────────────┬────────────────────┘
│
▼  [ Syncs Activity & Updates State ]
┌────────────────────────────────────────┐
│                  CRM                   │
└────────────────────────────────────────┘

CRM Trigger: An event (e.g., deal stage updated to "Proposal Sent", lead status set to "Follow-Up Needed") fires inside the CRM.
Payload Compilation: The CRM or an intermediary middleware service extracts contact parameters and builds a structured JSON payload.
RCS Gateway Dispatch: The payload is sent via an HTTPS POST call to the RCS Provider API (e.g., Google RBM, Twilio, Sinch).
Interactive Delivery: The customer receives a verified, branded card containing structured action buttons.
Inbound Webhook Execution: When the user taps a button (e.g., "Accept Proposal"), the RCS gateway posts an event payload to a webhook listener, which updates the CRM record automatically.

What Customer Data Can Be Used?

To build rich messaging flows, map your CRM data entities directly to RCS card components:

Contact Variables: First name, phone number, language preference, assigned account representative.
Transaction Metrics: Order IDs, shipment tracking numbers, invoice amounts, payment link URLs.
Appointment Metadata: Booking timestamps, service location maps, provider profiles.
Pipeline Attributes: Lead score, lifecycle stage, deal status, recent page visits.

Connecting a CRM to an RCS API

To connect a CRM (such as HubSpot, Salesforce, or a custom Laravel/Node.js CRM) to an RCS API, establish an outbound HTTP client service.

Example: Compiling & Sending an RCS Rich Card from CRM Data

Below is a Node.js / Express snippet demonstrating how a backend CRM workflow controller fetches customer attributes and dispatches an RCS card via an API gateway:

const axios = require('axios');

// Triggered by a CRM Webhook when a deal moves to 'Contract Sent'
async function sendRcsContractNotification(crmContactData) {
  const { phoneNumber, firstName, dealName, contractUrl, dealId } = crmContactData;

  const rcsPayload = {
    to: phoneNumber,
    agentId: "your-brand-agent-id",
    content: {
      richCard: {
        standaloneCard: {
          cardOrientation: "VERTICAL",
          cardContent: {
            title: `Hello ${firstName}! Your Proposal is Ready 📄`,
            description: `Review the details for "${dealName}". Tap below to inspect or sign the contract directly.`,
            media: {
              height: "MEDIUM",
              contentInfo: {
                fileUrl: "[https://cdn.yourdomain.com/assets/contract-banner.jpg](https://cdn.yourdomain.com/assets/contract-banner.jpg)"
              }
            },
            suggestions: [
              {
                action: {
                  text: "Review Proposal",
                  postbackData: `action=review_proposal&deal_id=${dealId}`,
                  openUrlAction: {
                    url: contractUrl
                  }
                }
              },
              {
                reply: {
                  text: "Request Revision",
                  postbackData: `action=request_revision&deal_id=${dealId}`
                }
              }
            ]
          }
        }
      }
    }
  };

  try {
    const response = await axios.post('[https://api.messaging-provider.com/v1/rcs/messages](https://api.messaging-provider.com/v1/rcs/messages)', rcsPayload, {
      headers: {
        'Authorization': `Bearer ${process.env.RCS_API_TOKEN}`,
        'Content-Type': 'application/json'
      }
    });

    // Log outbound message ID back to CRM Timeline
    await logCrmActivity(dealId, `Outbound RCS Contract Sent. Message ID: ${response.data.messageId}`);
  } catch (error) {
    console.error('RCS Dispatch Error:', error.response ? error.response.data : error.message);
  }
}

Using Webhooks for RCS Responses

When a customer taps an action button or types a reply, the RCS gateway issues an asynchronous HTTP POST request to your application's public webhook endpoint.
Server-Side Webhook Receiver Endpoint:

const express = require('express');
const app = express();
app.use(express.json());

app.post('/webhooks/rcs-crm-sync', async (req, res) => {
  const { sender, userResponse } = req.body;

  if (userResponse && userResponse.postbackData) {
    const params = new URLSearchParams(userResponse.postbackData);
    const action = params.get('action');
    const dealId = params.get('deal_id');

    if (action === 'request_revision') {
      // 1. Update CRM Deal Stage
      await updateCrmDealStage(dealId, 'Revision Requested');

      // 2. Assign follow-up task to Deal Owner inside CRM
      await createCrmTask({
        dealId: dealId,
        taskName: `Client requested contract revision via RCS. Phone: ${sender}`,
        priority: 'HIGH'
      });

      // 3. Send automated confirmation back to the user via RCS
      await sendRcsReply(sender, {
        text: "We've notified your account manager about the requested changes. They will reach out shortly!"
      });
    }
  }

  // Acknowledge webhook reception immediately
  res.status(200).send('EVENT_RECEIVED');
});

app.listen(3000, () => console.log('RCS-CRM Webhook Sync running on port 3000'));

RCS CRM Automation Use Cases

Connecting RCS APIs to CRM workflow engines enables event-driven triggers across the customer lifecycle:

Automated Customer Follow-Ups When a web lead remains inactive for 48 hours, the CRM fires an automated RCS carousel displaying recommended services, accompanied by a "Schedule Call" action button.
Order & Shipment Notifications When an Order Management System (OMS) linked to the CRM updates an order status to "Out for Delivery", an automated RCS rich card delivers live driver tracking and drop-off instructions.
Appointment Reminders & Rescheduling Send a booking confirmation 24 hours prior to an appointment. Tapping a "Reschedule" button initiates an automated quick-reply flow that queries CRM calendar availability directly inside the messaging thread.
Interactive Support Escalations If a customer submits a high-priority ticket, the CRM sends a verified RCS notification containing a "Upload Photo" button, allowing users to submit proof of issue directly into the ticket record.

Common RCS CRM Integration Challenges

Database Identity Matching: Phone numbers in CRMs must be stored in standardized E.164 format (e.g., +14155552671) to properly match inbound webhook events to contact profiles.
Handling Offline / Non-RCS Devices: Not every phone or carrier supports RCS. Ensure your integration architecture includes automated SMS/MMS fallback rules within the API pipeline.
Rate Limits & Batch Broadcasting: Sending high-volume RCS campaigns directly from a CRM can trigger API rate limits. Queue outbound requests using redis-backed job queues (e.g., BullMQ, Celery).

Best Practices for RCS CRM Integration

Keep postbackData Clean & Structured: Format postback strings as query parameters (action=confirm&ticket_id=901) for simple server-side string parsing.
Verify Webhook Hashes: Cryptographically validate incoming webhook headers (e.g., X-RBM-Signature) against your API secret to prevent unauthorized payload injection.
Maintain Bi-Directional State Sync: Ensure every outbound message and inbound response is appended to the CRM record so sales and support teams have a complete transcript.
Enforce Fallback Routing: Always configure an explicit fallback template (SMS/WhatsApp) within your CPaaS gateway to guarantee message delivery when RCS is unreachable.

Conclusion
Integrating RCS messaging with CRM software transforms messaging from a passive, unverified broadcast channel into an interactive, real-time extension of your application. By pairing rich media cards and postbacks with automated CRM workflows, developers can build responsive communication systems that drive engagement, accelerate pipeline velocity, and streamline support workflows.

About the Author
This technical integration overview was published by the engineering team at Software Solutions — specializing in custom backend systems, enterprise web applications, and multi-channel API integration architecture.

How to Implement RCS Messaging for Your Business: Architecture, APIs, & Webhooks

Software Solutions — Tue, 22 Sep 2026 08:38:07 +0000

As enterprise communication evolves beyond legacy SMS, Rich Communication Services (RCS) has become the gold standard for native mobile messaging. For developers, systems architects, and technical product managers, implementing RCS Business Messaging (RBM) is fundamentally different from sending basic transactional text messages over SS7 signaling. RCS requires configuring verified brand agents, establishing RESTful API pipelines, building structured JSON card payloads, and handling asynchronous inbound webhooks.

This guide provides a comprehensive technical breakdown for architecting, integrating, and deploying RCS Business Messaging within your application infrastructure.

Before diving into the code and API calls, it helps to understand the broader ecosystem, features, and commercial trade-offs. You can review our overview on RCS Messaging for Businesses: A Complete Guide.

What You Need Before Implementing RCS

To build and deploy a production-ready RCS messaging pipeline, your application stack and organization require several core prerequisites:

A Verified RCS Agent (Business Profile): Unlike standard SMS where you rent a numeric long code or shortcode, RCS relies on an official Business Profile (Agent). This profile includes your brand name, logo (PNG/JPEG, 1:1 ratio, min 224x224px), hero banner image, privacy policy URL, and terms of service.
CPaaS / Aggregator API Credentials: Direct access to Tier-1 carrier networks or CPaaS messaging providers (e.g., Google RBM, Twilio, Infobip, Sinch) that expose REST APIs and webhook interfaces.
Public HTTPS Webhook Endpoint: An SSL-secured web server endpoint capable of receiving and parsing asynchronous JSON POST payloads sent by the RCS API gateway.
Backend Event Triggers: An application infrastructure (Node.js, PHP/Laravel, Python, Java) connected to your database, CRM, or Order Management System (OMS) to fire outbound notification events.

How RCS Business Messaging Works

Under the hood, RCS operates over IP data networks (WiFi or 4G/5G mobile data) using SIP/SIMPLE protocols and HTTPS REST APIs rather than legacy cellular signaling channels.

When your application initiates an RCS message, the request travels via HTTPS to an RCS CPaaS API gateway. The gateway queries the carrier’s Capability Discovery API to check if the target recipient's phone number and active messaging client support RCS.

[ Outbound App Event ]
             │
             ▼
 [ RCS Provider Gateway ]
             │
  (Capability Discovery)
   /                   \
(RCS Supported)       (RCS Unsupported)
/

[ Rich Card Delivered ]   [ Automatic SMS Fallback ]

If RCS is active: The gateway renders the full rich card, carousel, or button payload on the recipient's device.
If RCS is unavailable: The gateway automatically falls back to standard SMS/MMS, delivering a plain-text version of your message to guarantee receipt.

Choosing an RCS Messaging Provider

When evaluating an RCS API gateway or CPaaS vendor, assess these core technical capabilities:

Direct RBM Carrier Connectivity: Ensures high message throughput (TPS) and low latency.
Granular Fallback Logic: Configurable automated fallback to SMS, WhatsApp, or email if the user is offline or lacks RCS device support.
Webhook Reliability: Retry mechanisms and cryptographic signature headers (e.g., HMAC SHA-256) for inbound webhook event processing.
Template & Payload Validation: Visual design studios or raw JSON schema builders for configuring card layouts and action buttons.

Technical Workflow & System Architecture

Architecturally, RCS sits as a bi-directional communication layer between your internal software services and the customer's native messaging application.

End-to-End System Data Flow:

┌────────────────────────────────────────┐
│          Business Application          │ ◄── (Cron Jobs / Event Listeners)
└───────────────────┬────────────────────┘
│
▼
┌────────────────────────────────────────┐
│         CRM / Business Software        │ ──► Triggers Outbound Event
└───────────────────┬────────────────────┘
│
▼  [ HTTPS POST API Call ]
┌────────────────────────────────────────┐
│                RCS API                 │
└───────────────────┬────────────────────┘
│
▼  [ IP Carrier RBM Gateway ]
┌────────────────────────────────────────┐
│        RCS Business Messaging          │
└───────────────────┬────────────────────┘
│
▼  [ IP Delivery ]
┌────────────────────────────────────────┐
│                Customer                │
└───────────────────┬────────────────────┘
│
▼  [ User Taps Button / Replies ]
┌────────────────────────────────────────┐
│            Response / Event            │
└───────────────────┬────────────────────┘
│
▼  [ Asynchronous Payload ]
┌────────────────────────────────────────┐
│                Webhook                 │
└───────────────────┬────────────────────┘
│
▼  [ Update Database State ]
┌────────────────────────────────────────┐
│          Business Application          │
└────────────────────────────────────────┘

Setting Up RCS Business Messaging

Setting up RBM requires registering your brand agent within your chosen CPaaS partner portal.

Submit Agent Details: Upload your high-resolution logos, brand color schemes, and legal policy URLs.
Domain Verification: Prove ownership of your brand domain by adding TXT records to your DNS settings.
Carrier Review & Approval: Brand agents undergo manual verification by mobile network operators to prevent spam and phishing (smishing).

Integrating an RCS API

Outbound messages are triggered by issuing authenticated HTTPS POST requests to the provider's endpoint. You pass recipient details, sender agent IDs, and structured card components within the request body.

Creating RCS Message Templates

RCS supports three primary UI structures:

Text Messages with Suggested Chip Replies: Standard text accompanied by quick-reply buttons (e.g., "Confirm", "Reschedule").
Standalone Rich Cards: Single cards containing a header image/video, title, description, and up to 4 action buttons.
Carousels: Horizontal swipeable arrays containing up to 10 rich cards, ideal for product catalogs or multi-item order tracking.

Sending RCS Messages Through an API

Below is an example payload for sending a standalone rich card containing an order update and interactive action buttons.

Example REST API Request (cURL):

curl -X POST [https://api.messaging-provider.com/v1/rcs/messages](https://api.messaging-provider.com/v1/rcs/messages) \
  -H "Authorization: Bearer YOUR_API_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+12345678901",
    "agentId": "your-brand-agent-id",
    "content": {
      "richCard": {
        "standaloneCard": {
          "cardOrientation": "VERTICAL",
          "cardContent": {
            "title": "Order #89201 Shipped! 📦",
            "description": "Your package has been dispatched via Express Delivery and is estimated to arrive tomorrow.",
            "media": {
              "height": "MEDIUM",
              "contentInfo": {
                "fileUrl": "[https://cdn.yourdomain.com/assets/shipping-preview.jpg](https://cdn.yourdomain.com/assets/shipping-preview.jpg)"
              }
            },
            "suggestions": [
              {
                "action": {
                  "text": "Track Package",
                  "postbackData": "action=track&order_id=89201",
                  "openUrlAction": {
                    "url": "<a href="https://yourdomain.com/track/89201">https://yourdomain.com/track/89201</a>"
                  }
                }
              },
              {
                "reply": {
                  "text": "Change Address",
                  "postbackData": "action=change_address&order_id=89201"
                }
              }
            ]
          }
        }
      }
    }
  }'

Using Webhooks for Responses and Events

When a user taps an action button, selects a suggested reply, or types a text response, the RCS gateway posts an event to your HTTP webhook listener.
Inbound Webhook JSON Payload Example:

{
  "eventId": "evt_987654321",
  "agentId": "your-brand-agent-id",
  "sender": "+12345678901",
  "timestamp": "2026-09-22T14:32:10Z",
  "userResponse": {
    "type": "POSTBACK",
    "postbackData": "action=change_address&order_id=89201",
    "text": "Change Address"
  }
}

Server-Side Webhook Handler (Node.js / Express Example):

const express = require('express');
const app = express();
app.use(express.json());

app.post('/webhooks/rcs-inbound', async (req, res) => {
  const { sender, userResponse } = req.body;

  if (userResponse && userResponse.postbackData) {
    const params = new URLSearchParams(userResponse.postbackData);
    const action = params.get('action');
    const orderId = params.get('order_id');

    if (action === 'change_address') {
      // 1. Trigger internal CRM / OMS workflow
      await initiateAddressChangeWorkflow(sender, orderId);

      // 2. Dispatch a follow-up response via RCS API
      await sendRcsReply(sender, {
        text: `Please reply with your new delivery address for Order #${orderId}.`
      });
    }
  }

  // Acknowledge receipt immediately with HTTP 200
  res.status(200).send('OK');
});

app.listen(3000, () => console.log('RCS Webhook Server running on port 3000'));

Connecting RCS With Your CRM or Business Software

Integrating RCS into systems like HubSpot, Salesforce, or custom internal admin portals requires two-way data synchronization:

Outbound Trigger Sync: Map CRM lifecycle stage changes (e.g., "Lead Created", "Ticket Resolved") to API POST calls.
Inbound Conversation Logging: Ensure incoming webhook messages are attached directly to the contact's activity timeline inside your database so support agents have full context.

Testing an RCS Integration

Before releasing your RCS integration to production:

Register Test Devices: Add developer phone numbers to your CPaaS partner dashboard to test unapproved agent profiles.
Validate Payload Schemas: Test card aspect ratios, image file sizes, and URL protocol formatting (https:// is mandatory).
Verify SMS Fallback: Send test messages to non-RCS devices (or disable data connections) to ensure fallback SMS templates render cleanly.
Stress Test Webhook Listeners: Ensure your HTTP webhook endpoints handle high-volume event bursts during batch broadcasts.

Common RCS Implementation Challenges

Unverified Agent Rejection: Brand profiles submitted with poor-resolution logos or missing legal terms will fail carrier verification.
Payload Truncation: Button text labels have strict length limits (typically 25 characters max); exceeding limits causes rendering errors.
Session Management: Treating RCS as one-way bulk SMS ignores postback events, resulting in ignored user button taps.

RCS Implementation Best Practices

Keep Postbacks Structured: Store state in postbackData strings using key-value query parameters (action=verify&user_id=882).
Optimize Media Assets: Compress header images and use standard 16:9 or 2:1 aspect ratios for fast rendering.
Implement Robust Signature Verification: Verify the signature header on inbound webhooks to prevent spoofing attacks.
Graceful Human Handoff: Ensure automated quick-reply trees include an explicit "Talk to Agent" option.

Conclusion

Implementing RCS messaging bridges the gap between passive SMS notifications and interactive app experiences. By establishing clean REST API pipelines, structured JSON card templates, and event-driven webhook handlers, developers can build reliable, verified, and interactive communication systems for modern applications.

About the Author

This technical implementation guide was written by the engineering team at Software Solutions — specializing in custom backend systems, enterprise web application development, and API integration services.

Why Skill Differs from Tool: Self-Evolution and Dynamic Loading Mechanism for Agent Skill Libraries

Tidiane Stano — Tue, 22 Sep 2026 08:31:46 +0000

Introduction

Many developers confuse Tool and Skill in AI Agent development. A simple analogy helps clarify this distinction: a Tool is a screwdriver, while a Skill is the assembly manual that tells you which screw to turn and in what order.

Tools are atomic, generic capabilities with no inherent business context. A Skill, by contrast, is a human-curated package that encapsulates business standard operating procedures (SOPs). Simply stuffing 50 Tools into an Agent will overwhelm it and easily trigger dead loops. The correct way to handle complex long-running tasks is to dynamically load high-quality Skills by matching task routing rules.

In the middle-to-late phases of self-built Agent projects, teams often run into a frustrating problem. Engineers build dozens or even hundreds of refined MCP Tools, including Git operations, Kubernetes scheduling, database CRUD and network diagnosis. But when assigning real business tasks, for example, investigating slow user login latency and generating a full incident report, Agents frequently fall into three typical failure modes.

Failure Phenomenon	Specific Manifestation	Root Architectural Cause
Tool Paralysis	When faced with roughly 50 candidate tools, the model repeatedly picks incorrect tools or generates invalid parameter sets	Excessive tool descriptions pollute the prompt, causing severe attention dilution
Lack of Business SOP	The Agent has access to log query tools, but it does not know whether to first inspect gateway metrics or database records; it attempts blind trial and error	Only atomic execution functions exist, with no expert-guided workflow for diagnosis
No Skill Evolution	After successfully completing a complex troubleshooting workflow once, the Agent restarts from scratch for identical follow-up incidents	Tools are stateless one-off operations, with no mechanism for experience solidification and skill iteration

To resolve these three core pain points, the architecture must introduce the Skill layer. This article explains the essential boundary between Tool and Skill, and demonstrates how Skill serves as code-backed procedural memory for Agents. It also covers the standardized SKILL.md specification, two-phase dynamic loading design, and production-grade Python implementation for self-evolving skill engines.

1. Tool vs Skill: Core Definition and Comparison Matrix

We need to fully separate Tool and Skill at the conceptual level of system design.

Comparison Dimension	Tool (Atomic Primitive)	Skill (Expert Knowledge / SOP)
Core Definition	Atomic operation instruction without business semantics	Knowledge package that contains domain rules and execution workflows
Typical Format	Executable functions, MCP Server API	Markdown specification, prompt templates, scripts
Cognitive Layer	Execution (hands and limbs)	Cognition (muscle memory and operational norms)
State & Evolution	Static, hardcoded by engineers	Dynamically accumulated and self-evolved by the Agent
Complexity & Scope	Single discrete action, such as `exec_sql(query)`	Multi-step workflow, e.g., full K8s pod troubleshooting SOP
Metaphor	Surgical knife, suture thread	Complete step-by-step surgical operation manual

A Tool executes atomic operations. It accepts input A and returns output B, with no understanding of business objectives.
A Skill defines domain procedures. It guides the Agent to combine multiple Tools sequentially and conditionally once a trigger condition is matched. It also includes pitfall warnings and acceptance criteria.

In short: Tools define what can be done, while Skills specify how to do it. A K8s troubleshooting Skill, for example, chains together kubectl_get_pods, kubectl_logs, and query_prometheus tools in a fixed order, with pre-check rules and validation gates built into the workflow.

2. Standard SKILL.md Specification and On-Demand Dynamic Loading Architecture

For enterprise-grade Agents such as Hermes Agent and Claude Code, developers cannot inject all Skills fully into the global prompt. Instead, a two-phase dynamic loading architecture is required.

2.1 Industrial Template for SKILL.md

Each Skill is defined in a Markdown frontmatter file named SKILL.md. The frontmatter stores metadata, while the main body contains SOP steps, trigger scenarios, pitfalls, and verification standards.

---
name: k8s-pod-troubleshooting
description: "Use when K8s pods are in CrashLoopBackOff, Pending, or OOMKilled states."
version: 1.0.0
author: Alben
category: devops
---
# K8s Pod Troubleshooting Standard Operating Procedure
## Trigger Scenario
When users report service exceptions, pod restarts, or health check failures, load this skill.
## Standard Procedure
1. **Initial Inspection**: Run `kubectl get pods` to identify pods with abnormal status.
2. **Event Diagnosis**: Use `kubectl describe` to check pod events and capture OOM or scheduling warnings.
3. **Log Retrieval**: Fetch container logs and compare pod resource limits with Prometheus metrics.
## Pitfall Guidance
- Never run `kubectl delete pod` before confirming root causes.
- When liveness probe failures occur, inspect ReadinessProbe configuration first.
## Acceptance Criteria
After intervention, continuously observe pod status for 30 seconds. Confirm the pod enters READY state and restart counts stop increasing.

2.2 Two-Phase Loading Mechanism

The two-phase design is the key to scaling up to hundreds of Skills without blowing up prompt context.

Indexing Phase: The engine scans the skill directory, parses metadata and short descriptions of every SKILL.md file. It builds a lightweight summary index and embeds only these short summaries into the system prompt. The full SOP content is not loaded at this stage.
Full Load Phase: When the Agent judges the current task matches a Skill’s trigger condition, the engine loads the complete SOP, pitfalls and validation rules from the corresponding SKILL.md into the prompt for that task session.

This design avoids prompt bloat. The model only sees brief skill summaries most of the time. Full skill details are fetched only when relevant.

3. Production-Grade Code Implementation: Skill Dynamic Management and Self-Evolution Engine

The following Python 3.11 implementation builds a skill runtime lifecycle engine. It handles metadata parsing, lightweight index generation, on-demand skill loading, and automatic skill crystallization after successful troubleshooting.

"""
skill_runtime_engine.py
Production-grade Skill dynamic loader and self-evolution manager
Modules: YAML frontmatter parser, metadata extraction, index generation, on-demand loading, auto crystallization
"""
import os
from typing import Dict, Optional
from pydantic import BaseModel
import yaml
import re

class SkillMetadata(BaseModel):
    """Skill metadata model"""
    name: str
    description: str
    version: str = "1.0.0"
    category: str = "general"
    file_path: str

class SkillRuntimeManager:
    """Runtime lifecycle manager for enterprise Skills"""
    def __init__(self, skills_dir: str):
        self.skills_dir = skills_dir
        self.skill_cache: Dict[str, SkillMetadata] = {}
        self._scan_and_index_skills()

    def _scan_and_index_skills(self):
        """Scan directory and build lightweight skill index by parsing Markdown frontmatter"""
        self.skill_cache.clear()
        if not os.path.exists(self.skills_dir):
            os.makedirs(self.skills_dir, exist_ok=True)
            return
        for root, _, files in os.walk(self.skills_dir):
            for filename in files:
                if not filename.endswith(".md"):
                    continue
                full_path = os.path.join(root, filename)
                meta = self._parse_frontmatter(full_path)
                if meta:
                    self.skill_cache[meta.name] = meta

    def _parse_frontmatter(self, file_path: str) -> Optional[SkillMetadata]:
        """Parse Markdown YAML frontmatter"""
        try:
            with open(file_path, "r", encoding="utf-8") as f:
                md_text = f.read()
            match = re.search(r"^---\s*\n(.*?)\n---", md_text, re.DOTALL)
            if not match:
                return None
            fm_data = yaml.safe_load(match.group(1))
            return SkillMetadata(
                name=fm_data.get("name", os.path.basename(file_path)),
                description=fm_data.get("description", ""),
                version=fm_data.get("version", "1.0.0"),
                category=fm_data.get("category", "general"),
                file_path=file_path
            )
        except Exception as e:
            print(f"Failed to parse skill at {file_path}: {e}")
            return None

    def generate_system_prompt_index(self) -> str:
        """Generate lightweight system prompt index without loading full skill content"""
        if not self.skill_cache:
            return ""
        lines = ["<available_skills>"]
        for name, meta in self.skill_cache.items():
            short_desc = meta.description[:60] if len(meta.description) >60 else meta.description
            lines.append(f"- {name}: {short_desc}")
        lines.append("</available_skills>")
        lines.append("Use `skill_view(name)` to load full SOP when task matches skill trigger.")
        return "\n".join(lines)

    def load_skill_content(self, skill_name: str) -> str:
        """Load full skill SOP by skill name"""
        meta = self.skill_cache.get(skill_name)
        if not meta:
            return f"Error: Skill `{skill_name}` not found."
        try:
            with open(meta.file_path, "r", encoding="utf-8") as f:
                return f.read()
        except Exception as e:
            return f"Reading skill failed: {e}"

    def auto_crystallize_skill(self, name:str, description:str, category:str, skill_body:str):
        """Auto-generate new SKILL.md after completing a successful task workflow"""
        skill_file = os.path.join(self.skills_dir, f"{name}.md")
        full_doc = f"""---
name: {name}
description: {description}
category: {category}
version: 1.0.0
---
{skill_body}
"""
        with open(skill_file, "w", encoding="utf-8") as f:
            f.write(full_doc)
        self._scan_and_index_skills()
        return f"Successfully crystallized skill `{name}`"

The core auto_crystallize_skill function implements the self-evolution mechanism. Once the Agent successfully finishes a complex task, it can summarize the complete workflow, pitfalls and validation rules, then persist this experience into a new SKILL.md file. The new skill will be indexed and available for future tasks. This turns one-off successful execution into reusable procedural memory.

4. Architecture Summary and Engineering Takeaways

Three core conclusions can be drawn from this practice:

Lightweight Indexing + Full Content On-Demand Loading: This is the critical architecture that enables scaling to hundreds of Skills. It avoids prompt inflation and attention collapse caused by loading all 50+ Tools and SOPs into context at once.
Standardized SKILL.md specification: Each skill must define triggers, execution steps, pitfalls, and acceptance criteria. This standardizes knowledge entry and version control for domain workflows.
Skill Auto-Crystallization: Agents can solidify successful task workflows into new skills automatically. This gives the system continuous self-improvement capability.

This separation of Tool and Skill fundamentally changes Agent design. Tools provide raw action primitives, while Skills package human domain expertise. For enterprise Agent pipelines that integrate multiple model backends and tool endpoints, 4sapi can act as an API gateway to standardize request routing and credential management across different services.

When building production Agent systems, teams should stop adding more Tools blindly. Instead, they should encapsulate proven multi-step workflows into Skills, adopt two-phase dynamic loading, and enable auto-crystallization to let the Agent accumulate experience continuously. This approach drastically reduces tool paralysis and stabilizes long complex task execution.

Conclusion

Tool and Skill serve two distinct layers in Agent architecture. Tools are atomic executable primitives. Skills are domain SOP packages that define sequential logic, failure prevention and acceptance standards. The two-phase dynamic loading architecture prevents prompt overflow, while auto-crystallization allows Agents to grow their own skill library as they complete real-world tasks. This pattern is especially suitable for enterprise scenarios such as devops troubleshooting, data analysis and business process automation.

When orchestrating multi-model, multi-tool Agent services, a unified API gateway simplifies endpoint management and traffic control. 4sapi provides a unified interface to manage distributed model and tool API endpoints for complex Agent workflows.

International access: https://4sapi.com
Domestic access: https://4sapi.cn

How to Resize an Image by Exact Pixels or by Percentage via API

PDF4me — Tue, 22 Sep 2026 08:09:52 +0000

Ask five developers what "resize an image" means and you will get five different answers. One means shrinking a 4000px camera photo down to a 200px thumbnail. Another means scaling every image in a batch to 80% so a page loads faster. A third means forcing every upload into an exact 600x600 box regardless of what came in. These are not the same operation, and the way most image libraries handle them (one function, a dozen optional flags, an aspect ratio bug waiting to happen) is exactly why resize logic tends to accumulate as one of those "someone wrote this three years ago and nobody wants to touch it" corners of a codebase.

PDF4me's Resize Image endpoint treats the two real-world resize methods as two real, distinct parameters instead of one overloaded function, and that distinction is worth understanding before you wire it into anything.

Percentage and pixels are different problems, not two flags on the same one

Resize by percentage answers "make this smaller, proportionally, without me having to know its exact dimensions." A batch of product photos that arrive at wildly different resolutions can all get scaled down by 50% and come out proportionally consistent, whatever their starting size.

Resize by exact pixel dimensions answers a completely different question: "I need this to be 600 by 400, full stop." That is the shape of the problem when a destination has a hard requirement: a thumbnail grid that expects uniform tiles, a CMS field with a fixed image slot, a print template with a defined canvas. Asking a single "resize" call to guess which one you meant is how you end up with distorted product photos or under-sized thumbnails nobody caught in code review.

PDF4me's endpoint exposes both as first-class options via the ImageResizeType field, which takes either Percentage or Specific. You pick the method that matches the actual problem instead of coercing one flag to do both jobs.

What the request actually looks like

The REST call is a POST to /api/v2/ResizeImage, and the request body is unremarkable in the way a well-designed document API's request body should be. Here is the field-by-field shape, verified against the official pdf4me-api-samples Python sample rather than just the marketing-page example:

import os
import base64
import requests
import time

def resize_image():
    api_key = "get the API key from https://dev.pdf4me.com/dashboard/#/api-keys"
    image_file_path = "sample.jpg"
    output_path = "Resize_image_output.jpg"

    base_url = "https://api.pdf4me.com"
    url = f"{base_url}/api/v2/ResizeImage"

    with open(image_file_path, "rb") as f:
        image_content = f.read()
    image_base64 = base64.b64encode(image_content).decode('utf-8')

    payload = {
        "docName": os.path.basename(image_file_path),
        "docContent": image_base64,
        "ImageResizeType": "Percentage",       # or "Specific"
        "ResizePercentage": "50.0",
        "Width": 800,                          # used when ImageResizeType is "Specific"
        "Height": 600,                         # used when ImageResizeType is "Specific"
        "MaintainAspectRatio": True,
        "isAsync": True
    }

    headers = {
        "Authorization": f"Basic {api_key}",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers, timeout=300)

    if response.status_code == 200:
        # Synchronous completion
        with open(output_path, "wb") as f:
            f.write(response.content)
    elif response.status_code == 202:
        # Asynchronous processing: poll the Location header until done
        location_url = response.headers.get('Location')
        for attempt in range(20):
            time.sleep(10)
            poll = requests.get(location_url, headers=headers)
            if poll.status_code == 200:
                with open(output_path, 'wb') as out_file:
                    out_file.write(poll.content)
                break
            elif poll.status_code != 202:
                print(f"Error: {poll.status_code} - {poll.text}")
                break

A few things worth calling out that are easy to miss from the docs page alone: ImageResizeType takes exactly two values, Percentage or Specific, and both Width/Height and ResizePercentage are sent in every request regardless of which mode you use, the endpoint just ignores whichever pair doesn't apply to the selected type. The isAsync flag is real and documented in the official sample even though the docs page's own JSON example doesn't show it: set it to true and a large or slow-processing image returns a 202 with a Location header to poll instead of holding the connection open, the same async pattern used across PDF4me's other endpoints. Authentication uses Authorization: Basic {api_key}, and the response for a synchronous 200 is the resized image itself as binary content, not a JSON wrapper.

The aspect ratio setting nobody reads until something looks wrong

Here is the setting that actually decides whether your resized image looks correct or looks broken: MaintainAspectRatio. Leave it true and a percentage or pixel resize scales width and height together, so a landscape photo stays a landscape photo, just smaller. Set it false, or set a target width and height that do not match the source's proportions, and you get exactly what you asked for: an image forced into a box, stretched or squashed to fit.

There is a real use case for both. A thumbnail grid that needs every tile to be a literal square wants MaintainAspectRatio: false and a fixed pixel target, because uniformity is the point. A hero image being scaled down for a slower connection wants it locked true, because nobody wants a portrait photo of a person rendered as a slightly wider person. The mistake is not knowing which one your integration is set to, and finding out only when a customer screenshots a warped logo.

Resizing without touching the REST payload yourself

Not every team wants to own a resize function, and PDF4me's no-code integrations exist for exactly that reason. In Power Automate, the Resize Image action takes the same percentage-or-dimensions choice and the same aspect ratio control, dropped into a flow alongside whatever triggers it: a new file landing in SharePoint, an email attachment, a form submission, with batch processing built into the action itself.

Zapier ships the same capability under the name "Smart Scaler," and if you are already routing images through a Zap, inserting resize as a middle step means you never write image-processing code at all.

n8n's node is aimed squarely at the two jobs developers actually reach for it for: generating thumbnails on the fly and normalizing a pile of inconsistent uploads into one predictable size, both by percentage or exact pixels, with aspect ratio preserved by default.

Make covers percentage-based scaling for jobs like thumbnail generation and web optimization, with one caveat worth knowing before you build around it: there is no dedicated batch-resize mode in the module itself. Resizing more than one file means wrapping the module in a Make Iterator and letting it run the same percentage setting once per image. If your scenario already resizes a folder of files one at a time, this is not a limitation you will notice. If you are picturing a single-step bulk operation, plan for the Iterator up front rather than after your first test run comes up short.

Test the exact request before you build a pipeline around it

Before wiring resize into a workflow that will run unattended, confirm the exact request and response shape against a real image using PDF4me's feature-specific Resize Image API Tester (or the general API Tester for any other endpoint), which sends live requests from the browser and returns the actual response, no code required. This matters more for resize than it might for a simpler endpoint: aspect ratio behavior, the decimal format for percentage values, and how a non-square pixel target actually renders are all things that are faster to see once, live, than to debug after the fact in a batch job.

Where this actually gets used

The pattern that comes up most is normalization: user uploads arrive in every resolution imaginable, and a fixed downstream requirement (a thumbnail grid, a storage budget, a CMS image slot) needs them all brought to one size before anything else touches them. The second most common pattern is bandwidth: serving a smaller percentage-scaled version of a large photo to a page that does not need full resolution. A third, less obvious pattern is compliance with a third-party spec: marketplaces, print vendors, and ad networks routinely publish exact pixel requirements for submitted images, and rejecting a file for being the wrong size is a worse customer experience than resizing it automatically before it ever gets uploaded.

None of these needs a dedicated image-processing service, a native image library your team now has to patch for security updates, or a homegrown wrapper around one. It is one endpoint, or one no-code action, doing one job correctly, and because it sits on the same PDF4me surface as the rest of the image and document toolset, a resize step chains naturally into a larger pipeline without switching services or re-authenticating partway through.

Getting started

If you have not connected to the PDF4me API before, the Connect to PDF4me API guide covers authentication, API keys, and response codes, and is the fastest path to your first successful call, resize or otherwise.

Website: pdf4me.com
Documentation: docs.pdf4me.com

CF7 to Freshdesk Tickets Not Being Created - Four Causes and Exact Fixes

Rahul Sharma — Tue, 22 Sep 2026 08:08:23 +0000

A developer documented a real CF7 to Freshdesk integration in a 2017 tutorial. The comments section became a support thread multiple people following the same code reported that "it just sends the email but no activity on Freshdesk." The author's debugging advice: use Postman to test the API directly, then print the PHP array to check the values.

That debugging advice is still the right starting point. Here are the four causes that produce the same symptom form submits, email arrives, no Freshdesk ticket.

Cause 1: Wrong Domain Format in the API URL

Freshdesk API calls require your full subdomain in the URL:

https://YOUR_SUBDOMAIN.freshdesk.com/api/v2/tickets

Where YOUR_SUBDOMAIN is the part before .freshdesk.com in your Freshdesk URL. If your Freshdesk account is at acmehelp.freshdesk.com, the API endpoint is:

https://acmehelp.freshdesk.com/api/v2/tickets

The most common mistake is entering just the subdomain (acmehelp) without the full domain, or using freshdesk.com without the subdomain prefix. Both produce a URL that either does not resolve or returns a generic Freshdesk error page rather than an API response.

Test your domain format:

curl -v "https://YOUR_SUBDOMAIN.freshdesk.com/api/v2/tickets" \
  -u "YOUR_API_KEY:X"

If you see a JSON response (even a 401), the URL format is correct. If you see an HTML page or a connection error, the domain format is wrong.

Cause 2: API Key Authentication Format Is Not Standard

Freshdesk uses HTTP Basic authentication but with a non-standard format. The username is your API key and the password is literally the letter X:

Authorization: Basic base64(YOUR_API_KEY:X)

This is different from most APIs. The API key goes in the username position. The password is always X not your actual password, not empty, literally the single character X.

Using wp_remote_post in PHP:

'headers' => [
    'Authorization' => 'Basic ' . base64_encode(FRESHDESK_API_KEY . ':X'),
    'Content-Type'  => 'application/json',
],

If you enter your email and password in the Basic auth fields instead of API_KEY:X, authentication will fail with a 401 even if the credentials are otherwise correct.

Find your Freshdesk API key: In Freshdesk, click your profile picture at the top right, then Profile Settings. The API Key appears at the bottom right of the page.

Cause 3: Required Fields Missing from the Ticket Payload

Freshdesk requires at minimum two fields for every ticket:

subject — the ticket title/subject line
email — the requester's email address

Without both, Freshdesk returns a 422 Unprocessable Entity with a validation error. If your plugin or custom code maps only the message body or only the name, the ticket creation fails.

A minimal valid Freshdesk ticket payload:

{
  "subject": "Enquiry from Jane Smith",
  "description": "Message content here",
  "email": "jane@example.com",
  "status": 2,
  "priority": 1
}

Freshdesk status codes:

1 = Open
2 = Pending
3 = Resolved
4 = Closed

Freshdesk priority codes:

1 = Low
2 = Medium
3 = High
4 = Urgent

For most CF7 contact form tickets, status: 2 (Pending) and priority: 1 (Low) are appropriate defaults.

Cause 4: The Hook `wpcf7_mail_sent` vs `wpcf7_before_send_mail`

The original tutorial code used wpcf7_mail_sent as the hook:

add_action('wpcf7_mail_sent', 'cf7_create_freshdesk_ticket', 0, 1);

wpcf7_mail_sent fires only after CF7 has successfully sent its notification email. If CF7's mail sending fails for any reason — SMTP configuration issues, email blocked by spam filter, missing mail configuration — this hook never fires and no Freshdesk ticket is created.

The more reliable hook for API integrations is wpcf7_before_send_mail:

add_action('wpcf7_before_send_mail', 'cf7_create_freshdesk_ticket');

This fires during the submission processing regardless of whether the email is sent successfully. For use cases where the Freshdesk ticket should always be created when someone submits the form, not only when the email delivery succeeds, use wpcf7_before_send_mail.

Complete CF7 to Freshdesk Implementation

add_action('wpcf7_before_send_mail', 'cf7_create_freshdesk_ticket');

function cf7_create_freshdesk_ticket($contact_form) {
    if ((int) $contact_form->id() !== YOUR_FORM_ID) return;

    $submission = WPCF7_Submission::get_instance();
    if (!$submission) return;

    $data      = $submission->get_posted_data();
    $name      = sanitize_text_field($data['your-name']    ?? '');
    $email     = sanitize_email($data['your-email']        ?? '');
    $message   = sanitize_textarea_field($data['your-message'] ?? '');

    if (empty($email)) return;

    $api_key   = defined('FRESHDESK_API_KEY')    ? FRESHDESK_API_KEY    : '';
    $subdomain = defined('FRESHDESK_SUBDOMAIN')  ? FRESHDESK_SUBDOMAIN  : '';

    $response  = wp_remote_post(
        "https://{$subdomain}.freshdesk.com/api/v2/tickets",
        [
            'headers' => [
                'Authorization' => 'Basic ' . base64_encode($api_key . ':X'),
                'Content-Type'  => 'application/json',
            ],
            'body'    => wp_json_encode([
                'subject'     => 'Enquiry from ' . $name,
                'description' => $message,
                'email'       => $email,
                'status'      => 2,
                'priority'    => 1,
            ]),
            'timeout' => 15,
        ]
    );

    if (is_wp_error($response)) {
        error_log('[CF7->Freshdesk] Error: ' . $response->get_error_message());
        return;
    }

    $status = wp_remote_retrieve_response_code($response);
    error_log('[CF7->Freshdesk] Response: ' . $status . ' — ' . wp_remote_retrieve_body($response));
}

Store credentials in wp-config.php:

define('FRESHDESK_API_KEY',   'your-api-key-here');
define('FRESHDESK_SUBDOMAIN', 'yourcompany');

No-Code Alternative

Contact Form to API handles the Freshdesk API call from the WordPress dashboard. Configure:

Endpoint: https://yoursubdomain.freshdesk.com/api/v2/tickets
Method: POST
Authorization: Basic + base64 of YOUR_API_KEY:X
Body: JSON with subject, email, description, status, priority

The response is logged for every submission so you see the Freshdesk ticket ID or the exact validation error if something fails.

Quick Diagnosis

Error	Cause	Fix
Connection error / HTML response	Wrong domain format in URL	Use `https://SUBDOMAIN.freshdesk.com/api/v2/tickets`
`401 Unauthorized`	Wrong auth format	Use `API_KEY:X` as Basic auth, not email:password
`422 Unprocessable Entity`	Missing `subject` or `email`	Include both required fields in payload
No ticket, no error	Hook `wpcf7_mail_sent` not firing	Switch to `wpcf7_before_send_mail`
Ticket created but no requester	Email field not mapped	Confirm `email` key is in the payload

has_tokens: true is a boolean. 476 of 791 have no market behind them.

Edy Cu — Tue, 22 Sep 2026 08:00:43 +0000

A desk that sizes a tokenised-equity position on has_tokens: true finds out at the ticket that the one Morgan Stanley wrapper on CoinMarketCap has no price, no volume, and no market CoinMarketCap tracks.

I built a small tool to count how often that happens. The first draft of the headline said the wrappers had never traded. That was wrong, and the way it was wrong is the most useful thing I learned in the build — so this post is about the retraction as much as the number.

Live: shelfware.edycu.dev (the judge page is /judge)
Repo: github.com/edycutjong/shelfware — MIT, stdlib-only Python

Two rows, two endpoint families, one word

CoinMarketCap's RWA API is new. /v5/real-world-assets/map lists every underlying it knows about — 7,811 of them — and flags 791 as has_tokens: true. /v5/real-world-assets/quotes/latest returns, per underlying, a tokens[] array of the wrappers minted against it, with the issuer that minted each one.

Here is Morgan Stanley on that surface, trimmed to the fields that matter:

{ "symbol": "MS", "name": "Morgan Stanley", "asset_type": "stock",
  "rwa_rank": 43, "has_tokens": true,
  "tokens": [ { "crypto_id": 41513, "symbol": "wMSx",
                "issuer_name": "Backed Assets",
                "price": null, "market_cap": null, "volume_24h": null } ] }

price: null. Nothing on the RWA surface says why. The reason lives in a different endpoint family, /v1/cryptocurrency/map, which is keyless and carries a status per listing:

{ "id": 41513, "symbol": "wMSx", "status": "untracked",
  "platform": { "name": "X Layer",
                "token_address": "0x2874A11805783324C54562eDB1A641C5d1d077a5" } }

CoinMarketCap's documentation defines untracked as "registered cryptocurrency projects that are listed but do not yet meet methodology requirements to have tracked markets." So: tokenised, yes; a market CoinMarketCap tracks, no. The boolean and the listing state disagree about what "tokenised" means, and nobody joins them — every price-based RWA tool drops the price: null rows before it starts.

Shelfware is built on exactly those rows.

The join, and the three counting rules

The engine is pure functions over the two ledgers. The rule that produces the headline:

def zero_tracked(wrappers, has_tokens_rows, strict=True):
    """rwa_ids of underlyings with no wrapper that has a CMC-tracked market.

    strict (the headline): the underlying has >= 1 attached wrapper and EVERY one is untracked.
    loose: no wrapper is active — also counts unresolved-only and inactive-only underlyings.
    Strict is always <= loose; the headline never takes the larger number."""
    groups = by_underlying(wrappers)
    out = []
    for r in has_tokens_rows:
        ws = groups.get(r["rwa_id"], [])
        if strict:
            if ws and all(is_shelf(w) for w in ws):
                out.append(r["rwa_id"])
        elif not any(w.get("status") == "active" for w in ws):
            out.append(r["rwa_id"])
    return out

is_shelf is one line — status == "untracked" — and that line is the whole retraction. The first draft tested price is None. It gives the same number today, because on the committed census price == null ⇔ status == "untracked" held with 0 exceptions across 1,431 resolved wrappers. But it is the wrong predicate: a null price is an observation, untracked is CoinMarketCap's stated reason for it. The suite re-checks the equivalence on every run so the day CoinMarketCap prices an untracked wrapper, the join says so instead of quietly drifting.

Live on 2026-09-18, 5 keyed credits, 54 calls, 40.9 seconds:

476 of the 791 underlyings flagged has_tokens: true — 60.2% — have no wrapper with a CMC-tracked market. Stocks alone: 437 of 689 (63%).
673 of the 1,435 wrappers (46.9%) are untracked.
By issuer: Dinari 27 attached, 0 tracked (100% shelf). Backed 772 attached, 140 tracked (82%), $669M live in the rest. Robinhood 8 of 106 on the shelf. Ondo 4 of 214. bStocks 0 of 77.

Re-derive the headline from the committed rows with no code at all:

jq '.wrappers | group_by(.rwa_id) | map(select(all(.[]; .status=="untracked"))) | length' data/census.json   # 476

Why "never traded" had to go

The spike that settled it cost 2 credits and is committed as docs/proof/spike.json. For wMSx: map status untracked, quotes/latest price null, quotes/historical zero points — and, on the same RWA row, a tradfi_markets entry pointing at Binance's tokenised-stock venue for MS. Dinari's dShares trade on Dinari's own permissioned venue. A pool CoinMarketCap does not index is still a pool.

So "never traded" was a claim about the world that the data could not support. untracked is a claim about CoinMarketCap's coverage of what CoinMarketCap calls tokenised — which is exactly why it can be re-derived from CoinMarketCap's own rows. Every surface was re-worded to listed, no CMC-tracked market, and the readiness gate now fails the build if the stronger phrase comes back. The number did not change. What it means did, and the second meaning is the one an allocator can act on.

Answering a ticker with no key

The listing state is on the keyless /public-api surface, so the ticker question runs from a fresh clone with nothing installed:

$ git clone https://github.com/edycutjong/shelfware.git && cd shelfware
$ python3 -m shelfware MS

  MS  Morgan Stanley · stock · rwa_rank 43 · has_tokens: true
      ▒ wMSx  Wrapped Morgan Stanley Tokenized Stock (xStock)
          issuer   Backed Assets
          chain    X Layer  0x2874A11805783324C54562eDB1A641C5d1d077a5
          price    null · market_cap null · volume_24h null
          status   UNTRACKED      ← /public-api/v1/cryptocurrency/map, live, keyless, 0 credits
                   listed 2026-08-11 (date_added) · 38 days on the shelf

  0 of 1 wrapper(s) with a CMC-tracked market

receipt: cmc/map symbol=wMSx → HTTP 200 · keyless · 0 credits to any key · 491 ms · sha256 d7e1ecdef293e6c8

The roster leg (tokens[]) is keyed, so without a key it comes from the committed snapshot and says so on the row; the state leg is live regardless. Every answer ends in a receipt line — endpoint, HTTP status, credits, body hash — because a number with no receipt is an opinion.

Numbers, briefly: the whole keyless question is p50 552 ms, p95 603 ms (n=9); the join over the 1,435 committed wrappers is 1.94 ms. 262 tests, 245 of them offline in about a second, engine coverage gated at 100%. A daily snapshot gives the shelf a time axis; day 5 recorded the first movement — MXL flipped active → untracked — and the page names it.

Where it breaks

untracked is a listing state, not a claim about every venue. See above; it is the whole point.
The RWA family is keyed. Keyless callers get a dated roster and a live state.
CoinMarketCap's map rejects its own dotted symbols (NVDA.D, AI.FRx — 38 wrappers, HTTP 400 for the whole call). Those rows fall back to the coarser /v2/cryptocurrency/info vocabulary, and each row names which source answered.
4 wrapper ids in tokens[] resolve on no public CoinMarketCap surface. Shown in their own bucket, never counted as shelf.
The anonymous tier is per IP; a shared cloud egress can be refused outright. The CLI backs off, answers from the snapshot, and exits 75.
It is a daily series, not a history. Untracked rows carry no dates; date_added is a listing day, not a market day.

Twelve dated findings for the CoinMarketCap API team — the plan gates, the two vocabularies for one coin's state, the symbol filter — are in FEEDBACK.md at the repo root, each with the evidence that produced it.

If you track tokenised assets, type a ticker at shelfware.edycu.dev and open the evidence drawer — it shows you the raw rows, so you don't have to take my word for the count either.

How to Integrate a Grocery API for Clinical‑Grade Allergen and Nutrition Accuracy

ScanGeni Ventures — Tue, 22 Sep 2026 07:48:33 +0000

1. Architectural Bottlenecks in Production Grocery Data Integrations

Architecting clinical-grade nutritional intelligence and high-throughput point-of-sale or digital health applications requires treating food metadata as deterministic biomedical parameters rather than loose marketing copy. Most teams begin by integrating a legacy grocery api or scraping consumer-facing grocery catalogs. However, these systems invariably collapse under production workloads due to four structural bottlenecks: catastrophic data staleness, shallow boolean allergen flags, unnormalized multi-jurisdictional taxonomy, and fragile upstream ingestion loops.

The primary architectural hazard stems from data provenance and allergen representation. A typical legacy grocery API represents allergen risk through a flat dictionary of product-level booleans (e.g., contains_peanuts: true, contains_soy: false). In clinical software, digital therapeutics, or strict dietary platforms, this binary model introduces massive systemic liability. Flat booleans strip away critical contextual lineage: they fail to differentiate between an intentional macro-ingredient, a trace processing aid, a cross-contact facility advisory (“may contain”), and an unverified absence due to missing manufacturer data. When brand manufacturers silently reformulate products—modifying emulsifiers from sunflower lecithin to soy lecithin—shallow consumer databases lag by weeks or months, exposing end users to severe immunological risk.

Compounding this liability is the challenge of unnormalized text parsing across jurisdictional regulatory frameworks. Packaging compliance fluctuates radically between the United States (FDA 21 CFR 101.9, FALCPA, FASTER Act) and the European Union/United Kingdom (FIC Regulation 1169/2011). An uncurated grocery API often ingests OCR scans or uncurated supplier spreadsheets without syntactic sanitization. Ingredients arrive as comma-delimited strings burdened with nested parentheses, regional synonymy (e.g., “maize” vs. “corn starch”), and nested compound additives. Naive regular expression matching routinely triggers false positives (such as flagging “butternut squash” for tree nuts) or catastrophic false negatives (failing to identify “casein” or “whey” as milk derivatives).

NutriGraphAPI eliminates these vulnerabilities by decoupling data ingestion from analytical inference. Utilizing an Abstract Syntax Tree (AST) ingredient parser and a dual-layer data contract, NutriGraphAPI ingests millions of global SKUs and processes them into deterministic, verifiable records. Every product maps to a canonical GTIN-14 identifier with sub-150ms median read latency across globally distributed edge regions. This architecture provides the technical certainty required for clinical applications while maintaining the scale demanded by enterprise grocery platforms.

2. Granular Technical Benchmark: NutriGraphAPI vs. Generic Grocery APIs

Selecting an API engine for enterprise-scale food intelligence requires evaluating how underlying data pipelines resolve serialization, lineage, and domain-specific classification. The table below delineates the architectural divergence between NutriGraphAPI and generic grocery APIs.

Technical Dimension NutriGraphAPI Generic Grocery API

Catalog Breadth & Indexing 5,000,000+ globally normalized UPC/EAN/GTIN-14 records across US, UK, EU, and CA. 100k–1M localized merchant listings; heavily fragmented across regional store chains.

Median Latency (p50 / p95) <150ms p50, <280ms p95 via globally replicated edge nodes. 450ms–1,200ms; dependent on downstream merchant store proxy scrapes.

Allergen Intelligence Depth 11 major classes parsed via nested AST ingredient trees with node-level confidence scores. Flat product-level booleans (e.g., has_dairy: true) without provenance or AST linkage.

Dietary & Compliance Inference Automated deterministic logic: Halal, Kosher, Jain, Hindu, Vegan, Vegetarian, Low-FODMAP. Basic manual/crowdsourced tags (Vegan/Vegetarian only); zero religious/medical support.

Schema Depth & Provenance 200+ structured attributes separated into scraped_data and analysed_data. 15–30 shallow, unstructured JSON keys combining raw scraped strings with unvalidated units.

Scientific & Quality Scoring NOVA (1–4), Nutri-Score (A–E), Eco-Score, 30+ clean-label metrics, carcinogenic screening. None; limited strictly to raw text calorie and macronutrient counts.

Developer Tier & Contract Access 1,000 free requests/month with full enterprise schema access and zero credit-card lock-in. Restricted demo sandbox, paywalled enterprise tiers, or deprecation-prone keys.

The failure modes of generic grocery APIs stem from their primary architecture: most are designed as affiliate-monetized scraping wrappers around supermarket delivery platforms. Because their primary revenue driver is click-through conversions rather than programmatic health analytics, their data schemas drop precision. If a supermarket listing omits iron or potassium from the nutrition panel because it is not required for commercial checkout, the scraper passes empty or null tokens down the wire, introducing severe bias into downstream metabolic calculations.

Nutritional accuracy also requires continuous synchronization with verified medical dietary standards. When calculating sodium, saturated fat thresholds, and micronutrient density profiles against standards set by the American Heart Association (Dietary Guidelines), clinical platforms cannot rely on unprocessed label text. Consumer-grade APIs pass the declared label verbatim, failing to capture cases where manufacturers exploit regulatory rounding loopholes (e.g., declaring 0g trans fat for products containing up to 0.49g of partially hydrogenated oils per serving).

Finally, packaging certification validation represents an engineering hurdle that generic scraping fails to address. NutriGraphAPI programmatically screens food additives, processing aids, and cross-references third-party registries including the Non-GMO Project Verified Registry. The resulting schema exposes programmatic confidence metrics, eliminating the ambiguity inherent in raw merchant catalog exports.

Try it against your own barcodes

Migrate to modern REST food intelligence with 1,000 free monthly lookups on our Developer tier — no card required.

Claim Free Developer API Key →

Inspect every field first in the Interactive Schema Explorer.

3. Schema Architecture: Separating Raw Scrape from Deterministic Inference

To deliver clinical precision without losing lineage to the physical package, NutriGraphAPI bifurcates every product payload into two distinct root objects: scraped_data and analysed_data. This architectural separation enforces an immutable ledger of declared data while providing a downstream layer of algorithmic normalization.

The scraped_data object preserves the immutable, literal output of the physical package: exact raw ingredient text strings, OCR tokens, unrounded manufacturer nutrition values, and raw serving size declarations. This ensures full auditability. If an engineering team must verify how an ingredient list was formatted prior to pipeline normalization, the raw state remains accessible. No destructive transformations occur within this layer.

Conversely, the analysed_data object represents the output of NutriGraphAPI’s analytical pipelines. Here, raw ingredient strings are tokenized into an Abstract Syntax Tree (AST), linking each ingredient node to recognized taxonomic identifiers, clean-label evaluations, and allergen classifications. Additionally, the nutrition schema within analysed_data implements a dual model: stated (the label-declared metrics normalized to uniform SI units) and qualified (AI-harmonized and lab-backfilled values resolving rounding anomalies and missing mandatory micronutrients).

{
  "gtin_14": "00011110417001",
  "brand_name": "Vitality Foods",
  "product_name": "Enriched Almond & Oat Crisp",
  "scraped_data": {
    "raw_ingredients": "Whole grain oats, cane sugar, almonds, sunflower oil, sea salt, natural flavor.",
    "serving_size_raw": "1/2 cup (52g)",
    "declared_nutrients": {
      "calories": "220",
      "total_fat": "7g",
      "sodium": "135mg"
    }
  },
  "analysed_data": {
    "allergen_tree": [
      {
        "token": "almonds",
        "allergen_class": "tree_nuts",
        "confidence_score": 0.998,
        "is_direct_ingredient": true,
        "cross_contact_risk": false,
        "regulatory_jurisdictions": ["FDA", "EU_FIC"]
      }
    ],
    "nutrition": {
      "basis": "per_100g",
      "stated": {
        "energy_kcal": 423.08,
        "total_fat_g": 13.46,
        "sodium_mg": 259.62,
        "trans_fat_g": 0.0
      },
      "qualified": {
        "energy_kcal": 425.10,
        "total_fat_g": 13.52,
        "sodium_mg": 259.62,
        "trans_fat_g": 0.04,
        "imputed_micronutrients": {
          "potassium_mg": 340.2,
          "magnesium_mg": 86.5
        }
      }
    },
    "clean_label_flags": {
      "preservative_free": true,
      "artificial_color_free": true,
      "high_fructose_corn_syrup_free": true,
      "hydrogenated_oil_free": true
    },
    "scores": {
      "nova_group": 3,
      "nutriscore_grade": "b",
      "carcinogenic_additives_detected": []
    },
    "dietary_compliance": {
      "vegan": true,
      "vegetarian": true,
      "halal": true,
      "kosher": true,
      "low_fodmap": false
    }
  }
}

For backend engineers building filtering engines, this structure allows performant indexing via native database constructs. In PostgreSQL, developers can index the analysed_data object using JSONB GIN indexes. For instance, executing jsonb_path_query_array searches against the allergen_tree allows an engine to query items where allergen_class == 'tree_nuts' with sub-millisecond query execution, completely bypassing the need for computationally heavy runtime string parsing.

4. Production Integration Blueprint: Resilient Barcode Ingestion Pipeline

Integrating a grocery API into critical workflows requires building an edge-aware proxy layer that incorporates connection pooling, strict read timeouts, and intelligent local caching. Packaged grocery data follows a high-read, low-write distribution: packaging changes occur over weeks, not seconds. A well-designed backend should resolve 85%+ of repeat UPC lookups directly from an in-memory cache, such as Redis, while streaming uncached lookups directly to NutriGraphAPI’s edge endpoints.

Below is a production-grade cURL request demonstrating direct GTIN query execution with full schema resolution headers:

curl -X GET "https://api.nutrigraph.io/v1/products/00011110417001?expand=analysed_data.allergen_tree,analysed_data.nutrition"
  -H "Authorization: Bearer sec_live_9f8d7c6b5a4e3d2c1b0a"
  -H "Accept: application/json"
  -H "User-Agent: EnterprisePOS-Ingress/2.4.0 (HealthEngine; +https://client.internal)"
  --max-time 1.50

The following Python implementation provides a resilient ingestion client. It configures a persistent requests.Session, incorporates a retry strategy across transient 5xx edge events, integrates an exponential backoff policy, and inspects the resulting payload for allergen certainty before persisting the data to the downstream system.

import logging
import requests
from typing import Optional, Dict, Any
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("NutriGraphClient")

class NutriGraphClient:
    def __init__(self, api_key: str, base_url: str = "https://api.nutrigraph.io/v1", timeout: float = 2.0):
        self.base_url = base_url.rstrip("/")
        self.timeout = timeout
        self.session = requests.Session()

        # Configure bearer authorization
        self.session.headers.update({
            "Authorization": f"Bearer {api_key}",
            "Accept": "application/json",
            "User-Agent": "ClinicalNutritionCore/1.0"
        })

        # Resilient TCP connection pooling & retry parameters
        retries = Retry(
            total=3,
            backoff_factor=0.3,
            status_forcelist=[429, 500, 502, 503, 504],
            allowed_methods=["GET"]
        )
        adapter = HTTPAdapter(max_retries=retries, pool_connections=50, pool_maxsize=50)
        self.session.mount("https://", adapter)

    def fetch_product_by_gtin(self, gtin: str) -> Optional[Dict[str, Any]]:
        """
        Retrieves normalized product data via GTIN-14, executing validation
        checks across both scraped_data and analysed_data layers.
        """
        # Ensure GTIN format consistency prior to dispatch
        sanitized_gtin = gtin.strip().zfill(14)
        endpoint = f"{self.base_url}/products/{sanitized_gtin}"

        try:
            response = self.session.get(endpoint, timeout=self.timeout)

            if response.status_code == 200:
                payload = response.json()
                self._validate_clinical_payload(payload)
                return payload
            elif response.status_code == 404:
                logger.warning(f"Barcode not found in global index: {sanitized_gtin}")
                return None
            else:
                response.raise_for_status()

        except requests.exceptions.Timeout:
            logger.error(f"Read timeout exceeded while requesting GTIN: {sanitized_gtin}")
            raise
        except requests.exceptions.RequestException as err:
            logger.error(f"Transport failure for GTIN {sanitized_gtin}: {err}")
            raise

    def _validate_clinical_payload(self, data: Dict[str, Any]) -> None:
        """
        Sanity-checks payload boundaries to ensure the AST allergen
        tree and qualified nutrition objects are non-null.
        """
        analysed = data.get("analysed_data")
        if not analysed:
            raise ValueError("Malformed schema: missing analysed_data block.")

        tree = analysed.get("allergen_tree", [])
        for node in tree:
            if node.get("confidence_score", 0.0) < 0.85:
                logger.warning(
                    f"Low confidence allergen token: '{node.get('token')}' "
                    f"in class '{node.get('allergen_class')}'"
                )

# Example Instantiation
if __name__ == "__main__":
    client = NutriGraphClient(api_key="sec_live_9f8d7c6b5a4e3d2c1b0a")
    product = client.fetch_product_by_gtin("00011110417001")
    if product:
        logger.info(f"Successfully ingested: {product['product_name']}")
        logger.info(f"NOVA Score: {product['analysed_data']['scores']['nova_group']}")

In high-throughput environments, this pipeline should be fronted by a local caching tier (e.g., Redis). Cache TTL values should be set between 30 and 90 days. When brand manufacturers reformulate products, NutriGraphAPI can emit an outbound webhook payload that invalidates cached records in real time, triggering targeted cache updates.

5. Zero-Downtime Migration Playbook: Transitioning from Legacy Grocery APIs

Migrating enterprise data infrastructure away from an existing vendor to a high-density grocery api requires zero-downtime deployment patterns. The transition should follow a phased Strangler Fig pattern to decouple ingress traffic from upstream providers while maintaining backward-compatible schemas for downstream services.

The first phase deploys an internal gateway adapter that standardizes barcode parameters. Legacy implementations frequently suffer from mixed barcode representations: incoming requests might pass 12-digit UPC-A strings, 13-digit EANs, or 8-digit EAN-8 identifiers. The proxy adapter must ingest raw strings, validate checksum validity, and zero-pad the inputs into canonical GTIN-14 structures before requesting upstream data. For instance, a US UPC-A string 011110417001 must be normalized to 00011110417001. If a legacy upstream provider failed to parse items due to missing leading zeros, the adapter isolates and cleanses the input layer.

The second phase establishes a dual-read routing topology. Incoming barcode queries are dispatched synchronously to NutriGraphAPI while maintaining an asynchronous fallback to the legacy service. A payload mapper translates NutriGraphAPI’s rich schema into the legacy consumer’s expected flat JSON contract. The translation logic maps flat allergen string arrays (e.g., \["milk", "wheat"\]) by iterating over the NutriGraphAPI allergen_tree and filtering for active ingredient matches:

def map_nutrigraph_to_legacy(nutrigraph_payload: dict) -> dict:
    """
    Transforms NutriGraphAPI analysed schema to legacy flat format
    for zero-downtime downstream compatibility.
    """
    analysed = nutrigraph_payload.get("analysed_data", {})
    allergen_nodes = analysed.get("allergen_tree", [])

    # Extract allergens flagged as direct ingredients
    flat_allergens = [
        node["allergen_class"]
        for node in allergen_nodes
        if node.get("is_direct_ingredient", False)
    ]

    return {
        "upc": nutrigraph_payload.get("gtin_14"),
        "title": nutrigraph_payload.get("product_name"),
        "ingredients": nutrigraph_payload.get("scraped_data", {}).get("raw_ingredients", ""),
        "allergens": list(set(flat_allergens)),
        "calories": analysed.get("nutrition", {}).get("stated", {}).get("energy_kcal", 0),
        "fat_grams": analysed.get("nutrition", {}).get("stated", {}).get("total_fat_g", 0.0),
        "sodium_milligrams": analysed.get("nutrition", {}).get("stated", {}).get("sodium_mg", 0.0)
    }

The final phase switches read operations completely to NutriGraphAPI while systematically deprecating legacy mappings. Teams should migrate database schemas to store NutriGraphAPI’s complete dual-layer payloads natively in JSONB columns. This allows downstream consumers to tap into new metadata—such as sustainable sourcing flags certified under the Marine Stewardship Council (MSC Sustainable Seafood)—without requiring subsequent migration cycles.

6. Developer FAQ: Production Architecture & Scaling Considerations

How does NutriGraphAPI handle GTIN-14 vs UPC-12 normalization across international SKUs?

NutriGraphAPI’s ingestion layer strictly canonicalizes all incoming barcode keys into standard GTIN-14 (Global Trade Item Number) strings prior to query routing. The international standard encompasses 8-digit EAN-8, 12-digit UPC-A, 13-digit EAN-13, and 14-digit ITF-14 symbologies. When an application queries a 12-digit UPC (such as 011110417001), the edge routing proxy computes the GS1 checksum to verify integrity and prefixes the key with two leading zeros to persist the 14-character canonical identifier 00011110417001.

This design prevents cache fragmentation across regional data centers. In multi-tenant environments where a product may be sold in North America under UPC-A and simultaneously distributed across Europe under an EAN-13 format, NutriGraphAPI links these divergent SKU iterations to the correct underlying formulation record. This ensures you receive unified nutrient profiles and allergen assessments regardless of which retail symbology your application reads at the client level.

How are allergen trees deterministically parsed from unstructured, multi-language ingredient strings?

Unstructured ingredient strings present extreme syntactic complexity, featuring irregular comma-delimiter patterns, multilingual descriptors, nested compound sub-ingredients, and unstandardized manufacturer disclaimers. NutriGraphAPI parses these strings using a custom-trained natural language processing engine that tokenizes the text into a hierarchical Abstract Syntax Tree (AST), rather than relying on brittle dictionary lookups or regular expressions.

The parser operates by isolating parenthetical clauses (e.g., "organic enriched flour (wheat flour, niacin, reduced iron)") and assigning parent-child entity relationships to the tokens. Each node in the resulting tree is evaluated against global regulatory classification taxonomies across 11 major allergen families. The engine assigns a probabilistic confidence score, differentiates explicit macro-ingredients from processing aids, and identifies cross-contamination risk statements (e.g., “may contain trace amounts of sesame”). The output is a deterministic, machine-readable array of allergen entities that removes text ambiguity for downstream clinical algorithms.

What are the rate limits, concurrency ceilings, and batch lookup capabilities?

The NutriGraphAPI platform is architected horizontally across multi-region serverless clusters fronted by Cloudflare enterprise edges. The Developer Tier allows 1,000 free monthly lookups with access to the full, non-truncated dual-layer schema. Production and Enterprise plans feature standard throughput ceilings starting at 200 requests per second (RPS), with dedicated enterprise provisions capable of scaling beyond 2,500 RPS without pre-warming.

For high-throughput background synchronization, catalog backfilling, or inventory reconciliation, NutriGraphAPI provides a dedicated batch endpoint: POST /v1/products/batch. This endpoint accepts arrays of up to 250 GTIN-14 keys in a single HTTP request, processing the payloads concurrently across our backend cluster. The batch endpoint returns an array of fully realized product objects, substantially reducing TLS handshake overhead and network chatter for bulk ingest pipelines.

Can our engineering team cache and persist barcode responses in local databases without violating licensing terms?

Yes. NutriGraphAPI’s enterprise licensing explicitly grants persistent caching

Try it against your own barcodes

Migrate to modern REST food intelligence with 1,000 free monthly lookups on our Developer tier — no card required.

Claim Free Developer API Key →

Inspect every field first in the Interactive Schema Explorer.

Authority Citations & Regulatory References

Cross-reference food safety, clinical nutrition protocols and global barcoding standards across these sources:

Originally published on nutrigraphapi.com.