<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: LEI QIN</title>
    <description>The latest articles on DEV Community by LEI QIN (@lei_qin_486b5f0693eac8f66).</description>
    <link>https://dev.to/lei_qin_486b5f0693eac8f66</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2013928%2F9cb2e937-5043-4e32-baf2-0e6be94f3595.png</url>
      <title>DEV Community: LEI QIN</title>
      <link>https://dev.to/lei_qin_486b5f0693eac8f66</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lei_qin_486b5f0693eac8f66"/>
    <language>en</language>
    <item>
      <title>Install an agent skill with the SourceWeft CLI, and check that the files match what was scanned</title>
      <dc:creator>LEI QIN</dc:creator>
      <pubDate>Sat, 21 Nov 2026 14:51:41 +0000</pubDate>
      <link>https://dev.to/lei_qin_486b5f0693eac8f66/install-an-agent-skill-with-the-sourceweft-cli-and-check-that-the-files-match-what-was-scanned-2m17</link>
      <guid>https://dev.to/lei_qin_486b5f0693eac8f66/install-an-agent-skill-with-the-sourceweft-cli-and-check-that-the-files-match-what-was-scanned-2m17</guid>
      <description>&lt;p&gt;&lt;em&gt;Last updated: 2026-11-20. Originally published on the &lt;a href="https://sourceweft.com/blog/install-an-agent-skill-with-the-sourceweft-cli" rel="noopener noreferrer"&gt;SourceWeft blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What you will do
&lt;/h2&gt;

&lt;p&gt;You will search the SourceWeft skills directory from a terminal, read a skill's details before installing, install it for a local agent, and then check that what is on disk is what the directory scanned.&lt;/p&gt;

&lt;p&gt;Every command below was run with the published CLI, version 0.3.0, in an empty folder on 2026-11-20. You need Node.js and a terminal. You do not need a SourceWeft account.&lt;/p&gt;

&lt;p&gt;If you want to understand how to judge whether a skill is appropriate for you, read &lt;a href="https://sourceweft.com/blog/how-to-check-an-agent-skill-before-installing" rel="noopener noreferrer"&gt;How to check an agent skill before you install it&lt;/a&gt; first. This tutorial is about the mechanics.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "verified" means here
&lt;/h2&gt;

&lt;p&gt;The directory records, for each skill, the source repository, the exact commit it was scanned at, and a SHA-256 hash for every file. The CLI downloads the files from that repository at that commit and compares each one to the recorded hash. If anything differs, it installs nothing.&lt;/p&gt;

&lt;p&gt;That check proves one thing: &lt;strong&gt;the files you install are the files that were scanned.&lt;/strong&gt; It does not prove the files are harmless. For that, you read the skill and its scan notes. The two checks do different jobs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Search
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills search &lt;span class="s2"&gt;"pdf forms"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each line shows the install name, a &lt;code&gt;scripts&lt;/code&gt; marker if the skill ships scripts, and a short description. A few of the results we got:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@github/pdftk-server                                       Skill for using the command-line tool pdftk (PDFtk Server) for workin…
@claude-office-skills/pdf-form-filler-0a843e12             Fill out PDF forms programmatically and extract form data
@jwynia/pdf-generator                             scripts  Create and manipulate PDF files programmatically. Use when the user n…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The install name, such as &lt;code&gt;@github/pdftk-server&lt;/code&gt;, is what you pass to the other commands.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Read the details before you install
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills info @google/gke-kueue-troubleshooting
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the output for a skill that ships scripts and carries scan flags:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@google/gke-kueue-troubleshooting  (Gke Kueue Troubleshooting)
Diagnose a GKE batch or AI workload (…

Version:    e96f2070b42c11e184e662cfe1fda8b20c9e2140
License:    not stated
Source:     https://github.com/google/skills/tree/e96f2070b42c11e184e662cfe1fda8b20c9e2140/skills/cloud/gke-kueue-troubleshooting
Trust:      Featured — a featured publisher, not reviewed by a SourceWeft admin (files are still hash-checked)
Files:      24
Scripts:    yes — this skill includes executable files
Scan flags: egress:fetch, script:out-of-bundle-path, secrets:env-access
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read these lines in this order:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Source&lt;/strong&gt; is the repository and commit. The commit is the version you will get.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trust&lt;/strong&gt; says whether a SourceWeft admin reviewed the skill. "Featured" and "Community" mean no admin did.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scripts&lt;/strong&gt; tells you whether anything in the skill is code. If so, it can run on your machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scan flags&lt;/strong&gt; are the automated scan's notes. What they mean is explained in the article linked above.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Install
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills install @google/gke-kueue-troubleshooting
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The CLI prints the same details and asks you to confirm. If the skill ships scripts, the question is "This skill includes scripts your agent may run. Install it?"&lt;/p&gt;

&lt;p&gt;In a terminal that is not interactive (a script or a CI job), the CLI will not guess. It stops with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Error: Confirmation required. Re-run with --yes to proceed without asking.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and exits with a non-zero status, having written nothing. Use &lt;code&gt;--yes&lt;/code&gt; only when you have already read the details.&lt;/p&gt;

&lt;p&gt;By default the skill is installed for Claude Code, into &lt;code&gt;~/.claude/skills/&amp;lt;name&amp;gt;/&lt;/code&gt;. To choose another agent, use &lt;code&gt;--agent&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills install @github/pdftk-server --agent codex
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;npx @sourceweft/cli skills agents&lt;/code&gt; prints the full list with the folder each agent uses. For example, &lt;code&gt;claude-code&lt;/code&gt; uses &lt;code&gt;~/.claude/skills&lt;/code&gt;, &lt;code&gt;cursor&lt;/code&gt; uses &lt;code&gt;~/.cursor/skills&lt;/code&gt;, and &lt;code&gt;codex&lt;/code&gt; uses &lt;code&gt;~/.agents/skills&lt;/code&gt;. Add &lt;code&gt;--scope project&lt;/code&gt; to install into the current project's folder instead of your home directory.&lt;/p&gt;

&lt;p&gt;To try an install without touching your real agent folders, point it at an empty folder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills install @google/gke-kueue-troubleshooting --dir ./out --yes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We ran this, and it wrote 24 files into &lt;code&gt;./out/gke-kueue-troubleshooting/&lt;/code&gt;: the skill's own files plus one file the CLI adds, &lt;code&gt;.sourceweft.json&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: See what was recorded
&lt;/h2&gt;

&lt;p&gt;Open &lt;code&gt;.sourceweft.json&lt;/code&gt;. It holds the registry address, the skill's install name, the version, the source repository and commit, the license, and a hash for every file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"installRef"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"@google/gke-kueue-troubleshooting"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"e96f2070b42c11e184e662cfe1fda8b20c9e2140"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"repoUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://github.com/google/skills"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"commitSha"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"e96f2070b42c11e184e662cfe1fda8b20c9e2140"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"subpath"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"skills/cloud/gke-kueue-troubleshooting"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"files"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"references/attribution.md"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"b419a7d3…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"…"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"…"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We recomputed the SHA-256 of all 24 files with a separate script, and every one matched the recorded value. You can do the same with &lt;code&gt;shasum -a 256&lt;/code&gt; and compare.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Check the install later
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills doctor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;doctor&lt;/code&gt; compares what is on disk with what was recorded. Right after the install above, with &lt;code&gt;--dir ./out&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OK — 1 installed skill(s), no problems found.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We then appended one line to &lt;code&gt;scripts/snapshot.py&lt;/code&gt; and ran it again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;note     …/gke-kueue-troubleshooting: 'scripts/snapshot.py' was edited
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;doctor&lt;/code&gt; reports edited files as a note, missing files as a problem, and extra files as a note. It tells you that something changed. It cannot tell you whether the change was yours.&lt;/p&gt;

&lt;p&gt;Two more commands behave the same way around local edits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;skills list&lt;/code&gt; shows the install with the state &lt;code&gt;modified&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Running &lt;code&gt;install&lt;/code&gt; again over a modified folder stops with an error and asks for &lt;code&gt;--force&lt;/code&gt;, rather than overwriting your changes. With &lt;code&gt;--force&lt;/code&gt; it restores the recorded files and prints &lt;code&gt;Updated …&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 6: Update and remove
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @sourceweft/cli skills update --dry-run
npx @sourceweft/cli skills update
npx @sourceweft/cli skills remove @google/gke-kueue-troubleshooting
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--dry-run&lt;/code&gt; shows what an update would do without doing it. A new version can bring new scripts, so read the new version's details the same way you read the first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Without the CLI: ask your agent to do it
&lt;/h2&gt;

&lt;p&gt;Each skill page has an Agent tab with a prompt you can paste into your agent. It tells the agent to read the skill's &lt;code&gt;install.md&lt;/code&gt;, show you the source, license, and whether it ships scripts, wait for your OK, and ask before changing anything else on your machine. Use this if you would rather not run commands yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not do
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;It does not review the skill for you. A file that matches its hash can still be a bad idea to run.&lt;/li&gt;
&lt;li&gt;It does not protect you from a skill that was already problematic when it was scanned. That is what reading the skill and its scan notes is for.&lt;/li&gt;
&lt;li&gt;It does not watch the upstream repository for changes after you install. &lt;code&gt;skills update&lt;/code&gt; is how you find out about a new version.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How do I install an agent skill from the command line?&lt;/strong&gt;&lt;br&gt;
Run &lt;code&gt;npx @sourceweft/cli skills install &amp;lt;name&amp;gt;&lt;/code&gt;. It prints the skill's source, license, and scan notes, asks you to confirm, and writes the files for your agent. By default it installs for Claude Code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which agents does the CLI support?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;npx @sourceweft/cli skills agents&lt;/code&gt; lists them, with each agent's skills folder. The list includes Claude Code, Codex, Cursor, Gemini CLI, Windsurf, and GitHub Copilot, among others.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does the CLI check before it writes a file?&lt;/strong&gt;&lt;br&gt;
It downloads the skill from its source repository at the recorded commit and compares every file to the SHA-256 hash recorded when the skill was scanned. If the download does not match, it installs nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do I need a SourceWeft account?&lt;/strong&gt;&lt;br&gt;
No. The commands in this tutorial ran without signing in.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I try an install without changing my agent's folders?&lt;/strong&gt;&lt;br&gt;
Yes. Use &lt;code&gt;--dir &amp;lt;folder&amp;gt;&lt;/code&gt; to install into an empty folder, and &lt;code&gt;skills remove&lt;/code&gt; or delete the folder when you are done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What if I edit an installed skill?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;skills doctor&lt;/code&gt; and &lt;code&gt;skills list&lt;/code&gt; will show that it was edited. Installing over it requires &lt;code&gt;--force&lt;/code&gt;, so your changes are not overwritten by accident.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related articles
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://sourceweft.com/blog/how-to-check-an-agent-skill-before-installing" rel="noopener noreferrer"&gt;How to check an agent skill before you install it: what scan flags tell you, and what they don't&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;I maintain SourceWeft, which publishes this CLI and directory. This article was drafted with an AI assistant and reviewed by me.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>cli</category>
      <category>agents</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>MCP server directories compared: the official registry, Glama, mcp.so, MCP Market, Smithery, PulseMCP, and SourceWeft</title>
      <dc:creator>LEI QIN</dc:creator>
      <pubDate>Sat, 21 Nov 2026 14:41:19 +0000</pubDate>
      <link>https://dev.to/lei_qin_486b5f0693eac8f66/mcp-server-directories-compared-the-official-registry-glama-mcpso-mcp-market-smithery-53p</link>
      <guid>https://dev.to/lei_qin_486b5f0693eac8f66/mcp-server-directories-compared-the-official-registry-glama-mcpso-mcp-market-smithery-53p</guid>
      <description>&lt;p&gt;&lt;em&gt;Data as of: 20 November 2026. Originally published on the &lt;a href="https://sourceweft.com/blog/mcp-server-directories-compared" rel="noopener noreferrer"&gt;SourceWeft blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The short answer
&lt;/h2&gt;

&lt;p&gt;There is no single list of every MCP server. There is one official registry that publishers submit to, and several directories that add search, categories, and their own checks on top.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;To &lt;strong&gt;publish&lt;/strong&gt; a server, or to build on a neutral data source, use the &lt;strong&gt;official MCP Registry&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;To &lt;strong&gt;browse the largest catalogue&lt;/strong&gt;, use &lt;strong&gt;Glama&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;To &lt;strong&gt;browse by category with setup guides&lt;/strong&gt;, use &lt;strong&gt;mcp.so&lt;/strong&gt;, &lt;strong&gt;MCP Market&lt;/strong&gt;, or &lt;strong&gt;SourceWeft&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;If you &lt;strong&gt;used Smithery to connect to servers&lt;/strong&gt;, that service ended on 31 October 2026. Smithery is still a directory, and its verified server pages now show each vendor's own server URL.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whichever you use, a directory helps you find a server. It does not vouch for what the server's tools do.&lt;/p&gt;

&lt;h2&gt;
  
  
  Registry or directory?
&lt;/h2&gt;

&lt;p&gt;The two words mean different things here.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;registry&lt;/strong&gt; is a source of record. Publishers submit their server's metadata, and the registry confirms that the publisher controls the namespace it is published under. The official MCP Registry describes itself as "like an app store for MCP servers", and it offers a public API that directories and marketplaces can read from.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;directory&lt;/strong&gt; is a website for people. It gathers servers from the registry and from other sources, then adds things a registry does not: categories, search, descriptions, tool lists, and trust labels of its own.&lt;/p&gt;

&lt;p&gt;The official registry is designed to feed directories: its documentation says it is "intended to be consumed primarily by downstream aggregators". Directories also take GitHub repositories, either submitted by authors or indexed directly. That is why the same server appears on several of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changed at Smithery
&lt;/h2&gt;

&lt;p&gt;Smithery used to do two jobs: list servers, and run a proxy that connected you to them through Smithery's own URLs and accounts.&lt;/p&gt;

&lt;p&gt;On 31 October 2026 the second job ended. According to Smithery's own notice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What stays:&lt;/strong&gt; smithery.ai remains a public directory of MCP servers, now operated as part of Arcade. Pages for verified servers show the vendor's direct MCP URL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What stopped:&lt;/strong&gt; server URLs on &lt;code&gt;run.tools&lt;/code&gt; and &lt;code&gt;server.smithery.ai&lt;/code&gt;, the Smithery API, Connect and Toolbox, Smithery accounts and API keys. The Smithery CLI and SDKs are deprecated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What to do:&lt;/strong&gt; switch to the vendor's official MCP URL, which Smithery shows on its verified server pages.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If a client of yours still points at a Smithery URL, it no longer connects. Replace the URL with the vendor's own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The directories at a glance
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Directory&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;th&gt;Size it states&lt;/th&gt;
&lt;th&gt;Also lists&lt;/th&gt;
&lt;th&gt;Operated by&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://registry.modelcontextprotocol.io" rel="noopener noreferrer"&gt;Official MCP Registry&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Source of record with a public API&lt;/td&gt;
&lt;td&gt;Not shown on the home page&lt;/td&gt;
&lt;td&gt;Servers only&lt;/td&gt;
&lt;td&gt;MCP contributors, in the open&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://glama.ai/mcp/servers" rel="noopener noreferrer"&gt;Glama&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory with many filters and sort orders&lt;/td&gt;
&lt;td&gt;124,343 servers&lt;/td&gt;
&lt;td&gt;Connectors, tools, clients&lt;/td&gt;
&lt;td&gt;Glama&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://mcpmarket.com" rel="noopener noreferrer"&gt;MCP Market&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory with categories&lt;/td&gt;
&lt;td&gt;55,745 servers&lt;/td&gt;
&lt;td&gt;Agent skills, clients&lt;/td&gt;
&lt;td&gt;MCP Market&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://mcp.so" rel="noopener noreferrer"&gt;mcp.so&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory with categories, rankings, and collections&lt;/td&gt;
&lt;td&gt;20,828 servers, 1,551 of them remote&lt;/td&gt;
&lt;td&gt;Clients, skills, agents&lt;/td&gt;
&lt;td&gt;mcp.so&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://smithery.ai" rel="noopener noreferrer"&gt;Smithery&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory; connection services ended 31 Oct 2026&lt;/td&gt;
&lt;td&gt;Not shown on the home page&lt;/td&gt;
&lt;td&gt;Servers; its home page also links to Arcade's tools&lt;/td&gt;
&lt;td&gt;Smithery, now part of Arcade&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.pulsemcp.com" rel="noopener noreferrer"&gt;PulseMCP&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory with classifications and usage estimates&lt;/td&gt;
&lt;td&gt;21,523 servers&lt;/td&gt;
&lt;td&gt;Not checked beyond the server list&lt;/td&gt;
&lt;td&gt;PulseMCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://sourceweft.com/mcp" rel="noopener noreferrer"&gt;SourceWeft&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Directory with categories and connect guides&lt;/td&gt;
&lt;td&gt;69,749 servers&lt;/td&gt;
&lt;td&gt;Agent skills&lt;/td&gt;
&lt;td&gt;SourceWeft&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sizes are the figures each site displayed on 20 November 2026. They are not comparable one-to-one: directories count versions, forks, and remote and local variants differently, and a bigger number does not mean more usable servers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Each one in brief
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Official MCP Registry.&lt;/strong&gt; Run in the open by MCP contributors through a working group. Publishers submit servers under a namespace they have proved they control, such as a domain or a GitHub account. It launched in preview in September 2025, froze its API at version 0.1 in October 2025, and was still labelled a preview on 20 November 2026. Use it to publish, or as the base data for your own tooling. Its website is a plain search over that data, without categories or reviews.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Glama.&lt;/strong&gt; The largest catalogue here by its own count. It filters by language, remote or local, and capability type, and sorts by measures such as GitHub stars and package downloads. It marks servers as "Official" and as "Claimed" by their authors. It also has an API for the directory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;mcp.so.&lt;/strong&gt; A browsable directory with top categories, featured and trending lists, rankings, and collections. It separates remote servers from the rest and also lists clients, skills, and agents. It describes its listings as including setup guides for Claude, Cursor, and other clients.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MCP Market.&lt;/strong&gt; A directory of servers, clients, and agent skills, organized by category. It also offers a way to sell skills.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Smithery.&lt;/strong&gt; Now a public directory without the connection services, as described above. Its verified server pages give the vendor's direct URL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;PulseMCP.&lt;/strong&gt; A directory that classifies each server as Official, Reference, or Community, and shows an estimated weekly visitor count and a release date for each. One of its team is a member of the official registry's working group. On 20 November 2026 the site carried a notice that new submissions and listing changes were paused while it reworks how listings are ingested.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SourceWeft.&lt;/strong&gt; A directory of MCP servers and, separately, of agent skills. Each server page shows the publisher and what was verified about it, the transport and runtime, the authentication method, the license, the version history, and the tool list where it could be read. Each server also has a machine-readable connect guide at &lt;code&gt;/mcp/&amp;lt;identifier&amp;gt;/install.md&lt;/code&gt; for an agent to follow. Servers are grouped into 17 categories. SourceWeft is also an AI workspace, so a listed server can be added to a workspace directly. For finding agent skills rather than servers, see &lt;a href="https://sourceweft.com/blog/where-to-find-and-install-agent-skills" rel="noopener noreferrer"&gt;Where to find agent skills for Claude, and how to install one from GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to choose
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Publishing a server?&lt;/strong&gt; Submit it to the official registry. Directories that read the registry pick it up from there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Looking for one specific server?&lt;/strong&gt; Search any of them, then go to the vendor's own documentation for the URL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exploring what exists for a task?&lt;/strong&gt; Use a directory with categories and filters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Comparing servers for trust?&lt;/strong&gt; Use a directory that shows the publisher, the authentication method, and the tool list on the page, so you can apply your own checks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Building tooling?&lt;/strong&gt; Start from the official registry's API.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What no directory can tell you
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;That a server's tools are safe.&lt;/strong&gt; Labels such as "verified" or "official" identify the publisher. They do not mean the code was reviewed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;That a listing is current.&lt;/strong&gt; A server can change after it was indexed. Check the version and update date.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;That a popular server is a good fit.&lt;/strong&gt; Stars and downloads measure attention.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read the tool list and the authentication method yourself before you connect anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What are the alternatives to Smithery for MCP servers?&lt;/strong&gt;&lt;br&gt;
For finding servers: the official MCP Registry, Glama, mcp.so, MCP Market, PulseMCP, and SourceWeft. For connecting to them: use each vendor's own MCP URL directly, which is what Smithery itself now recommends.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is Smithery shut down?&lt;/strong&gt;&lt;br&gt;
No. Smithery's connection services ended on 31 October 2026: its proxy URLs, API, accounts, and API keys. The website remains as a public directory, now part of Arcade.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is there an official MCP server registry?&lt;/strong&gt;&lt;br&gt;
Yes, at registry.modelcontextprotocol.io. It is run in the open by MCP contributors and has a public API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which MCP directory is the biggest?&lt;/strong&gt;&lt;br&gt;
By the numbers each site displayed on 20 November 2026, Glama, with 124,343 servers. Counts are not directly comparable between directories.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where can I find MCP servers by category?&lt;/strong&gt;&lt;br&gt;
Glama, mcp.so, MCP Market, and SourceWeft all group servers by category. SourceWeft has 17 categories, such as Databases, Browser Automation, and Web Search &amp;amp; Scraping.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which directories have install guides?&lt;/strong&gt;&lt;br&gt;
mcp.so describes its listings as including setup guides. SourceWeft has a connect guide on every server page and a machine-readable version for agents.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do directories host the servers?&lt;/strong&gt;&lt;br&gt;
Generally no. They list servers and link to where each one runs or to its source. Smithery used to proxy connections and no longer does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related articles
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://sourceweft.com/blog/how-to-choose-a-trustworthy-mcp-server" rel="noopener noreferrer"&gt;How to choose a trustworthy MCP server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://sourceweft.com/blog/where-to-find-and-install-agent-skills" rel="noopener noreferrer"&gt;Where to find agent skills for Claude, and how to install one from GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://sourceweft.com/blog/what-is-mcp" rel="noopener noreferrer"&gt;What is the Model Context Protocol (MCP)?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;I maintain SourceWeft, which publishes this directory. This article was drafted with an AI assistant and reviewed by me.&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>ai</category>
      <category>opensource</category>
      <category>agents</category>
    </item>
    <item>
      <title>From PDF to Markdown: Why Document Parsing is Important For RAG.</title>
      <dc:creator>LEI QIN</dc:creator>
      <pubDate>Sun, 15 Mar 2026 17:15:13 +0000</pubDate>
      <link>https://dev.to/lei_qin_486b5f0693eac8f66/from-pdf-to-markdown-why-document-parsing-is-important-for-rag-4ao</link>
      <guid>https://dev.to/lei_qin_486b5f0693eac8f66/from-pdf-to-markdown-why-document-parsing-is-important-for-rag-4ao</guid>
      <description>&lt;p&gt;RAG (Retrieval Augmented Generation) is quickly becoming the default pattern for grounding LLMs in your own data. But the quality of your RAG system depends heavily on a step many teams overlook: &lt;strong&gt;how you turn documents into text before they ever hit the vector store&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If your source is PDF-heavy—technical docs, reports, contracts—the parsing layer can make or break retrieval. Here’s why it matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Parsing Quality Matters for Retrieval
&lt;/h2&gt;

&lt;p&gt;RAG works by embedding chunks of text, storing them in a vector DB, and retrieving the most relevant chunks at query time. The better those chunks reflect the document’s structure and meaning, the better the model can answer questions.&lt;/p&gt;

&lt;p&gt;Bad parsing (raw text extraction, naive PDF-to-text):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Broken tables&lt;/strong&gt; → numbers and headers get mixed into paragraphs; retrieval returns incomplete or nonsensical rows&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lost headings&lt;/strong&gt; → no semantic hierarchy; chunk boundaries ignore section logic&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Garbled layout&lt;/strong&gt; → multi-column or complex docs produce a jumbled reading order&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Noise&lt;/strong&gt; → headers, footers, page numbers pollute chunks and dilute relevance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Good parsing&lt;/strong&gt; (structured Markdown with layout-aware extraction):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Clean tables&lt;/strong&gt; → preserved as HTML or structured blocks; chunks can contain coherent rows and cells&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Preserved hierarchy&lt;/strong&gt; → headings become natural chunk boundaries; sections stay intact&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logical flow&lt;/strong&gt; → reading order follows the document’s intended structure&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Less noise&lt;/strong&gt; → boilerplate can be filtered; chunks are more representative of actual content&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The result: embeddings capture real meaning, and retrieval returns chunks that actually answer the question instead of fragments of garbage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Markdown as an Ideal RAG Ingestion Format
&lt;/h2&gt;

&lt;p&gt;Markdown is a natural fit for RAG pipelines because it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keeps &lt;strong&gt;structure&lt;/strong&gt; (headings, lists, tables) explicit instead of hidden in layout&lt;/li&gt;
&lt;li&gt;Is &lt;strong&gt;easy to chunk&lt;/strong&gt; — split on &lt;code&gt;##&lt;/code&gt; or by block type without arbitrary character cuts&lt;/li&gt;
&lt;li&gt;Preserves &lt;strong&gt;semantic units&lt;/strong&gt; — a table row, a code block, or a section stays together&lt;/li&gt;
&lt;li&gt;Plays well with &lt;strong&gt;embeddings&lt;/strong&gt; — models typically train on Markdown; structured text tends to embed more predictably&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Raw PDF text, by contrast, often looks like one long run-on with no clear boundaries. You either chunk by fixed token count (losing context) or try to infer structure yourself (expensive and fragile). Starting from clean Markdown gives you structure “for free.”&lt;/p&gt;

&lt;h2&gt;
  
  
  What to Look For in a Parser
&lt;/h2&gt;

&lt;p&gt;When building a document ingestion pipeline for RAG, look for parsers that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Preserve structure&lt;/strong&gt; — headings, tables, lists, code blocks&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Produce Markdown&lt;/strong&gt; — or structured JSON you can convert to Markdown&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handle layout&lt;/strong&gt; — multi-column, complex formatting, reading order&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expose block metadata&lt;/strong&gt; — block types, bounding boxes, reading order for custom chunking&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>nlp</category>
      <category>rag</category>
    </item>
    <item>
      <title>Claude Code Curl Alternative: When Websites Block Your Requests</title>
      <dc:creator>LEI QIN</dc:creator>
      <pubDate>Wed, 11 Mar 2026 17:36:28 +0000</pubDate>
      <link>https://dev.to/lei_qin_486b5f0693eac8f66/claude-code-curl-alternative-when-websites-block-your-requests-491d</link>
      <guid>https://dev.to/lei_qin_486b5f0693eac8f66/claude-code-curl-alternative-when-websites-block-your-requests-491d</guid>
      <description>&lt;h1&gt;
  
  
  Claude Code Curl Alternative: When Websites Block Your Requests
&lt;/h1&gt;

&lt;p&gt;You've got Claude Code doing research or pulling data—and it leans on &lt;code&gt;curl&lt;/code&gt; to fetch pages. Works like a charm at first. Then you hit a site that slams the door: 403 Forbidden, access denied, or nothing but a blank screen. "Fine," you think, "I'll spin up Playwright locally." Real browser, real JavaScript—what could go wrong? Turns out: plenty. Local Playwright adds serious overhead and often runs into the same anti-bot walls. There's a smoother path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Claude Code Curl Fails on Certain Websites
&lt;/h2&gt;

&lt;p&gt;Claude Code fetches pages with curl: lightweight, straightforward GET requests. Nothing fancy. The catch? Modern sites are built to spot and block exactly that kind of traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Sites Say No to Curl
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Missing or suspicious headers&lt;/strong&gt; — Curl keeps it minimal. Sites expect &lt;code&gt;User-Agent&lt;/code&gt;, &lt;code&gt;Accept-Language&lt;/code&gt;, &lt;code&gt;Accept-Encoding&lt;/code&gt;, and more. Skip those, and you look like a bot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No JavaScript execution&lt;/strong&gt; — Plenty of sites ship skeleton HTML and load the real content via JS. Curl grabs the skeleton. You get empty shells.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IP-based rate limiting&lt;/strong&gt; — One IP, lots of requests. Whether it's your machine or Claude's backend, you burn through limits fast.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bot detection&lt;/strong&gt; — TLS fingerprints, header order, timing, behavior. Curl's signature is easy to read.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When any of that kicks in, you see &lt;strong&gt;403 Forbidden&lt;/strong&gt;, &lt;strong&gt;429 Too Many Requests&lt;/strong&gt;, &lt;strong&gt;503 Service Unavailable&lt;/strong&gt;, or blank pages. No content means Claude Code has nothing to work with.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Local Playwright Trap
&lt;/h2&gt;

&lt;p&gt;You might think: real browser, real everything—problem solved. Playwright &lt;em&gt;does&lt;/em&gt; handle JavaScript and feels more like a real browser. But it comes with trade-offs:&lt;/p&gt;

&lt;h3&gt;
  
  
  The Heavier Stack
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Heavy dependencies&lt;/strong&gt; — Chromium, Firefox, or WebKit. Installation and updates eat hundreds of MB.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory and CPU&lt;/strong&gt; — Each headless instance chews RAM and CPU. Parallel crawls get expensive fast.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Environment headaches&lt;/strong&gt; — Different OSes, Node versions, Playwright versions. "Works on my machine" becomes your new catchphrase.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Blocked Anyway
&lt;/h3&gt;

&lt;p&gt;Even with a real browser, many high-value targets—e-commerce, news, travel, finance—use bot detection that still catches you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cloudflare challenges&lt;/strong&gt; — Turnstile, JS challenges, "Checking your browser" screens. They stop curl, and often stop local headless browsers too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Headless detection&lt;/strong&gt; — &lt;code&gt;navigator.webdriver&lt;/code&gt;, missing plugins, odd resolutions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Behavioral analysis&lt;/strong&gt; — No mouse moves, no natural scroll, inhuman click speeds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proxy and VPN detection&lt;/strong&gt; — Datacenter IPs? Often flagged.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Result: same &lt;strong&gt;403s&lt;/strong&gt;, &lt;strong&gt;Cloudflare&lt;/strong&gt;, and &lt;strong&gt;captchas&lt;/strong&gt;. But now you're lugging a heavier stack.&lt;/p&gt;

&lt;h2&gt;
  
  
  AnyCrawl: Crawl for AI
&lt;/h2&gt;

&lt;p&gt;AnyCrawl is &lt;strong&gt;crawl for AI&lt;/strong&gt;—a cloud scraping API built for AI agents and LLMs. It gets around these limits, &lt;strong&gt;including Cloudflare challenges&lt;/strong&gt;, without running browsers on your machine. You get &lt;strong&gt;LLM-optimized Markdown&lt;/strong&gt; back, ready for Claude and others. When curl or local Playwright hits a wall, this is the drop-in replacement.&lt;/p&gt;

&lt;h3&gt;
  
  
  How It Handles What Curl Can't
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Curl Problem&lt;/th&gt;
&lt;th&gt;AnyCrawl Solution&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Minimal headers, easy to fingerprint&lt;/td&gt;
&lt;td&gt;Full browser-like headers and request patterns&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No JavaScript execution&lt;/td&gt;
&lt;td&gt;Cheerio, Playwright, and Puppeteer engines in the cloud&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single IP, easily rate-limited&lt;/td&gt;
&lt;td&gt;Rotating proxy support to distribute traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cloudflare challenges, 403, anti-bot&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Bypass Cloudflare&lt;/strong&gt; and managed anti-detection&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;AnyCrawl &lt;strong&gt;bypasses Cloudflare challenges&lt;/strong&gt; (Turnstile, JS checks, and more), so sites that block curl or local browsers are still reachable. Send a URL, get clean Markdown. No local browser, no proxy setup, no maintenance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Claude Code Integration via AnyCrawl CLI Skill
&lt;/h3&gt;

&lt;p&gt;The best way to use AnyCrawl with Claude Code is the &lt;a href="https://github.com/any4ai/anycrawl-cli" rel="noopener noreferrer"&gt;AnyCrawl CLI&lt;/a&gt; and its &lt;strong&gt;skill&lt;/strong&gt; for AI coding agents:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Install AnyCrawl CLI&lt;/span&gt;
npm &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; anycrawl-cli
&lt;span class="c"&gt;# Or: npx -y anycrawl-cli init&lt;/span&gt;

&lt;span class="c"&gt;# Authenticate (get API key at anycrawl.dev/dashboard)&lt;/span&gt;
anycrawl login &lt;span class="nt"&gt;--api-key&lt;/span&gt; &amp;lt;your-api-key&amp;gt;

&lt;span class="c"&gt;# Install skill for Claude Code (and Cursor, Codex, etc.)&lt;/span&gt;
anycrawl setup skills
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The skill teaches Claude Code when and how to use AnyCrawl for scraping, search, mapping, and crawling—swapping in AnyCrawl when curl gets blocked.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MCP integration&lt;/strong&gt; — Add the AnyCrawl MCP server to Claude Code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;claude mcp add &lt;span class="nt"&gt;--transport&lt;/span&gt; stdio anycrawl &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="nv"&gt;ANYCRAWL_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YOUR_API_KEY &lt;span class="nt"&gt;--&lt;/span&gt; npx &lt;span class="nt"&gt;-y&lt;/span&gt; anycrawl-mcp@latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Swap &lt;code&gt;YOUR_API_KEY&lt;/code&gt; with your key from &lt;a href="https://anycrawl.dev/dashboard" rel="noopener noreferrer"&gt;anycrawl.dev/dashboard&lt;/a&gt;. Or run &lt;code&gt;anycrawl setup mcp&lt;/code&gt; after installing the CLI.&lt;/p&gt;

&lt;h3&gt;
  
  
  API Usage (Curl-Compatible)
&lt;/h3&gt;

&lt;p&gt;Want to call the API directly from scripts or another AI? Same interface:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.anycrawl.dev/v1/scrape"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer YOUR_API_KEY"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "url": "https://example.com/page-that-blocks-curl",
    "engine": "playwright"
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Response includes &lt;code&gt;markdown&lt;/code&gt; or &lt;code&gt;html&lt;/code&gt; (or both), &lt;strong&gt;optimized for AI&lt;/strong&gt;—ready for Claude or any LLM.&lt;/p&gt;

&lt;h2&gt;
  
  
  1,500 Credits: Enough for Daily Use
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Free tier&lt;/strong&gt; gives you &lt;strong&gt;1,500 credits per month&lt;/strong&gt;—no card required. Credit use depends on page size and engine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cheerio&lt;/strong&gt; (static HTML) — lowest cost, ideal when JS isn't needed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Playwright&lt;/strong&gt; (JS-heavy sites) — moderate cost, handles most anti-bot cases&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Puppeteer&lt;/strong&gt; — similar to Playwright for Chrome-specific needs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For everyday research, docs lookup, and small-scale extraction—dozens to a few hundred pages a month—1,500 credits usually covers it. Need more? Paid tiers start at 2,500 credits (Hobby) and scale from there.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Use Each Approach
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Best Tool&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Simple public pages, no anti-bot&lt;/td&gt;
&lt;td&gt;Claude Code curl&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JS-heavy but not protected&lt;/td&gt;
&lt;td&gt;Local Playwright (if you're fine with the setup)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Protected sites, Cloudflare, 403/429, captchas&lt;/td&gt;
&lt;td&gt;AnyCrawl&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High volume, many domains&lt;/td&gt;
&lt;td&gt;AnyCrawl (cloud, proxies, concurrency)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code integration&lt;/td&gt;
&lt;td&gt;AnyCrawl CLI + skill (&lt;code&gt;anycrawl setup skills&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Sign up at &lt;a href="https://anycrawl.dev" rel="noopener noreferrer"&gt;anycrawl.dev&lt;/a&gt; 
Promo code: STARTUPY&lt;/li&gt;
&lt;li&gt;Grab your API key from the &lt;a href="https://anycrawl.dev/dashboard" rel="noopener noreferrer"&gt;dashboard&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Install the CLI and skill: &lt;code&gt;anycrawl setup skills&lt;/code&gt; (see &lt;a href="https://github.com/any4ai/anycrawl-cli" rel="noopener noreferrer"&gt;anycrawl-cli&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Or call the Scrape API directly from your own scripts&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;No credit card for Free. Try scraping a URL that blocks curl—the difference shows up right away.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Claude Code's curl is perfect for open, static pages. When you run into 403s, Cloudflare, blank responses, or rate limits, local Playwright adds complexity and often fails anyway. &lt;strong&gt;AnyCrawl is crawl for AI&lt;/strong&gt;—cloud-based, bypasses Cloudflare and anti-bot, uses rotating proxies and multiple engines, and returns LLM-ready Markdown. For daily use, 1,500 free credits cover most research and extraction—without running browsers locally or wrestling anti-bot systems yourself.&lt;/p&gt;




</description>
      <category>anycrawl</category>
      <category>claudecodeskill</category>
      <category>crawlforai</category>
      <category>firecrawlalternative</category>
    </item>
  </channel>
</rss>
