DEV Community: Paul Works

Stop Hand-Holding Your AI: How to Build a Real-World Web Scraping Agent with Claude Tools 🕷️

Paul Works — Sun, 26 Apr 2026 08:31:01 +0000

Let’s be honest. Large Language Models (LLMs) are incredibly smart, but they suffer from one crippling weakness: they are trapped in a box.

If you ask Claude to summarize a breaking news article or read documentation for a brand-new library released yesterday, it will apologize and say it doesn't have real-time internet access.

But what if you could give Claude the ability to browse the web itself?

Enter Claude Tools (Anthropic's version of Function Calling). By giving Claude tools, you transform it from a conversational chatbot into a powerhouse autonomous agent.

Instead of showing you a boring "Get Current Time" example, we are going to build something you can actually use in production today: A Web-Scraping Assistant.

🎭 How Tool Use Actually Works

Using tools with Claude isn't magic; it's a 3-step conversation loop:

You: "Here is a prompt, and here is a list of Python scripts (tools) you can ask me to run if you need help."
Claude: "I need to read this URL. Please run the fetch_webpage tool for me."
You: "Here is the raw text from the website!"
Claude: "Here is the summary of the article..."

Claude never executes the code itself. It just outputs a JSON payload telling your server to run it.

💻 Let's Build It: Giving Claude the Internet

First, let's write a real-world Python function using requests and BeautifulSoup to scrape any webpage and extract the text.

import requests
from bs4 import BeautifulSoup

# 1. The actual Python function
def fetch_webpage(url: str):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        # Parse HTML and extract just the text
        soup = BeautifulSoup(response.text, 'html.parser')
        text = soup.get_text(separator=' ', strip=True)

        # Truncate to avoid blowing up the context window!
        return text[:8000] 
    except Exception as e:
        return f"Failed to fetch webpage: {str(e)}"

Next, we tell Claude that this function exists by defining a JSON Schema:

# 2. Tell Claude about it
fetch_webpage_schema = {
  "name": "fetch_webpage",
  "description": "Fetches the raw text content of a given URL. Use this tool when you need to read an article, documentation, or any web page to answer the user's question.",
  "input_schema": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The exact, fully qualified URL to scrape (e.g., https://example.com)"
      }
    },
    "required": ["url"]
  }
}

Pro-Tip 💡: The description field here is incredibly important. Claude reads it to decide when and whether it should use this tool! Outline exactly when the AI should reach for it.

Wiring it up to the API

Now let's ask Claude a question about recent news that it absolutely cannot answer without scraping the web.

import anthropic

client = anthropic.Anthropic() # Relies on ANTHROPIC_API_KEY in your .env

messages = [{
    "role": "user", 
    "content": "Can you read this article and give me 3 bullet points summarizing it? https://dev.to/about"
}]

# 3. Make the API Call
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1000,
    messages=messages,
    tools=[fetch_webpage_schema] # Pass the tool schema here!
)

Instead of returning a standard text message, Claude's stop_reason will be "tool_use". It is asking us to scrape the website for it!

# Claude's response looks like this:
[
  ToolUseBlock(
    type='tool_use', 
    id='toolu_01...xyz', 
    name='fetch_webpage', 
    input={'url': 'https://dev.to/about'}
  )
]

Closing the Loop

Claude is waiting for the data! It gave us the ToolUseBlock telling us exactly what tool to execute, and grabbed the url from our prompt. Let's finish the job:

# A. Save Claude's tool request to our message history
messages.append({"role": "assistant", "content": response.content})

# B. Actually execute the Python function locally
tool_request = response.content[0]
scraped_text = fetch_webpage(**tool_request.input)

# C. Send the scraped text back to Claude
messages.append({
    "role": "user",
    "content": [{
        "type": "tool_result",
        "tool_use_id": tool_request.id,
        "content": scraped_text,
        "is_error": False
    }]
})

# D. Make the final API call so Claude can read the text and answer
final_response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1000,
    messages=messages,
    tools=[fetch_webpage_schema]
)

print(final_response.content[0].text)

Output:

"Based on the article at that URL, here are 3 key takeaways about DEV:

DEV is a community of software developers getting together to help one another out.

The platform is built on open-source software called Forem.

They focus on fostering an inclusive, decentralized, and positive environment for developers of all backgrounds."

🚀 Where to go from here?

You've just built a fully functioning web-scraping AI Agent. But you don't have to stop at reading data. You can define tools that let Claude:

Create calendar invites via the Google Calendar API
Query your PostgreSQL database to generate custom SQL reports
Execute Bash commands to manipulate files on your machine (just like Devin)

By bridging the gap between LLM reasoning and real-world execution, you are no longer building chatbots—you're building agents.

Take off the training wheels and give Claude a real tool today!

I Cut My Rails Hosting Costs by 70%: Migrating from Heroku to Railway

Paul Works — Sun, 08 Mar 2026 14:59:09 +0000

I Cut My Rails Hosting Costs by 70%: Migrating from Heroku to Railway

For a while, I was hosting a small Rails API on Heroku with JawsDB as the MySQL database.

It worked perfectly. Heroku’s developer experience is still one of the best out there.

But for a small personal project, I realized I was paying more than necessary.

So I migrated the app and database to Railway and reduced my hosting costs from roughly ~$20/month to ~$6/month — about a 70% reduction.

The best part: the migration took less than 30 minutes.

Here’s exactly how I did it.

TL;DR

I migrated a Rails API from Heroku + JawsDB to Railway.

Results:

Hosting cost reduced from ~$20/month → ~$6/month
Migration took ~20–30 minutes
No downtime
Database migrated using one streaming command

My Original Setup

My stack looked like this:

Rails API
   │
Heroku Dyno
   │
JawsDB (MySQL)

Approximate monthly cost:

Service	Provider	Cost
App hosting	Heroku	~$7–10
MySQL	JawsDB	~$10–12
Total		~$20/month

Again, not huge — but for side projects, I prefer keeping infrastructure simple and inexpensive.

The New Setup

After migrating:

Rails API
   │
Railway App
   │
Railway MySQL

New cost breakdown:

Service	Provider	Cost
App hosting	Railway	~$3–4
MySQL	Railway	~$2
Total		~$6/month

That’s roughly a 70% cost reduction.

Migration Strategy

The migration consisted of four steps:

Link the Railway project
Export the database from JawsDB
Import the data into Railway MySQL
Update the Rails configuration

No downtime was required.

Step 1: Link Your Railway Project

First install and authenticate with the Railway CLI.

Then run:

railway link

What this does:

Connects your local repo to your Railway project
Allows the CLI to run commands against that project
Creates a .railway folder locally

This step only needs to be done once per repository.

Step 2: Dump Data from JawsDB and Import into Railway

Instead of creating a dump file and importing it later, you can stream the database directly.

Here’s the command I used:

mysqldump \
-h [jawsdb-host] \
-u [jawsdb-user] \
-p[jawsdb-password] \
--no-tablespaces \
--set-gtid-purged=OFF \
--single-transaction \
[jawsdb-database] \
| railway run -s MySQL mysql \
-h $MYSQLHOST \
-u $MYSQLUSER \
-p$MYSQLPASSWORD \
$MYSQLDATABASE

This command:

Exports the database from JawsDB
Pipes the output directly into Railway MySQL

Why These Flags Matter

`--no-tablespaces`

JawsDB runs on AWS RDS, which restricts certain privileges required for dumping tablespaces.

This flag avoids that issue.

`--set-gtid-purged=OFF`

Prevents replication metadata from causing conflicts during import.

`--single-transaction`

Creates a consistent snapshot of the database while exporting.

This works well for InnoDB tables, which most Rails apps use.

Why Streaming the Database Is Powerful

The pipe (|) streams the SQL output directly into Railway.

JawsDB → mysqldump → pipe → Railway MySQL

Benefits:

No intermediate dump files
Faster migration
Lower disk usage
Works well for large databases

Step 3: Verify the Migration

After the import, verify the migration with:

railway run -s MySQL mysql \
-h $MYSQLHOST \
-u $MYSQLUSER \
-p$MYSQLPASSWORD \
$MYSQLDATABASE \
-e "SHOW TABLES;"

If the migration worked correctly, you should see your tables:

users
transactions
posts
comments

Step 4: Update Rails Configuration

Previously, my Rails app used the JawsDB DATABASE_URL.

Railway automatically provides environment variables:

MYSQLHOST
MYSQLPORT
MYSQLUSER
MYSQLPASSWORD
MYSQLDATABASE
MYSQL_URL

The simplest Rails configuration is:

production:
  adapter: mysql2
  url: <%= ENV['MYSQL_URL'] %>

After pushing the code, Railway automatically redeployed the application.

Heroku vs Railway (Quick Comparison)

Here’s a quick comparison based on my experience migrating a small Rails API.

Feature	Heroku + JawsDB	Railway
Hosting model	App dynos + add-ons	App + services in one platform
Database	JawsDB MySQL add-on	Native managed MySQL
Cost (small project)	~$20/month	~$6/month
Deployment	Git push	Git push / GitHub integration
Environment variables	Supported	Supported
CLI tools	Heroku CLI	Railway CLI
Database migration	Manual dump/import	Easy via `railway run`
Infrastructure management	Managed	Managed
Static outbound IP	Not guaranteed	Not guaranteed
Best suited for	Mature production apps	Side projects & small apps

Final Thoughts

Heroku still offers one of the best developer experiences.

But for small projects or side tools, simplifying infrastructure and reducing costs can make sense.

For my Rails API, moving to Railway:

simplified my stack
reduced hosting costs by ~70%
required minimal migration effort

If you're running a small Rails project with MySQL on Heroku, exploring Railway might be worth considering.

Question for Other Developers

If you're hosting side projects:

Are you still using Heroku?
Or have you moved to platforms like Railway, Fly.io, or Render?

I'd love to hear what your stack looks like.