DEV Community: Harun Mahmud

I Built a Job Aggregator That Scrapes 10 Sources Every Morning — Here's How It Works

Harun Mahmud — Fri, 05 Jun 2026 18:30:47 +0000

Why I Built It

Job hunting is exhausting — not because jobs don't exist, but because they're scattered everywhere. LinkedIn, Indeed, RemoteOK, We Work Remotely, Remotive… you end up checking six tabs every morning just to see what's new.

I wanted one place that quietly does that work for you. Browse it on the site, or just get a clean email digest every morning with only the roles that match what you're looking for.

That's DailyJobFeed.com — a read-only job aggregator. No job posting, no accounts, no applications. Just a fresh, filtered feed of real jobs updated daily.

How It Works

1. Scraping — every morning at 6 AM

A Python cron job runs daily and pulls listings from 10 sources. Each source uses a different method:

RSS feeds — We Work Remotely exposes 9 category feeds
Public JSON APIs — RemoteOK, Remotive, Arbeitnow, Himalayas, Jobicy
Keyed REST APIs — Adzuna (12 countries), USAJobs (US federal), Reed UK, The Muse

Each source is wrapped in its own module. If one fails, it's caught and the rest continue. Nothing blocks the run.

2. Normalization

Every source speaks a different language. Remotive calls it "Software Development". RemoteOK uses freeform tags. WWR uses RSS feed names. The scraper normalizes all of this into a consistent canonical schema:

{
  "slug":             make_slug(title, company, url),
  "title":            title,
  "company":          company,
  "category":         canonical_category(title, tags),  # Engineering, Design, Marketing…
  "experience_level": parse_experience(title),           # entry, mid, senior, lead
  "remote_type":      "remote" | "hybrid" | "on-site",
  "job_type":         "full-time" | "part-time" | "contract",
  "country":          ...,
  "salary_min":       ...,
  "salary_max":       ...,
  "source_url":       ...,
  "posted_at":        ...,
}

canonical_category() is a keyword-matching function that maps any free-text label into one of ~20 fixed buckets. Getting this right — especially across sources that use completely different taxonomies — took more iteration than anything else in the project.

3. Deduplication

Jobs get cross-posted across sources constantly. Deduplication is slug-based: a SHA-256 hash of the source URL is truncated to 12 hex characters and appended to the slug. That gives 281 trillion unique combinations with no collision risk. The database has a UNIQUE constraint on source_url as a second safety net.

4. Storage

Jobs live in PostgreSQL. Only the last 6 months of listings are kept — a cleanup job runs before each scrape to purge anything older.

Job descriptions are sanitized with bleach before storage — dangerous tags (script, iframe, event handlers) are stripped and only safe formatting tags (p, ul, li, strong, etc.) are kept. Descriptions are capped at 50KB.

5. The Frontend

The site is built with Nuxt 3 (Vue 3, SSR, TypeScript). Server-side rendering matters here because every job page needs to be crawlable.

The job listing page supports filtering by:

Category (Engineering, Design, Marketing, Data, DevOps, etc.)
Work mode (Remote, Hybrid, On-site)
Experience level (Entry, Mid, Senior, Lead)
Country (40+ countries)
Job type (Full-time, Part-time, Contract)
Date posted (last 24h, 3 days, week, month)
Salary range
Visa sponsorship

Filters hit a Supabase API endpoint with faceted aggregation via a custom PostgreSQL RPC — so sidebar counts update accurately with every filter combination without expensive full-table scans.

One small detail: on the default unfiltered view, results are round-robin interleaved across all 9 scrapers so no single source dominates the top of the feed.

6. The Email Digest

Subscribers enter their email and set preferences — categories, work mode, send time. That's it. No password.

Every morning after the scraper finishes, the email sender:

Queries all active subscribers
Filters the last 24 hours of jobs against each subscriber's preferences
Builds a digest — headline with total match count, 6 preview job cards, a CTA linking to the full filtered feed
Sends via Resend API
Updates last_emailed_at on each subscriber row

Unsubscribe is a single link click — no confirmation screen, no login. It sets is_active = FALSE. Rows are never deleted.

The Tech Stack

Layer	Technology
Frontend	Nuxt 3 (Vue 3, SSR, TypeScript)
Styling	SCSS + CSS custom properties (no Tailwind)
Database	PostgreSQL
DB hosting	Supabase
Frontend hosting	Vercel
Scraper	Python
Cron runner	Railway
Email	Resend API

The Biggest Technical Challenge

Normalization. Every source has its own category system, experience labels, and remote-type conventions. Building canonical_category() — a reliable function that maps any combination of title, tags, and source category into one of ~20 fixed buckets — required reading through thousands of real job titles and edge cases.

A sample of what the mapping handles:

("data scientist",     "Data"),
("ml engineer",        "Data"),
("platform engineer",  "DevOps"),
("entwickler",         "Engineering"),  # German: developer
("product owner",      "Product"),
("devrel",             "Marketing"),

Without this, filters would be useless — a "show me Engineering jobs" query would miss half the results because some source filed them under "Software Development" or "Tech".

What I Skipped (Intentionally)

No Indeed scraping — ToS risk, and the only reliable approach requires a paid proxy that wasn't in scope
No user accounts — subscribers only provide email + preferences. No passwords, no sessions, no JWTs
No job applications — every listing links directly to the original source. DailyJobFeed is read-only

If you're building something similar or have questions about any part of the architecture, happy to discuss in the comments.

🔗 dailyjobfeed.com

Text Case Converter: The Complete Guide to camelCase, snake_case, kebab-case and More

Harun Mahmud — Sat, 16 May 2026 03:47:09 +0000

Every developer has been there — you copy a variable name from one codebase and it's in the wrong case for your project. Or you're converting a database column name to a JavaScript variable. Or you need to turn a human-readable label into a URL slug.

Text case conversion is one of those small tasks you do dozens of times a day. Here's a complete guide to every case format and when to use each one.

The Most Common Cases

camelCase

Words are joined together, first word lowercase, subsequent words capitalized.
helloWorld
firstName
getUserById
myVariableName

Used for: JavaScript/TypeScript variables, function names, object properties, JSON keys.

PascalCase (UpperCamelCase)

Same as camelCase but the first letter is also capitalized.
HelloWorld
FirstName
GetUserById
MyComponent

Used for: React components, TypeScript interfaces, class names, constructors.

snake_case

Words are separated by underscores, all lowercase.
hello_world
first_name
get_user_by_id
my_variable_name

Used for: Python variables, database column names, Ruby, file names in some projects.

SCREAMING_SNAKE_CASE

All uppercase with underscores.
HELLO_WORLD
MAX_RETRY_COUNT
API_BASE_URL
DATABASE_HOST

Used for: Constants, environment variables, config values.

kebab-case

Words separated by hyphens, all lowercase.
hello-world
first-name
get-user-by-id
my-component

Used for: CSS class names, HTML attributes, URL slugs, file names, CLI flags.

Title Case

Each word starts with a capital letter.
Hello World
First Name
Get User By Id
My Variable Name

Used for: Page titles, headings, button labels, UI text.

dot.case

Words separated by dots, all lowercase.
hello.world
first.name
config.database.host

Used for: Config file keys, logging namespaces, some API response formats.

Quick Reference Table

Case	Example	Common Use
camelCase	`myVariable`	JS variables, JSON keys
PascalCase	`MyComponent`	Classes, React components
snake_case	`my_variable`	Python, databases
SCREAMING_SNAKE	`MY_CONSTANT`	Constants, env vars
kebab-case	`my-variable`	CSS, URLs, HTML
Title Case	`My Variable`	UI labels, headings
dot.case	`my.variable`	Config, namespaces

How to Convert in JavaScript

const str = "hello world example";

// to camelCase
const camel = str.replace(/\s(.)/g, (_, c) => c.toUpperCase());
// "helloWorldExample"

// to PascalCase
const pascal = str.replace(/(^\w|\s\w)/g, c => c.trim().toUpperCase());
// "HelloWorldExample"

// to snake_case
const snake = str.replace(/\s+/g, '_').toLowerCase();
// "hello_world_example"

// to kebab-case
const kebab = str.replace(/\s+/g, '-').toLowerCase();
// "hello-world-example"

// to SCREAMING_SNAKE_CASE
const screaming = str.replace(/\s+/g, '_').toUpperCase();
// "HELLO_WORLD_EXAMPLE"

How to Convert in Python

import re

text = "hello world example"

# to snake_case
snake = text.replace(" ", "_").lower()
# "hello_world_example"

# to kebab-case
kebab = text.replace(" ", "-").lower()
# "hello-world-example"

# to PascalCase
pascal = "".join(word.capitalize() for word in text.split())
# "HelloWorldExample"

# to camelCase
words = text.split()
camel = words[0].lower() + "".join(w.capitalize() for w in words[1:])
# "helloWorldExample"

Stop Converting Manually

Writing a one-off conversion function every time gets old fast. A dedicated tool lets you paste any text and instantly see all formats side by side — camelCase, PascalCase, snake_case, kebab-case, SCREAMING_SNAKE, Title Case, dot.case and more in one click.

👉 ToolsCraft Text Case Converter — paste any text, convert to any case instantly. Free, works in your browser, no account needed.

When You're Working Across Languages

The case confusion gets worse when you're working across different parts of a stack:

Your database uses snake_case columns
Your API response uses camelCase JSON keys
Your CSS uses kebab-case classes
Your constants use SCREAMING_SNAKE_CASE
Your React components use PascalCase

All from the same underlying concept — just named differently depending on where it lives. A quick converter saves you the mental overhead of doing it by hand every time.

Which case format trips you up most often when switching between languages or frameworks? Drop it in the comments 👇

JSON Formatter: 5 Ways to Pretty-Print JSON (Online, CLI, VS Code, JavaScript & Python)

Harun Mahmud — Mon, 11 May 2026 17:55:52 +0000

If you've ever stared at a wall of minified JSON like this:

{"user":{"id":1,"name":"John","email":"john@example.com","roles":["admin","editor"],"active":true}}

You know how painful it is to read. Here are 5 ways to format it instantly depending on where you're working.

1. Online JSON Formatter (fastest, no install)

If you just need to quickly format a JSON response from an API or debug a config file, an online tool is the fastest option.

The important thing is to use one that runs in your browser and doesn't send your data to a server — especially if your JSON contains tokens, user data, or anything sensitive.

👉 ToolsCraft JSON Formatter — formats, validates, and minifies JSON entirely client-side. Nothing leaves your machine.

2. VS Code (best for files)

VS Code has JSON formatting built in. Open your .json file and press:

Windows/Linux: Shift + Alt + F
Mac: Shift + Option + F

Or right-click → Format Document.

To auto-format on save, add this to your settings.json:

{
  "[json]": {
    "editor.formatOnSave": true
  }
}

3. JavaScript (in your code)

const ugly = '{"name":"John","age":30,"city":"New York"}';
const pretty = JSON.stringify(JSON.parse(ugly), null, 2);
console.log(pretty);

The third argument in JSON.stringify is the indentation level. Use 2 or 4 depending on your preference.

4. Python (scripts & data processing)

import json

ugly = '{"name":"John","age":30,"city":"New York"}'
parsed = json.loads(ugly)
pretty = json.dumps(parsed, indent=2)
print(pretty)

Or directly from the terminal on any JSON file:

python -m json.tool yourfile.json

5. Command Line with jq (power users)

# Install
brew install jq  # Mac
apt-get install jq  # Linux

# Format a file
jq . yourfile.json

# Format API response directly
curl https://api.example.com/users | jq .

Quick comparison

Method	Best for	Install needed
Online tool	Quick debugging, API responses	No
VS Code	Working with JSON files	No
JavaScript	In your app code	No
Python	Scripts, data processing	No
jq	Terminal, pipelines	Yes

Bonus: Validate while formatting

A good formatter should also validate your JSON and tell you exactly where the error is. If you paste invalid JSON:

{"name": "John", "age": 30,}

That trailing comma makes it invalid. A proper formatter should highlight the exact line and character where the error is — not just silently fail.

The ToolsCraft JSON Formatter shows inline error highlighting, which saves a lot of head-scratching.

That's it! Use the online tool for quick jobs, VS Code for files, and jq if you live in the terminal.

What's your go-to method for formatting JSON? Drop it in the comments 👇