Filtering Freelance Leads: Recency Windows and Job‑Board Rules in My Home‑Hosted Scanner

Introduction

When I started scanning freelance job boards for my own projects, I quickly ran into two problems: most listings were stale, and each board had its own quirks that made a naïve scraper return a lot of noise. In this post I walk through how I built a lightweight filter layer that sits on top of my home‑hosted lead scanner. The layer applies recency windows and board‑specific rules so that only fresh, relevant opportunities make it into my daily Slack digest.

Architecture Overview

The scanner runs on a modest Raspberry Pi 4 that I keep in my garage. A cron job triggers a Node.js script every 30 minutes. The script:

1. Fetches raw JSON feeds from the boards I monitor (Upwork, Freelancer, Guru, etc.).
2. Normalises the payload into a common shape.
3. Passes the normalised items through a filter pipeline.
4. Sends the surviving leads to a Slack webhook.

All of the filtering logic lives in a single filters.js module so I can tweak rules without touching the scraper.

Pulling Job Posts

Each board exposes a slightly different API. Here’s a trimmed example for Upwork:

// src/fetchers/upwork.js
const fetch = require('node-fetch');

module.exports = async function fetchUpwork() {
  const resp = await fetch('https://www.upwork.com/api/v3/search/jobs?category=web-development');
  const data = await resp.json();
  // Normalise to our internal shape
  return data.jobs.map(job => ({
    id: job.id,
    title: job.title,
    board: 'upwork',
    postedAt: new Date(job.created_at),
    budget: job.budget?.amount,
    description: job.description,
    url: job.url,
  }));
};

Similar fetchers exist for the other boards. The important part is that every job ends up with the same fields (id, title, board, postedAt, …) so the filter code can stay board‑agnostic.

Defining Recency Windows

A recency window is simply “only keep jobs posted within the last N minutes”. Different boards update at different rates, so I keep a per‑board configuration:

// config/recency.json
{
  "upwork": 60,
  "freelancer": 30,
  "guru": 120
}

The values are minutes. The filter reads this file at start‑up:

// src/filters/recency.js
const recencyConfig = require('../config/recency.json');

function withinWindow(job) {
  const minutes = recencyConfig[job.board] || 60; // default 60
  const age = (Date.now() - job.postedAt.getTime()) / 60000;
  return age <= minutes;
}

module.exports = withinWindow;

Because the config is a plain JSON file, I can adjust windows on the fly without redeploying.

Board‑Specific Rules

Beyond age, each board has its own “noise” patterns. For example:

• Upwork – many listings have a budget of $0 which usually means “hourly rate not disclosed”. I ignore those unless the title contains “fixed‑price”.
• Freelancer – the API returns duplicate entries for the same project; they share the same id but differ in url. I keep the first occurrence.
• Guru – some jobs are marked “closed” but still appear in the feed. The status field tells the truth.

I encode these as small predicate functions:

// src/filters/boardRules.js
function upworkRule(job) {
  if (job.budget === 0 && !/fixed-price/i.test(job.title)) return false;
  return true;
}

function freelancerRule(job, seen) {
  if (seen.has(job.id)) return false;
  seen.add(job.id);
  return true;
}

function guruRule(job) {
  return job.status !== 'closed';
}

module.exports = { upworkRule, freelancerRule, guruRule };

Putting It Together: The Filter Pipeline

The main pipeline composes the generic recency filter with the board‑specific rule set:

// src/filters/index.js
const withinWindow = require('./recency');
const { upworkRule, freelancerRule, guruRule } = require('./boardRules');

function filterJobs(jobs) {
  const seen = new Set(); // for duplicate detection on Freelancer
  return jobs.filter(job => {
    if (!withinWindow(job)) return false;

    switch (job.board) {
      case 'upwork':
        return upworkRule(job);
      case 'freelancer':
        return freelancerRule(job, seen);
      case 'guru':
        return guruRule(job);
      default:
        return true;
    }
  });
}

module.exports = filterJobs;

The pipeline runs after all fetchers have merged their results:

// src/main.js
const fetchUpwork = require('./fetchers/upwork');
const fetchFreelancer = require('./fetchers/freelancer');
const fetchGuru = require('./fetchers/guru');
const filterJobs = require('./filters');

(async () => {
  const raw = [
    ...(await fetchUpwork()),
    ...(await fetchFreelancer()),
    ...(await fetchGuru()),
  ];
  const leads = filterJobs(raw);
  // send leads to Slack …
})();

Running It at Home

On my Pi I use pm2 to keep the process alive and crontab to trigger it:

*/30 * * * * /usr/local/bin/node /home/kaleb/lead‑scanner/src/main.js >> /var/log/lead‑scanner.log 2>&1

All the code lives in a Git repo on the same machine, so I can git pull updates and restart pm2 with a single command. The only external dependency is the Slack webhook URL, which I store in an environment variable (SLACK_WEBHOOK).

Lessons Learned

• Keep the filter pure – By avoiding any I/O inside the predicate functions I can unit‑test them easily. A couple of jest tests now guard against regressions when I tweak the recency windows.
• Configuration over code – Moving the windows to JSON meant I could experiment with a tighter window for Upwork during a busy period without touching any source files.
• Board quirks change – APIs evolve, so I schedule a monthly “sanity check” that runs the fetchers and logs any new fields. When a board adds a visibility flag, I add a rule and I’m good.

Conclusion

Filtering out stale or noisy listings is the difference between a useful lead digest and a daily inbox full of dead ends. By separating the concerns—fetching, normalising, and filtering—I built a system that is easy to extend and runs reliably on a cheap home server.

I build sites and automations for small businesses at dreamfuturestech.com.