DEV Community: MJ API Development Pty LTD

How We Moved Our Docs to the Edge (And Stopped Rebuilding for Typos)

mobius-crypt — Tue, 16 Dec 2025 13:03:27 +0000

At Tenders SA, we recently undertook a migration that sounds minor on paper but had a massive impact on our developer experience and site performance: we moved our entire documentation system from local file reads to the Cloudflare Edge.

Here is why we did it, how we architected it using Cloudflare Workers and KV, and the code that makes it work.

The Problem: The "Monolithic" Content Trap

Like many Next.js projects, our documentation started as a folder of JSON files living in our repo (data/docs/*.json).

Our app would read these files at runtime (or build time) using standard file system calls:

// The old way (src/lib/docs/load-documentation.ts)
import fs from 'fs';
import path from 'path';

export function getDoc(slug) {
  // 🚫 This couples content to the build artifact!
  const filePath = path.join(process.cwd(), 'data/docs', `${slug}.json`);
  return JSON.parse(fs.readFileSync(filePath, 'utf8'));
}

The issues with this approach:

Rebuilds for Typos: Correcting a spelling mistake in the docs required a full CI/CD pipeline run and redeployment of the main application.
Platform Limitations: As we looked to move certain services to edge runtimes (like Cloudflare Workers), we hit a wall. Edge workers generally cannot read from the local file system.

The Solution: Cloudflare KV + Workers

We decided to decouple the content from the application code.

Storage: Cloudflare KV (Key-Value storage). It is eventually consistent, extremely fast for reads, and perfect for text data like documentation.
Delivery: A lightweight Cloudflare Worker acting as the docs-api.

The Architecture

Instead of fs.readFile, our frontend now fetches data from an API.

Namespace: TENDERS_DOCS
Key Strategy:
- doc:{slug} -> Stores the full content of a specific page.
- docs:tree -> Stores the navigation structure (sidebar categories, order, etc.).

Step 1: The Migration Script (Seeding KV)

We couldn't just drag-and-drop files. We needed a script to transform our local JSON files into a format Cloudflare KV accepts for bulk uploading.

We created scripts/generate-docs-kv-data.ts:

// scripts/generate-docs-kv-data.ts
import fs from 'fs';
import path from 'path';

const DOCS_DIR = path.join(__dirname, '../data/docs');
const OUTPUT_FILE = path.join(__dirname, 'docs_kv_bulk.json');

const generateKVData = () => {
  const files = fs.readdirSync(DOCS_DIR);
  const kvPairs = [];
  const navigationTree = [];

  files.forEach(file => {
    if (!file.endsWith('.json')) return;

    const content = fs.readFileSync(path.join(DOCS_DIR, file), 'utf8');
    const docData = JSON.parse(content);
    const slug = file.replace('.json', '');

    // 1. Prepare the content key
    kvPairs.push({
      key: `doc:${slug}`,
      value: JSON.stringify(docData) // Store as string
    });

    // 2. Build the navigation tree metadata (simplified)
    navigationTree.push({
      title: docData.title,
      slug: slug,
      category: docData.category
    });
  });

  // 3. Add the tree key
  kvPairs.push({
    key: `docs:tree`,
    value: JSON.stringify(navigationTree)
  });

  fs.writeFileSync(OUTPUT_FILE, JSON.stringify(kvPairs, null, 2));
  console.log(`✅ Generated ${kvPairs.length} KV pairs.`);
};

generateKVData();

Once generated, we upload the data via Wrangler:

npx wrangler kv:bulk put --namespace-id <YOUR_KV_ID> scripts/docs_kv_bulk.json

Step 2: The Worker Logic

The Worker is incredibly simple. It sits on the edge, intercepts the request, and queries KV.

// worker.ts
export interface Env {
  TENDERS_DOCS: KVNamespace;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);

    // 1. Handle Navigation Request
    if (url.pathname === '/docs/tree') {
      const tree = await env.TENDERS_DOCS.get('docs:tree', 'json');
      return new Response(JSON.stringify(tree), {
        headers: { 'Content-Type': 'application/json' }
      });
    }

    // 2. Handle Individual Pages
    // Pattern: /docs/{slug}
    const match = url.pathname.match(/^\/docs\/(.+)$/);
    if (match) {
      const slug = match[1];
      // Fetch directly from KV
      const doc = await env.TENDERS_DOCS.get(`doc:${slug}`, 'json');

      if (!doc) {
        return new Response('Doc not found', { status: 404 });
      }

      return new Response(JSON.stringify(doc), {
        headers: { 
          'Content-Type': 'application/json',
          'Cache-Control': 'public, max-age=3600' // Cache heavily!
        }
      });
    }

    return new Response('Not found', { status: 404 });
  },
};

Step 3: Decoupling the Frontend

Finally, we refactored our Next.js frontend components. Instead of importing a server-side file reader, we just fetch.

Before (Server Component):

const doc = await loadDocumentationData(slug); // ❌ Reads disk

After (Server Component):

const res = await fetch(`https://docs-api.tenders-sa.org/docs/${slug}`, {
  next: { revalidate: 3600 } 
});
const doc = await res.json(); // ✅ Fetches from Edge

The Benefits

Instant Content Updates: We can now update a specific KV key (via API or dashboard) and the documentation updates instantly without a redeploy.
Performance: Cloudflare KV is distributed globally. Our documentation data is now physically closer to the user than our origin server ever was.
Microservices Ready: Our documentation data is no longer trapped in the Next.js repo. Other services (like our chatbots or CLI tools) can hit the docs-api to retrieve the same standardized content.

Have you moved static content to the Edge? Let me know in the comments!

Tenders SA Documentations

Unlocking the Power of Async Programming in FastAPI (and Flask!)

mobius-crypt — Wed, 30 Apr 2025 16:29:53 +0000

Unlocking the Power of Async Programming

As Python developers, we often face the challenge of handling concurrent requests efficiently. FastAPI, a modern web framework built on Python's async capabilities, is a great solution. It allows us to handle large numbers of requests simultaneously without blocking, improving performance and responsiveness.

In this article, we’ll explore how to leverage async programming, covering key concepts and practical tips that will help you make the most of this powerful feature. We’ll also see how async programming is utilized in real-world projects like my job listing platform, JobFinders.site, which uses Flask with async support.

What is Async Programming?

Async programming allows us to run tasks concurrently rather than sequentially, making it possible to handle multiple tasks at once. In Python, async programming uses the async and await keywords to define asynchronous functions. These functions can pause execution while waiting for I/O-bound operations, such as database queries or API calls, to complete, without blocking the entire program.

Why Async Programming in FastAPI (and Flask)?

FastAPI is built with async in mind, and Flask can also support async operations with the help of extensions. By using async, you can:

Handle a large number of concurrent requests: FastAPI can process multiple requests at the same time, making it perfect for I/O-bound operations like reading from a database or querying external APIs.
Improve performance: Async operations allow your server to remain responsive, even when handling many requests at once.
Reduce latency: Async programming minimizes the waiting time for tasks, allowing your API to respond faster.

In my JobFinders.site project, Flask with async support has significantly improved performance, ensuring that hundreds of users can browse job listings and apply for positions simultaneously without delays.

Setting Up Async in FastAPI and Flask

To get started with async in FastAPI, we need to define async functions and use asynchronous libraries wherever applicable (e.g., for database access or API calls). In Flask, you can enable async by using libraries like gevent or eventlet.

Here’s an example of an async route handler in FastAPI:

from fastapi import FastAPI
import asyncio

app = FastAPI()

# Simulate a long-running task
async def long_task():
    await asyncio.sleep(3)
    return "Task Completed"

@app.get("/")
async def root():
    result = await long_task()
    return {"message": result}

In Flask, you can achieve async behavior using asyncio or similar libraries:

from flask import Flask
import asyncio

app = Flask(__name__)

# Simulate a long-running task
async def long_task():
    await asyncio.sleep(3)
    return "Task Completed"

@app.route('/')
async def root():
    result = await long_task()
    return {'message': result}

if __name__ == '__main__':
    app.run(debug=True)

Both examples achieve similar functionality: running an asynchronous task while keeping the server responsive.

Database Access with Async

FastAPI and Flask both support asynchronous database access. You can use async-compatible database libraries, such as databases or asyncpg for PostgreSQL.

Here’s an example using SQLAlchemy with async support in FastAPI:

from fastapi import FastAPI
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import sessionmaker

DATABASE_URL = "postgresql+asyncpg://user:password@localhost/dbname"
engine = create_async_engine(DATABASE_URL, echo=True)

async_session = sessionmaker(
    engine, class_=AsyncSession, expire_on_commit=False
)

@app.get("/items")
async def get_items():
    async with async_session() as session:
        result = await session.execute("SELECT * FROM items")
        items = result.fetchall()
        return {"items": items}

In Flask, you can use SQLAlchemy with async extensions like asyncpg or the databases library for asynchronous database operations.

Handling Background Tasks

FastAPI provides an easy way to handle background tasks, such as sending emails or processing large files, without blocking the main application thread. Flask also supports background tasks using packages like Celery or RQ.

Here’s an example using FastAPI’s BackgroundTasks:

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

def send_email(email: str):
    # Simulate email sending
    print(f"Email sent to {email}")

@app.post("/send-email")
async def send_email_task(background_tasks: BackgroundTasks, email: str):
    background_tasks.add_task(send_email, email)
    return {"message": "Email will be sent in the background"}

In Flask, you can implement similar background task functionality using libraries like Celery:

from flask import Flask
from celery import Celery

app = Flask(__name__)

# Configure Celery
celery = Celery(app.name, broker='redis://localhost:6379/0')

@celery.task
def send_email(email):
    # Simulate email sending
    print(f"Email sent to {email}")

@app.route("/send-email")
def send_email_task():
    email = "example@example.com"
    send_email.apply_async(args=[email])
    return {"message": "Email will be sent in the background"}

if __name__ == '__main__':
    app.run(debug=True)

Common Pitfalls in Async Programming

While async programming can greatly improve performance, it's important to avoid common pitfalls, such as:

Blocking the event loop: Avoid using synchronous functions (like time.sleep) in async code, as they block the event loop and can degrade performance.
Unmanaged exceptions: Async functions should properly handle exceptions to prevent the server from crashing.
Database connections: When using async with databases, make sure you're using an async-compatible library and managing connections efficiently.

How Async Powers Real-World Projects

In my projects, such as JobFinders.site (built with Flask + async) and Funeral Manager, async programming has allowed me to handle large amounts of traffic without sacrificing performance. These projects require processing large amounts of data quickly—whether it’s job listings or funeral service management—and async programming helps to ensure everything runs smoothly.

In particular, my job platform uses Flask with async support to process numerous concurrent users searching for job postings, applying for positions, and interacting with various services. Thanks to async capabilities, the application can handle this traffic while delivering real-time updates and interactions.

Conclusion

Async programming is a powerful tool for building high-performance applications, and both FastAPI and Flask support async operations. By using async functions and taking advantage of async capabilities, you can build fast, efficient APIs that handle many concurrent requests without blocking. With a little attention to best practices, you can optimize your applications for both speed and scalability.

Feel free to check out my projects, JobFinders.site and Funeral Manager, to see how async programming has been applied to real-world solutions.

The Power of a Portfolio Website for Job Seekers: Enhancing Your Career Opportunities

mobius-crypt — Sun, 10 Sep 2023 14:01:30 +0000

The Power of a Portfolio Website for Job Seekers: Enhancing Your Career Opportunities

This article is sponsored by https://jobfinders.site

In today's competitive job market, standing out from the crowd is crucial for landing your dream job.

Searching for a job

While a well-crafted resume and cover letter are essential, having a portfolio website can be a game-changer that significantly improves your chances of securing that coveted position.

In this article, we'll explore the reasons why job seekers should consider creating a portfolio website to enhance their career opportunities.

1. Showcase Your Work and Skills

A portfolio website provides a dedicated space to showcase your work, projects, and accomplishments. Whether you're a graphic designer, web developer, writer, or any other professional, having an online portfolio allows you to display your skills and expertise in a visually appealing and organized manner. Potential employers can easily assess your abilities and see tangible examples of your work, making a strong first impression.

2. Personal Branding

Creating a portfolio website allows you to establish and enhance your personal brand. You can customize the design and content to reflect your unique style, values, and professional identity. A well-branded portfolio website can help you stand out and leave a lasting impression on employers, making you a memorable candidate in a crowded job market.

3. Increased Credibility

A professional-looking portfolio website adds credibility to your job application. It demonstrates your commitment to your field and your willingness to go the extra mile to showcase your qualifications. Employers are more likely to trust candidates who invest in building an online presence, as it suggests a proactive and self-motivated approach to their career.

4. Easy Access to Information

When employers receive a large number of job applications, they appreciate easy access to information. Including a link to your portfolio website on your resume or LinkedIn profile streamlines the hiring process for recruiters. They can quickly access your work, saving time and ensuring that your qualifications are thoroughly reviewed.

5. Better Control Over Your Narrative

With a portfolio website, you have complete control over the narrative you present to potential employers. You can curate the content, highlight your most relevant experiences, and share your career goals and aspirations. This level of control allows you to shape the perception employers have of you, emphasizing your strengths and minimizing potential weaknesses.

6. Networking Opportunities

A portfolio website can also serve as a powerful networking tool. You can connect with professionals in your industry, share your work on social media, and engage with like-minded individuals. Building a strong online presence can lead to valuable connections and opportunities that may not have been possible otherwise.

7. Adaptability and Flexibility

A portfolio website is a versatile tool that can evolve with your career. You can continuously update it to include new achievements, projects, and skills as you gain more experience. This adaptability ensures that your online presence remains relevant and up-to-date, making it an asset throughout your career journey.

In conclusion, a portfolio website is a valuable asset for job seekers looking to improve their career opportunities.
It allows you to showcase your work, build your personal brand, increase your credibility, and take control of your narrative. By investing in a portfolio website, you position yourself as a proactive and highly capable candidate, increasing your chances of landing your desired job in today's competitive job market.
So, start building your online portfolio today and watch your career prospects soar.

Online Resources to Start Building Your Portfolio Website

Website Hosting.

You need a hosting Provider and for this NameCheap does not dissapoint so i would recommend you start using it today.

Website Developer.

You need a website developer who can create something that wont only serve as display but can actually be useful for employers and you, in this regard I recommend you give me a try,

find my github profile here for contact details and some popular work.

Start Using a Job Curation Site Such as jobfinders.site this will enable you to find jobs you are actually looking for easily.

jobfinders will create a daily listing of all the jobs available in South Africa for your easy picking.