DEV Community: Awaliyatul Hikmah

How to Secure Docusaurus for Internal Documentation using Vercel Middleware

Awaliyatul Hikmah — Thu, 15 Jan 2026 03:23:02 +0000

When building internal documentation with Docusaurus, standard static hosting exposes your sensitive content to the public internet. This guide provides a complete, step-by-step tutorial on how to secure your Docusaurus site using a serverless approach with Vercel.

The Challenge

Static site generators like Docusaurus are great for performance, but they produce simple HTML/JS/CSS files. Unlike a dynamic CMS, they don't have a backend to check "Is this user logged in?" before serving a page.

We need a solution that:

Protects all routes (not just client-side masking).
Verifies identity using a secure, stateless method.
Requires zero server management.

The Solution Architecture

We will implement a Two-Layer Security Model:

Vercel Edge Middleware: The "Bouncer" at the door. It intercepts requests at the CDN edge.
Serverless Functions: The "ID Card Issuer". Handles email OTP and issues secure cookies.

Step-by-Step Implementation

Prerequisites

A Docusaurus project.
A Vercel account.
An email provider API (we'll use Resend for this example).

Step 1: Install Dependencies

You'll need jose for lightweight JWT handling and resend for sending emails.

npm install jose resend

Step 2: Configure Vercel Project

Create a vercel.json file in the root to ensure Vercel builds Docusaurus correctly and serves the output.

{
    "buildCommand": "npm run build",
    "outputDirectory": "build"
}

Step 3: Configure Environment Variables

In your Vercel project settings, add the following variables:

AUTH_SECRET: A long random string (e.g., generated via openssl rand -hex 32).
RESEND_API_KEY: Your API key from Resend.

Step 4: Create the Edge Middleware (`middleware.ts`)

Create a middleware.ts file in the root of your project. This file runs on Vercel's Edge Network for every request.

// middleware.ts
import { jwtVerify } from 'jose';

// Paths that don't require authentication - be SPECIFIC
const PUBLIC_PATHS = [
    '/login',
    '/api/auth/', // Only auth-related APIs are public
];

// Static assets that are safe to serve without auth
// .js is needed for Docusaurus to render login page
// EXCLUDED: .json, .html, .xml, .txt (could leak doc content via prefetch)
const PUBLIC_EXTENSIONS = [
    '.js',      // Framework scripts needed for login page
    '.css',     // Stylesheets
    '.ico', '.png', '.jpg', '.jpeg', '.svg', '.webp', '.gif', // Images
    '.woff', '.woff2', '.ttf', '.eot', // Fonts
];

export const config = {
    // Match all paths (Vercel Edge Middleware)
    matcher: '/(.*)',
};

// Parse cookie using Web API (Vercel Edge compatible)
function getCookie(request: Request, name: string): string | null {
    const cookieHeader = request.headers.get('cookie');
    if (!cookieHeader) return null;

    const match = cookieHeader.match(new RegExp(`(^| )${name}=([^;]+)`));
    return match ? decodeURIComponent(match[2]) : null;
}

export default async function middleware(request: Request) {
    const url = new URL(request.url);
    const pathname = url.pathname;

    // 1. Allow specific public paths (login page and auth APIs only)
    if (PUBLIC_PATHS.some(path => pathname.startsWith(path))) {
        return undefined; // Continue to origin
    }

    // 2. Allow static assets (js, css, images, fonts - NOT json/html)
    if (PUBLIC_EXTENSIONS.some(ext => pathname.endsWith(ext))) {
        return undefined; // Continue to origin
    }

    // 3. Get auth token from cookie
    const token = getCookie(request, 'site_auth');

    if (!token) {
        return Response.redirect(new URL('/login', request.url), 302);
    }

    // 4. Verify JWT signature (stateless)
    try {
        const secret = new TextEncoder().encode(process.env.AUTH_SECRET);
        await jwtVerify(token, secret);
        return undefined; // Token valid, continue to origin
    } catch (error) {
        // Invalid or expired token - redirect to login
        return Response.redirect(new URL('/login', request.url), 302);
    }
}

Step 5: Create Authentication APIs

Create a specific folder structure for Vercel Functions: api/auth/.

A. Send OTP (`api/auth/send-otp.js`)

import { Resend } from 'resend';
import { SignJWT } from 'jose';

const ALLOWED_DOMAIN = '@yourcompany.com';
const ALLOWED_EMAILS = [
    'admin@yourcompany.com',
    'tester@gmail.com'
];
const OTP_EXPIRY_SECONDS = 600; // 10 minutes

export default async function handler(req, res) {
    if (req.method !== 'POST') {
        return res.status(405).json({ error: 'Method not allowed' });
    }

    try {
        const { email } = req.body;

        if (!email || typeof email !== 'string') {
            return res.status(400).json({ error: 'Email is required' });
        }

        const normalizedEmail = email.toLowerCase().trim();

        // Validate: must be allowed domain OR in whitelist
        const isAllowedDomain = normalizedEmail.endsWith(ALLOWED_DOMAIN);
        const isWhitelisted = ALLOWED_EMAILS.includes(normalizedEmail);

        if (!isAllowedDomain && !isWhitelisted) {
            return res.status(403).json({
                error: 'Email not authorized for access'
            });
        }

        // Generate 6-digit OTP
        const otp = Math.floor(100000 + Math.random() * 900000).toString();

        // Create a signed token containing the OTP (stateless verification)
        const secret = new TextEncoder().encode(process.env.AUTH_SECRET);
        const otpToken = await new SignJWT({
            email: normalizedEmail,
            otp: otp,
        })
            .setProtectedHeader({ alg: 'HS256' })
            .setIssuedAt()
            .setExpirationTime(`${OTP_EXPIRY_SECONDS}s`)
            .sign(secret);

        // Send email via Resend
        const resend = new Resend(process.env.RESEND_API_KEY);

        await resend.emails.send({
            from: 'Docs Auth <noreply@docs.yourcompany.com>',
            to: normalizedEmail,
            subject: 'Your Access Code',
            html: `
        <div style="font-family: sans-serif; max-width: 480px; margin: 0 auto; padding: 20px;">
          <h1 style="color: #1a1a1a;">Documentation Access</h1>
          <p>Your access code is:</p>
          <div style="background: #f5f5f5; padding: 20px; text-align: center; border-radius: 8px;">
            <span style="font-size: 32px; font-weight: bold; letter-spacing: 4px;">${otp}</span>
          </div>
          <p style="color: #888; font-size: 14px;">Expires in 10 minutes.</p>
        </div>
      `,
        });

        return res.status(200).json({
            success: true,
            message: 'OTP sent successfully',
            otpToken: otpToken, // Send token back to client for stateless verification
        });

    } catch (error) {
        console.error('Send OTP error:', error);
        return res.status(500).json({
            error: 'Failed to send OTP. Please try again.'
        });
    }
}

B. Verify OTP & Set Cookie (`api/auth/verify-otp.js`)

import { SignJWT, jwtVerify } from 'jose';

const SESSION_DURATION = 365 * 24 * 60 * 60; // 1 year session

export default async function handler(req, res) {
    if (req.method !== 'POST') {
        return res.status(405).json({ error: 'Method not allowed' });
    }

    try {
        const { email, otp, otpToken } = req.body;

        if (!email || !otp || !otpToken) {
            return res.status(400).json({ error: 'Email, OTP, and token are required' });
        }

        const normalizedEmail = email.toLowerCase().trim();
        const secret = new TextEncoder().encode(process.env.AUTH_SECRET);

        // Verify the OTP token
        let payload;
        try {
            const result = await jwtVerify(otpToken, secret);
            payload = result.payload;
        } catch (err) {
            return res.status(400).json({ error: 'OTP expired. Please request a new code.' });
        }

        // Check if email matches
        if (payload.email !== normalizedEmail) {
            return res.status(400).json({ error: 'Email mismatch. Please request a new code.' });
        }

        // Check if OTP matches
        if (payload.otp !== otp.trim()) {
            return res.status(400).json({ error: 'Invalid OTP. Please try again.' });
        }

        // OTP valid - generate session JWT
        const sessionToken = await new SignJWT({ email: normalizedEmail })
            .setProtectedHeader({ alg: 'HS256' })
            .setIssuedAt()
            .setExpirationTime(`${SESSION_DURATION}s`)
            .sign(secret);

        // Set Secure HttpOnly Cookie
        res.setHeader('Set-Cookie', [
            `site_auth=${sessionToken}; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=${SESSION_DURATION}`,
        ]);

        return res.status(200).json({
            success: true,
            message: 'Authentication successful'
        });

    } catch (error) {
        console.error('Verify OTP error:', error);
        return res.status(500).json({
            error: 'Verification failed. Please try again.'
        });
    }
}

C. Verify Session Endpoint (`api/auth/verify.js`)

This endpoint is Essential for the client-side to check if the user is authenticated. It reads the HttpOnly cookie using a helper function and verifying the JWT token.

import { jwtVerify } from 'jose';

// Helper to parse cookies from header
function getCookie(request, name) {
    const cookieHeader = request.headers.get('cookie');
    if (!cookieHeader) return null;

    const cookies = cookieHeader.split(';').reduce((acc, cookie) => {
        const [key, value] = cookie.trim().split('=');
        acc[key] = value;
        return acc;
    }, {});

    return cookies[name] || null;
}

export default async function handler(req, res) {
    try {
        // Robust cookie parsing for Node.js environment
        const cookieHeader = req.headers.cookie || '';
        const cookies = cookieHeader.split(';').reduce((acc, cookie) => {
            const [key, value] = cookie.trim().split('=');
            if (key && value) acc[key] = value;
            return acc;
        }, {});

        const token = cookies['site_auth'];

        if (!token) {
            return res.status(401).json({ error: 'Not authenticated' });
        }

        const secret = new TextEncoder().encode(process.env.AUTH_SECRET);
        // Verify token signature and expiration
        const { payload } = await jwtVerify(token, secret);

        return res.status(200).json({
            authenticated: true,
            email: payload.email,
        });

    } catch (error) {
        return res.status(401).json({ error: 'Invalid or expired token' });
    }
}

D. Logout Endpoint (`api/auth/logout.js`)

To log the user out, we simply clear the cookie by setting its Max-Age to 0.

export default async function handler(req, res) {
    // Clear the auth cookie
    res.setHeader('Set-Cookie', [
        'site_auth=; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=0',
    ]);

    // Redirect to login page
    res.writeHead(302, { Location: '/login' });
    res.end();
}

Step 6: Client-Side Integration

We need a wrapper to handle the UI state (redirecting if session is invalid).

A. Create `src/components/AuthWrapper.js`

import React, { useEffect, useState } from 'react';

export default function AuthWrapper({ children }) {
    const [isAuthenticated, setIsAuthenticated] = useState(false);

    useEffect(() => {
        // Skip check on login page
        if (typeof window !== 'undefined' && window.location.pathname === '/login') {
            setIsAuthenticated(true);
            return;
        }

        // Verify token with API (HttpOnly cookies are sent automatically)
        fetch('/api/auth/verify', {
            method: 'GET',
            credentials: 'include',
        })
            .then((res) => {
                if (res.ok) {
                    setIsAuthenticated(true);
                } else {
                    window.location.href = '/login';
                }
            })
            .catch(() => {
                window.location.href = '/login';
            });
    }, []);

    if (!isAuthenticated) return null; // Or a loading spinner
    return <>{children}</>;
}

B. Wrap the App in `src/theme/Root.js`

import React from 'react';
import AuthWrapper from '@site/src/components/AuthWrapper';

export default function Root({ children }) {
    return <AuthWrapper>{children}</AuthWrapper>;
}

Step 7: Create the Login Page (`src/pages/login.js`)

Here is a minimal React component to handle the login flow.

import React, { useState } from "react";

export default function Login() {
    const [step, setStep] = useState("email"); // 'email' or 'otp'
    const [email, setEmail] = useState("");
    const [otp, setOtp] = useState("");
    const [otpToken, setOtpToken] = useState("");
    const [message, setMessage] = useState("");

    const handleSendOtp = async (e) => {
        e.preventDefault();
        const res = await fetch("/api/auth/send-otp", {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({ email }),
        });
        const data = await res.json();

        if (res.ok) {
            setOtpToken(data.otpToken);
            setStep("otp");
            setMessage("Code sent!");
        } else {
            setMessage(data.error);
        }
    };

    const handleVerifyOtp = async (e) => {
        e.preventDefault();
        const res = await fetch("/api/auth/verify-otp", {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({ email, otp, otpToken }),
        });

        if (res.ok) {
            window.location.href = "/";
        } else {
            const data = await res.json();
            setMessage(data.error);
        }
    };

    return (
        <div style={{ padding: "2rem", maxWidth: "400px", margin: "0 auto" }}>
            <h1>Login</h1>
            {message && <p style={{ color: "red" }}>{message}</p>}

            {step === "email" ? (
                <form onSubmit={handleSendOtp}>
                    <input
                        type="email"
                        placeholder="Enter your email"
                        value={email}
                        onChange={(e) => setEmail(e.target.value)}
                        required
                    />
                    <button type="submit">Send Code</button>
                </form>
            ) : (
                <form onSubmit={handleVerifyOtp}>
                    <p>Enter code sent to {email}</p>
                    <input
                        type="text"
                        placeholder="123456"
                        value={otp}
                        onChange={(e) => setOtp(e.target.value)}
                        required
                    />
                    <button type="submit">Verify</button>
                </form>
            )}
        </div>
    );
}

Summary

By combining Docusaurus (Content) + Vercel Middleware (Security) + Serverless Functions (Auth Logic), we created a robust internal documentation hub that is:

Secure: Protected at the network edge.
Fast: Static content served from CDN (after auth check).
Cost-Effective: Uses serverless pricing models.

This setup ensures your internal knowledge base remains internal, without the complexity of VPNs or enterprise SSO servers.

Breaking Down Cost Functions in Linear Regression: A Conceptual Overview

Awaliyatul Hikmah — Thu, 12 Dec 2024 10:29:48 +0000

The cost function is a crucial concept in machine learning, helping us understand how well our models are performing. It's the tool that tells us how close our model's predictions are to the actual results and guides us in improving accuracy. In this post, we'll break down the cost function in simple terms, with a focus on linear regression.

Introduction to Cost Functions

The cost function tells us how well our model's predictions match the actual target values. Essentially, it measures the error between the predicted values and the true values. By minimizing this error, we can improve our model's accuracy.

Consider you have a training set with input features $x$ and output targets $y$ .

Size in feet² (x)	Price $1000s (y)
2104	460
1416	232
1534	315
852	178

The model you use to fit this training set can be represented by a linear function:

fw,b(x)=w⋅x+bf_{w,b}(x) = w \cdot x + b

For a training example $x^{(i)},y^{(i)} )$ , the function $f$ predicts $y^{(i)}$ as $y^(i)\hat{y}^{(i)}$ . Thus:

y^(i)=fw,b(x(i))=w⋅x(i)+b\hat{y}^{(i)} = f_{w,b}(x^{(i)}) = w \cdot x^{(i)} + b

The challenge is to find $w$ and $b$ that make the prediction $y^(i)\hat{y}^{(i)}$ close to the target $y^{(i)}$ for all training examples.

Here, $w$ and $b$ are the parameters of the model. These parameters are adjusted during training to enhance the model's performance.

Depending on the values chosen for $w$ and $b$ , we will get different functions $f (x)$ , which generate different lines on a graph. Writing $f (x)$ as shorthand for $f_{w,b}(x)$ , we can look at some plots to understand how $w$ and $b$ influence $f$ .

When $w = 0$ and $b = 1.5$

\cdot x + 1.5

The function is a horizontal line, predicting a constant value of 1.5.

When $w = 0.5$ and $b = 0$

\cdot x

The slope is 0.5, creating a line that increases steadily.

When $w = 0.5$ and $b = 1$

\cdot x + 1

This line has a slope of 0.5 and intersects the vertical axis at 𝑏=1.

Understanding Errors

The cost function calculates the error between the predicted prices $y^\hat{y}$ and the actual prices $y$ . This error is given by:

Error=y^−y\text{Error} = \hat{y} - y

Then, we square this error to avoid negative values. This squaring operation ensures that all errors are positive and emphasizes larger errors more than smaller ones. It will be:

Error=(y^−y)2\text{Squared Error} = (\hat{y} - y)^2

Here, if we have more training examples, the sum of the errors will naturally be larger. To normalize this, we use the average squared error instead of the total squared error to get a sense of the overall performance. This way, the cost function doesn't automatically get bigger just because we have more training examples. It makes it a fair comparison, no matter how big our dataset is. Dividing by the number of examples $m$ :

Mean Squared Error

=1m∑i=1m(y^(i)−y(i))2= \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2

To simplifies the derivative calculations during optimization (like gradient descent), we add factor of $12\frac{1}{2}$ in the cost function. Thus, the final of cost function formula will be:

\frac{1}{2m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)})^2

The extra division by 2 is a bit of a mathematical trick to make later calculations easier, especially when we use calculus to minimize the cost function.

Why Do We Square the Error?

Imagine we're trying to predict something—like the price of a house. Our model makes a prediction $(y^)(\hat{y})$ and we compare it to the actual price $(y)$ . The error is the difference between these two:

Error=y^−y\text{Error} = \hat{y} - y

But here's the thing: this error can be positive or negative.

If your prediction is higher than the actual value, the error is positive.
If your prediction is lower than the actual value, the error is negative.

Example:

Predicted price $(y^)(\hat{y})$ : $300,000
Actual price $(y)$ : $280,000

The error is 300,000 − 280,000 = 20,000 (positive error)

But:

Predicted price $(y^)(\hat{y})$ : $250,000
Actual price $(y)$ : $280,000

The error is 250,000 − 280,000 = −30,000 (negative error)

If we simply add up these errors, positive and negative values can cancel each other out, which wouldn't give us the real picture of how well our model is performing.

So, adding them as 20,000 + (-30,000) = -10,000 doesn’t accurately show the total error. Instead, by considering the total magnitude of errors (20,000 + 30,000), we get 50,000. This approach provides a more accurate representation of the errors, helping us better understand our model's performance and work on making better predictions.

Why Not Use Absolute Value to Avoid Negative Value?

When we square the errors, larger errors have a bigger impact. For example, an error of 10 becomes 100 when squared, while an error of 1 becomes 1. This helps the model focus on reducing larger mistakes more aggressively. The squared error function also smooth and differentiable everywhere. This smoothness is important for optimization algorithms like gradient descent because it allows for more efficient and predictable convergence to the minimum error.

Meanwhile, when we use absolute error, each error contributes linearly. An error of 10 remains 10, and an error of 1 remains 1. Both are treated equally without any extra emphasis on the larger ones. The absolute value function has a kink at zero, meaning it's not differentiable at that point. This can complicate the optimization process, making it harder to find the minimum error.

In summary, squaring the error puts more emphasis on larger mistakes, which helps in creating a better model overall by addressing those big errors more effectively. This is why squared errors are often preferred in many machine learning applications.

Why Divide by $2 m$ and Not $100 m$ ?

You may wonder, if our goal is to avoid larger numbers in the cost function as our data set gets bigger, why should we divide it by 2m? Why 2? Why not 100 or some other number? As I mentioned before, the extra division by 2 is a bit of a mathematical trick to make later calculations easier. Specifically, The factor of 2 cancels out when we take the derivative, simplifying our calculations.

When training the model, we often use optimization algorithms like gradient descent to minimize the cost function. Gradient descent involves taking the derivative (gradient) of the cost function with respect to the parameters $w$ and $b$ . The gradient tells us how to change the parameters to reduce the cost.

Consider a simple function:

\frac{1}{m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2

When we take the derivative of this function with respect to $w$ :

dJdw=1m∑i=1m2(wx(i)−y(i))x(i)\frac{dJ}{dw} = \frac{1}{m} \sum_{i=1}^{m} 2(wx^{(i)} - y^{(i)})x^{(i)}

The derivative produces a 2 from the squared term. This 2 can make the gradient calculations a bit cumbersome. By including a factor of 1/2 in the cost function, we simplify the gradient calculations. This adjustment doesn't change the ultimate goal (minimizing the cost), but it makes the math cleaner:

\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y{(i)})^2

Now, when we take the derivative, the factor of 2 cancels out:

dJdw=1m∑i=1m(w(x(i))−y(i))x(i)\frac{dJ}{dw}= \frac{1}{m} \sum_{i=1}^{m} (w(x^{(i)}) - y^{(i)})x^{(i)}

Why Not Another Number?

If we used a different number, like 100, in the cost function, the math wouldn't simplify as neatly:

\frac{1}{100m} \sum_{i=1}^{m} (wx^{(i)} - y{(i)})^2

Taking the derivative:

dJdw=150m∑i=1m(w(x(i))−y(i))x(i)\frac{dJ}{dw}= \frac{1}{50m} \sum_{i=1}^{m} (w(x^{(i)}) - y^{(i)})x^{(i)}

Here, the factor 1/50 doesn't simplify as nicely, and we end up with more complex expressions. This also means your model will learn very slowly. You would have to compensate by increasing the learning rate 𝛼, but this requires careful tuning to avoid making the model unstable.

Using $12m\frac{1}{2m}$ is a balanced choice. It simplifies the gradient calculations without making the steps too small or too large. It’s also a widely-accepted convention, which makes it easier to follow standard practices and compare results across different studies and implementations.

Visualizing the Cost Function

In linear regression, the objective is to find the optimal values for the parameters $w$ and $b$ that minimize the cost function $J (w, b)$ . This is typically achieved through an optimization algorithm, such as gradient descent, which iteratively adjusts $w$ and $b$ to reduce the difference between the predicted outputs and the actual target values.

To illustrate this concept, let's work with a simplified version of the linear regression model:

fw(x)=w⋅xf_w(x) = w \cdot x

In this model, we've eliminated the parameter $b$ . Now, the cost function looks like this:

\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2

The goal is to find the value of $w$ that minimizes $J (w)$ .

Let's visualize how the cost function changes with different values of $w$ . Consider that there are 3 data points. The graphs below show both the function $f_w(x)$ (left) and the corresponding cost function $J (w)$ (right) for four different values of $w$ :

When $w = 1$ : The function $f_w(x)$ is a line with a slope of 1, and the cost $J (w)$ is 0 for perfectly fitting data points and resulting in a lower cost.
- Function: $\cdot x$
- Cost: $\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2$ $\frac{1}{2\cdot3}((1\cdot1 - 1)^2+(1\cdot2 - 2)^2+(1\cdot3 - 3)^2) = 0$
- Graph:
When $w = 0.5$ : The function $f_w(x)$ has a slope of 0.5, leading to a higher cost due to the error between predicted and actual values. The line does not fit the data well.
- Function: $\cdot x$
- Cost: $\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2$ $\frac{1}{2\cdot3}((0.5\cdot1 - 1)^2+(0.5\cdot2 - 2)^2+(0.5\cdot3 - 3)^2) = 0.58$
- Graph:
When $w = 0$ : The function $f_w(x)$ is a horizontal line, resulting in a significant error and a higher cost.
- Function: $\cdot x$
- Cost: $\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2$ $\frac{1}{2\cdot3}((0\cdot1 - 1)^2+(0\cdot2 - 2)^2+(0\cdot3 - 3)^2) = 2.3$
- Graph: The line is a flat line, which does not fit the data points at all, which also results in a high cost.
When $w = - 0.5$ : The function $f_w(x)$ is a line with slopes downwards, showing an inverse relationship with the data points, resulting in the highest cost.
- Function: $\cdot x$
- Cost: $\frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2$ $\frac{1}{2\cdot3}((-0.5\cdot1 - 1)^2+(-0.5\cdot2 - 2)^2+(-0.5\cdot3 - 3)^2) = 5.25$
- Graph:

Here's how the plots would look:

These graphs help visualize how different values of $w$ affect the line that fits the data points and the corresponding value of the cost function $J (w)$ . The goal is to find the value of $w$ that results in the lowest cost, indicating the best fit for the data.

Please note that we can have different values for $w$ and $b$ that can reach the minimum cost $J$ . This depends on the distribution of the training data.

More specifically:

If the training data points have a clear linear trend and are tightly clustered, the cost function will have a narrow, bowl-shaped surface. In this case, there may only be a single unique combination of 𝑤 and 𝑏 that minimizes the cost.
However, if the training data points are more spread out and don't have a clear linear trend, the cost function will have a wider, bowl-shaped surface. This allows for multiple combinations of 𝑤 and 𝑏 to reach the global minimum of the cost function.

So, the distribution of the training data affects the shape of the cost function. This determines whether there is a single unique minimum or multiple possible $w$ and $b$ values that can reach the minimum cost. The training data distribution is a key factor.

Choosing the Optimal Parameters

The goal is to choose the value of $w$ that minimizes $J (w)$ . This is achieved by selecting the value of $w$ that results in the smallest possible value of the cost function. For instance, in our example, if choosing $w = 1$ results in the smallest $J (w)$ , then $w = 1$ is the optimal parameter for our model.

Different Cost Functions for Different Applications

While the mean squared error cost function is the most commonly used for linear regression, different applications may require different cost functions. The mean squared error is popular because it generally provides good results for many regression problems.

There’s no one-size-fits-all rule for defining the cost function. It truly depends on our unique case and what we’re trying to achieve with our model. For example, if we’re dealing with outliers in our data, a cost function that reduces their impact might be beneficial. On the other hand, if we’re interested in minimizing all errors equally, we might opt for a different approach.

In simple terms, the cost function we choose depends on the specifics of our application. Different problems have different requirements, so the best cost function is the one that aligns with our goal.

Additionally, we can follow the latest research to gain insights into which cost function might best suit our needs. Researchers often introduce new approaches and compare different cost functions in various scenarios, providing valuable guidance for our own application.

Understanding Supervised Learning: The Basics of Linear Regression

Awaliyatul Hikmah — Sun, 01 Dec 2024 15:42:26 +0000

In the world of machine learning, understanding the core concepts of how models are trained is essential. One fundamental approach is supervised learning, where we provide the algorithm with a training set comprising input features and their corresponding output targets. The learning algorithm then processes this information and produces a function, which we'll denote as f.

fw,b(x)=w⋅x+bf_{w,b}(x) = w \cdot x + b

Historically, this function f has been referred to as a hypothesis. The primary role of f is to take a new input, denoted as x, and output an estimate or prediction, which we'll call ŷ (y-hat).

Linear Regression: Fitting the Data

One common method within supervised learning is linear regression. The goal of linear regression is to determine the values for the parameters w (weights) and b (bias) so that the resulting straight line from the function f fits the data points well.

Here's a quick breakdown:

f is the model or function.
x is the input or input features.
ŷ (y-hat) is the output of the model, representing the prediction or estimate for y.

The model's prediction, ŷ, is an estimate of the true value y. When we simply use the symbol y, we refer to the target value, which is the actual true value within the training set. It's crucial to note that ŷ is merely an estimate and might not always align perfectly with the actual true value y.

The Role of Parameters w and b

It's important to understand that w and b are numbers, and the values chosen for them determine the prediction ŷ based on the input feature x. When we write $f_{w,b}(x)$ , it means f is a function that takes x as input, and depending on the values of w and b, f will output some prediction ŷ. For simplicity, we'll often write f(x) without explicitly including w and b in the subscript, though it means the same thing as $f_{w,b}(x)$ .

In simpler terms, this function f uses the input feature x to predict the output y.

Application: House Sizes and Prices

The image above provides a clear illustration of this concept. It shows how a linear regression model predicts house prices based on their sizes. The graph demonstrates a positive correlation between house size (in square feet) and price (in thousands of dollars). The blue regression line indicates the relationship, highlighting how larger houses tend to have higher prices.

To illustrate these concepts, let's consider the example of predicting house prices based on house sizes:

x represents the house size in square feet.
y is the actual house price in thousands of dollars.
ŷ (y-hat) is the predicted house price based on the input feature x.

For example, consider a house size of 1,250 square feet (x = 1250). The actual price of the house might be $220,000 (y = 220).

To make a prediction, we use the function $fw,b(x)=w⋅x+bf_{w,b}(x) = w \cdot x + b$ . This function helps us find the values of ŷ (predicted price) by minimizing the difference between y (actual price) and ŷ (predicted price). In a real scenario, the parameters w and b are calculated by the computer using the training data. For illustration, let's assume the model finds w = 0.15 and b = 100. Thus, the prediction function would be: f(x) = 0.15x + 100. This is just a hypothetical example to illustrate the concept.

Applying this function:

For a house size of 1,250 square feet (x = 1250), the predicted price (ŷ) would be:
ŷ = 0.15(1250) + 100 = 187.5 + 100 = 287.5
This would mean the model predicts the house price to be $287,500.

Comparing this prediction (ŷ = 287.5) to the actual price (y = 220), we can see there's a difference. The goal of the linear regression model is to minimize such differences across all data points to make the predictions as accurate as possible.

In summary, the linear regression model uses the input feature (x), applies the learned parameters (w and b), and provides a prediction (ŷ) that estimates the true value (y).

Why Use a Linear Function?

You might wonder, why use a straight-line (linear) function instead of a more complex curve or parabola (non-linear function)? Although non-linear functions can sometimes fit data more accurately, linear functions are simpler and easier to work with. They serve as a great starting point because they are easier to understand and interpret.

By using a linear function, we can gain a foundational understanding of the relationship between the input and output variables. Once this foundation is established, we can build on it and move to more complex non-linear models that might capture the intricacies of the data better. Essentially, mastering linear regression provides the stepping stones necessary to tackle more sophisticated machine learning models effectively.

This post is based on notes taken from Andrew Ng's Machine Learning Specialization course on Coursera.

Fine-Tuning vs. Retrieval-Augmented Generation (RAG): Enhancing LLMs for Specific Tasks

Awaliyatul Hikmah — Sun, 01 Dec 2024 11:05:11 +0000

Fine-tuning and Retrieval-Augmented Generation (RAG) are both powerful techniques used to enhance the capabilities of a large language model (LLM) beyond its initial pre-training. While they share a common goal of improving the model's performance, they employ fundamentally different approaches.

The Key Differences:

Fine-Tuning:

Direct Modification: Fine-tuning involves directly updating the internal parameters and weights of the LLM by training it on a specific dataset. This process allows the model to become specialized and perform better on a particular task or within a specific domain.
Customization: By exposing the LLM to a focused dataset, fine-tuning tailors the model to understand and generate outputs that are highly relevant and accurate for the task at hand. For example, a fine-tuned model on medical literature would excel in generating precise medical summaries or answering health-related queries.

Retrieval-Augmented Generation (RAG):

External Augmentation: Unlike fine-tuning, RAG does not modify the LLM's internal parameters. Instead, it augments the model with an external information retrieval system.
Dynamic Knowledge Integration: During the generation process, RAG dynamically pulls in relevant information from a knowledge base to supplement the LLM's output. This approach leverages external data to enhance the model's responses without altering its core structure.
Adaptability: RAG is particularly useful for tasks that require up-to-date or highly specific information. For instance, when generating responses to questions about current events, RAG can access and incorporate the latest news articles to provide accurate answers.

Choosing Between Fine-Tuning and RAG:

The choice between fine-tuning and RAG depends on the specific requirements of the task:

Fine-Tuning is ideal for scenarios where a high degree of customization is needed. It excels in tasks that demand in-depth knowledge of a particular domain or consistency in the output style.
RAG is better suited for tasks that benefit from integrating external, real-time information. It is especially useful when the LLM needs to provide answers based on the most recent data or when the knowledge base is too vast to be encapsulated within the model itself.

Conclusion:

Fine-tuning and RAG both offer unique advantages for enhancing the capabilities of LLMs. By understanding their differences and applications, we can choose the right technique to optimize performance for specific tasks and domains. Whether it's through the tailored expertise of fine-tuning or the dynamic augmentation of RAG, these techniques empower LLMs to achieve greater accuracy, relevance, and adaptability.

Fine-Tuning LLM: Transforming General Models into Specialized Experts

Awaliyatul Hikmah — Sun, 01 Dec 2024 11:02:22 +0000

Fine-tuning is a fascinating technique used to modify a large language model (LLM) by training it on an additional dataset. This process helps change its outputs or improve its performance on a specific task. Imagine having a versatile artist who can paint in any style; fine-tuning is like teaching this artist to excel in a particular genre, whether it's impressionism, surrealism, or hyperrealism. Let's delve deeper into the intricacies of fine-tuning, its applications, and the process involved.

Applications of Fine-Tuning

Fine-tuning can be applied to various tasks, some of which are challenging to define in a mere prompt:

Complex Tasks: These include tasks like summarizing customer service calls, mimicking a specific writing or speaking style, or even creating an artificial character. These tasks benefit from fine-tuning because they require the model to understand and replicate nuanced details that go beyond basic instructions.
Domain-Specific Knowledge: For tasks that demand a deep understanding of a particular domain, such as medical notes, legal documents, or financial documents, fine-tuning is invaluable. By training the LLM on domain-specific data, it can provide more accurate and contextually relevant outputs. For instance, a model fine-tuned on medical literature can generate precise summaries of patient records or assist in diagnosing conditions.
Efficiency and Cost-Effectiveness: Fine-tuning allows for the creation of smaller, more efficient models tailored to specific tasks. Instead of relying on a large, general-purpose model, a smaller, fine-tuned model can perform just as well, if not better, on certain tasks. This approach not only reduces computational costs but also speeds up response times.

The Fine-Tuning Process

The process of fine-tuning an LLM involves several key steps:

Starting Point: Begin with an LLM that has been pre-trained on a vast amount of general data, such as texts from the internet. This pre-training gives the model a broad understanding of language and various topics.
Target Dataset: Create or obtain a target dataset that contains examples of the desired outputs or specific domain knowledge. This dataset serves as the foundation for fine-tuning the model. For example, if the goal is to fine-tune a model for legal document analysis, the dataset would include numerous legal texts, court rulings, and contracts.
Training the Model: Train the LLM on the target dataset. During this phase, the model's parameters are adjusted to shift its behavior towards producing the desired outputs. This training process involves multiple iterations, where the model's performance is continually evaluated and refined.

Fine-tuning essentially customizes a general-purpose model to excel in specific applications, much like an artist specializing in a particular style. This technique leverages the broad knowledge gained during pre-training and hones it for specialized tasks, making LLMs versatile and powerful tools for a wide range of applications.

By understanding and applying fine-tuning, we can unlock the full potential of LLMs, enabling them to perform complex tasks with greater accuracy and efficiency. Whether it's handling domain-specific documents, mimicking unique writing styles, or performing cost-effective computations, fine-tuning paves the way for more advanced and tailored AI solutions.

Enhancing Language Models with Retrieval-Augmented Generation (RAG)

Awaliyatul Hikmah — Sun, 01 Dec 2024 10:40:08 +0000

Language models (LLMs) like GPT-3 have taken the world by storm with their ability to generate human-like text. However, their knowledge is limited to what they have been trained on, which often ends with the cut-off date of their training data. This is where Retrieval-Augmented Generation (RAG) comes in—a technique that can greatly expand what an LLM can do by giving it access to additional, up-to-date knowledge beyond its pre-existing data.

Understanding RAG

RAG is a method that enhances LLM capabilities by retrieving relevant documents or texts from external sources and incorporating them into the model's responses. This process not only enriches the LLM's output but also ensures that the information is accurate and current.

Steps of RAG

Here’s a closer look at how RAG works:

Retrieve Relevant Documents: The first step in RAG is to retrieve the most relevant document or text that may contain the answer to the question at hand. This involves searching databases, websites, or other resources to find the best possible match.
Incorporate Retrieved Text into the Prompt: Once the relevant text is found, it is integrated into an updated prompt. This new prompt instructs the LLM to use the retrieved context to generate a response. By doing this, the model can draw on fresh and relevant information rather than relying solely on its pre-existing training data.
Generate Answer Using Enriched Prompt: Finally, the LLM is prompted with the enriched input and generates an answer. The additional context provided by the retrieved documents allows the model to produce more informed and accurate responses.

Applications of RAG

RAG's ability to augment an LLM's knowledge makes it particularly useful in various applications. Here are a few notable examples:

Chat with a PDF File: Tools like PandaChat, AskYourPDF, and ChatPDF allow users to upload a PDF file and ask questions based on its content. RAG retrieves relevant sections from the PDF, enabling the LLM to provide accurate answers.
Chat with a Website Article: Platforms like Coursera Coach, SnapChat, and Hubspot use RAG to allow users to ask questions based on the content of a specific website. By retrieving relevant web pages, the LLM can offer insights and information grounded in the latest online content.
Chat with a Web Search Engine: Services like Microsoft Bing and Google integrate RAG to enhance their chat-like interfaces. Users can ask questions, and the system retrieves the most relevant web pages to provide up-to-date and precise answers.

The Big Idea: LLMs as Reasoning Engines

The core philosophy behind RAG is to treat LLMs not just as repositories of memorized facts but as reasoning engines. By providing relevant context, RAG enables LLMs to process and reason through information dynamically. This shifts the focus from relying on stored knowledge to leveraging real-time data, making interactions with LLMs more effective and reliable.

Conclusion

Retrieval-Augmented Generation represents a significant advancement in the field of artificial intelligence and natural language processing. By bridging the gap between static training data and dynamic real-world information, RAG transforms LLMs into more powerful and versatile tools. As we continue to explore and refine this technique, the potential applications and benefits will undoubtedly expand, leading to even more innovative and impactful uses of language models.

Limitations of Large Language Models: Unpacking the Challenges

Awaliyatul Hikmah — Sun, 01 Dec 2024 10:28:04 +0000

Large Language Models (LLMs) have revolutionized the field of artificial intelligence, pushing the boundaries of what machines can achieve in understanding and generating human-like text. However, despite their groundbreaking capabilities, LLMs come with several notable limitations that users and developers need to be aware of. In this blog post, we'll delve into some of the key challenges associated with LLMs.

Knowledge Cutoffs

One of the most significant limitations of LLMs is their knowledge cutoffs. An LLM's understanding of the world is essentially frozen at the time of its training. For example, a model trained on data scraped from the internet up until January 2022 will have no information about events or developments that occurred after that date. This limitation means that LLMs cannot provide insights or answers about recent events, making them less useful for tasks that require up-to-date information. As technology advances rapidly, this knowledge gap can become quite pronounced, impacting the model's relevance and reliability.

Hallucinations: Making Things Up

Another critical issue with LLMs is their tendency to hallucinate or generate information that is not based on the training data. This phenomenon occurs because LLMs are designed to predict the next word in a sequence, and sometimes they generate plausible-sounding but incorrect or nonsensical information. These hallucinations can be problematic, especially in applications where accuracy and reliability are paramount. Users must always exercise caution and verify the information provided by LLMs to avoid potential misinformation.

Input and Output Length Limitations

LLMs also face constraints regarding the length of input and output they can handle. Most LLMs have a maximum token limit, which restricts the amount of text they can process in a single instance. This limitation can be a significant drawback for tasks that require processing large documents or generating lengthy responses. Users often need to find creative solutions to work within these constraints, such as breaking down large texts into smaller, more manageable chunks.

Challenges with Structured Data

Generative AI, including LLMs, does not work well with structured (tabular) data. While LLMs excel at handling unstructured data such as text, images, audio, and video, they struggle with tasks that involve structured data. Structured data requires precise and accurate manipulation, which is not the strength of generative models. This limitation means that LLMs are less effective for applications that rely heavily on structured data, such as database management and spreadsheet analysis.

The Strength of Unstructured Data

On the flip side, Generative AI shines brightest with unstructured data. This type of data encompasses text, images, audio, and video—areas where LLMs can truly showcase their prowess. For instance, in natural language processing, content creation, and image generation, LLMs can produce remarkable results that often surpass human capabilities. Understanding this strength can help users maximize the potential of LLMs in the right contexts.

Conclusion

In conclusion, while LLMs have brought about significant advancements in the field of AI, they are not without their limitations. Knowledge cutoffs, hallucinations, input and output length constraints, and challenges with structured data are some of the key issues that users and developers need to consider. Understanding these limitations is crucial for effectively leveraging the capabilities of LLMs and mitigating their drawbacks. As the field of AI continues to evolve, addressing these limitations will be essential for the development of more robust and reliable models.

By acknowledging these challenges, we can better appreciate the impressive feats of LLMs while remaining mindful of their current boundaries. As always, the goal is to harness the strengths of these models while continuously striving for improvements and innovations.

How to Automounting Your VHDX on Windows 11 Without Mount-DiskImage Command

Awaliyatul Hikmah — Sun, 01 Dec 2024 09:16:15 +0000

If your Windows doesn't support the Mount-DiskImage -ImagePath command, follow these steps to automount your VHDX file.

Create The Script

Open Notepad and create a new file. Add this script:

select vdisk file="<path_to_your_vdhx_file>"
attach vdisk

Example:

select vdisk file="D:\MyVirtualDisk.vhdx"
attach vdisk

Save the file as .txt file

Create New Task Scheduler

Click the Windows Button and search for Task Scheduler
Look at the Action Panel on the right side, or click Action on Status Bar, then click Create Task
Fill in the Name field (it's up to you), Choose Run whether user is logged on or not, and check Run with highest privileges too
Go to Trigger tab, then click New
Choose At startup for Begin the task, and Check Enabled
Go to Action tab, then click New
- Choose Start a program for Action field.
- Type powershell.exe for Program/script
- Add arguments diskpart /s "<your_txt_script_path_you_created_in_the_first_step>" (e.g., diskpart /s "D:\script.txt")
- Click OK
Go to Conditions tab, make sure to uncheck Start the task only if the computer is on AC power, then click OK.
Enter your password, then click OK

Understanding the Evolution of Word Representation: Static vs. Dynamic Embeddings

Awaliyatul Hikmah — Thu, 28 Nov 2024 10:32:11 +0000

In the ever-evolving field of natural language processing (NLP), word embeddings have played a crucial role in how models understand and interpret human language. Two primary types of embeddings have emerged: static embeddings, such as those produced by GloVe, Word2Vec, and fastText, and dynamic (contextual) embeddings, like those generated by BERT, ELMo, and GPT.

Static Word Embeddings

Static embeddings, such as GloVe, Word2Vec, and fastText, assign a single, fixed vector representation to each word in the vocabulary. This means the embedding for a word remains the same regardless of the context in which it appears. Static embeddings capture some contextual information through techniques like a global co-occurrence matrix. For example, GloVe uses this matrix to track how often each word co-occurs with every other word in the corpus, learning word vectors that encode these relationships. The co-occurrence information is used to learn the initial embedding, but the embedding itself does not dynamically adapt.

Dynamic (Contextual) Embeddings

In contrast, dynamic embeddings are context-sensitive. The vector representation of a word can change depending on the surrounding words, allowing these embeddings to capture nuanced meanings. For instance, the word "bank" will have different embeddings in "bank account" and "river bank." Models like BERT, ELMo, and GPT use advanced neural network architectures to generate these dynamic embeddings, which adapt to the specific context a word appears in. Although more computationally intensive, dynamic embeddings often lead to better performance on NLP tasks that require understanding word meanings in context, such as text classification, question answering, and language generation.

Comparison and Conclusion

While GloVe captures some contextual signals through its co-occurrence matrix, it remains a static embedding approach. On the other hand, models like BERT and ELMo generate truly context-aware embeddings, representing a significant advancement over earlier techniques. The ability to produce embeddings that fluidly adapt to usage context is a key improvement, enhancing the performance of NLP applications.

In summary, the evolution from static to dynamic word embeddings marks a pivotal development in NLP, enabling models to better understand and process human language in its full complexity.

DEV Community: Awaliyatul Hikmah

How to Secure Docusaurus for Internal Documentation using Vercel Middleware

The Challenge

The Solution Architecture

Step-by-Step Implementation

Prerequisites

Step 1: Install Dependencies

Step 2: Configure Vercel Project

Step 3: Configure Environment Variables

Step 4: Create the Edge Middleware (middleware.ts)

Step 5: Create Authentication APIs

A. Send OTP (api/auth/send-otp.js)

B. Verify OTP & Set Cookie (api/auth/verify-otp.js)

C. Verify Session Endpoint (api/auth/verify.js)

D. Logout Endpoint (api/auth/logout.js)

Step 6: Client-Side Integration

A. Create src/components/AuthWrapper.js

B. Wrap the App in src/theme/Root.js

Step 7: Create the Login Page (src/pages/login.js)

Summary

Breaking Down Cost Functions in Linear Regression: A Conceptual Overview

Introduction to Cost Functions

Understanding Errors

Why Do We Square the Error?

Example:

Why Not Use Absolute Value to Avoid Negative Value?

Why Divide by 2m2m2m and Not 100m100m100m ?

Why Not Another Number?

Visualizing the Cost Function

Choosing the Optimal Parameters

Different Cost Functions for Different Applications

Understanding Supervised Learning: The Basics of Linear Regression

Linear Regression: Fitting the Data

The Role of Parameters w and b

Application: House Sizes and Prices

Why Use a Linear Function?

Fine-Tuning vs. Retrieval-Augmented Generation (RAG): Enhancing LLMs for Specific Tasks

The Key Differences:

Choosing Between Fine-Tuning and RAG:

Conclusion:

Fine-Tuning LLM: Transforming General Models into Specialized Experts

Applications of Fine-Tuning

The Fine-Tuning Process

Enhancing Language Models with Retrieval-Augmented Generation (RAG)

Understanding RAG

Steps of RAG

Applications of RAG

The Big Idea: LLMs as Reasoning Engines

Conclusion

Limitations of Large Language Models: Unpacking the Challenges

Knowledge Cutoffs

Hallucinations: Making Things Up

Input and Output Length Limitations

Challenges with Structured Data

The Strength of Unstructured Data

Conclusion

How to Automounting Your VHDX on Windows 11 Without Mount-DiskImage Command

Create The Script

Create New Task Scheduler

Understanding the Evolution of Word Representation: Static vs. Dynamic Embeddings

Static Word Embeddings

Dynamic (Contextual) Embeddings

Comparison and Conclusion

Step 4: Create the Edge Middleware (`middleware.ts`)

A. Send OTP (`api/auth/send-otp.js`)

B. Verify OTP & Set Cookie (`api/auth/verify-otp.js`)

C. Verify Session Endpoint (`api/auth/verify.js`)

D. Logout Endpoint (`api/auth/logout.js`)

A. Create `src/components/AuthWrapper.js`

B. Wrap the App in `src/theme/Root.js`

Step 7: Create the Login Page (`src/pages/login.js`)

Why Divide by $2 m$ and Not $100 m$ ?