DEV Community: Mishan Raj Shah

Why I still use NGINX over everything else

Mishan Raj Shah — Sat, 27 Jun 2026 16:10:32 +0000

People keep recommending Caddy. The config is cleaner, automatic HTTPS, three lines and you're done. They're not wrong. For a side project or a small team that doesn't want to think about infrastructure, Caddy is fine.

I still use NGINX. Here's why.

The setup that made it clear

Earlier this year I deployed a production system on a single VPS. Hostinger KVM2, 2 vCPU, 8 GB RAM. The kind of system that sits quiet for months and then gets hit hard in a short window, hundreds of concurrent users at the same moment.

The requirement was 1,000 requests per minute. I load tested to 2,500 requests per second. Zero errors. Zero timeouts. p99 on the result lookup was 308ms.

No autoscaling. No managed load balancer. No cloud bill. One NGINX config that I wrote by hand.

That's the argument.

What NGINX actually does in production

Here's the architecture that handled it:

Browser
    | HTTPS
    v
Cloudflare  (DDoS protection, CDN, TLS to browser)
    | HTTPS
    v
NGINX (host)  (TLS termination, rate limiting, cache)
    |-- /api/*  →  Express + Prisma container (127.0.0.1:5001)
    └-- /*      →  Next.js container (127.0.0.1:3001)
                        |
                 PostgreSQL (internal Docker network only)

Containers bind to localhost only. Nothing is reachable from the internet except through NGINX. That one decision eliminates an entire class of exposure.

The five things NGINX handles that people underestimate

1. Rate limiting before the app sees the request

Rate limiting in Express middleware still consumes a Node.js worker thread per rejected request. NGINX rejects rate-limited requests before they reach Node.js. No event loop, no DB query, no memory allocation. A 429 from NGINX costs almost nothing.

limit_req_zone $real_ip zone=api:10m rate=10r/s;

location /api/ {
    limit_req zone=api burst=20 nodelay;
    limit_req_status 429;
    proxy_pass http://127.0.0.1:5001;
}

2. Cache stampede protection

When the traffic spike hits, hundreds of users land on the same URL at the exact same moment. Without a cache, every request goes Next.js → Node.js → PostgreSQL. With proxy_cache_lock on, when the cache expires only one upstream request goes through. Every other concurrent request waits for that single response, then gets served from cache. Without the lock, every request that arrives during the cache miss window hits upstream simultaneously. That's the thundering herd problem, and it's what kills servers on high-traffic moments.

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=pages:10m
                 max_size=100m inactive=1m use_temp_path=off;

location / {
    proxy_cache pages;
    proxy_cache_valid 200 10s;
    proxy_cache_lock on;
    proxy_cache_lock_timeout 5s;
    proxy_pass http://127.0.0.1:3001;
}

10 seconds is short enough to feel live, long enough to absorb any realistic spike.

3. Cloudflare real-IP restoration

Cloudflare proxies all traffic, which means every request arrives at NGINX from a Cloudflare edge IP. Without fixing this, your rate limiting targets Cloudflare's shared IPs and blocks every user at once instead of individual abusers.

real_ip_header CF-Connecting-IP;

set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;

map $http_cf_connecting_ip $real_ip {
    default $http_cf_connecting_ip;
    ""      $remote_addr;
}

4. One domain, multiple apps, zero complexity

The entire system runs under one domain. Frontend on port 3001, backend on port 5001, both invisible to the internet, both served through the same NGINX vhost.

server {
    server_name yourdomain.com;

    location /api/ {
        proxy_pass http://127.0.0.1:5001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $real_ip;
    }

    location / {
        proxy_pass http://127.0.0.1:3001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $real_ip;
    }
}

No separate subdomains. No DNS juggling. One cert. One vhost.

5. Static file serving and compression with no extra layer

Gzip compression built in. Static assets served directly without hitting the app.

gzip on;
gzip_types text/plain text/css application/json
           application/javascript text/xml application/xml
           image/svg+xml;
gzip_min_length 1000;

location /_next/static/ {
    proxy_pass http://127.0.0.1:3001;
    proxy_cache_valid 200 1y;
    add_header Cache-Control "public, max-age=31536000, immutable";
}

Next.js static assets are immutable by design (content-hashed filenames). Tell the browser and Cloudflare to cache them for a year. Zero repeat fetches on return visits.

Why not Caddy or Traefik

Caddy's killer feature is automatic HTTPS with zero config. That's genuinely useful if cert management is your problem. But the same server was already running WordPress, MariaDB, and a PM2 process with existing certs. Caddy would mean either migrating all of that or running two proxies. NGINX was already there.

Traefik is the right choice if your services come and go dynamically, containers spinning up and down, labels driving routing automatically. When you own a fixed VPS with a known set of services, that auto-discovery is solving a problem you don't have. You end up with more moving parts for no benefit.

NGINX wins on control. When something goes wrong at 2 AM, I want to read a config file, not debug a label hierarchy.

What happens when none of this is in place

Failure under load is not isolated. It cascades.

The primary endpoint slows down. Users retry. The request queue builds. Database connections spike. Other parts of the app start timing out. Users see errors on pages that have nothing to do with the original problem. One block hits another, then another.

Every one of those config blocks above cuts a link in that chain. Rate limiting stops the retry spiral before it reaches the app. Cache lock stops the thundering herd before it reaches the container. Real-IP restoration stops rate limiting from blocking the wrong people entirely. These are not optimizations. They are the difference between a system that survives a spike and one that takes everything else down with it.

The actual numbers

2,500 requests per second. 147 times the required throughput. p99 on the primary endpoint at 308ms. Zero errors in 30 seconds of sustained load.

One VPS. One NGINX config. No managed services.

A VPS is not a limitation. It is an exam. Pass it and you understand distributed systems better than most people who have only ever scaled horizontally.

Have you hit a case where NGINX was not enough on a single server? Curious where the ceiling actually is before you have to scale out.

Building a Stock Trend Predictor for a Market That Has No API

Mishan Raj Shah — Sat, 30 May 2026 04:28:17 +0000

Nepal Stock Exchange has been running since 1994. It lists 359 companies. Thousands of traders use it daily. It has no public API.

That is the first wall you hit when you try to build anything data-driven for NEPSE. No official endpoint, no rate-limited free tier, no developer documentation. Just a website called MeroLagani that aggregates market data for human eyes.

This is the story of building around that constraint. The scraper, the data pipeline, the LSTM training runs, the Django REST backend, the Next.js frontend, and where the validation loss told me the truth I did not want to hear.

The project ran from March 2025 through May 2025, roughly three months from first commit to a working full-stack system.

The scraper

Clock on the wall shows 11:00 AM. Window outside shows Kathmandu cityscape.
Dark moody lighting, code visible on screen, realistic editorial style." width="800" height="640">

With no API available, I wrote a scraper that hits MeroLagani's live market table every 30 seconds during trading hours.

NEPSE trades Sunday through Thursday, 11:00 AM to 3:00 PM Nepali time. The scraper respects that schedule. It calculates the wait time to the next market open and sleeps, rather than polling and checking continuously.

market_open = dtime(11, 0)
market_close = dtime(15, 0)
allowed_weekdays = {6, 0, 1, 2, 3}  # Sunday to Thursday

Every 30 minutes it pauses for one minute to avoid triggering any server-side rate limiting. Each scrape fetches Symbol, LTP (last traded price), percent change, open, high, low, quantity, previous close, and diff, covering nine fields per stock across all listed symbols.

The data goes into a daily JSON file. One file per trading day.

def get_filename():
    date_str = datetime.now().strftime("%Y-%m-%d")
    return f"market_data_{date_str}.json"

The JSON problem

The daily files came out broken.

The scraper writes incrementally throughout the day. If anything interrupts the process, whether from a network hiccup, a keyboard interrupt, or a power cut,the JSON file ends mid-write and fails to parse. This happened repeatedly.

I ended up writing a repair script that runs json_repair on every raw file before importing:

from json_repair import repair_json

with open('./new-data/market_data_2025-04-29.json', 'r') as f:
    broken_json = f.read()

fixed_json = repair_json(broken_json)

with open('./new-fixed/repair_market_data_2025-04-29-data.json', 'w') as f:
    f.write(fixed_json)

This became a mandatory step in the pipeline. Scrape, repair, import. The repair step should have been unnecessary. The right fix is atomic writes or appending newline-delimited JSON instead of building one large JSON array throughout the day. I noted that and kept moving.

Getting it into PostgreSQL

The import script uses ijson for streaming. The daily files were large enough that loading them fully into memory caused problems on the local machine.

The data also needed cleaning before insert. NEPSE formats numbers with commas (1,234.56). Percentages come with a % suffix. Quantity fields are integers stored as comma-formatted strings.

numeric_cols = ['LTP', 'Open', 'High', 'Low', 'PClose', 'Diff.']
for col in numeric_cols:
    if col in df.columns:
        df[col] = pd.to_numeric(df[col].str.replace(',', ''), errors='coerce')

if '% Change' in df.columns:
    df['percent_change'] = pd.to_numeric(
        df['% Change'].str.replace('%', ''), errors='coerce'
    )

Inserts go in batches of 1000 rows using SQLAlchemy's multi method. The final dataset used for training had 1,109,081 rows across 359 symbols, covering around 20 trading days between February and April 2025.

Feature engineering

Before training, I added technical indicators on top of the raw OHLC data:

10-day and 50-day SMA
SMA crossover signal (1 when short SMA is above long SMA, 0 otherwise)
Position differencing to detect actual crossover events
EMA (10-period)
RSI (14-period)

The training pipeline then used RandomForestRegressor to select the top 50 features from all engineered columns, including one-hot encoded symbol columns. The idea was to let the forest figure out which features actually predicted price movement rather than guessing manually.

This decision created a problem I will get to.

Training

Three stacked LSTM layers, dropout 0.2 after each, Adam at lr=0.0005, MSE loss. Input windows of 60 timesteps, 50 features per timestep. Early stopping with patience=5 on validation loss. Training ran on Google Colab for GPU access since the local machine could not handle it.

I ran multiple training iterations between March 9 and April 29, 2025 as I cleaned and expanded the dataset. The loss plots from those runs tell the same story every time.

Train loss drops consistently across epochs. The model is learning. Validation loss oscillates. It spikes, drops, spikes again, and never converges with the training curve. In the worst runs it diverges completely after epoch 13 or 14, spiking to 3-4x the training loss before early stopping kicks in.

This is overfitting. The model memorizes the training data and fails to generalize.

The cause is straightforward: 20 trading days of intraday data is not enough to train an LSTM on time-series patterns across 359 symbols. The model sees each symbol only a handful of times in each window configuration. It has no real basis for generalization.

The feature mismatch

There is a second problem that compounds the first.

During training, RandomForest selects the top 50 features including one-hot encoded symbol columns. Different symbols get selected because different stocks have different volatility patterns and sector behaviors. The model learns symbol-specific patterns as part of its 50-feature input.

At inference time, I only have the 17 real market features for the stock being predicted. The symbol one-hot columns from the training dataset do not exist. So I pad the remaining 33 slots with zeros.

dummy_cols = np.zeros((60, 33))
full_features = np.concatenate([current_features.values, dummy_cols], axis=1)

The model runs, but it is partially blind. The symbol-specific patterns it learned during training contribute nothing at prediction time because those feature positions are always zero.

The README documents this honestly. The fix is to use a fixed symbol embedding at both training and inference. A lookup table of learned vectors per symbol, reconstructed from just the symbol string, without needing the full training dataset present.

The backend

The Django REST backend exposes prediction and market data through a clean set of endpoints under /nepse/. The prediction endpoint takes a stock symbol, pulls the last 60 records from PostgreSQL, engineers the 17 features in real time, pads to 50, runs the LSTM model, and returns a normalized trend value between 0 and 1. Above 0.5 signals an upward trend, below 0.5 signals downward.

@api_view(['POST'])
def predict_stock_price(request):
    stock_symbol = request.data.get("symbol")
    stock_data = list(
        Marketdata.objects.filter(symbol=stock_symbol)
        .order_by('-timestamp')[:60]
    )
    if len(stock_data) < 60:
        return JsonResponse({"error": "Not enough historical data"}, status=400)

Authentication uses JWT via djangorestframework-simplejwt. Admin users can manage market data records directly through the API. Regular users get read access to stock lists and predictions. The 60-record minimum is a hard requirement. New or thinly traded symbols simply cannot be predicted without enough history in the database.

The frontend

The Next.js 15 frontend gives users a stock search interface where they select a symbol, enter the current NEPSE index, and hit predict. The result comes back as a normalized percentage with a Chart.js line chart showing the predicted price movement over time.

There are two separate dashboards: one for regular users showing recent market data and trend indicators, and one for admins with full CRUD access to the market data records and a user management panel. Styling is Tailwind CSS 4. The whole frontend was built between April 5 and May 13, 2025, after the model training work was done.

What the data wall actually looks like

NYSE and NASDAQ have free API tiers through providers like Alpha Vantage and Polygon. Twenty years of daily OHLC for any listed stock, accessible with a single authenticated request.

NEPSE has the official website with a data table, a downloads section with PDFs, and nothing else.

The PDFs are worth mentioning. NEPSE publishes daily trading reports as downloadable PDFs. Parsing those is more work than scraping MeroLagani, but the data is authoritative and less likely to change HTML structure without notice. A more robust version of this scraper would target the PDFs as the primary source and use MeroLagani as a fallback.

What I would change

Atomic file writes in the scraper. Write each scrape snapshot to a separate file, or use newline-delimited JSON and append. Stop building one large JSON array that breaks on interruption.

More data before any training. I started training on weeks of data. That is not enough for an LSTM to learn market patterns across 359 symbols. The right approach is to scrape for at least 6 months before touching the model.

Fixed feature contract between training and inference. Encode each stock as a fixed embedding vector, not as one-hot columns selected by a forest that runs on the full training dataset. The symbol embedding approach means inference needs only the symbol string, not the entire historical dataset.

Separate the scraper from everything else. The scraper should run on a schedule, write to the database, and know nothing about the model. Mixing concerns early made the pipeline harder to maintain.

The full codebase including the scraper, training pipeline, Django REST API, and Next.js frontend is at github.com/Mishansavy/nepse-trend-analayis. The known limitations are in the README. Pull requests open.