Why Moving Only the Web Server to a Neighboring Region Is a Mistake — 5ms vs 37ms DB Round Trips

#aws #database #networking #devops

The request

On a production e-commerce site (EC-CUBE), the request came in: "we'd like the production server to live in the Japan region." Understandable — the users are in Japan, so the server should be too, right?

Except the managed database for this system lives in the Seoul region. What happens if you move only the web server to Tokyo? As it turned out, we already had a live example in-house: the dev server in Tokyo (with its DB in Seoul) felt about 4x slower than production. For a long time everyone blamed "dev is just a cheaper instance." The real cause was somewhere else entirely.

The symptom — only the dev server is slow

Production (Seoul) and dev (Tokyo) run the same code against similar-sized data. Still, only dev takes seconds to render a page. Comparing instance plans showed no meaningful difference. Time to stop guessing and measure: specifically, the TCP connect time from the web server to the DB.

Amazon Linux 2023 doesn't ship nc, so bash's /dev/tcp does the job:

HOST=db.example.internal   # managed DB endpoint
for i in 1 2 3; do
  s=$(date +%s%N)
  timeout 5 bash -c "exec 3<>/dev/tcp/$HOST/3306" && r=ok || r=ng
  e=$(date +%s%N)
  echo "try$i: $(( (e - s) / 1000000 ))ms ($r)"
done

The numbers spoke for themselves:

Path	TCP connect time (3 tries)
Seoul web → Seoul DB	4–9ms
Tokyo web → Seoul DB	37–42ms

The cause — 30ms × dozens of queries = seconds per page

A single 37ms round trip is nothing. The problem is multiplication. A framework like EC-CUBE fires dozens to a hundred DB queries per page view, and each query pays the round trip:

30ms (added cross-region round trip) × 50–100 queries = 1.5–3 seconds per page

That explained the slow dev server completely. Not CPU, not RAM — the 30ms between web and DB was the culprit.

The 30ms between a user and your server is paid once per request. The 30ms between your web server and your DB is paid once per query. The same 30ms costs roughly 100x more between web and DB.

The fix — lay out the options with numbers attached

With measurements in hand, the options reduced to three:

A: Move both web and DB to Tokyo — summing information_schema.tables put the DB at 6.8MB. A one-minute migration, ~5 minutes of maintenance, same monthly cost.
B: Move only the web server — 1–3 seconds slower per page, per the measurements. Not recommended.
C: Stay where we are — do nothing, lose nothing.

We went with C. Faced with "we want the Japan region," showing the two numbers — 5 and 37 — ended the meeting in about a minute.

SELECT table_schema,
       ROUND(SUM(data_length + index_length)/1024/1024, 1) AS mb
FROM information_schema.tables
GROUP BY table_schema;

Bonus — a DB nobody uses was sitting in Tokyo

During the investigation I found a 1GB managed DB already running in the Tokyo region. "Could we just use that?" — first question: who's using it? Grepping the code, .env files, git history, and server configs turned up zero references. It was debris from some past experiment, so it was reported as a deletion candidate.

One caveat: if you judge "unused" by connection-count metrics, the managed service's own monitoring can register as connections, so the metric may never read zero even when nothing uses it. Combine it with the zero-references check.

Takeaways

When picking a region, "is it in the same region as the DB?" comes before "is it close to the users?"
Don't guess latency — measure it with three TCP connects via /dev/tcp, no nc required
30ms between web and DB gets amplified by query count; it's a different beast from 30ms between user and server
Judge mystery resources as unused via "zero references + connection metrics," remembering the metrics can be inflated by monitoring

"Japanese customers, so Japan region" is intuitive, but the 30ms between your web server and your DB costs about 100x more than the 30ms between your customers and your server. Measure one number before you decide.

DEV Community