DEV Community: Kimusta

Google Indexed 258 of My 406 Pages. The Sitemap Was Never the Problem.

Kimusta — Mon, 19 Oct 2026 09:50:49 +0000

I run Kimusta, a marketplace for local services in Turkey. 406 URLs in the sitemap, every one returning HTTP 200, every one with its own title, description and canonical.

Google indexed 258 of them and left 140 in Discovered — currently not indexed.

The obvious suspects were all clean: no crawl errors, no manual actions, robots.txt correct, sitemap read successfully. So I crawled all 406 URLs myself, with JavaScript disabled, and built a link graph from the raw HTML. The answer was embarrassing.

The homepage linked to four pages

Distance from `/`	Pages
0	1
1	4
2 or more	0
Unreachable without JS or the sitemap	401

Four links. On a site with 406 pages.

They pointed at /hizmetler, /hizmet-verenler, /iletisim and /yasal — and all four were single-page app shells. Their HTML body was empty; the content arrived after the JavaScript bundle did.

The cause: navigation was mounted by JavaScript

Every server-rendered page started like this:

<header id="commonSiteHeaderMount" class="site-header"></header>
<main id="main"> ...real content... </main>
<footer id="commonSiteFooterMount" class="site-footer"></footer>
<script src="/js/common-site-header.js" defer></script>
<script src="/js/common-site-footer.js" defer></script>

The content was server-rendered. The navigation was not. Menu and footer links — the most important internal links on any site — only existed after the defer scripts ran.

Total internal links across the whole site, as reported by Search Console: 970. That is 2.4 per page.

The catalogue hub had zero links

/hizmetler was the worst case. After the SPA rendered, the page had 323 words and 10 links — all of them menu items. None pointed at any of the 119 service pages it was supposed to list.

The 119 service pages, in turn, received a median of one internal link each.

What I changed

Three server-side changes. No framework, no build step.

1. Render the chrome on the server

One PHP partial returns exactly the markup the JavaScript was already producing:

// lib/site-chrome.php
function k_ust_chrome_header(): string {
    return <<<'HTML'
<header class="site-nav">
  <a class="brand" href="/">KIMUSTA</a>
  <nav class="desktop-nav">
    <a href="/hizmetler">Hizmetler</a>
    <a href="/hizmet-veren-basvurusu">Hizmet Veren Ol</a>
  </nav>
</header>
HTML;
}

The templates print it straight into the mount:

<header id="commonSiteHeaderMount">
  <?= k_ust_chrome_header() ?>
</header>

The client script stays. When it runs it writes the same markup into the same element, so there is no hydration mismatch — it only swaps the session-dependent buttons (login / account).

2. Give the hub a real body

I replaced the /hizmetler shell with a server-rendered A–Z index: 119 services, each linked, grouped by letter, with its parent category as a label. 46 KB of HTML that works with JavaScript switched off.

3. Measure the same thing again

Metric	Before	After
Links on a city page	17	38
Links on `/hizmet`	44	78
Links on `/hizmetler`	2	142

The other 140 pages are a different problem

Those 140 URLs are city pages: 20 categories × 12 cities, generated from one template. Measured across a category, the text overlap is 91%. Only the city name changes.

That is not a crawling bug. It is a content verdict, and no amount of internal linking fixes it.

What I did not do is delete them. They are linked, they return 200, and the catalogue is still growing — a category that has one service today will have ten next quarter, and its city pages become useful on their own. Deleting URLs to please a crawler is how you lose the pages you will want later.

The honest plan is slower: write content that actually differs per city (provider counts, price ranges, seasonal demand) and let the thin ones stay unindexed until they earn it.

Five things I took away

Measure crawl depth, not sitemap size. A 406-URL sitemap tells you nothing about what Google can reach by following links.
If a component is navigation, render it on the server. Hydration is for interactivity, not for structure.
Never ship an empty mount for anything you want indexed. An SPA shell returns 200 and looks fine in a browser, which is exactly why it survives review.
Discovered — currently not indexed is a quality signal. 140 pages said it at once; that was the content model talking, not a technical fault.
Programmatic pages are cheap to generate and expensive to earn. Generating them took an afternoon. Everything since has been the real work.

The site is at kimusta.com; the catalogue index that replaced the empty shell is at kimusta.com/hizmetler.

Running a two-sided marketplace on shared hosting: PHP 7.4, no framework, no cron

Kimusta — Sun, 04 Oct 2026 00:08:57 +0000

I run Kimusta, a marketplace for local services in Turkey. Someone posts what they need (a move, a deep clean, a broken boiler), vetted service providers send quotes, and the customer compares them.

The interesting part is not the product. The interesting part is that it runs on cheap shared hosting: PHP 7.4 behind LiteSpeed, MariaDB 10.3, no framework, no Node runtime, no Redis, no queue worker, and no guarantee that cron will actually fire. Here is what that constraint forced me to build, and the bugs it produced.

1. No trust in cron → "trigger on request" background jobs

Shared hosting cron is unreliable (and sometimes not available at all). Instead of a scheduler, the app runs background work inside normal web requests, gated so that only roughly 1 in 20 requests actually executes it:

// called early in the request lifecycle; cheap gate, no locks needed at this volume
if (random_int(1, 20) === 1) {
    match_auto_tick();        // pair new requests with suitable providers
    davet_hatirlatma_tick(5); // remind providers whose invitations expire soon
}

Two rules make this safe:

Every job must be idempotent. If the same tick runs twice, nothing changes twice. The reminder job writes a reminder_sent_at marker so an invitation is reminded exactly once.
Jobs must be short. If a tick takes two seconds, one unlucky user pays for it. All queries are indexed and the batches are small.

2. Never send email inside a database transaction

This was a real, user-visible bug. Matching wrote invitation rows and updated the request status inside a transaction — and notifications were simply missing from the flow. Providers had invitations sitting in the database that they never saw.

The fix was structural: collect the affected provider IDs inside the transaction, COMMIT, then notify.

$etkilenenServisler = [];
$pdo->beginTransaction();
// ... insert invitations, update request status, collect provider ids ...
$pdo->commit();

foreach ($etkilenenServisler as $servisId) {   // outside the transaction, on purpose
    davet_bildirimlerini_gonder($talepId, $servisId);
}

If the mail server is slow or down, the data is already committed. The user never waits on SMTP.

3. Deliverability: a missing header can put you in spam

mail() is disabled on this host, so everything goes through SMTP on localhost. Mail was being accepted (250 OK) — but landing in spam. The spam scanner report was blunt:

MISSING_MID 1.60
DOS_BODY_HIGH_NO_MID 4.00
=> score 9.06 (required 5)  → spam

The app never set Message-ID. Adding Message-ID, Date and To headers dropped the score to 3.461 — not spam, and the subject lost its {Spam?} prefix.

Deliverability is not "did SMTP return 250". It is "did the message arrive, and does the scanner like it". Test with real external addresses, and read the actual headers.

4. Single-page apps hand out soft 404s for free

The client-side router meant the fallback rule sent every unknown URL to the app shell with HTTP 200. Google sees an infinite supply of URLs that all return the front page — the classic soft 404.

The fix was a small server-side router that returns honest status codes, driven by the same route manifest the client uses:

known route or matching pattern → 200
unknown → 404 (still rendering the shell, but with the correct status)
if the manifest cannot be read → fail open, never fail closed

One source of truth for both sides means the server and the SPA cannot drift apart.

5. Programmatic pages: volume is easy, uniqueness is the work

The catalogue has 139 service categories, plus city-level pages, which produced a 406-URL sitemap. Generating the pages took an afternoon; making them not thin content is the ongoing part:

unique title, description and canonical per page
breadcrumbs and real internal links between parent/child categories
city pages only for cities that actually have districts in the database (example: cleaning services in Istanbul)
non-existent combinations return 404 instead of an empty page

A page that nobody can rank for is not neutral — it dilutes the crawl budget for the pages that matter.

6. Instant indexing (IndexNow) has one annoying prerequisite

IndexNow is easy once ownership is verified and infuriating before that: submissions fail with 403 SiteVerificationNotCompleted and no further explanation. The key file has to be reachable at the root, and verification has to be completed on each search engine's side first. After that, 406 URLs submitted in one go.

7. The cold-start bug nobody warns you about

A two-sided marketplace can be technically perfect and still dead, because the supply side never learns that demand exists. Our first real customer request produced three invitations — and zero quotes, because none of the providers ever opened the panel to look.

Matching is not the feature. Notifying is the feature. The notification has to arrive where the provider already is (email), and it has to say something concrete: what the job is, which district, what time window, when the invitation expires — without leaking personal data.

What I would do differently

Build notifications before the matching engine. Data in the database that nobody is told about is worthless.
Return honest HTTP status codes from day one. Retrofitting a router is more work than writing it first.
Test email deliverability against a real mailbox before launch, not after the first customer asks why nothing arrived.
Assume there is no cron. "Trigger on request" is not a hack if you design the jobs to be short and idempotent — it is a deployment model.

None of this needs a framework, a VPS, or a queue. It needs deciding what the constraints are and then designing inside them.

Kimusta is live at kimusta.com. The whole catalogue — from cleaning to boiler repair — is browsable at kimusta.com/hizmetler.