DEV Community: New Way Capital Advisory

Google indexed 2 of our 9,200 pages. ChatGPT answered questions about 3 of them the same day

New Way Capital Advisory — Fri, 10 Apr 2026 16:28:18 +0000

Last week I ran site:https://nwc-advisory.com on Google. It returned 2 results.

The same day, our nginx logs showed ChatGPT fetching 3 of our pages in real time to answer user questions. One of them was a Burnham-on-Sea postcode. I don't know who asked.

Here is what 15 hours of crawler traffic looked like across our 12 domains:

Source Hits What it means

Googlebot 298 Crawling aggressively
Bingbot 217 Crawling aggressively
OAI-SearchBot 20 Building ChatGPT's search index
GPTBot 14 OpenAI training data
ClaudeBot 10 Anthropic crawler
ChatGPT-User 3 Real users getting our data via ChatGPT
PerplexityBot 0
Applebot 0

Google crawled us 298 times in 15 hours and has indexed 2 of our 9,152 canonical pages in the six weeks since we launched. OpenAI's three bots together crawled us 37 times and served 3 live answers from our pages the same day. Anthropic's ClaudeBot added another 10. Perplexity and Apple did not show up at all.

For a small SaaS with no domain authority and few backlinks, these are not the same distribution channel anymore.

The receipts:

Three log lines (I have removed the client IPs but kept everything else):

[10/Apr/2026:08:03:43 +0000] "nwc-advisory.com"
"GET / HTTP/2.0" 200 38145 "-"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko);
compatible; ChatGPT-User/1.0; +https://openai.com/bot"
[10/Apr/2026:09:06:21 +0000] "property.nwc-advisory.com"
"GET /prices/sg7-5aa HTTP/2.0" 200 6831 "-"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko);
compatible; ChatGPT-User/1.0; +https://openai.com/bot"
[10/Apr/2026:09:46:47 +0000] "property.nwc-advisory.com"
"GET /prices/ta8-1aa HTTP/2.0" 200 6816 "-"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko);
compatible; ChatGPT-User/1.0; +https://openai.com/bot"

OpenAI operates three distinct crawlers and our logs show all three in the same window:

GPTBot:training data collection
OAI-SearchBot:building the search index used inside ChatGPT
ChatGPT-User:real-time fetches triggered when a user asks ChatGPT a question

The first two are indexers, like Googlebot or Bingbot. The third is different. A ChatGPT-User hit means someone asked ChatGPT a question, and ChatGPT decided to fetch your page to answer them.

The three pages our logs:

Homepage (https://property.nwc-advisory.com)
/prices/sg7-5aa, Hertfordshire. 134 transactions last year, median £348,750.
/prices/ta8-1aa -- Burnham-on-Sea, Somerset. 223 transactions, median £275,000.

Someone asked ChatGPT about UK house prices in Baldock. Someone else asked about Burnham-on-Sea.

They got numbers. They never visited our site.

If you want to see the raw pages ChatGPT fetched, they are here:

Why our pages are AI-readable?

Nothing exotic. Three boring things.

robots.txt whitelists AI bots

Most of the "how to block ChatGPT" posts got this backwards. We do the opposite:

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent:
Disallow: /v1/
Disallow: /api/

The API endpoints are protected. The landing pages are wide open. Block what is monetized, allow what you want distributed.

Server-rendered HTML, no SPA

Every landing page is a complete HTML document at request time. No React, no Vue, no client-side fetch() for the data. ChatGPT-User does not execute JavaScript -if your data is populated client-side, it is invisible to the fetch. We use a Python script that renders Jinja2 templates into static HTML files at build time. 9,152 pages across 12 domains, all pre-rendered, all around 6KB each.

IndexNow, not just Google Search Console

IndexNow is a free push protocol that notifies Bing, Yandex, and a few others instantly when you publish a URL. Bing's index directly feeds ChatGPT Search, Grok, and Copilot. We submit every new page via IndexNow within seconds of deploying it. Waiting for Googlebot to re-crawl is slower by orders of magnitude, and for a low-authority domain, Google may never index the page at all.

Here is the Python snippet we use to classify bot traffic when we parse our nginx logs:

python
AI_BOTS = {
'ChatGPT-User': 'Live query - user asking ChatGPT now',
'OAI-SearchBot': 'ChatGPT search index',
'GPTBot': 'OpenAI training data',
'ClaudeBot': 'Anthropic crawler',
'PerplexityBot': 'Perplexity search',
'Bytespider': 'ByteDance / TikTok',
'Applebot': 'Apple / Siri',
}

SEARCH_BOTS = {
'Googlebot': 'Google Search',
'bingbot': 'Bing Search',
'DuckDuckBot': 'DuckDuckGo',
}

def classify(user_agent: str) -> str:
ua = user_agent.lower()
for bot, purpose in AI_BOTS.items():
if bot.lower() in ua:
return f"AI: {purpose}"
for bot, purpose in SEARCH_BOTS.items():
if bot.lower() in ua:
return f"Search: {purpose}"
return "Human"

Run that across your access log and you will see AI traffic showing up in categories you do not have dashboards for yet.

What we built that Google ignored

Because I want to be precise about this: we did the SEO work. Not a rushed job. Over six weeks we shipped:

9,152 canonical tags across 12 domains
hreflang tags linking en to fr on the bilingual property apps
Sitemaps on all 12 domains, 9,157 URLs total
IndexNow batch submissions, 9,157 URLs pushed to Bing and Yandex
Google Search Console verified on every domain, sitemaps submitted
Disallow: /v1/ and Disallow: /api/ on all property apps so crawlers do not waste their budget on API endpoints
FAQPage JSON-LD on every landing page for rich snippets
Open Graph and Twitter Card meta on every domain

Result after six weeks: 2 URLs indexed on Google.The homepage (nwc-advisory.com/) and one NYC property page (property-nyc.nwc-advisory.com/comps/11225). That is the entire Google index for our brand.

This is not a Google bug. It is a domain authority floor -- Google rate-limits indexing for new domains without backlinks. The old playbook was: wait three to six months, build backlinks, eventually get indexed. The new playbook is: while you are waiting for Google, OpenAI and Microsoft are already crawling you and serving your data to their users.

Six takeaways if you are building something data-heavy

Allow AI bots in robots.txt. Block sensitive paths, not the crawlers themselves.
Server-render every data page.ChatGPT-Userdoes not execute JavaScript. If your data is in a fetch() call, it is invisible.
Submit to IndexNow.Bing feeds ChatGPT Search, Grok, and Copilot. Waiting for Google is slower and less likely to work for new domains.
Write long-tail pages for long-tail queries. ChatGPT's live fetches in our logs were for specific UK postcodes. One page per specific question.
Parse your access logs weekly. Grep for ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot. You will see AI distribution working before it shows up in any dashboard.
Keep doing SEO for Google anyway. Do both. But calibrate your expectations for the first six months - if your domain is new, Google is not a near-term channel.

We Added Rental Yield to a Free Property Comps API — Here's How It Works

New Way Capital Advisory — Mon, 06 Apr 2026 14:29:18 +0000

Last month I shared a free property comparable sales API covering 11 markets.
Since then, we shipped two updates worth talking about: a new US market (Washington DC) and rental yield data baked into the stats endpoint.

The Problem

If you're building real estate tools, getting comparable sales data is step one. But investors always ask the same follow-up: "What's the yield?"

Answering that requires a second data source — rental data — and a calculation layer. Most developers either hardcode national averages (inaccurate) or pay for expensive rental APIs (overkill for a yield estimate).

What We Built

The /v1/stats endpoint now returns gross rental yield for all 6 US markets, derived from two federal data sources:

Census ACS 5-Year Survey** (primary) — median rents by ZIP and bedroom count
HUD Small Area Fair Market Rents (SAFMR) FY2026** (fallback) — published annually per ZIP

No extra API call. No separate subscription. Yield comes back in the same stats response you're already using.

Example


bash
  curl "https://api.nwc-advisory.com/v1/stats?market=dc&zip_code=20001&months=12" \
    -H "X-RapidAPI-Key: YOUR_KEY"

Response:

  {
    "area": "20001",
    "period": "12 months",
    "stats": {
      "median": 760000,
      "count": 745
    },
    "monthly_rent_estimate": 3167,
    "gross_yield_pct": 5.0,
    "yield_source": "safmr",
    "beds_used_for_yield": 2,
    "median_price_per_sqft": 639,
    "by_type": {
      "Row House": { "median": 913000, "count": 178 },
      "Condo": { "median": 635000, "count": 456 }
    }
  }

One call gives you: median price, transaction count, price per sqft, breakdown by property type — plus monthly rent estimate, gross yield percentage, and the data source.

Bedroom-Specific Yields

Pass min_beds to refine the yield to a specific segment:

  3-bedroom yield in DC
  curl "https://api.nwc-advisory.com/v1/stats?market=dc&zip_code=20001&min_beds=3&months=12"

This does two things:
  1. Filters comparable sales to 3+ bedroom properties
  2. Uses the 3-bedroom rent bracket from SAFMR/Census instead of the default 2-bedroom

The response includes beds_used_for_yield: 3 so you know which bracket was applied.

How the Yield Is Calculated

gross_yield = (monthly_rent × 12) / median_sale_price × 100

  - monthly_rent comes from Census ACS (preferred) or HUD SAFMR (fallback), matched by ZIP + bedroom count
  - median_sale_price comes from actual government-recorded transactions in the search radius
  - This is gross yield — before expenses, vacancy, or taxes

The formula is intentionally simple. Investors apply their own expense ratios on top.

New Market: Washington DC

We added DC as our 12th market (6th US market). The data comes from the DC Office of Tax and Revenue CAMA system:

  - 256,000+ transactions (1979–2026)
  - Full property detail: beds, baths, sqft, year built, neighborhood, ward
  - 86% of records have complete bedroom/bathroom data
  - Updated from DC government's ArcGIS REST API

All 12 Markets

  ┌────────────────┬──────┬──────────────┬────────────────┐
  │     Market     │ Code │ Transactions │ Location Param │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ United Kingdom │ uk   │ 31M+         │ postcode       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ France         │ fr   │ 8.3M+        │ code_postal    │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Singapore      │ sg   │ 970K+        │ postal_code    │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ New York City  │ nyc  │ 500K+        │ zip_code       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Chicago        │ chi  │ 580K+        │ zip_code       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Miami-Dade     │ mia  │ 417K+        │ zip_code       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Philadelphia   │ phl  │ 290K+        │ zip_code       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Connecticut    │ ct   │ 210K+        │ town           │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Washington DC  │ dc   │ 256K+        │ zip_code       │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Dubai          │ dxb  │ 1.3M+        │ area_name      │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Ireland        │ ie   │ 230K+        │ county         │
  ├────────────────┼──────┼──────────────┼────────────────┤
  │ Taiwan         │ tw   │ 150K+        │ city_en        │
  └────────────────┴──────┴──────────────┴────────────────┘

Rental yield is available on the 6 US markets (NYC, CHI, MIA, PHL, CT, DC). Yield data for UK, France, and Dubai is available in PDF valuation reports — API exposure for international markets is on the roadmap. 

Data Sources

Every transaction comes from an official government registry — not MLS, not scraped listings, not estimates. The key difference: these are actual recorded sale prices, not asking prices or algorithmic valuations.

Yield data comes from federal sources (Census Bureau + HUD) that are updated annually and publicly available. We cache and serve them per-ZIP so you don't have to parse XLSX files from government portals.

Try It

  - Free tier: 50 requests/month, no credit card
  - Docs: https://api.nwc-advisory.com/docs
  - RapidAPI: https://rapidapi.com/NWCA/api/property-comparable-sales

We're adding more US markets — if you're building something that needs specific coverage, let us know in the comments.

All data sourced from government registries. Yield estimates are informational — not financial advice.

● Live Apps

What to see the API/data in action ?? — free property search with PDF reports

US locations:

- Chicago: https://property-chi.nwc-advisory.com
- New York City: https://property-nyc.nwc-advisory.com
- Miami-Dade:  https://property-miami.nwc-advisory.com
- Philadelphia: https://property-phl.nwc-advisory.com
- Connecticut:  https://property-ct.nwc-advisory.com
- Washington DC: https://property-dc.nwc-advisory.com

International locations:

- United Kingdom: https://property.nwc-advisory.com
- France: https://property-fr.nwc-advisory.com
- Singapore: https://property-sg.nwc-advisory.com
- Dubai: https://property-dxb.nwc-advisory.com
- Ireland: https://property-ie.nwc-advisory.com
- Taiwan: https://property-tw.nwc-advisory.com

We caught ChatGPT answering property questions with our data -- here's the nginx log proof

New Way Capital Advisory — Mon, 30 Mar 2026 21:43:07 +0000

The moment we noticed

Yesterday, while running our routine nginx log analysis, we spotted something unusual:

51.107.70.192 - [30/Mar/2026:19:42:35 +0000] "property.nwc-advisory.com"
"GET /prices/sk10-1ae HTTP/2.0" 200 4623 "-"
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible;
ChatGPT-User/1.0; +https://openai.com/bot"

A ChatGPT-User bot fetched one of our UK property landing pages. Not GPTBot (the training crawler). Not OAI-SearchBot (the indexer). The ChatGPT-User agent -- the one that fetches pages in real time to answer a user's question.

Someone asked ChatGPT about house prices near Macclesfield (SK10 postcode), and ChatGPT pulled the answer from our page.

What's the difference between OpenAI's bots?

OpenAI operates three distinct crawlers. In the same 2-hour window, we saw all three:

Bot	User-Agent	Purpose	Our logs
GPTBot	`GPTBot/1.3`	Training data collection	Crawling sitemaps, homepages
OAI-SearchBot	`OAI-SearchBot/1.3`	Search index building	Crawling UK landing pages (SK postcodes), robots.txt across all 11 domains
ChatGPT-User	`ChatGPT-User/1.0`	Live query answering	Fetched `/prices/sk10-1ae` for a real user

The ChatGPT-User hit is the interesting one. It means our data is being served to end users in real time through ChatGPT's search feature. The user never visits our site -- they get the answer inside their chat.

The numbers

In a 2-hour window on March 30, 2026:

OpenAI bots:  22 requests (GPTBot + OAI-SearchBot + ChatGPT-User)
Googlebot:     4 requests

OpenAI is crawling our pages 5x more frequently than Google.

Here's what OAI-SearchBot was systematically crawling:

property.nwc-advisory.com       GET /prices/sk1-1aa
property.nwc-advisory.com       GET /prices/sk10-1ae
property.nwc-advisory.com       GET /prices/sk11-0aa
property.nwc-advisory.com       GET /prices/sk12-1aa
property.nwc-advisory.com       GET /prices/sk13-0aa
property.nwc-advisory.com       GET /prices/sk14-1aa
property.nwc-advisory.com       GET /sitemap.xml
property.nwc-advisory.com       GET /robots.txt
property-chi.nwc-advisory.com   GET /robots.txt
property-dxb.nwc-advisory.com   GET /robots.txt
property-ie.nwc-advisory.com    GET /robots.txt
property-miami.nwc-advisory.com GET /robots.txt
property-phl.nwc-advisory.com   GET /robots.txt
property-sg.nwc-advisory.com    GET /robots.txt
property-tw.nwc-advisory.com    GET /robots.txt

It's reading the sitemaps, then crawling individual landing pages. Systematically.

What we built (and why AI can read it)

We operate property comparable sales apps across 11 markets (UK, France, Singapore, NYC, Chicago, Miami, Philadelphia, Connecticut, Dubai, Ireland, Taiwan), backed by 35M+ government-source transactions.

For SEO, we generated 9,100+ static landing pages -- one per postcode/ZIP/area:

/prices/sw1a-1aa    → London Westminster
/prices/sk10-1ae    → Macclesfield
/prix/75001         → Paris 1er
/comps/10001        → Midtown Manhattan

Each page contains:

Median price, average price, price range
Price per sqft/m2 statistics
Recent comparable sales with addresses
Stamp duty / notary fee calculators
FAQ schema (JSON-LD)
Nearby area links

The key design decision: no login walls, no JavaScript-rendered content, no gated data. Every landing page is server-rendered HTML with structured data that any crawler can parse.

The page structure that AI loves

<!-- Clean semantic HTML -->
<h1>Property Prices in SK10 1AE, Macclesfield</h1>

<div class="stats-grid">
  <div class="stat">
    <span class="label">Median Price</span>
    <span class="value">£285,000</span>
  </div>
  <div class="stat">
    <span class="label">Price per sqft</span>
    <span class="value">£198</span>
  </div>
</div>

<!-- JSON-LD structured data -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "WebApplication",
  "name": "Property Comparable Sales - SK10",
  "description": "Recent property sales near SK10 1AE, Macclesfield",
  "url": "https://property.nwc-advisory.com/prices/sk10-1ae"
}
</script>

<!-- FAQ schema for rich snippets -->
<script type="application/ld+json">
{
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is the average house price in SK10?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "The average house price in SK10 is £312,450..."
      }
    }
  ]
}
</script>

The SEO stack

nginx (SSL, bot-block with search engine whitelist)
  │
  ├── /prices/{postcode}  →  Static HTML (2,308 UK pages)
  ├── /prix/{code_postal} →  Static HTML (5,851 FR pages)
  ├── /comps/{zip}        →  Static HTML (per US market)
  │
  ├── sitemap.xml          →  All page URLs
  ├── robots.txt           →  Allow everything except /v1/ and /api/
  └── IndexNow key         →  Instant Bing/Yandex notification

Key technical decisions:

1. robots.txt allows AI bots

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: *
Disallow: /v1/
Disallow: /api/

Many sites block GPTBot. We deliberately allow it. The API endpoints are protected (disallowed), but the landing pages are open.

2. Server-rendered, not SPA

Each landing page is a complete HTML document. No client-side rendering, no JavaScript required to see the data. ChatGPT-User fetches the HTML and can immediately extract the statistics.

3. Canonical URLs and sitemaps

Every page has <link rel="canonical"> and is listed in the sitemap. This tells crawlers exactly which pages exist and which URL is authoritative.

4. Structured data (JSON-LD)

FAQPage schema means the questions and answers are machine-readable. When ChatGPT needs "What is the average house price in SK10?", it can extract the answer directly from the structured data.

How we detect AI bot traffic

Our visitor analysis pipeline (Python, runs against nginx logs) classifies every request:

# Bot classification
AI_BOTS = {
    'ChatGPT-User':   'Live query - someone asked ChatGPT',
    'GPTBot':         'Training data collection',
    'OAI-SearchBot':  'Search index building',
    'ClaudeBot':      'Anthropic training',
    'PerplexityBot':  'Perplexity search',
    'Bytespider':     'TikTok/ByteDance',
}

SEARCH_BOTS = {
    'Googlebot':      'Google Search indexing',
    'bingbot':        'Bing Search indexing',
    'Applebot':       'Apple/Siri search',
    'DuckDuckBot':    'DuckDuckGo indexing',
}

def classify_request(user_agent: str) -> str:
    ua_lower = user_agent.lower()
    for bot, purpose in AI_BOTS.items():
        if bot.lower() in ua_lower:
            return f"AI Bot: {purpose}"
    for bot, purpose in SEARCH_BOTS.items():
        if bot.lower() in ua_lower:
            return f"Search: {purpose}"
    return "Human visitor"

We run this daily against our nginx logs. The trend over the past few weeks: AI bot traffic is growing faster than search bot traffic.

What this means for developers

If you're building a data-heavy application, the old playbook was:

Build app → SEO → Rank on Google → Users find you → They visit

The new playbook is:

Build app → Structured data → AI crawls you → Users get your data via ChatGPT
                                              (they may never visit your site)

This is both exciting and unsettling. Exciting because your data reaches users through a completely new channel. Unsettling because those users never see your UI, your brand, or your conversion funnel.

Practical takeaways

If you want AI to use your data:

Don't block AI bots in robots.txt (unless you have a reason to)
Server-render your pages -- ChatGPT-User can't execute JavaScript
Use structured data (JSON-LD, schema.org) -- makes extraction trivial
Generate landing pages for long-tail queries -- AI searches the same way humans do
Keep data ungated -- if it's behind a login, AI can't reach it
Maintain sitemaps -- AI bots follow them just like Googlebot

If you want to track it:

Parse your nginx/access logs for ChatGPT-User, GPTBot, OAI-SearchBot
ChatGPT-User = your data is being served to real users right now
OAI-SearchBot = your pages are being indexed for future queries
GPTBot = your content may be used for model training

The broader picture

Google is still the primary search engine. But in our 2-hour sample, OpenAI's crawlers outnumbered Googlebot 5:1. And while Google has been crawling our 12 domains for weeks without indexing most pages (domain authority problem), ChatGPT is already serving our data to users.

For a small startup with no domain authority, no backlinks, and a brand-new domain, this is significant. The traditional SEO path (build authority -> get indexed -> rank -> get traffic) takes months to years. The AI search path (structured data -> get crawled -> get cited) can happen in weeks.

We're not saying Google doesn't matter. We're saying there's a new, parallel distribution channel, and it favors clean data over domain authority.

Our stack: Python (FastAPI) + SQLite + nginx + static HTML generators. 11 property markets, 35M+ transactions from government open data sources. Free to search at property.nwc-advisory.com.

The API is also available on RapidAPI and as an MCP server for AI agents.

What AI bot traffic are you seeing in your logs? Have you caught ChatGPT-User fetching your pages? Drop a comment -- I'm curious if this is a broader trend or we're just early.*

I Built a Free Property Comparable Sales API Covering 11 Global Markets (44M+ Transactions)

New Way Capital Advisory — Sat, 28 Mar 2026 15:15:49 +0000

Every property valuation starts with the same question: what did similar properties sell for nearby? The data exists — government
land registries publish it — but it is scattered across different formats, languages, and access methods. UK Land Registry uses CSV
dumps. France publishes DVF files. Singapore has a government portal. New York City releases rolling Excel files. None of them
talk to each other.

I built a unified API that normalises all of it into a single interface. One endpoint, one schema, eleven markets.

What It Covers
Market Source Transactions Location Param
UK (England & Wales) HM Land Registry 31M+ postcode

France DVF 8.3M+ code_postal

Singapore HDB Resale Data 973K+ postal_code

New York City Dept of Finance 51K+ zip_code Chicago Cook County 480K+ zip_code

Dubai DLD Transactions 1.3M+ area_name

Miami Miami-Dade County 190K+ zip_code

Philadelphia OPA Sales Data 350K+ zip_code

Connecticut Town Clerk Records 280K+ town

Ireland Property Price Reg 450K+ county

Taiwan Ministry of Int 180K+ city_en

All data is government-sourced. No scraped listings. No estimates. Actual recorded sale prices.

Three Endpoints

Comparable Sales

GET /v1/comps?market=nyc&zip_code=10001&months=12&limit=20

Returns recent sales near a location: address, sale price, date, property type, size, price per square foot/metre. Filter by
property type, bedroom count, and date range.

Area Statistics

GET /v1/stats?market=uk&postcode=SW1A1DA

Median price, price per area unit, transaction volume, price trends, property type breakdown. The numbers you need to understand a micro-market at a glance.

Monthly Trends

GET /v1/trends?market=fr&code_postal=75001&months=24

Month-by-month median prices and volumes. Useful for spotting whether a market is heating up or cooling down.

How It Works

Each market has its own backend API (FastAPI + SQLite) running on a dedicated port. A unified gateway at api.nwc-advisory.com routes requests by the market parameter to the correct backend, normalises the response schema, and returns consistent JSON regardless of which country you are querying.

Client request
→ api.nwc-advisory.com (gateway, port 8080)
→ market backend (ports 8060-8070)
→ SQLite database (government data, indexed)
← normalised JSON response
← consistent schema

The SQLite databases are built from government bulk downloads. Each has compound indexes on location + property type + date for
fast radius searches. The densest queries (central London postcodes covering thousands of transactions) return in under 5 seconds.

The Hardest Parts

Address normalisation across 11 countries. UK postcodes, French postal codes, Singapore 6-digit codes, US ZIP codes, Dubai area
names, Irish counties, Taiwanese districts in both English and Traditional Chinese. Each market has its own geocoding logic and
search radius behaviour.

Currency and unit consistency. Price per square foot in the US, price per square metre in Europe and Asia. GBP, EUR, USD, SGD, AED,NTD. The API returns values in each market's native currency and unit — no forced conversions that would confuse local users.

Data freshness. Each market publishes on a different schedule. UK Land Registry is monthly with a 6-week delay. France DVF is semi-annual. Singapore HDB is monthly with minimal delay. The update pipeline checks each source, downloads new data, ingests it into SQLite, deploys to the server, and verifies the health endpoint — all scripted.

Free Tier

The API is available on RapidAPI with a free tier:

Free: 50 requests/month, 10 results per search, 3 markets (UK, NYC, Singapore)
Pro ($29/mo): 1,000 requests, 100 results, all 11 markets
Ultra ($99/mo): 10,000 requests, 500 results, all 11 markets

Interactive docs at api.nwc-advisory.com/docs (Swagger UI). You can test queries directly in the browser.

MCP Server (For AI Agents)

If you are building with Claude, GPT, or any LLM agent framework, there is an MCP server that wraps the API into three tools:
search_property_comps, get_area_stats, and list_markets. The repo is open source on GitHub: Tianning-lab/property-comps-mcp-server.

This means an AI agent can answer questions like "What did 3-bedroom flats sell for near Zurich Bahnhofstrasse in the last year?" by calling the tool directly — no prompt engineering around raw API calls needed.

What I Would Do Differently

Start with fewer markets. I launched UK and France first, then added nine more. Each new market took 2-3 days of data pipeline work plus frontend localisation (French, Traditional Chinese, Spanish for NYC). If I did it again, I would ship three markets and validate demand before building the rest.

Billing from day one. The free property apps get traffic but almost nobody signs up for paid tiers. Building the RapidAPI listing earlier and focusing on API-first distribution (developers, agents, fintech integrations) would have been a better monetisation path than consumer web apps.

Stack

Backend: Python 3.13, FastAPI, SQLite (one DB per market)
Gateway: FastAPI proxy with OpenAPI 3.1 spec
Frontend: Static HTML/CSS/JS (11 localised versions)
Server: Single Ubuntu box, nginx, Let's Encrypt
Data pipeline: Python ingest scripts per market, cron-scheduled checks

No cloud services, no managed databases, no Kubernetes. One server handles all 11 markets comfortably.

The API docs are at api.nwc-advisory.com/docs. The MCP server is at github.com/Tianning-lab/property-comps-mcp-server. If you are building anything in proptech or real estate analytics, I would like to hear what data you wish existed.

How I made a 31M-row SQLite query go from 168 seconds to 5 seconds

New Way Capital Advisory — Tue, 24 Mar 2026 14:24:20 +0000

My property comparison app was working great in development. Then a user in Norwich searched for nearby sales and my API took 168 seconds to respond. nginx gave up at 60 seconds and returned a 504.

Here's how I fixed it -- and what I learned about SQLite performance at scale.

The Setup

I built a property comparables API that serves recent sales data from government open datasets. The UK version uses HM Land Registry Price Paid Data -- every residential property transaction in England and Wales since 1995.

That's 31 million rows in a single SQLite database. 8.7GB on disk.

The API is simple: give it a postcode, it finds nearby sales within a radius, filtered by property type and date range. The stack is Python + FastAPI + SQLite, running on a single $20/month VPS.

For most postcodes, it was fine. London postcodes with a few hundred results came back in under a second. But Norwich postcode NR1 has 1,941 postcodes in its search radius -- and that's where everything broke.

The Query

SELECT *
FROM transactions
WHERE postcode IN (?, ?, ?, ... )  -- up to 1,941 postcodes
  AND property_type = ?
  AND date >= ?
ORDER BY date DESC
LIMIT 100

Nothing exotic. But SQLite was doing a full table scan on 31M rows for each query.

The Wrong Fix I Tried First

My instinct was to add an index on postcode:

CREATE INDEX idx_postcode ON transactions(postcode);

This helped -- queries dropped to about 40 seconds. Still unusable. The problem is that after finding rows by postcode, SQLite still had to scan all matching rows to filter by property_type and date.

The Actual Fix: Compound Indexes

The key insight is that SQLite can only use one index per table per query. If your WHERE clause filters on three columns, a single-column index only helps with the first filter. The rest are scanned.

The fix is a compound index that matches your query pattern:

CREATE INDEX idx_comp ON transactions(postcode, property_type, date DESC);

The column order matters -- it must match the query's filter order. SQLite walks the B-tree left to right: first it narrows by postcode, then by property type, then by date range. All three filters are served by the same index lookup.

After creating the index:

ANALYZE;  -- update the query planner's statistics

Result: NR1 went from 168 seconds to 4.8 seconds. Every other postcode dropped to under 1 second.

The Same Fix, Different Country

I had the same problem with the France version -- 8.3 million transactions from the DVF open dataset. Same fix:

CREATE INDEX idx_comp ON transactions(code_postal, type_local, date_mutation DESC);
ANALYZE;

All major French cities went to under 100ms.

What Else I Learned

1. ANALYZE is not optional. Without it, SQLite's query planner doesn't know about the index distribution and may choose a suboptimal plan. I spent an hour wondering why my new index wasn't helping before realizing I'd skipped this.

2. nginx timeout matters. Default proxy_read_timeout is 60 seconds. Dense queries that take 5-10 seconds are fine, but if you ever regress, nginx will kill the connection before the user sees an error. I bumped it to 300 seconds as a safety net:

location /v1/ {
    proxy_read_timeout 300s;
    proxy_pass http://127.0.0.1:8060;
}

3. SQLite handles 31M rows better than you'd think. There's a bias in the industry that SQLite is "for small stuff." With proper indexing, it serves production traffic on a single VPS without connection pooling, replication, or any of the Postgres/MySQL infrastructure. My entire hosting cost is $20/month for 11 country databases.

4. Test with your densest data. I had been testing with London postcodes (small radius, few matches). The bug only appeared in Norwich (huge radius, 1,941 postcodes). Now I always test with the worst case after any database or query change.

The Numbers

Metric	Before	After
NR1 (worst case)	168s	4.8s
SW1A (London)	12s	0.3s
NR1 via HTTPS	504 timeout	5.1s
Index size overhead	--	+1.2GB
Total DB size	8.7GB	9.9GB

The 1.2GB index overhead is a good trade for 30x query speedup.

Live Demo

The API serves comparable sales across 11 countries, all using the same SQLite + compound index pattern. If you want to see the result: property.nwc-advisory.com for UK data.

The API is also available on
RapidAPI(https://rapidapi.com) if you want to integrate property comps into your own tools, and there's an open-source MCP server if you use Claude or other AI agents.

Have you hit similar SQLite performance walls? I'm curious what scale other people are running SQLite at in production.

I built a free Morningstar X-Ray replacement for European/US ETF investors

New Way Capital Advisory — Mon, 23 Mar 2026 15:33:22 +0000

tags: fintech, python, api, opensource

When Morningstar paywalled their Portfolio X-Ray tool in 2025 ($249/year), European investors got hit the hardest. Most alternatives are US-ticker-only. If you hold iShares, Vanguard, Xtrackers, or Synchrony funds with ISINs, your options were basically zero.

So I built a free replacement: nwc-advisory.com/xray

What it does

Enter your fund ISINs and amounts (no login, no account). It fetches public factsheet data, decomposes each fund into underlying holdings, and shows 6 analysis views:

Asset Allocation -- before and after look-through
Geographic Exposure -- economic exposure, not fund domicile (your IE-domiciled S&P 500 ETF shows as US, not Ireland)
Sector Breakdown -- GICS sectors across your entire portfolio
Stock Overlap -- stocks appearing in 2+ funds, with fund similarity percentages
Fee Analysis -- weighted TER vs cheapest passive alternative, with annual savings in your currency
Top Holdings -- your 30 largest underlying positions

Tech stack

Backend: Python/FastAPI
Data: 13 async factsheet fetchers (iShares CSV exports, JustETF, FT.com, Gerifonds PDF parsing, and 9 more)
Cache: 24h disk-backed JSON + in-memory LRU (500 entries). Popular funds return in under 100ms, cold starts take 15-30s
PDF: ReportLab for 6-page PDF reports (email-gated)
Frontend: Vanilla HTML/CSS/JS, no framework
Hosting: Single Ubuntu server, systemd service, nginx reverse proxy

The interesting technical bits

Name normalization was harder than expected. The same company appears as "Nestle SA", "Nestle AG", "NESTLE S.A.", and "Nestlé" across different data sources. I strip accents, remove corporate suffixes (AG, SA, Ltd, Inc, Corp, PLC, Group, Holdings), and normalize whitespace before deduplicating.

Fund-pair overlap uses a Jaccard-like calculation on the intersection of underlying holdings, weighted by position size. This tells you "your MSCI World and S&P 500 ETFs are 78% identical" -- meaning you're paying two sets of fees for nearly the same exposure.

ISIN-based routing determines which fetchers to try first. CH-prefix ISINs go to Swiss sources (Gerifonds, SwissFundData), IE/LU go to JustETF and iShares, and everything falls back through 9 legacy fetchers.

What I couldn't replicate

Morningstar's style box (3x3 value/growth/blend grid) -- proprietary classification
P/E and P/B ratios -- would need a real-time data feed
Performance tracking -- out of scope for a free tool

Also: Property Comps API

Separately, I built an API serving 4.2M+ government-recorded property transactions across 11 markets (UK, France, Dubai, Singapore, NYC, and 6 more). If you're building anything in proptech:

Swagger docs: api.nwc-advisory.com/docs
Free tier available

Happy to answer questions about the architecture or data sources.