DEV Community: Malavnaik

70% of Ireland's Rent Table Is Empty. Here's How I Cleaned It

Malavnaik — Mon, 02 Nov 2026 11:42:33 +0000

Ireland has some of the best open rent data in Europe. The Residential Tenancies Board (RTB) records the rent of every new tenancy, and the Central Statistics Office (CSO) publishes the averages, every half-year since 2008, for counties, towns and small areas.

I wanted to use it for a quick analysis and found the raw table hard to work with. So I cleaned it properly and published the result as an open dataset on Kaggle and Hugging Face.

What's wrong with the raw table

The CSO table (RIH02) has 674,352 rows. Three things make it awkward:

Every category appears twice, once as a code and once as a label, plus columns that never change.
Places of very different sizes are mixed together: counties, towns, Dublin postal districts and small areas sit in one column with nothing saying which is which.
70% of the rent cells are empty.

Empty doesn't mean zero

The empty cells are the most important part. The RTB leaves a cell empty when too few new tenancies were registered to give a reliable average. It's mostly small areas and specific bedroom counts: 86% of "one bed" cells are empty, against 50% for "all bedrooms".

If you load the file and fill those gaps with zeros, or let a tool do it quietly, every average you compute is wrong. So I dropped them and wrote down why in the data card. 201,912 published rents remain.

Labelling places: my first try was wrong

I first guessed each place's level from the CSO's location codes, assuming counties ended in four zeros. The check I printed found only 4 "counties", and one of them was Christchurch, an area of Dublin city. So the guess was wrong, and the printout caught it before anything was published.

I switched to matching the 26 county names. Everything without a comma ("Carlow Town", "Dublin 8") is a town or district, and everything with one ("Tullow, Carlow") is a smaller area: 26 counties, 36 towns and districts, 375 areas.

Totals, flagged

"All bedrooms", "1 to 2 bed" and "All property types" are sums of other rows. Leaving them unmarked invites double counting, so each row has two true/false columns that say whether it's a total.

Checks, then publish

Before anything goes out, the pipeline asserts one row per half-year, place, bedrooms and property type, and that every rent is between €50 and €20,000. A GitHub Actions workflow downloads the official table, cleans it, publishes the files with a data card, and runs the same code as a Kaggle notebook. Both produce the same 201,912 rows. The code is on GitHub.

What 18 years of rents show

Average monthly rent for a new tenancy, all homes:

Dublin fell from €1,325 in early 2008 to €1,004 in early 2011, then more than doubled to €2,162 by late 2025.
Over the last ten years, rents at least doubled in 17 of 26 counties. The biggest rises were in Longford (+156%), Limerick (+146%) and Roscommon (+133%). Dublin rose 67%, from a much higher base.
Late 2025, the most expensive counties were Dublin (€2,162), Wicklow (€1,739) and Kildare (€1,719). The least expensive were Donegal (€992), Leitrim (€1,010) and Monaghan (€1,034).

These are averages for new tenancies, not what all tenants pay, and they aren't adjusted for inflation. The chart is on the project page.

What I'd do next

Adjust for inflation using the CSO's consumer price index, so 2008 and 2025 rents can be compared fairly.
Add tenancy counts, if the RTB publishes them, so users can see how reliable each average is.

The dataset is free to use under CC BY 4.0. If you build something with it, I'd love to see it.

Featured product: Cadentiq · habits, walks, workouts and focus in one Android app.

I Built a Search Tool for the EU Digital Services Act, and Tested It First

Malavnaik — Mon, 02 Nov 2026 11:42:13 +0000

The EU's Digital Services Act (DSA) sets the rules for how online platforms handle illegal content, appeals, ads and transparency. If you work in trust & safety, you keep needing to find the article that covers a situation. But you think in everyday words ("can they remove my post without saying why?"), and the law uses its own ("statement of reasons").

So I built a small search tool: type a question, get the three articles most likely to answer it, each with the matching paragraph quoted and a link to the official text. You can try it on the project page. It runs in your browser, so nothing you type is sent anywhere.

Not legal advice. The tool only quotes the law.

Why it quotes the law instead of answering

The popular approach is "RAG": find the relevant text, then have an AI model write an answer from it. For law, I chose to stop at the first half. A generated answer can sound confident and still get the law wrong; a quoted paragraph with a link can always be checked. It also meant the whole thing could run for free, with no model server to pay for.

1. Split the law into searchable pieces

I took the official English text from EUR-Lex (the EU's law website, whose documents may be reused with credit) and split it into one chunk per paragraph of each article. Long lists, like the definitions in Article 3, are split into their points. Each chunk keeps its article number, title, chapter and link. That gave 638 chunks (93 articles plus 156 recitals), which I published as an open dataset on Hugging Face.

2. Write the test before choosing the method

Before building the demo, I wrote 25 questions in everyday language, each with the article(s) that answer it, checked against the text. Then I compared three ways of searching:

Keyword search (BM25): the classic approach, matching words.
Semantic search: a small open model (all-MiniLM-L6-v2) turns text into vectors, so it can match meaning, not just exact words.
Hybrid: both rankings combined.

	Keyword	Semantic	Hybrid
Right article ranked first	48%	52%	56%
Right article in the top 3	68%	84%	80%
Right article in the top 5	72%	92%	84%

The demo shows three articles, so I shipped semantic search: it had the right article in the top 3 most often. The full evaluation is in a Kaggle notebook.

3. Run it in the browser

The demo uses a compressed copy of the same model (about 23 MB), downloaded once by the visitor's browser, plus the pre-computed vectors for every chunk. I re-ran the 25 questions with the compressed model, and it is weaker: the right article was in the top 3 for 72% of questions (56% first, 92% in the top 5). That's the honest cost of a free, private demo, and why it always shows three results with links rather than one "answer".

Where it fails

The misses are telling. "Does a platform have to explain why it removed my post?" should find Article 17, but the law calls this a statement of reasons and never uses the word "explain". The everyday question and the legal wording barely overlap, and a small model doesn't always bridge the gap.

What I'd do next

Write more test questions, ideally with people who work with the DSA, because 25 questions is a small test.
Try a stronger embedding model and see whether the gains survive compression for the browser.

What it costs

€0: a free static Hugging Face Space, a free dataset, and GitHub Actions and Kaggle on their free tiers. The code is on GitHub.

Source of the law: EUR-Lex, © European Union. Only the Official Journal text is authentic.

Featured product: Cadentiq · habits, walks, workouts and focus in one Android app.

Why '2 Hours of Maintenance' Becomes 3 Days: The Engineering Behind Failed Deployments and Rollbacks

Malavnaik — Tue, 27 Oct 2026 14:53:15 +0000

(https://malavnaik.com/blog/maintenance-window-failed-deployment-rollback/)

I Built a Tested Data Pipeline for Dublin Bikes, for €0

Malavnaik — Mon, 26 Oct 2026 18:48:31 +0000

The Dublinbikes feed tells you how many bikes are at each station right now. It doesn't keep history. So if you want to ask "how do weather and time of day change how many bikes are free?", you first need a pipeline that collects the data, keeps it safe and tells you when something breaks.

I built one, end to end, using only free tiers. The code is on GitHub and the project page has a live dashboard.

1. Collect: small, boring and append-only

A Python script runs on GitHub Actions every 15 minutes. It reads the Dublinbikes GBFS feed (all 115 stations) and, once an hour, the Met Éireann observations for Dublin Airport. Both are open data under CC BY 4.0.

Each run writes a new Parquet file, partitioned by date, to a public Hugging Face dataset. Nothing is ever overwritten, so a bad run can't damage history.

Two lessons from this step:

GitHub's own scheduler is unreliable for frequent jobs. My 15-minute schedule didn't fire once in its first hour. A free external timer (cron-job.org) that calls GitHub's API fixed it.
Fail loudly, then stop. After three failed runs in a row the workflow switches itself off and sends me a Telegram message, instead of failing silently for days.

2. Model: dbt on DuckDB

Every morning, dbt builds the models inside a DuckDB file:

Staging: proper types, UTC timestamps (the weather comes in Irish local time) and de-duplication. Each hourly weather file repeats the whole day so far, so only the latest copy of each observation is kept.
Intermediate: one row per station per hour, with average, minimum and maximum bikes, and the share of the hour the station was empty or full.
Marts: a small star schema. fct_station_hour sits in the middle, with dim_station (name, location, capacity) and dim_hour (Dublin local time, weekday and that hour's weather).

The fact table is incremental: each run only recomputes the last three hours. It is also idempotent: running it twice gives exactly the same table. That matters, because reruns are the first thing you reach for when something goes wrong.

DuckDB writes everything to a single file, so I store it in a second Hugging Face dataset between runs. That's how the incremental table "remembers" what it already built without a database server.

3. Test: 14 checks, every run

dbt runs 14 data tests on every build:

keys are unique and never empty;
every row in the fact table links to a station and an hour;
bike and dock counts are never negative;
freshness: the newest snapshot is less than 2 hours old;
volume: at least 75% of the expected snapshots arrived in the last 24 hours.

The last two catch the real-world failures: a feed that goes quiet, or a collector that silently stops. If any check fails, I get a Telegram alert.

The data also taught me something: one "station" in the feed is a placeholder that never has a bike. Left in, it tops every "most often empty" list. The dashboard now leaves out stations that never had a bike in the period.

4. Publish: a dashboard with no backend

Each run publishes the tables as Parquet, a run log and a small dashboard.json summary. The project page fetches that file in the browser and draws the chart. There's no server to keep running, and the dashboard updates as soon as the morning run finishes. The dbt docs and lineage graph are published as a free static page.

What it costs

€0: public-repo GitHub Actions minutes, free Hugging Face datasets and a static Space, and one free cron-job.org job.

What I'd do differently

Compact the small files. About 100 small Parquet files a day is fine for now, but over a year I'd merge them into daily files.
Add a "late data" test per station, not just overall, to spot a single broken dock.

What's next

The pipeline is the easy part. The question needs weeks of data across different weather before any answer is honest. Once that has built up, I'll publish the analysis and write up what the data says.

What 1.8 Million Comments Taught Me About Fair Content Moderation

Malavnaik — Mon, 26 Oct 2026 18:46:25 +0000

Content warning: this project uses a dataset of real toxic comments. This post doesn't quote any of them.

You can't share real moderation data from work, and you shouldn't. But you can practise every core trust and safety analytics skill on public data. So I did: I trained two content moderation models on 1.8 million public comments and measured them the way a trust and safety team would.

Everything is public: the Kaggle notebook, the model and its model card, a demo that runs in your browser and the code. The project write-up is on my site.

The data

The Civil Comments dataset has 1.8 million comments from news sites, each rated by several people for toxicity, insults, threats and more. It's released under CC0, so anyone can use it. Crucially, a subset is also labelled with the identity groups each comment mentions, which makes a fairness check possible. 8% of comments are toxic.

Two models

A simple baseline: TF-IDF word features with logistic regression. Fast, cheap and surprisingly strong.
A modern model: DistilBERT, a small transformer that reads words in context, fine-tuned on Kaggle's free GPU.

Every number below comes from 180,488 test comments that neither model saw during training.

	Simple baseline	DistilBERT
Ranking quality (ROC-AUC, 1 = perfect)	0.941	0.957
Toxic comments auto-actioned at ≥ 90% precision	30.9%	36.6%
Comments sent to human review (to catch ~80% of toxic ones)	10.7%	8.5%

Thresholds are policy, not maths

A model only gives scores. Someone has to decide where the lines go. I used two:

Auto-action only where the model is at least 90% precise, because wrongly removing a comment is costly.
Send to review low enough to catch about 80% of toxic comments, with humans checking everything in between.

With the modern model, that means about 3% of comments are actioned automatically and 8.5% go to reviewers. That's about 20% less review work than the baseline for the same coverage.

Don't count the model's flags

How much toxic content is there? Counting the model's flags would say 11.7%. A random sample of 1,000 comments, the way you'd do it with human reviewers, estimates 7.4% (95% interval 5.9–9.2%), and the true figure is 8.0%. The small sample wins.

The fairness check

Here's the finding that matters most. Among harmless comments that simply mention a group, how many get wrongly flagged?

Comment mentions	Simple baseline	DistilBERT
(average, identity-annotated comments)	12.8%	9.6%
Black	49.6%	35.6%
white	44.3%	31.0%
gay or lesbian	38.6%	29.9%
Muslim	37.2%	25.8%
transgender	31.6%	24.5%

The modern model is fairer for every group, but harmless comments mentioning some groups are still flagged up to about four times more often than average. The model has partly learned that naming a group is a sign of toxicity.

What I learned

What surprised me most: harmless comments that simply mention a group get flagged far more often. With the simple model, half of the harmless comments mentioning "Black" were sent to review (50%), against 13% of harmless comments overall. The modern model shrank that gap, but it didn't close it. A single headline score (above 0.95 out of 1) hides this completely, so I would always break results down by group before trusting a moderation model.

What I'd do next: tackle that bias directly. I'd add more harmless training examples that mention these groups, track the false-positive rate for each group separately, and only call the model "better" when those gaps shrink.

Try it

The demo runs a compressed copy of the model in your browser, so nothing you type leaves your device. Move the threshold and watch decisions change. It's a portfolio demo, not a production moderation system.

For the record, a late submission to the original Kaggle competition scored 0.915 on its bias-aware metric. The competition closed years ago, so that's a score only, with no ranking.

Migrating On-Prem SQL Server to Azure SQL PaaS: How We Cut Infrastructure Costs by 25%

Malavnaik — Thu, 15 Oct 2026 11:43:47 +0000

At S&P Global Market Intelligence I helped migrate on-premise SQL Server databases to Azure SQL PaaS. Our team moved between 10 and 50 databases to Azure SQL Managed Instance. The headline result was a reduction in infrastructure costs of about 25%. The more important result was that the people using those databases barely noticed it happened.

Here's what I learned about doing a migration like that well.

1. Inventory before you plan

Before choosing a target, list everything that touches the database: ETL jobs, reports, linked servers, SQL Agent jobs, cross-database queries, CLR assemblies. PaaS options don't support every on-prem feature in the same way, and you want to find the gaps on a spreadsheet, not in production.

Microsoft's Data Migration Assistant and Azure Migrate help here, but they don't know about the Excel workbook someone in finance refreshes every Monday. Talk to users.

2. Pick the right PaaS flavour

Azure SQL Database and Azure SQL Managed Instance trade compatibility against simplicity. Managed Instance keeps more instance-level features (Agent jobs, cross-database queries), so it suits lift-and-shift. Single databases are simpler and can be cheaper, but may need refactoring. Decide per workload, not per estate. We went with Managed Instance, which kept most instance-level features in place and meant fewer changes to existing code.

3. Right-size, don't copy

Most of the savings came from not paying for capacity we didn't use. On-prem servers are sized for peak load plus growth plus fear. In the cloud you can measure actual usage and scale. Look at real CPU, memory and IO over several weeks before choosing service tiers.

4. Migrate in waves

Start with low-risk databases: development, reporting copies, internal tools. Each wave teaches you something about networking, permissions or performance that makes the next wave smoother.

5. Test the boring things

Performance gets attention. In our migration, the real problems came from elsewhere:

Connection strings hard-coded in old applications and reports
SSIS packages and SQL Agent jobs that needed reworking to run against Azure
Jobs that relied on local file paths on the old servers

Also worth checking: time zone and collation differences, firewall rules and private endpoints.

6. Define "done" with the users

A migration isn't finished when the data moves. It's finished when every report, job and application works as before and the old servers are switched off. Agree that checklist with stakeholders up front.

The takeaway

Cloud migration savings come mostly from right-sizing and retiring infrastructure. Keeping users happy comes from inventory, testing and communication. Neither is glamorous, and both matter more than the migration tooling.

Hi, I'm Malav Naik: from Gujarat to Dublin, building AI systems and VEDASRA

Malavnaik — Wed, 14 Oct 2026 20:06:04 +0000

I'm Malav Naik, an AI systems builder based in Dublin, Ireland, and the creator of VEDASRA, an anime-style project. This is my first post here, so here's who I am, what I build, and what I'll be writing about.

From Gujarat to Dublin

I grew up in Gujarat, India, and studied for a Bachelor of Computer Applications at Veer Narmad South Gujarat University. My career started in data: for over five years I worked at S&P Global Market Intelligence as a Data Analyst & ETL Developer, building SQL pipelines, migrating databases to the cloud, and making sure business intelligence reports were built on data people could trust.

In September 2024 I moved to Dublin to do an MSc in Data Analytics at the National College of Ireland. I graduated in 2025 with a 2:1. My dissertation looked at how knowledge graphs and prompt engineering can support AI-driven longitudinal analytics: in short, giving language models structured facts to reason over instead of letting them guess.

What I build now

Today I'm a BIS Analyst / Applications Analyst & Developer at Trinity Care Nursing Homes in Dublin. I design AI-driven automation and analytics that support 13 nursing homes: consolidating operational reporting, integrating business systems, and testing an AI integration with Oracle NetSuite that lets authorised users ask financial questions in plain English, with role-based access and human approval for anything sensitive.

The lesson I keep relearning: AI is only as reliable as the data engineering underneath it. Most of the work is validation, audit logs, exception handling and deciding where a human should stay in the loop.

A project I'm proud of: SkillBot

During my MSc I built SkillBot, a skill-recommendation chatbot that combines a fine-tuned GPT-3.5 model, retrieval-augmented generation (RAG) and a knowledge graph of about 2.3 million RDF triples. Grounding answers in the graph got it to 97% factual accuracy in its reported evaluation and noticeably reduced hallucinations. I'll write a full breakdown soon.

VEDASRA

Outside my day job I'm building VEDASRA, an anime-style creative project, here in Dublin. More on that in a future post.

What I'll write about

Every week I'll share something real from my work:

How SkillBot combines RAG with a knowledge graph
What my dissertation taught me about prompt engineering
Lessons from five years of enterprise ETL
Designing AI automations that ask before they act
Behind the scenes of VEDASRA

If you're working on AI agents, RAG or data engineering, especially in Ireland, I'd love to connect.

Malav Naik · Dublin · creator of VEDASRA
Portfolio: malavnaik99.github.io · GitHub: malavnaik99