DEV Community: Sergey Laptick

How We Automated a Production-Grade Traccar Deployment

Sergey Laptick — Mon, 20 Jul 2026 14:28:48 +0000

GPS tracking systems have become essential infrastructure for fleet management, logistics, delivery services, and field operations. Companies that manage vehicles, assets, or mobile workers need reliable visibility into where things are and where they have been.

Traccar is a mature open-source platform that handles such functionality well. The challenge is how to get it running in a production environment with monitoring, security, and the ability to customize it without spending weeks on setup.

We built an automated deployment script to solve this. We use it to deploy production-ready Traccar environments, then customize them according to our clients’ needs. This article explains how it works and what we deliver.

What Available Traccar Deployment Options Lack

Traccar is a system for tracking anything that moves: couriers, drivers, fleets, assets, etc. It includes a server, a frontend interface, and mobile applications that can function as GPS trackers themselves. The open-source version is available on GitHub, and anyone can download and try it. However, launching it and running it in production for years are two different things.

Most companies that need GPS tracking software do not want to spend time figuring out which database works best with Traccar, how to set up production environment monitoring so they know when something breaks, or how to configure a firewall to prevent data leaks. They want a system that works, stays running, and alerts them before it breaks down.

The default deployment options available today have gaps:

AWS Marketplace offers a basic Traccar instance at about $0.03 per hour. You get the software running, but the monitoring is whatever AWS provides by default. Limited styling, no tailored alerts, no ability to change how the system behaves;
Traccar's own hosted server for 50 devices costs around $49.95 per month in the US. It includes some branding options (logo, name and colors) with no UX or functional changes available. The server runs in Traccar's environment, not on the client's infrastructure. All data lives on Traccar's servers. For companies with a preference to keep their operational data on their own hardware, this is a non-starter;
Deploying Traccar yourself on your infrastructure from the open-source repository means your company owns and manages the entire stack. This option includes figuring out deployment, database configuration, reverse proxy setup, monitoring, alerting, and security. It is doable, but it takes time and expertise that most operations teams do not have to spare.

Here is how the available options compare and where we bring our expertise to the table:

What We Built: Automated Production Environment Deployment

When a client came to us needing a production-ready Traccar deployment without the usual headaches, we delivered it quickly and reliably. Here is what we built and how it works.

Instead of treating Traccar as a standalone application to be installed manually, XB Software built an automated deployment script that creates a complete production environment around it. The script runs on a Linux machine and handles everything needed to get Traccar running in a production-ready state.

Here is what the deployment includes:

Traccar backend and frontend;
PostgreSQL database;
Prometheus for metrics collection;
Alertmanager for sending notifications when things go wrong;
Nginx as a reverse proxy for monitoring interfaces;
Node exporter for Linux machine metrics;
PostgreSQL exporter for database metrics;
Firewall configuration using UFW/iptables to expose only necessary ports.

The script checks environment variables, installs Docker and Docker Compose, and generates configuration files from templates. It then sets up the firewall, clones the frontend repository (either standard Traccar frontend or a custom one), builds the stack, and starts everything. The logic strictly separates application, production environment monitoring, and data storage layers. When we update the Traccar interface or Alertmanager settings, the script performs smart, targeted restarts and rebuilds only the affected containers while leaving others untouched. This protects data from corruption, preserves active GPS tracker sessions, and maintains database uptime.

The deployment process with the script is fully idempotent. It never damages the system or creates duplicate resources. All original configuration templates remain on the server.

If a client needs to change monitoring passwords, update email alert settings, or modify any other configuration, there is no need for developers to edit complex files manually. They can update the corresponding variable in the centralized .env file and rerun the script. The infrastructure applies the changes without risking production stability.

For production-level reliability, we separated deployment logic from service management. Systemd handles stack availability, ensuring all containers restart automatically after a server reboot, while build and update tasks remain isolated within the deployment script.

System Architecture: Simple, Flexible, and Production-Capable

All the components listed above work together in a simple but production-capable architecture. Everything runs in Docker containers on a single server. For small to medium deployments this is more than sufficient. If tracking volume grows, the client can simply move the deployment to a more powerful server.

The Traccar backend and frontend are packaged in a single container. This is how Traccar's official Docker image works, and it keeps deployment straightforward. The frontend is written in React, and the backend runs on a custom Java server. The mobile applications (Traccar Client for drivers and Traccar Manager for fleet supervisors) are available from the official app stores.

For the database, we went with PostgreSQL. Traccar supports multiple databases, but PostgreSQL handles geospatial queries better than MySQL and manages high loads more effectively. For most fleets, the difference between databases is not dramatic, but PostgreSQL gives more headroom if tracking volume grows.

Prometheus and Alertmanager run in their own container. Prometheus scrapes data from two exporters. The Node exporter exposes such Linux machine metrics as CPU, memory, disk, network (these are the basic ones that can be extended if needed). The PostgreSQL exporter exposes database metrics. Together, they provide visibility into the health of the entire stack.

Nginx sits in front of everything. It routes traffic to Traccar, Prometheus, and Alertmanager. We added basic authentication to the monitoring interfaces so that metrics are not publicly accessible.

Why Traccar Integration Matters for Fleet Operations

Companies don’t buy tracking software and logistics solutions because they enjoy looking at maps. They buy them because they need to know where their vehicles are, how long drivers spend on routes, whether deliveries are on time, and whether assets are being used efficiently.

When the tracking system goes down, operations suffer. Dispatchers cannot assign jobs. Customers cannot get delivery updates. Managers cannot verify that routes were followed. The business loses visibility, and visibility is the whole point.

Our deployment approach addresses this directly.

Monitoring and Alerting: Knowing Before It Breaks

We built a deployment that not only works today but also tells you when it might stop working tomorrow. More vehicles get added. More trips get tracked. The database grows. Memory usage creeps up. CPU spikes during peak hours. Without monitoring, the first sign of trouble is often the system going down.

Our observability stack provides early warning. The alerts we configured track:

Host health: CPU, memory, disk usage, network activity;
Database health: PostgreSQL connections, query performance, storage;
Service availability: whether Traccar and its components are responding.

When metrics cross predefined thresholds, Alertmanager sends notifications. The client can see exactly what is happening on dashboards and plan upgrades before the system fails.

Security: Basic But Solid

Security follows the same principle as the rest of the stack: simple and effective. The script configures iptables (a traditional utility used to configure and manage IP packet filtering and NAT rules in the Linux kernel) to block all ports except those needed for the application to function.

This is essentially the same approach major cloud providers use with their security groups, just without the fancy UI. All internal traffic (database queries, exporter metrics collection) is fully isolated within Docker's private virtual networks and physically hidden from the outside world.

Customization: Frontend and Backend Modifications

A production environment also has to reflect the client's needs. One of the reasons companies choose Traccar is that they can customize it. In our experience, clients typically prioritize UX improvements. So we built flexibility into the deployment script accordingly. The repositories it uses are defined in environment variables. The frontend repo variable can point to the standard Traccar frontend on GitHub, a private repository with custom styles, or even a local directory.

This means we can:

Apply branding: replace the Traccar logo with the client's logo, adjust color palettes;
Modify UI elements: add, remove, or rearrange features in the frontend;
Update the frontend: pull new versions from the repository without changing the deployment script.

Most clients do not ask for backend changes. But for those who do need backend modifications (custom business logic, integrations with existing systems, specialized tracking features) we can handle those as well.

Support: Ongoing Maintenance, Adjustments, and Troubleshooting

Deploying the system is only the first step. When something drifts out of spec (memory spikes, database connection bloat, or unusual latency) we can diagnose it, and adjust the server or Traccar configuration before it becomes a problem.

Future Enhancements for Traccar Production Deployments

The deployment script is a foundation. As we work with more clients, we will refine the monitoring rules, expand the customization options, and improve the deployment process.

One area we are exploring is using AI-assisted development to accelerate frontend customization. Instead of manually modifying React components for each client, we could generate custom interfaces based on client requirements.

This is still in the exploration phase, but the potential is significant.

We are also considering how to handle larger deployments. Docker Swarm or Kubernetes would provide better scaling for fleet management software that works with hundreds or thousands of vehicles. The current script is a starting point, and we can build more complex architectures as client needs grow.

Conclusions

Traccar is a powerful GPS tracking system. But power without reliability is useless, and reliability without visibility is dangerous. Our automated deployment creates a production environment that combines Traccar's capabilities with the monitoring, security, and customization that businesses actually need.

Senior Developers Don’t Write More Code. They Prevent Disasters

Sergey Laptick — Thu, 09 Jul 2026 13:10:47 +0000

Imagine this. A team celebrates a massive release with dozens of features, thousands of commits, and months of late nights. Then, six months later, that same system is a maintenance nightmare. Bugs crop up everywhere. Onboarding new engineers takes weeks. Every small change breaks something else.

The problem is that many companies still measure senior engineers the same way they measure juniors: by output (more features delivered, more code in the repository, etc.). But that misses the real layer of seniority and results in production systems that fail from decisions that were never challenged.

Hidden Work of Senior Engineers

A senior developer’s day rarely looks like a productivity dashboard. They’re not constantly pushing commits or closing tickets. Instead, they’re doing things that look like they’re doing nothing:

Sitting in architecture discussions;
Mapping out failure modes;
Pushing back on vague requirements;
Asking “why” five times before anyone writes a spec.

None of this “ships.” But everything becomes safer because of it.

What actually separates a mid-level from a senior engineer in 2026 isn’t syntax mastery or knowing the latest build tool. It’s strategic thinking, technical influence, and the ability to balance trade-offs between speed, scalability, and maintainability. You can teach someone React in a week. Teaching them to see around corners takes years.

This invisible work is why senior engineers prevent problems that junior engineers cannot yet see:

Scaling bottlenecks before they exist;
Data consistency issues that only appear under load;
Dependency risks in third-party integrations;
Over-engineering disguised as “future-proofing”;
Under-engineering disguised as “MVP speed”.

A production outage is rarely a surprise. It’s a delayed decision that a senior engineer might have spotted six months earlier if anyone had listened.

The Paradox of Deleting Code or Why More Code Isn't Always Better

Here’s a counterintuitive truth that keeps coming up in engineering circles: your value as a developer becomes inversely proportional to the amount of code you write. Here’s how this so-called Deletion Paradox works. Junior engineers are busy adding features while senior engineers are quietly removing thousands of lines that never should have existed.

Why does that matter? Because every line of code is a future maintenance liability. It adds to the cognitive load, increases the surface area for bugs, and creates hidden coupling that will snap under pressure.

In mature systems, more code rarely means more value. It often means more operational complexity, more maintenance overhead, and more places where things can go wrong. The best way to improve a system is often to remove code, because you understand what happens after it ships.

Outcome Vs. Output: The Real Measuring Stick

This is where the shift from output to outcomes becomes concrete:

Output is what your team produces: a shipped feature, a release, a line of code. It’s easy to count, easy to celebrate, and easy to fake as progress;
Outcome is the measurable change in user behavior that creates business value: higher activation, better retention, more expansion revenue.

There’s this concept of “Feature Factories”, describing a situation where teams that celebrate activity and features delivered instead of business value and changed system behavior. The issue is that building a feature without an expected outcome is just a guess shipped at full cost.

Senior developers look at the backlog through the lens of hypotheses rather than blind commitments. They ask: “What are we trying to learn?” “What will change if this works?” “What’s the metric that moves if we get this right?”

This is why their most valuable contribution is often saying “don’t build this yet.” Not because they’re lazy or risk-averse, but because they can see that a feature will create long-term maintenance cost, or that an abstraction will break under real usage patterns, or that a shortcut will become technical debt faster than anyone expects.

At XB Software, we’ve seen this play out repeatedly in legacy app modernization projects. Sometimes the most valuable thing you can do is to know which parts of an old system to preserve and which to retire.

A UI redesign that strips familiar workflows can break user habits that took years to build. A senior engineer’s job is to see that coming and push back before anyone writes a line of code. That’s why we emphasize careful planning. The absence of a bad redesign often delivers more value than a dozen new screens.

What Changes When AI Generates Half Your Code

The Sonar 2026 State of Code Developer Survey found that 42% of production code is now AI-generated or assisted. 72% of developers who’ve tried AI coding tools now use them every day. That changes what “coding” means.

Anyone can generate massive amounts of code now. However, the hard part is knowing what not to generate, what to delete, and what to constrain.

As the VibeCoder Engineering Blog points out, a senior developer’s value in the AI era is building the guardrails. They write .cursorrules or CLAUDE.md files that encode project conventions, architecture patterns, and forbidden anti-patterns. The AI then generates code that follows their standards instead of generic best practices.

This is a form of systemic decision-making that happens before AI-generated code exists. AI cannot reliably make engineering trade-offs under business constraints. It doesn’t know when to choose simplicity over software scalability, or when to exchange speed for correctness. Those are still human judgments, and they’re exactly what senior engineers get paid for.

So while AI makes code cheaper, it makes engineering decision-making more valuable. The bottleneck here is editing, deleting, and deciding what not to build. The senior engineer’s job has shifted from writing code to writing rules that constrain the machines.

So What Does Seniority Actually Look Like?

Junior engineers optimize for “correct implementation.” They want to build the thing right. Senior engineers optimize for “sustainable system behavior.” They want to build the right thing, and sometimes that means not building anything at all.

The most important contribution a senior engineer makes is often invisible: shaping the system before it becomes expensive to fix. That’s the work that doesn’t show up in commit logs. But it’s the work that separates a stable product from a sinking one.

Senior developers don’t “do more.” They make sure the system doesn’t accumulate problems faster than the team can understand them. And in production, that is what defines success.

AI Can Write Code, But It Can't Own Production Systems

Sergey Laptick — Thu, 02 Jul 2026 13:36:27 +0000

AI is getting scary good at writing code. Drop a prompt into Lovable or Claude Code, and minutes later you have a working prototype with a modern tech stack. It feels like we solved all big problems in AI-assisted software development.

Then production happens. And production does not care how clean your code looks. It cares whether it survives reality. Can it handle traffic spikes or recover when a dependency fails at 2 AM? Can a person who inherits it six months from now actually understand what it does?

The uncomfortable truth is AI can generate code at scale, but it cannot own what happens after you deploy it.

The Illusion: "Perfect" Code from AI

What AI actually produces is impressive. Clean structure. Readable functions. Fast prototypes. Even decent architecture suggestions if you prompt it right. From the outside, it looks like senior-level output.

But here is the thing. AI does not understand business risk. It does not know what happens when your database connection pool runs out under load. It cannot tell you which of two equally valid solutions will survive a 10x traffic spike. It generates what looks right based on what it has seen before, not what is right for your specific system with its specific constraints.

Andy Anderson, a researcher at IBM, spent four months building a Kubernetes dashboard from scratch using Claude Code. No team. Just him and the AI. The first two weeks were exhilarating. Code poured out at a pace he had never experienced. Features that would normally take days appeared in hours. It felt like having a tireless junior developer who typed at the speed of thought.

Then the limitations hit. All at once: broken builds, wrong architectural patterns, scope creep, the AI trying to modify files Andy did not ask it to touch, etc. The problems were cascading: fix one thing, three others break. The researcher ended up spending more time reviewing and reverting than he would have spent writing the code himself.

We saw the same pattern with a scheduling app we built recently. Lovable generated a working prototype in hours with React frontend, Node.js backend, and PostgreSQL database. The calendar looked fine in the demo. But when we tested it, the AI-generated scheduler could not handle recurring events or drag-and-drop reliably. It was a concept car made of clay. So we swapped that component for DHTMLX Scheduler, a production-grade library, and rebuilt the backend for real-world scale. The prototype was useful, but the production system needed actual engineering.

This pattern shows up everywhere. The tools are great at generating code but terrible at understanding what the code is for.

Where Things Break: Production Reality

In production, systems behave differently. Traffic is unpredictable. Dependencies fail. Data grows. Users do things you never expected. Integrations become fragile. And suddenly, that “perfect” AI-generated code becomes just one layer in a very messy environment. The problem is not the code but everything around it

A large-scale study analyzing 302,600 verified AI-authored commits across 6,299 GitHub repositories found something sobering. AI-generated code introduces real issues, such as code smells, correctness problems, or security vulnerabilities, and 22.7% of those issues survive long-term, silently accumulating as technical debt in production codebases. More than 15% of commits from every AI coding assistant introduce at least one issue. Code that looks fine in a PR can quietly rot in production for years.

Stéphane Maes, who writes about software engineering, calls this the “Great Toil Shift.” The time you save generating code gets entirely consumed by the downstream work. This includes architectural review, security auditing, code understanding, documentation, and ongoing maintenance. The efficiency gains are not what they seem.

This is a common thing for teams that go all in on AI-generated code. The prototypes ship fast. Everyone is excited. Then the first production incident hits. The AI-generated code does something unexpected under load. Nobody fully understands the system because nobody wrote it. The team spends days debugging something that would have taken hours if they had built it themselves.

The Missing Piece: Engineering Judgment

Here is where the conversation needs to shift. What AI does is options generation. It gives you suggestions, often good ones. But someone still has to decide which suggestion to take. Someone still has to understand the trade-offs.

And they are the hard part:

Architecture trade-offs: Do we use a message queue or direct calls?
Failure handling: What happens when this service goes down?
Performance constraints: How many requests per second can this handle?
Security decisions: Is this endpoint properly authenticated?
Long-term maintainability: Will someone understand this in two years?

This is where experience matters more than output. Two solutions can both "work" in AI terms. But only one survives production load.

Andy Anderson's central finding from his experience report is worth quoting directly: "The intelligence of an AI-driven development system resides not in the AI model itself, but in the infrastructure of instructions, tests, metrics, and feedback loops that surround it."

In other words, AI is just a tool. The real system is everything you build around it. The tests. The review process. The deployment pipeline. The monitoring. The on-call rotation. The institutional knowledge about why things are the way they are.

In production environments, organizations fundamentally need what they call a "throat to choke," a human engineer who holds ultimate accountability, who can assess blame, who possesses the systemic intuition to troubleshoot complex, real-world failures. A machine cannot sign off on a Service Level Agreement (SLA). It cannot face a compliance audit. It cannot absorb liability for downtime.

This is not a philosophical point. Researchers who analyzed the terms of service for nine AI coding assistants found a consistent pattern: while users own the generated output, full liability, responsibility for correctness, and downstream production risk rests on the human user. The providers explicitly disclaim warranties and allocate responsibility to the developer.

If your AI-generated code causes a production outage, the AI company is not getting the pager duty alert. You are.

What This Means for How We Build Software

None of this means the tools are useless. They are incredibly useful. They can generate boilerplate, write tests, refactor code, and help you explore solutions faster than ever before. But the tools do not replace judgment. They do not eliminate the need for testing. They do not make architecture decisions for you. And they definitely do not own the systems they help you build.

The real skill in modern AI-assisted development is learning how to integrate generated code into a system that you fully understand, can debug, and evolve over time. It is knowing when to accept a suggestion and when to rewrite it.

Some teams are figuring this out. They use AI for rapid prototyping, like the Lovable-generated scheduler that we built, which validated the concept before replacing the AI-generated components with a production-ready solution. The prototype was useful for testing and validation. But the production system required real engineering: understanding the domain, handling edge cases, and ensuring the component could actually survive real-world use.

AI changes how fast we build software. But it has not changed what it means to be responsible for it. And in production, responsibility is still the hardest part.

How to Build a Local LLM Agent to Automate Work List Generation from Monthly Reports (With Jira Integration)

Sergey Laptick — Thu, 21 May 2026 11:39:00 +0000

Our management team spent hours manually extracting work items (“bug fix”, “released version 1”, etc.) from dozens of developer reports. The task was repetitive, error‑prone, and a security risk when using cloud‑based AI tools, since it means exposing internal activity to external servers.

To solve this, we built a local LLM‑powered agent that runs entirely on our own servers, normalizes chaotic report data, filters out useless noise, enriches descriptions from Jira, and generates a clean list of actual accomplishments. In this article, we break down the architecture and explain why a CPU‑only, on‑premise approach is practical for enterprise clients who prioritize data privacy.

The Problem: Manual Work List Generation Is Slow, Inconsistent, and Insecure

Usually, our managers followed the same routine: collect a month’s worth of developer reports, manually scan through hundreds of entries, and pick out the items that actually represented completed work. This process was straightforward but flawed.

The first issue was data quality. Developers write reports in wildly different formats. Some include detailed Jira ticket IDs and descriptions, others are cryptic one‑liners like “fixed issue”. When a manager who wasn’t deeply involved in the project later reviews these reports, the meaning is often lost. What does “adjusted header” refer to? Which feature did “refactored code” touch? What we really needed was an AI-powered task management approach that could process this unstructured data automatically.

The second issue was duplicate work. Managers would occasionally include tasks that had already been declared in previous months, creating overlaps. Another example is a task that spans several days. In this case, the same activity could be logged repeatedly, producing many near-identical entries. There was no automated way to compare new reports against historical data.

The third issue was security. Initially, we experimented with feeding entire monthly reports into ChatGPT, asking it to clean up the data and suggest a final list. It worked reasonably well, but we were handing over a full month of internal project activity to a cloud service. For many enterprise businesses, especially those in finance or healthcare, that level of exposure is unacceptable.

The Solution: A Secure, On‑Premise AI Agent for Task Extraction from Reports

Our approach was to implement a console‑based application that converts reports into tasks automatically. It runs on our internal server, triggered by a cron job (or an optional API call) at the end of each monthly reporting cycle. The AI agent processes raw reports for each active project, applies a series of transformations, and outputs a polished list of work items.

The entire pipeline runs on a CPU‑only server using Ollama to serve a local instance of the Gemma 4 E2B model. For embedding generation (used in duplicate detection), we use the tiny nomic‑embed‑text model, which is only a few megabytes in size. Here’s a high‑level view of the process flow:

Let’s walk through each stage in detail.

1. Normalization: Making Chaos Readable

A single project might receive 80+ individual reports per month with varying levels of detail. The first task for our AI agent was to normalize these disparate inputs into a consistent, machine‑readable format. This step alone turns a jumble of free‑form text into structured data that the rest of the pipeline can reliably process.

2. Chunking: Working Within Token Limits

This is where we hit our first major technical constraint. Running on CPU via Ollama, our Gemma 4 model is limited to a context window of 4,096 tokens. That’s not a lot. A single month of reports from a busy project can easily exceed that.

We solved this by chunking. The AI system calculates the approximate token count of the combined report text and splits it into batches of about 20 reports each. This ensures that the LLM never runs out of context space and that each chunk receives full attention.

Within each chunk, we also further split entries that contain multiple tasks in a single line (e.g., “Did A, did B, did C”). After this splitting, 22 raw reports became 94 individual work items in one of our test runs.

3. Jira Enrichment: Adding Missing Context

One of the most valuable features of our AI agent is its ability to automatically fetch additional context from Jira. When the system detects a Jira ticket ID in a report, it calls the Jira API to retrieve the ticket description.

Developers often write terse reports assuming the ticket ID is enough. But when that report later appears as “AAA‑123 – done”, it tells nothing. By pulling the full, manager‑written description from Jira, our AI agent replaces the vague entry with a clear, professional summary of what was actually accomplished.

4. Filtering Out the Noise

Not every report entry is worth including. Generic statements like “working on…” or “following up” don’t convey meaningful work. We built a bad‑word filter, one of key components of our intelligent document processing (IDP) pipeline. It flags entries containing these vague phrases.

The LLM processes each chunk and identifies data that match our exclusion list. In our test, this filter removed 69.1% of entries and only 29 items out of 94 survived the cut. What remained were concrete, specific descriptions of completed tasks.

5. Selecting the Best Candidates

Once we have a clean set of candidates, we need to choose the top N entries to present. The number N varies by project and is stored in our internal reporting database. To account for further filtering in the next step, we typically select a larger pool, say, 80 items.

6. Vector Duplicate Detection: Ensuring We Never Repeat Ourselves

This is the secret sauce that prevents duplicate entries. Before finalizing the list, the AI agent compares each candidate against a historical database of all work items we’ve ever submitted for that project. Here’s how it works:

Embedding Generation. Each work item is converted into a vector (a list of numbers) using the nomic‑embed‑text model. This vector captures the semantic meaning of the text;
Similarity Calculation. The system compares the new candidate’s vector against the vectors of all previously stored data for that project;
Threshold Decision. If the similarity score exceeds 0.85 (85%), the candidate is flagged as a duplicate and removed. This threshold catches not just exact matches but also near‑duplicates where the phrasing or word order has changed while the underlying idea remains the same.

The historical data is stored in a lightweight PostgreSQL table with just a few fields: project_id, text (the final description), embedding (the vector), and created_at (date of creation).

After duplicate removal, we’re left with a set of truly unique, high‑quality work items. These are then formatted for final delivery to the project manager.

Real‑World Performance: What Test Run Tells Us

Let’s walk through an actual test run to see the numbers in action. These test run results demonstrate how an AI report analysis tool can summarize reports into tasks even with noisy, inconsistent input.

Technical Deep Dive: Why CPU‑Only Deployment Works

One of the most common objections to running local LLMs is the perceived need for expensive GPU hardware. We deliberately chose a CPU‑only deployment to keep costs manageable and to prove that on‑premise AI doesn’t require significant infrastructure investments.

Model Selection: Gemma 4 E2B

We evaluated several local models and settled on Gemma 4 E2B. Here’s why:

Size. At 5 billion parameters, it fits comfortably in RAM without needing a GPU. Our server has extra memory allocated specifically for the model;
Performance. It’s fast enough for batch processing;
Quality. The model handles JSON output reliably, and follows detailed prompts with minimal hallucination.

NOTE: If you work with a multilingual team, make sure that the model you use understands target languages natively.

Proper Model Settings and Prompt Engineering for Consistency

Each pipeline stage has its own carefully crafted prompt that includes:

A clear role definition (e.g., “You are a specialized Data Parsing Engine”);
Good examples and bad examples of expected output;
Explicit formatting rules (JSON structure, field names);
Instructions to avoid creativity (temperature set to 0).

For the bad‑word filter, we provide a list of prohibited terms and their synonyms: “working on,” “following up,” “in progress,” “discussed,” etc. The LLM simply acts as a pattern matcher with semantic understanding. It can recognize that “still working on the header” is conceptually similar to “in progress” and flag it accordingly.

Also, for data‑processing tasks like this, we always disable “thinking” or “chain‑of‑thought” modes. Those are useful for complex reasoning but introduce unnecessary variability and output length in structured extraction tasks.

Extra Challenges We Overcame

Challenge 1: LLM Unpredictability. Even with temperature set to 0, LLMs can occasionally produce unexpected output. We added timeout limits to prevent the model from getting stuck in a loop, and we structured our prompts to request strictly formatted JSON that is easy to validate programmatically.All LLMs glitch. We can never fully trust them. They help, but everything still needs to be reviewed and rephrased. They assist, but they don’t solve the task 100%.

Challenge 2: CPU Processing Speed. Processing 94 items across multiple LLM calls takes time. We solved this by running the AI agent as an overnight cron job, so speed is never a bottleneck. The manager arrives in the morning to a ready‑to‑review list.

Why This Approach Matters for Enterprise Clients

1. Complete Data Sovereignty

When you use on-premise Artificial Intelligence solutions, no data ever leaves your infrastructure. The LLM runs locally, the embedding model runs locally, and the historical database resides on your own PostgreSQL server.

2. No Vendor Lock‑In

Cloud AI services change their pricing, deprecate models, or alter their APIs without notice. By using local AI agents and Ollama, you retain full control over the entire stack. Need to switch to a different model tomorrow? Just pull a new one and update the configuration.

3. Predictable Costs

The only ongoing cost is the electricity to run the server. There are no per‑token API fees, no monthly subscriptions, and no surprise bills after a particularly busy month of processing. For organizations that process thousands of reports annually, the savings are substantial.

4. Customizable to Your Workflow

Because we own the code, we can adapt the pipeline to fit your specific reporting format, integrate with your existing project management tools, and fine‑tune the prompts to match your industry’s terminology. This enables using AI for business process automation across diverse sectors, from construction to healthcare.

From Manual Chore to Automated Precision

Before, turning chaotic developer notes into clean reports meant choosing between tedious manual work or exposing sensitive data to cloud AI. Our private AI agent for document analysis offers a third way. Namely, secure, on‑premise automation.

By combining Gemma 4 on standard CPU hardware with vector‑based duplicate detection and direct Jira enrichment, we’ve turned hours of monthly review into a hands‑off process. The system normalizes vague entries, filters out noise, and guarantees you never repeat a task description.

How Spec-Driven Development Brings Structure to AI-Assisted Engineering and How We Put It to the Test

Sergey Laptick — Mon, 02 Mar 2026 13:13:18 +0000

AI coding assistants have made developers incredibly fast since the start of the AI boom, but this new speed often comes at a hidden cost. The IT industry is realizing that generating code is the easy part. The real challenge is building systems that are coherent, maintainable, and actually do what they were supposed to do.

This is where Spec-Driven Development (SDD) comes in. This methodology shifts the focus from vibe coding to following the general intent, using specifications as the new source of truth for AI-assisted engineering.

Why Vibe Coding Isn’t Built to Last

The term “vibe coding” perfectly captures the current experimental phase of AI-assisted development. You describe what you want, get a block of code back, and if it looks right and seems to work, you move on. This approach is undeniably powerful for prototypes and small scripts, allowing for unprecedented velocity. However, when applied to serious, mission-critical applications, the cracks begin to show:

The code might compile and even function, but the underlying architecture becomes an afterthought;
New features create unexpected conflicts;
Documentation is sparse or non-existent;
The codebase transforms into a collection of disjointed components that are hard to maintain, debug, and evolve.

The problem isn’t the AI’s coding ability. It’s rather the workflow where developers treat AI like a search engine when they should be treating it like a literal-minded, but exceptionally talented, pair programmer who needs unambiguous instructions.

What Is Spec-Driven Development

Spec-Driven Development is the practice of writing clear, structured, and testable specifications before a single line of code is generated. In the context of AI-assisted development, SDD provides the blueprint that guides AI agents to generate code that is consistent, architecturally sound, and perfectly aligned with business goals.

Unlike traditional waterfall requirements that gather dust, an SDD spec is a living, executable artifact. It becomes the shared source of truth for both humans and AI, driving development, software testing, validation, and even documentation. By moving architectural decisions, constraints, and clarity upstream, SDD directly addresses the shortcomings of vibe coding. It replaces guesswork with a clear contract for how your app should behave.

How Spec-Driven Development Works: A Step-by-Step Guide

The SDD process is structured into distinct, sequential phases. Each phase produces a key artifact that feeds into the next, ensuring a clear, traceable path from a high-level idea to production-ready code.

Specify (The “What” and “Why”). You start with a high-level description of what you’re building and why. The AI then generates a detailed functional specification. This phase is purely about business intent: user journeys, success criteria, and edge cases. It explicitly excludes technical details, forcing clarity on the problem before jumping to a solution;
Plan (The “How”). With the functional spec locked in, you provide the AI with your desired stack, architecture, and constraints. The AI generates a comprehensive technical plan, including technology choices, system design, integration patterns, and security considerations. This ensures the new code feels native to your project and aligns with your technical strategy;
Tasks (The Breakdown). The AI takes the spec and plan and breaks them down into small, reviewable, and actionable tasks. Each task is specific enough to be implemented and tested in isolation, like “create a user registration endpoint that validates email format.” This decomposition prevents the “big bang” coding approach that overwhelms both AI and reviewers;
Implement (The Execution). Finally, the AI tackles the tasks one by one. Instead of reviewing thousand-line code dumps, developers review focused changes that solve specific problems, verifying that the implementation matches the specification.

A Quick Look at the Tools Enabling SDD

The SDD ecosystem is maturing rapidly. While GitHub’s Spec Kit is a powerful open-source example, other platforms offer different interpretations of the model:

Spec Kit

An open-source CLI and template-based toolkit that integrates with your existing AI assistants like Copilot, Claude Code, and Gemini CLI. It introduces the concept of a constitution.md, a file that encodes your project’s immutable principles, such as stack versions, naming conventions, and architectural patterns.

Kiro

An agentic AI with an IDE and CLI that add structure to an existing editor. Kiro has SDD built directly into its core. When starting a new feature, its agents automatically generate requirements, design documents, and create task lists, guiding the developer through an opinionated workflow. It’s designed for developers who want a deeply integrated, automated, and context-aware environment for moving from concept to code.

BMAD Method

An open-source framework that simulates an entire agile team using specialized AI agents. With over 12 distinct agent personas, including a Product Manager, Architect, Scrum Master, etc., it manages the entire project lifecycle.

Putting Spec Kit to the Test

Theory is one thing, but practice is where the real lessons are learned. Our team recently dove into GitHub’s Spec Kit to understand its practical applications and limitations. Here’s what we found trying to recreate the app shown below. It’s a small fleet management dashboard with a scheduler, a map, and vehicle tables:

How Spec Kit Works (main commands)

/constitution:
- When to use: At the very beginning of a project.
- Purpose: Establishes your project’s foundational rules, defining the tech stack, architectural patterns, and coding conventions that AI-generated code must follow.
/specify:
- When to use: After the constitution is set.
- Purpose: Takes your high-level, plain-language description of a feature (the “what” and “why”) and expands it into a detailed functional specification.
/plan:
- When to use: After the specification is reviewed and approved.
- Purpose: Generates a technical implementation plan based on the spec and the project’s constitution, defines frameworks, libraries, etc.
/tasks:
- When to use: After the plan is finalized.
- Purpose: Breaks down the specification and plan into a list of small, concrete, and actionable tasks. Each task is designed to be implemented and tested independently.
/implement:
- When to use: After the tasks are defined.
- Purpose: Instructs the AI agent to start writing the actual code, working through the generated task list one by one.

Our web development team started by testing the limits of the process. The first attempt was to generate a complete demo application, in one go. The high-level description was fed into the /specify command, and the results quickly revealed the AI’s contextual limitations. The application generated by Spec Kit was a mess: the scheduler rendered strangely, filters didn’t work, and the statistics pulled random data not connected to the users or vehicles:

This shows that feeding too much information at once simply overwhelms the model. It gets confused and starts losing track of early requirements halfway through implementation. You shouldn’t feed a neural network a very large amount of data at once because it will get confused

The failure illustrated a core principle of SDD: decomposition is not optional. The AI’s context window, while large, has finite capacity. Handling a complex, multi-page application in one go leads to forgotten requirements and inconsistent results.

A better strategy is to build the project feature by feature. On the second attempt, our team started with the core layout (the header and collapsible menu) with a highly detailed spec that included exact styles and components:

Next came the “vehicles” page, specified down to the placement of inputs, instructing the AI to match the layout of a provided demo. While it wasn’t perfect, and the AI still decided to paint a button a different color, it was a manageable, high-quality chunk of work that could be easily corrected with a follow-up prompt or a quick manual tweak.

The final step was a tiny, well-scoped feature: adding sorting to two columns in an existing table. The simple command create client-sorting for type and year columns on Vehicle Stable proved perfectly suited for the SDD workflow, demonstrating that the methodology’s value extends to changes of any size, ensuring even small updates are implemented correctly and consistently:

Our experiments with Spec Kit revealed that successful AI-assisted development hinges on proper decomposition. Attempting to generate an entire application in one pass overwhelmed the model and produced unusable results, while breaking the project into small, well-scoped features consistently delivered high-quality, reviewable code. The key insight: specifications must match the size of the task, ensuring AI can maintain focus and consistency throughout implementation.

When SDD Makes the Most Sense

It is not a silver bullet for every coding task, but it provides immense value in specific scenarios:

Enterprise & Production Systems. For long-lived applications where maintainability, stability, and compliance are critical;
Complex Architectures. For projects with multiple services, APIs, and integration points where a lack of clarity can lead to catastrophic failure;
Team Development. When multiple developers (and AI agents) need to collaborate on a shared codebase, a central source of truth is invaluable;
Legacy Modernization. When rebuilding an old system, you can use SDD to capture the essential business logic in a modern spec before letting AI regenerate a clean, new implementation.

Conclusion: From Code-Centric to Spec-Centric Thinking

We are moving from an era where “code is the source of truth” to one where “intent is the source of truth.” AI is making specifications executable, turning our documented intent directly into working software. Spec-Driven Development allows small teams to build robust systems and large organizations to move with coherence and speed.

How AI Search Solves the Problem of Working with Unstructured Data

Sergey Laptick — Mon, 27 Oct 2025 11:57:03 +0000

Are you struggling with unstructured data, like support tickets, employee feedback, and documents? Many businesses face this challenge, leading to wasted time and missed insights. Unstructured datasets contain up to 90% of all enterprise-generated data, yet most systems are optimized for structured, field-based records. AI-powered search can interpret intent and context, find conceptually similar content, and improve results over time based on user behavior. Today, we’ll explore how AI search can transform the way you interact with data.

What’s Unstructured Data and Why It’s Hard to Work With It

Unstructured data refers to any information that doesn’t have a predefined format and does not conform to fixed schemas of databases. Common examples in enterprise environments include:

Text-heavy documents. Emails, meeting notes, support tickets, contracts, and project briefs;
Files and reports. PDFs, Word documents, PowerPoint presentations, and scanned forms;
Multimedia content. Customer call recordings, interview videos, and voice memos;
Collaboration tools. Slack/Teams messages, comment threads, and internal wikis.

Unlike structured data, such as a CRM system’s "Customer Name" field, unstructured content doesn’t have fixed labels. As a result, traditional database queries can’t easily retrieve it, which leads to multiple difficulties:

Keyword Search Falls Short. Users must guess the exact phrasing (e.g., searching for "order delay" won’t surface an email saying "shipment stuck in customs");
No Consistent Formatting. A contract clause might appear in a PDF, an email attachment, or a scanned handwritten note, each requiring different extraction methods;
Lack of Metadata and Context. Without AI, a customer complaint buried in a support ticket won’t be tagged unless manually labeled;
Scalability Issues. Manual categorization and tagging become slow at large scale. Imagine HR reviewing hundreds of resumes, or legal teams parsing years of contracts.

When employees can’t quickly find the information they need, businesses suffer:

Slower decision-making. Sales reps waste time digging for client details instead of closing deals;
Increased errors. Misinterpreted notes or overlooked contract terms lead to compliance risks;
Frustration and inefficiency. 30% of an employee’s time is spent searching for information;
Missed opportunities. Hidden insights in customer feedback or employee surveys go unused.

AI to the Rescue: Solving Issues with Unstructured Data

When traditional search methods become increasingly ineffective, AI-powered search changes the game. It can understand context, intent, and relationships within data. Let’s explore how it works and why it outperforms conventional approaches.

How AI Search Processes Unstructured Data

AI-driven search relies on several advanced technologies to make sense of unstructured content:

Natural Language Processing helps understand user intent. For example, a user's query "Find customer complaints about late deliveries" retrieves emails, support tickets, and notes even without exact keywords;
Vector Embeddings and Semantic Search. An AI-based system converts text from unstructured datasets into numerical vectors, which allows capturing meaning rather than just rely on word matching. AI search enables similarity-based retrieval to find documents with related ideas, even if phrasing differs ("order delay" vs. "shipment backlog");
Machine Learning for Continuous Improvement. AI software can learn from user interactions to prioritize relevant results over time. It automatically tags and categorizes documents, for example, to group contracts by clause type;
Multimodal Processing. AI software enables extracting text from images, PDFs, and scanned docs (OCR) and transcribe and index audio/video files for searchability.

Transforming Business Outcomes with Server-Side AI Search

Integrating a semantic search engine on the server side allows extracting value from unstructured data. By moving beyond literal keyword matching to a deeper conceptual understanding, businesses can convert unstructured text into a dynamic, intelligent asset that drives both growth and operational efficiency.

The real estate sector offers a compelling example of this transformation. Let’s see how a modern property platform uses server-side AI semantic search to make property discovery more intuitive and aligned with user intent.

From Query to Match: How Our AI Search Validates Results

Vector-Based Matching

Converts user queries and property descriptions into mathematical vectors;
Finds conceptual matches even without exact keyword overlap;
Example: Searching "education" returns properties mentioning "family seeking home near top-rated schools" and "Investor purchasing a property for student housing".

Intelligent Threshold Filtering

Applies similarity threshold to ensure relevant results;
Automatically excludes matches below confidence level;
Example: Query "quiet neighborhood" triggers matches for:"Retired teacher moving to a quiet suburb" (84% match)"Empty nester looking for a quiet neighborhood with walking trails" (77% match)"Single parent looking for a safe neighborhood and good schools" (76% match)

Contextual Understanding

Interprets related concepts through embedding relationships;
Recognizes "kids" → family amenities, playgrounds, safety features;
Identifies "investment potential" → rental yields, appreciation areas.

How Semantic Search Powers Property Discovery

This AI-powered property search directly addresses critical pain points for real estate businesses:

Capturing Buyer Intent Beyond Keywords. Many homebuyers use vague phrases like "good schools" or "quiet area" that traditional searches miss. The AI solution matches "family-friendly neighborhood" to listings mentioning "top-rated elementary school" or "playground access" even when exact terms differ. Business Impact: 30% more relevant leads by understanding implicit requirements;
Reducing Missed Opportunities. Valuable properties often buried due to keyword mismatches (e.g., "serene" vs "quiet"). AI solution surfaces conceptually aligned listings, which increases agent productivity by eliminating manual matchmaking;
Converting Ambiguous Queries. Another problem with traditional systems is when investors search for things like "high-growth areas" but listings describe "appreciation hotspots" or "development zones". Vector relationships can help by connecting investment terms across descriptions, leading to faster deal flow through precision matching.

AI Search Query Examples Across Industries

Customer Support

AI search uncovers recurring support issues by recognizing patterns across various channels, regardless of how customers describe their problems. This enables faster troubleshooting and better product insights.

Query: "Find tickets mentioning API integration issues"

What AI Search Does:

Retrieves support tickets, forum posts, and chat logs;
Understands related terms like "API connection errors," "webhook failures," or "authentication problems";
Groups similar cases for trend analysis.

Traditional Search Limitation: Fails if the ticket says "system won’t connect" without the exact term "API."

Legal Department

Legal teams save time reviewing contracts and compliance docs by having AI identify relevant clauses and terms, even in PDFs or scanned documents. No need to manually tag or search exact legal phrasing.

Query: "Show contracts discussing 2024 delivery terms"

What AI Search Does:

Analyzes PDFs, scanned agreements, and email attachments;
Identifies clauses about "delivery timelines," "shipment deadlines," or "FOB terms";
Filters by year without manual metadata.

Traditional Search Limitation: Can’t parse handwritten edits or complex legal phrasing without exact matches.

Sales & Marketing

Sales and marketing teams can get a clearer view of customer sentiment around campaigns and launches by analyzing unstructured feedback in real time. AI search connects surveys, emails, and social mentions to reveal trends and pain points.

Query: "Find customer feedback about the new service launch in April"

What AI Search Does:

Aggregates survey responses, social media mentions, and support tickets;
Detects sentiment (e.g., "loved the update" vs. "had issues with the rollout");
Focuses on April-timeframe discussions, even if undated (using contextual clues).

Traditional Search Limitation: Requires pre-tagged categories and dates, missing informal feedback.

Enterprise-Wide Real-World Example

A travel-fintech firm, Super.com, integrated an AI-powered enterprise search (powered by Glean) across Slack, Confluence, GitLab, and Google Drive to manage its remote-first workflow. The solution:

Saved over 1,500 employee hours per month;
Reduced onboarding effort by 20%;
Supported AI-driven, personalized task lists and email drafts;
Ensured rigorous access controls to maintain security boundaries.

AI-Powered Search: Advantages and Limitations

Advantages of Adopting AI-Based Search for Unstructured Data

Employee Time Savings. AI search enables instant, context-aware retrieval that eliminates manual digging through emails, documents, and legacy systems. Automated tagging and categorization reduce administrative overhead, while unified search across CRM, HRM, and ERP systems cuts time wasted switching between apps;
Improved Customer Experience. Frustrated clients often repeat themselves when agents can't locate previous interactions. By linking all support tickets, calls, and emails to customer profiles, Artificial Intelligence creates a complete interaction history. Companies using AI see first-contact resolution rates increase significantly;
Reduced Risk of Data Loss. AI software penetrates deep into attachments, images, and audio files to surface buried content. The system preserves decades of knowledge through intelligent organization and retrieval. Organizations using AI maintain permanent access to historical IP, R&D notes from years past remain instantly discoverable.

Potential Challenges and Limitations of AI-Powered Search

Data Quality Requirements. Common issues include poor OCR (Optical Character Recognition) accuracy in scanned documents, inconsistent file formats (e.g., handwritten notes vs. digital PDFs), and duplicate or outdated versions of documents. Solutions include pre-implementation data cleansing, use of OCR enhancement tools, and metadata enrichment that helps supplement raw content with organizational context;
Security and Privacy Considerations. Accidental exposure of confidential information can occur through broad search results. Also, there’s a risk of over-permissioned access when searching across sensitive documents. Mitigation strategies may include implementing role-based access controls or using redaction capabilities for sensitive content;
User Training Requirements. Employees accustomed to keyword searches may struggle with formulating natural language queries or understanding semantic search capabilities. For example, a search for "drilling" could return unrelated results from oil industry reports and dental clinic records. You can plan a phased rollout and use interactive tutorials showing query examples (instead of: "invoice 2023" try: "Show me unpaid vendor invoices from Q1 2023");
Additional Limitations. Language support limitations for multilingual organizations; Computational costs for processing large document volumes; Integration complexity with legacy systems.

Unlocking Tomorrow’s Insights Today: The AI Search Revolution

As Artificial intelligence models grow more sophisticated, we’ll see systems that don’t just find documents but anticipate user needs. They’ll be able to surface relevant contracts before negotiations begin or highlight customer pain points before they escalate. Multimodal capabilities will bridge text, voice, and even visual data. For businesses ready to begin this transformation, success lies in phased adoption paired with strategic data preparation. Companies can focus first on cleaning and connecting core data sources, then expand as users adapt to conversational queries and confidence-based results.

Data Migration in Software Modernization. Balancing Automation and Developer’s Expertise

Sergey Laptick — Wed, 30 Jul 2025 12:40:37 +0000

When business owners think about modernizing a legacy application, they often focus on the most visible part: a sleek new user interface. However, the real challenge often lies beneath the surface. It’s data migration strategy. Moving data from an outdated system isn’t just a simple copy-paste job. It requires deep planning and expert execution. While automated data migration tools promise speed and cost savings, they are not a silver bullet. In this article, we’ll explore why automated tools alone aren’t enough, when developer expertise remains irreplaceable, and how a hybrid approach can save time and money.

How Databases Evolve During Legacy Software Modernization

When we talk about how data usually changes in the context of modernizing legacy software, we typically mean the following key processes.

Schema Migration includes incremental changes to the database structure itself, such as:

Adding, modifying, or dropping tables, columns, indexes, or constraints;
Adjusting data types or relationships between tables;
Essentially, this is about evolving the database design while keeping the same underlying system.

Data Migration as a process of moving data from one database system to another, such as:

Extracting data from an old database;
Transfering to a different DBMS (database management system), a new version of the same DBMS, migrating from on-premise to cloud database, etc.

A Real-World Challenge: From One Table to Multiple Tables

Legacy system modernization often means completely restructuring how data is stored. Let’s consider a real-world example of legacy system migration. Take a 20-year-old system, for example, where all user information was crammed into a single users table, which is pretty typical for old software. We need to deliver a new software version, and the new system usually provides much more detailed functionality. Therefore, that same data might be spread across multiple normalized tables, with relationships, constraints, and additional fields that didn’t exist before.

For example, In a legacy application, user data can be stored in a single table called users, which includes the following fields:

id
email
nickname
username
Several fixed columns representing the user’s interests (e.g., interest_1, interest_2, interest_3)

This structure is common in older applications that treated interests as fixed, hardcoded fields in the main user table. Such a table can be useful for e-commerce and marketplace platforms, for example to recommend products, collections, or promotions. A user who marks "home decor" and "kitchen gadgets" as interests may receive targeted recommendations and offers. This model is also common in eLearning platforms and EdTech apps to recommend relevant courses or learning materials based on interests. When user interested in "machine learning" and "data visualization" will see courses, webinars, or learning paths tailored to those topics.

Why Dividing User-related Data Into Multiple Tables

Dividing user-related data into multiple tables, such as separating interests, profile data, and relationships, introduces structural and functional advantages to a modern application. This approach follows the principles of relational database normalization, and it offers both technical benefits and business value. Here's what it enables:

Scalability and Flexibility. Instead of being limited to a fixed number of interest fields (e.g., interest_1, interest_2), a separate interests table allows for unlimited, dynamically added interests. Proper data migration strategy helps support future features, like tagging, user-generated topics, or hierarchical categorization, without redesigning the schema;
Efficient Queries and Personalization. With a proper schema, you can quickly query for all users interested in a specific topic or all interests of a given user. It enables fast personalization features like interest-based recommendations, user segmentation, and content filtering;
Localization and Internationalization. With a separate interests table, it's easy to localize interest names into different languages;
Better Maintainability. Adding a new feature (e.g., allowing users to follow interests or subscribe to updates) only requires changes in the interests or user_interests tables;
Rich Metadata and Behavior Modeling. You can add attributes to interests, like popularity, category, or trendiness, or track user behavior (e.g., last_engaged_at). This feature improves user experience through trend-based suggestions and behavioral analytics.

Modernized Schema

To reap the described benefits, In the new system, user interests can be treated as a flexible, scalable relationship:

A separate interests table holds all possible interest categories;
A junction table user_interests maps users to their selected interests.

Therefore, in a modernized app, we’ll have three tables instead of a single users table from the outdated app. Additionally, the new system no longer stores a single username field. Instead, it splits this into:

first_name
last_name
gender (to support features like salutation: Mr., Ms., in an AI chatbot, for instance)

However, the original system does not store gender. Therefore, it must be inferred based on the user's name if no other information is available.

The Cost of Fully Manual Data Migration

The process of data migration is often far more labor-intensive than expected. We've only described a few basic features, and even implementing this little set requires splitting a single legacy table into three normalized tables. In real-world scenarios, the number of such transformations is often significantly higher.

Additionally, consider the volume of data handled by applications that have been on the market for decades. Migrating such data structures is a major task. The amount of custom logic a developer must implement to ensure data integrity and correct representation can be substantial.

This naturally raises the question:

Is there a way to reduce the workload, cut modernization costs, and accelerate the release of the new software version?

The answer is yes, and this is exactly where automated data migration tools come into play.

How Hybrid Data Migration Help Developers Modernize Legacy Systems

How Automated Tools Help Migrate Data

While complex transformations, such as splitting one legacy table into multiple modern relational tables, require custom logic and developer-written scripts, automated data migration tools can still significantly accelerate the overall process when applied strategically. Here are some examples:

Schema Discovery and Mapping. Automated data migration tools, such as Talend, can inspect and map source schemas to target schemas, highlighting structural differences. This is especially useful during initial planning, since it helps developers understand which parts of the schema need custom transformation and which can be migrated directly;
Bulk Data Transfer. Once transformation logic is implemented, automated tools, such as Fivetran, can execute bulk data loads to transfer millions of records reliably and with built-in error handling, logging, and retry mechanisms. This reduces human error and dramatically shortens the time required to move large data sets;
Validation and Monitoring. Automated systems like Datafold can verify that row counts and data types are consistent between source and target. Such data validation tools free developers from writing boilerplate verification logic and allows them to focus on business-critical transformation code.

By offloading tasks like extraction, loading, and validation, automated tools can save developers hours or even days on each test iteration of the migration process. For business owners modernizing decades-old software, this hybrid approach offers:

Lower development costs, as automation reduces the number of custom scripts required;
Faster iteration, making it easier to test, refine, and re-run migrations before final deployment;
Reduced risk, thanks to automated validation and rollback capabilities.

How Developers Handle Tasks That Are Beyond the Capabilities of Automated Tools

Automated data migration tools can help developers migrate to a different database management systems or to a new version of the DBMS in use, applying the required data manipulations to ensure accurate representation. Also, they can copy the id, email, and nickname fields with little trouble. Possibly, there will be no issues with replicating the old users table into a staging environment.

Automated data migration tools can’t successfully perform tasks required for the use case we described earlier. For instance, infer gender from names (e.g., determine "Sarah" is female, "John" is male), or populate the interests table dynamically from user-provided values. Also, there could be issues with deduplicating shared interests across users (e.g., don’t insert "kitchen gadgets" twice) or creating the correct many-to-many relationships in user_interests.

To address these limitations and handle the data migration process, developers must first copy to a new system all transferable data “as-is,” without any changes. This can be done using SQL queries or automated migration tools. Then, for data requiring additional processing, developers write custom migration scripts that:

Iterates through each row in the legacy users table;
Splits the username into first and last names;
Uses predefined logic or an external service to infer gender based on the first name;
Normalizes user interests. It includes adding new interests to the interests table (if not already present) and adding corresponding entries in user_interests linking users to their interests.

Example:

User 1: Sarah Jones → first_name = Sarah, inferred gender = Female
- Interests: A, B, C → Insert A, B, C into interests (if new), and link Sarah to them in user_interests
User 2: John Chivers → first_name = John, inferred gender = Male
- Interests: B, C, D → B and C already exist; insert D; create mappings for all three in user_interests

The Human-Automation Partnership in Data Migration

While automated tools bring undeniable speed and reliability to repetitive migration tasks, they cannot fully replace the insight, logic, and domain expertise. Effective data migration requires a hybrid approach. Automation can handle the heavy lifting (bulk transfers, change tracking, and validation) while developers focus on high-value tasks like transforming data models, preserving business logic, and ensuring consistency across evolving schemas.