DEV Community: Paul-S

I Connected an LLM to Production. Here’s What Broke

Paul-S — Tue, 08 Sep 2026 10:54:30 +0000

The demo looked ready.

The application could search company documents, answer support questions, classify requests, and create tickets through an API. It handled every test prompt we gave it.

Then real users arrived.

They asked incomplete questions. They pasted entire email threads. They used terms that did not exist in our documentation. Some requests matched several policies, while others required information the system could not access.

The LLM was still producing fluent answers. The workflow around it was falling apart.

The Output Was Valid Until It Wasn’t

During testing, the model returned predictable JSON:

{
"category": "billing",
"priority": "high",
"requires_human": true
}

Production inputs were less predictable. Sometimes the model changed a field name, returned an unsupported category, or added an explanation outside the JSON object.

A response that looks reasonable to a person can still break an application.

The first improvement was to treat model output as untrusted input. Every response had to pass schema validation before the application could use it.

from typing import Literal
from pydantic import BaseModel

class TicketAction(BaseModel):
category: Literal["billing", "technical", "account"]
priority: Literal["low", "medium", "high"]
requires_human: bool

def validate_action(model_output: dict):
return TicketAction.model_validate(model_output)

Structured output reduced parsing failures, but it did not prove that the selected category was correct. Format validation and business validation became separate steps.

RAG Retrieved Something Relevant but Not Correct

The knowledge base contained current policies, archived documents, internal notes, and several pages describing similar processes.

The retrieval system often returned a document related to the question. That did not mean it returned the document needed to answer it.

One customer asked about cancelling a subscription. The system retrieved an older cancellation policy because it shared more words with the question than the current policy did.

The solution was not simply increasing the number of retrieved chunks.

Documents needed version metadata, ownership, effective dates, and access rules. Archived content had to be removed from normal retrieval. The application also needed to recognise conflicting evidence and stop instead of asking the LLM to choose a convenient answer.

RAG improved the model’s access to information. It did not remove the need to manage that information.

Tool Calling Turned Small Errors Into Real Actions

A wrong answer is harmful. A wrong action can be worse.

Once the LLM could create tickets, update CRM records, and trigger notifications, every uncertain decision had operational consequences.

Retries created duplicate records. Incorrect parameters sent requests to the wrong queue. A broad service account gave the agent access to functions it never needed.

We reduced this risk by giving each tool one narrow purpose. Tool arguments were validated outside the model, write operations used idempotency keys, and high-impact actions required confirmation.

This follows the principle behind OWASP’s guidance on excessive agency: limit available tools, permissions, functionality, and autonomy.

The model could recommend an action. The application decided whether that action was allowed.

One User Request Became Eight Model Calls

The original demo made one request to an LLM.

The production version classified the user’s intention, rewrote the search query, retrieved documents, reranked the results, generated an answer, checked the answer, selected a tool, and summarized the result.

Each step appeared reasonable on its own. Together, they created noticeable latency and unpredictable costs.

Agents made the problem harder because the number of steps could change for every request. A simple question might finish immediately, while an ambiguous request could enter a loop of repeated searches and tool calls.

We added limits for execution time, model calls, retries, retrieved context, and total tokens. Smaller models handled basic classification, while stronger models were reserved for decisions that needed deeper reasoning.

The goal was not to minimize every model call. It was to ensure that each call justified its cost and delay.

Our Logs Said Everything Was Successful

The API returned 200. The workflow was still wrong.

Traditional logs showed that the request completed, but they did not explain which documents were retrieved, why a tool was selected, or where the answer changed.

Production debugging required a trace of the complete workflow. We recorded the prompt version, model version, retrieved document IDs, tool arguments, tool results, latency, token usage, validation failures, and final outcome.

Sensitive customer data was removed or masked before storage. Observability should help investigate failures without creating a new privacy problem.

The NIST Generative AI Profile also emphasizes ongoing monitoring, documented responsibilities, incident handling, and human oversight for generative AI systems.

The Evaluation Set Was Too Polite

Our early tests asked clear questions with known answers.

Real users did not behave like an evaluation dataset.

They submitted vague instructions, conflicting requests, spelling mistakes, old account details, pasted web content, and questions requiring permissions they did not have.

The evaluation set had to include those conditions. We added cases involving stale documents, conflicting sources, API timeouts, repeated actions, missing information, prompt injection, and requests that should be refused or escalated.

Every production failure became a new regression test.

That changed evaluation from a task completed before launch into a process that continued throughout the product’s life.

The LLM Was Only One Part of the Product

The biggest lesson was simple: connecting an LLM to an application is not the same as building a production AI system.

Reliable AI requires structured outputs, managed knowledge, secure tools, permission checks, human approvals, observability, cost controls, and realistic evaluations.

This is also what teams should examine when they plan to hire AI engineers. Prompting skills matter, but production AI development also requires backend engineering, API design, security, data management, testing, and operational thinking.

The LLM did not suddenly become less capable after deployment. Production simply exposed every assumption that the demo never tested.

What broke first when you moved an LLM application into production?

The Laravel App Worked Fine Until the Customer Base Started Growing

Paul-S — Tue, 01 Sep 2026 12:43:55 +0000

At launch, the Laravel application felt fast. Pages loaded quickly, orders moved through the system, and the support team heard few complaints. Then the customer base grew. Reports took longer to open, checkout occasionally stalled, and background tasks began to pile up. The application had not suddenly become badly written. Its original assumptions simply no longer matched the workload.

This is a common stage in a successful product. More customers do not just create more page views. They create more data, simultaneous sessions, uploads, notifications, searches, reports, API calls, and support activity. A design that works comfortably for hundreds of users may reveal bottlenecks when thousands arrive at the same time.

Growth changes the shape of the workload

Teams often respond to slower performance by adding a larger server. That can provide temporary relief, but it does not explain where time is being spent. One customer action may trigger several database queries, an email, a webhook, an audit entry, and a file operation. As traffic rises, each small cost is multiplied.

The first task is therefore measurement. Request duration, error rates, memory use, database load, slow queries, queue depth, and third-party response times help reveal the actual constraint. Without that visibility, scaling becomes an expensive guessing exercise.

The database often feels the pressure first

Database problems can stay hidden while tables are small. Missing indexes, repeated queries, and loading more records than a page needs may barely be noticeable during early development. With years of orders or a rapidly growing product catalogue, the same patterns become costly.

Laravel teams should examine slow-query logs and common user journeys. N+1 queries can often be reduced through appropriate eager loading. Large lists should use pagination, and searches should avoid scanning unnecessary columns or rows. Indexes should support real filtering and sorting patterns, not be added blindly. Optimizing the query behind a busy endpoint is usually more valuable than increasing hardware without investigation.

Synchronous work makes customers wait

A web request should finish the work the customer needs immediately and move suitable follow-up tasks elsewhere. Sending emails, generating reports, processing imports, resizing images, or notifying external systems can often run through queues. This keeps the user-facing response short even when the total amount of work grows.

Queues are not a place to hide unreliable code. Jobs need sensible timeouts, controlled retries, failure monitoring, and protection against running the same action twice. Workers must also scale with demand. A fast website paired with a six-hour queue backlog is still a poor customer experience.

Caching helps when its rules are clear

Frequently read information can be cached to reduce repeated database and API work. Configuration, catalogue summaries, and expensive calculations are common candidates. However, caching introduces a new question: when does the stored value become outdated?

A cache strategy should define what is stored, how long it remains valid, and which event refreshes or removes it. Caching everything can trade a performance issue for stale prices, incorrect permissions, or confusing customer data. The safest approach starts with measured hot paths and clear invalidation rules.

External services become part of reliability

Payment providers, shipping platforms, CRMs, and AI services may work perfectly in a low-volume test. At scale, rate limits, network timeouts, and intermittent failures become normal operating conditions. Integrations should fail gracefully, use appropriate retries, record enough context for diagnosis, and avoid blocking an entire page when a non-essential service is unavailable.

Scaling requires operational discipline

Adding application instances works best when the application is stateless. Sessions, cache data, and user uploads should not depend on the local disk of one server. Deployments should be repeatable, and database changes should be planned so old and new application versions can operate safely during a release.

Businesses investing in Custom Laravel Development should plan for expected traffic, data growth, background processing, integrations, failure handling, and observability before adding infrastructure. These decisions make later growth less disruptive and help teams scale the parts of the system that genuinely need it.

Start with evidence, not a rewrite

A growing application does not automatically need microservices or a complete rebuild. A well-structured Laravel monolith can support substantial demand when its database access, queues, caching, and deployment model are designed carefully. Splitting a system too early can add network failures, duplicated data, and operational complexity without improving the customer experience.

Begin with profiling and realistic load tests. Fix the most expensive query, move slow non-essential work to queues, add monitoring, and test again. Repeat until the system meets a defined performance target. Architecture should change only when evidence shows that a clear boundary needs independent scaling, ownership, or release cycles.

Customer growth is not proof that Laravel failed. It is proof that the product reached a workload its early design never had to handle. The strongest teams treat that moment as a signal: measure what changed, remove the real bottlenecks, and build the operational habits needed for the next stage of growth.

AI Chatbot vs Traditional Mobile App: What Do Customers Actually Need?

Paul-S — Tue, 01 Sep 2026 11:51:41 +0000

When businesses discuss digital customer experience, the conversation often becomes a contest: should they build an AI chatbot or a traditional mobile app? That framing sounds simple, but it overlooks the most important person in the decision—the customer.

Customers rarely care which technology sits behind an experience. They want to complete a task quickly, understand what is happening, and feel confident that their information is safe. Sometimes a conversation is the easiest route. In other situations, a familiar screen with clear buttons is faster and more dependable.

What a traditional mobile app does well

A mobile app is strong when customers perform the same structured tasks regularly. Banking customers may want to check balances, review transactions, transfer money, or manage cards. Fitness users may want dashboards, progress charts, saved routines, and device tracking. These actions benefit from a stable visual layout that users can learn over time.

Apps are also useful when an experience depends on phone features such as the camera, GPS, biometric login, push notifications, or offline access. A well-designed app gives users direct control. They can see available options, compare information, move backward, and confirm an action before submitting it.

That predictability matters for high-value or sensitive tasks. Customers may prefer a visible form and a clear confirmation screen when making a payment, changing account settings, or uploading personal documents.

Where an AI chatbot can reduce friction

Chatbots are more helpful when customers do not know where to begin. Instead of searching through menus, users can explain what they need in their own words. The chatbot can ask follow-up questions, clarify the request, and guide them toward a relevant answer or action.

This approach works well for product discovery, appointment scheduling, onboarding, troubleshooting, and questions that involve several possible routes. A customer could describe the type of insurance cover they need, ask which subscription suits their team, or explain an unusual delivery problem without first learning the company's navigation structure.

Conversation can also make digital services easier for people who struggle with complex menus or unfamiliar terminology. Voice input, translation, and step-by-step explanations can improve access when they are designed carefully.

A chatbot is not automatically the simpler choice

Natural language feels easy, but it can introduce uncertainty. A customer may not know what the chatbot can do, how to phrase a request, or whether an answer is accurate. Long conversations can also become slower than tapping a few familiar buttons.

Chatbots are weak when users need to scan, compare, or control several items at once. Choosing airline seats, reviewing financial charts, editing a detailed profile, or comparing product specifications usually works better through a visual interface. Customers should not have to conduct a lengthy conversation for a task that a simple screen can complete in seconds.

Trust is another concern. If a chatbot can access accounts or perform actions, users need clear confirmation, visible limits, and an easy way to reach a person. A confident but incorrect answer can damage the experience faster than a confusing menu.

The practical answer is often a hybrid experience

Businesses do not always need to choose one interface. A mobile app can provide the stable structure, while an AI assistant helps users navigate it. The chatbot might explain a feature, find a transaction, summarize account activity, or prepare an action. The app can then display the details and ask the user to review and confirm.

This combination uses conversation for discovery and guidance while keeping visual controls for comparison, editing, and approval. It also allows customers to switch methods. Someone may start with a question, move to a form, and return to the conversation if they need help.

Start with customer tasks, not technology

Businesses considering AI chatbot app development services should begin by studying the tasks customers are trying to complete. The right design depends on how often the task occurs, how much information it involves, how serious an error would be, and whether customers need to compare options visually.

Teams should map the complete journey, including what happens when the system does not understand, when data is missing, or when human judgment is required. They should also test the experience with real customers rather than assuming that a conversational interface is naturally easier for everyone.

Useful measures include completion rate, time required, abandonment, repeated attempts, customer satisfaction, and the quality of human handoffs. These reveal whether the interface is solving a genuine problem or simply adding a fashionable AI layer.

Give customers the shortest reliable path

AI chatbots and traditional mobile apps serve different needs. Apps provide structure, visibility, and control. Chatbots provide flexibility, guidance, and a natural starting point for uncertain requests. Neither is universally better.

What customers actually need is the shortest reliable path to their goal. The best digital products choose the interfac

Five Signs Your Python Application Needs an Experienced Engineer

Paul-S — Wed, 26 Aug 2026 11:56:55 +0000

A Python application can appear healthy while problems are quietly growing underneath it.

Features are being released. Customers can log in. The API responds. Nothing seems urgent.

Then traffic increases, a dependency is updated, or a new developer joins the project. Suddenly, simple changes take days, errors become difficult to reproduce, and nobody wants to touch certain parts of the code.

These problems do not necessarily mean Python was the wrong choice. They usually mean the application has grown beyond the engineering practices used to build its first version.

Here are five signs that your Python application may need a more experienced engineer.

1. Every Small Change Breaks Something Else

Adding a field to a form should not break the reporting system. Updating a payment method should not affect user registration.

When unrelated features repeatedly fail after small changes, the code probably contains tight dependencies. One function may be handling validation, database operations, business rules, and external API calls at the same time.

This often happens in early-stage products. The team moves quickly because proving the idea matters more than creating perfect architecture.

That approach can work for a prototype. It becomes dangerous once customers depend on the application.

An experienced Python engineer can separate responsibilities, introduce clearer boundaries, and reduce the chance that one change creates failures elsewhere. The objective is not to rewrite everything. It is to make future changes safer.

2. Nobody Trusts the Test Suite

A test suite should give developers confidence before a release. Instead, some teams have tests that fail randomly, take too long, or cover only the easiest parts of the application.

The team eventually stops paying attention to failures. Developers rerun the same test until it passes or disable it to complete a deployment.

At that point, testing becomes decoration rather than protection.

A senior engineer will first identify which workflows create the greatest business risk. Authentication, payments, permissions, data processing, and third-party integrations usually deserve attention before minor interface details.

Good testing is not about reaching an impressive coverage percentage. It is about detecting failures that could affect customers or business operations.

3. Production Errors Are Difficult to Investigate

Consider this error handling:

try:
process_payment(order)
except Exception:
pass

The application does not crash, but the payment may fail without leaving useful evidence. The customer sees an incomplete order, while the support team has no information about what happened.

Changing pass to a log statement is not enough if the system still lacks request IDs, structured logs, alerts, performance metrics, and relevant business context.

Experienced engineers think about how software will be investigated after deployment. They design applications to answer practical questions:

Which customer was affected?
Which request failed?
Did an external service time out?
Can the operation be retried safely?
Did the same error affect other users?

If finding the cause of an error requires hours of guesswork, the application needs better observability, not just more debugging.

4. Performance Problems Are Solved by Adding Servers

Adding infrastructure can temporarily hide inefficient code, but it does not always solve the underlying problem.

A slow Python application may be making repeated database queries, loading too much data into memory, calling external services one after another, or performing heavy work inside a web request.

An experienced engineer measures the system before changing it. The real bottleneck might be a database index, an inefficient query, a blocking network call, or a task that belongs in a background queue.

This distinction matters because each problem requires a different solution. Adding servers to compensate for a poor database query can increase cloud costs without providing reliable performance.

Optimization should start with evidence.

5. One Developer Holds the Entire System Together

Sometimes only one person knows how deployments work, why a particular workaround exists, or which background job must be restarted manually.

That person becomes the project’s unofficial documentation.

This situation creates a serious business risk. If the developer is unavailable, releases slow down and production incidents become harder to resolve. New team members may avoid important parts of the application because they do not understand the consequences of changing them.

An experienced engineer can reduce this dependency through clearer documentation, automated deployments, code reviews, architecture notes, and repeatable operational processes.

The goal is not to make every developer know everything. It is to ensure that critical knowledge belongs to the team rather than one individual.

Experience Is More Than Writing Advanced Python

Experienced Python engineers do not simply write more complicated code. In many cases, they make the application simpler.

They know when a small refactor is enough and when an architectural change is necessary. They consider security, testing, deployment, monitoring, and maintainability alongside feature delivery.

If you plan to hire Python developers for an existing application, evaluate more than framework knowledge. Ask candidates how they have diagnosed production failures, improved legacy code, reduced deployment risk, and handled systems that grew beyond their original design.

At Spaculus Software, this is often the first step when joining an existing Python project: understand the current system before recommending changes. A careful technical review can reveal whether the application needs focused improvements, gradual modernization, or a larger architectural update.

Final Thought

A working application is not always a healthy application.

Frequent regressions, unreliable tests, unclear production errors, rising infrastructure costs, and knowledge concentrated in one person are signs that technical risk is accumulating.

The right engineer will not begin by rebuilding everything. They will identify the most important risks, protect what already works, and help the application become easier to change as the business grows.

Which of these warning signs have you encountered in a Python project?

The Chatbot Worked in Testing, Then Real Users Arrived

Paul-S — Tue, 25 Aug 2026 11:53:23 +0000

The demo looked ready.

The chatbot answered every question from the testing document. It explained product features, found account information, and politely handed difficult conversations to a human.

The team tested it again before launch.

“Where can I download my invoice?”

“Can I change my subscription?”

“What is your refund policy?”

Every answer was correct.

Then real customers arrived.

One user wrote, “charged twice pls fix.” Another sent three messages instead of one complete question. Someone pasted an entire email thread into the chat. A customer referred to “the plan I had before,” although the chatbot had no access to that history.

By the end of the first day, the team had discovered something its test script never showed:

The chatbot understood the test cases. It did not yet understand the messiness of real conversations.

Testing Questions Were Too Clean

Development teams often test chatbots with complete, well-written questions because those cases are easy to review.

Real users do not behave that way.

They misspell words, change topics halfway through a message, use internal product names, and assume the chatbot remembers information from earlier sessions. They may provide too little context or far more context than the system can process effectively.

A chatbot tested only with ideal questions is like a payment form tested only with valid cards. It proves that the happy path works, not that the product is ready.

A stronger evaluation set should include incomplete messages, spelling mistakes, conflicting requests, long conversations, unsupported languages, angry customers, and questions with no verified answer.

OpenAI’s official evaluation guidance describes evals as an essential part of checking whether model outputs meet defined content and style expectations. The important word is defined. “The answer looks good” is not a measurable production requirement.

Retrieval Worked Until the Wording Changed

The chatbot used retrieval-augmented generation to answer from company documents. During testing, the user’s wording closely matched the documentation.

The knowledge base said “subscription cancellation.” Testers asked, “How do I cancel my subscription?”

Customers asked:

“How do I stop getting billed next month?”

The intent was the same, but retrieval did not always return the correct document.

This is why teams should inspect more than the final answer. They need to know which documents were retrieved, how relevant those documents were, and whether the model had enough evidence to respond.

A useful production trace might record:

{
"conversation_id": "conv_1842",
"intent": "cancel_subscription",
"retrieved_documents": 3,
"top_relevance_score": 0.62,
"response_time_ms": 2480,
"used_fallback": false,
"human_handoff": true
}

The exact fields will vary, but the principle remains: if the chatbot produces a bad answer and the team cannot reconstruct what happened, debugging becomes guesswork.

The Bot Answered When It Should Have Stopped

One customer asked about a policy that was not present in the approved knowledge base.

The chatbot still responded.

The answer sounded reasonable, confident, and completely invented.

This is one of the most dangerous production failures because fluent language can hide missing evidence. Anthropic’s official guidance on reducing hallucinations recommends allowing the model to express uncertainty and grounding responses in direct source material.

A production chatbot should have a clear refusal or escalation rule:

if retrieval_score < MIN_CONFIDENCE:
return {
"answer": "I don’t have enough verified information to answer that.",
"action": "handoff_to_human"
}

The threshold should be tested using real conversations. Setting it too low increases unsupported answers. Setting it too high sends too many customers to human support.

Some Users Tested the Boundaries on Purpose

Not every unexpected input is accidental.

Users may ask the chatbot to ignore previous instructions, reveal its system prompt, expose private information, or perform actions outside their permissions. If the chatbot reads uploaded files, webpages, emails, or support tickets, malicious instructions may also enter indirectly through that content.

The OWASP GenAI Security Project lists prompt injection as a major risk for LLM applications. It also makes an important point: retrieval and fine-tuning do not completely remove the problem.

Teams therefore need controls outside the prompt. Tools should enforce user permissions independently. Sensitive actions should require confirmation. Retrieved content should be treated as untrusted input, and the chatbot should never receive broader system access than the task requires.

Production Quality Is More Than Answer Accuracy

A chatbot can answer correctly and still create a poor experience.

A response that arrives after twelve seconds may cause the customer to leave. A correct answer written in five dense paragraphs may be useless on mobile. A bot that forgets the previous message forces the customer to start again. A handoff that loses the conversation history makes human support repeat the same questions.

Production monitoring should therefore cover:

Answer correctness and source support
Retrieval quality
Response time and failures
Cost per conversation
Fallback and handoff rates
Repeated questions after an answer
Customer feedback and unresolved conversations

These metrics should be reviewed by intent. A chatbot may perform well for opening hours and order tracking while failing badly on billing or account access. One overall success rate can hide those differences.

Real Conversations Should Become New Tests

The most valuable evaluation set is not created once before launch. It grows from production.

Failed searches, poor answers, unusual wording, escalated conversations, and negative feedback should become new test cases. Before changing the prompt, model, retrieval settings, or knowledge base, teams can run those cases again and check whether the update fixes one problem without creating another.

That feedback loop is what separates a chatbot demo from a maintained product.

It is also the work businesses should examine when choosing an AI chatbot development company. Spaculus Software supports chatbot architecture, RAG pipelines, integrations, evaluation, security controls, deployment, and ongoing monitoring—not only the chat interface customers see.

The Real Launch Begins After Launch

The chatbot did not suddenly become less intelligent when customers arrived.

The environment changed.

Testing gave it clean questions, known answers, and predictable conversations. Production introduced ambiguity, missing context, unusual language, security risks, latency, integration failures, and genuine consequences for being wrong.

The team’s mistake was not launching too early.

It was treating launch as the end of testing.

For an AI chatbot, real users do more than use the product. They reveal the test cases the development team never knew it needed.

AI Developers Can Build the Demo in a Week. Production Is Where the Real Work Starts

Paul-S — Fri, 21 Aug 2026 10:39:55 +0000

The demo looks perfect.

A user asks a question. The AI finds the right information, writes a clear answer, and returns it within seconds.

Everyone in the meeting is impressed.

Then the application goes live.

Real users ask unclear questions. Some upload broken files. Others try prompts nobody expected. Responses become slower, API costs rise, and the AI occasionally gives a confident answer that is completely wrong.

The demo proved that the idea could work.

Production reveals whether it can keep working.

Why Is an AI Demo Easy to Build?

Modern AI APIs make it possible to create a working prototype quickly. A developer can connect a language model, add a simple interface, provide a few instructions, and have something impressive within days.

That is valuable. A quick demo helps a company test an idea before spending heavily on it.

But a demo normally works with:

Selected questions
Clean data
Limited users
Controlled conditions
Little concern about cost or scale

Production removes all those protections.

What Changes When Real Users Arrive?

Real users do not follow the demo script.

They misspell words, leave out important details, switch topics, upload unexpected formats, and sometimes ask the AI to do things it should never do.

This is where AI development becomes less about prompts and more about engineering.

A production AI system needs to know:

When it has enough information to answer
When it should search company data
When it should ask another question
When it should refuse a request
When a human should take control

A better prompt may improve the response, but it cannot solve every production problem.

Production AI Must Be Measured, Not Trusted

Traditional software usually gives the same result when it receives the same input. AI systems can behave differently, even when a request looks similar.

That means testing a few successful examples is not enough.

AI developers need evaluation sets containing normal questions, difficult cases, incomplete requests, unsafe prompts, and examples collected from real usage. They must measure whether answers are correct, grounded in approved data, useful, fast, and affordable.

Logging is equally important. If an answer goes wrong, the team should be able to see what the user asked, what information was retrieved, which model responded, and where the process failed.

Without that visibility, improving the system becomes guesswork.

Reliability Is More Than Preventing Hallucinations

Wrong answers receive the most attention, but production AI can fail in quieter ways.

A response may be correct but arrive too late. An AI agent may repeat a tool call and increase costs. A retrieval system may find an outdated document. A model update may change behaviour that worked yesterday.

Production engineering therefore includes:

Data quality and retrieval
Security and access control
Response evaluation
Monitoring and alerts
Cost and latency control
Fallbacks and human review

This is often the point when companies decide to hire AI engineers instead of treating AI as one more API integration.

When Should You Hire AI Developers?

You probably do not need a large AI team to test an early idea. One focused prototype can answer an important question: Does this solve a real user problem?

Once the answer is yes, the requirements change.

If you plan to Hire AI engineers, look beyond model knowledge and prompt writing. Ask how they test output quality, protect private data, control costs, handle model failures, and monitor the complete user request.

While exploring production AI at Spaculus Software, one lesson keeps coming up: getting the first answer is easy, but making the complete system reliable takes real engineering. The goal is not simply to make AI answer once. It is to make the complete system useful, measurable, and dependable when real people start using it.

A demo earns attention. Production earns trust.

*For developers who have shipped an AI feature: what failed first when real users arrived? *

Can I Hire MEAN Stack Developers to Upgrade an Existing Angular or Node.js Application?

Paul-S — Thu, 20 Aug 2026 13:17:30 +0000

Yes. You can hire MEAN Stack developers to upgrade an existing Angular frontend, Node.js backend, or complete JavaScript application.

A MEAN developer works across MongoDB, Express.js, Angular, and Node.js. This full-stack knowledge is useful when an upgrade affects both the user interface and backend APIs.

However, the MEAN Stack label alone does not guarantee a successful upgrade. The developer should have practical experience with your current Angular or Node.js version, dependencies, testing environment, database, deployment process, and application architecture.

Why Should You Upgrade an Existing Application?

Framework upgrades are not only about accessing new features. They help applications continue receiving security fixes, maintain compatibility with third-party packages, and avoid growing technical debt.

As of August 2026, Angular 22 is under active support. Angular 21 and Angular 20 are receiving long-term support, although Angular 20’s LTS period ends on November 28, 2026. Angular 19 and earlier versions are no longer supported.

Angular provides approximately 12 months of active support followed by 12 months of long-term support. Once a version becomes unsupported, it no longer receives regular framework fixes or security patches. Angular release policy

Node.js has a different release model. Node.js 26 is the current release, while Node.js 24 and Node.js 22 are supported LTS versions. Node.js 20 and earlier versions have reached end of life.

The Node.js project recommends using only Active LTS or Maintenance LTS releases in production. Node.js release schedule

If your application uses an unsupported version, postponing the upgrade can increase security, compatibility, and maintenance risks.

When Is a MEAN Stack Developer the Right Choice?

A MEAN developer is a strong fit when an Angular upgrade also affects Node.js APIs, Express middleware, authentication, MongoDB queries, or shared TypeScript models.

For example, upgrading Angular may require a newer TypeScript version. That change can affect shared packages used by the Node.js backend. A full-stack developer can review these dependencies together instead of treating the frontend and backend as separate systems.

MEAN developers can also help when the project needs more than a version update. They can replace deprecated packages, improve API performance, review MongoDB queries, modernize Angular components, and strengthen automated testing.

If the application contains only a small Angular interface without Node.js or MongoDB, a dedicated Angular developer may be more efficient. A complex Node.js microservices platform may similarly require a backend specialist with distributed-systems experience.

The right choice depends on the actual architecture, not the name of the stack.

What Should Be Checked Before the Upgrade?

An upgrade should begin with an application audit rather than immediately changing package versions.

For an Angular application, developers can start with:

ng version
ng update
npm outdated
npm audit

For a Node.js application:

node --version
npm outdated
npm audit
npm test

These commands identify the current versions, outdated dependencies, known package vulnerabilities, and the condition of the existing test suite.

The audit should also cover custom build configurations, deprecated APIs, UI libraries, authentication packages, database drivers, environment variables, and CI/CD workflows.

This assessment helps determine whether the project needs a routine update, a broader modernization effort, or selected parts of the application to be rebuilt.

Can Angular Be Upgraded Across Several Versions at Once?

Angular recommends upgrading one major version at a time.

An application moving from Angular 19 to Angular 22 should normally follow this path:

*Angular 19 → Angular 20 → Angular 21 → Angular 22
*
Angular’s migration tooling supports updates between adjacent major versions. The official guidance says that the version being upgraded should be within one major version of the target. Angular Update Guide

This staged approach makes problems easier to isolate. Jumping across several versions at once can combine dependency conflicts, TypeScript changes, removed APIs, and test failures into one difficult debugging process.

What Happens During an Upgrade Project?

The team should first document the current versions, architecture, package dependencies, business-critical workflows, and known defects.

Next, developers identify compatible versions of Angular, Node.js, TypeScript, MongoDB drivers, and third-party packages. Unsupported libraries may need to be replaced before the main framework upgrade can continue.

The application is then upgraded in controlled stages. After each stage, developers run unit tests, integration tests, build checks, and important user journeys.

The updated application should be released to a staging environment before production. Performance, errors, API behaviour, authentication, database operations, and customer-facing workflows should be monitored after deployment.

An upgrade is complete only when the application works reliably in production. Successfully installing new package versions is not enough.

Should You Upgrade or Rewrite the Application?

In most cases, an incremental upgrade is safer than a complete rewrite.

A rewrite creates new risks around feature parity, data migration, testing, delivery time, and business continuity. Existing business rules that took years to develop may be overlooked during reconstruction.

A rewrite may be appropriate when the current architecture blocks essential changes, core dependencies have no supported upgrade path, serious security problems are deeply embedded, or maintaining the existing code costs more than replacing it.

The decision should be based on a technical audit rather than the age of the application alone.

What Should You Ask Before Hiring MEAN Stack Developers?

Ask developers how they would assess the application before providing a final estimate.

They should be able to explain their approach to incremental Angular upgrades, Node.js LTS migration, dependency conflicts, automated testing, rollback planning, staging, and production monitoring.

Request examples of previous upgrade projects and ask what unexpected problems occurred. Experience resolving real dependency, testing, and deployment issues is often more valuable than familiarity with a long list of technologies.

How Can Spaculus Help?

Spaculus Software provides MEAN Stack developers for Angular interfaces, Node.js and Express APIs, MongoDB applications, performance optimization, and legacy-system modernization.

The team can audit an existing application, prepare a staged upgrade plan, replace unsupported dependencies, improve testing, optimize APIs, and support production deployment. Spaculus also provides custom software, SaaS, Cloud and DevOps, UI/UX, QA, mobile, and AI development services when the upgrade is part of a broader product roadmap.

Businesses can hire MEAN Stack developers from Spaculus without automatically rebuilding the complete application.

Key Takeaways

MEAN Stack developers can upgrade Angular-only, Node.js-only, or complete MEAN applications.
Angular 19 and earlier versions are unsupported as of August 2026.
Node.js 20 and earlier versions have reached end of life.
Angular upgrades should normally move through one major version at a time.
A technical audit should determine whether the application needs an update, modernization, or rewrite.

Upgrading an application is rarely just an npm install command. A safe project begins by understanding what the current system does, which workflows users cannot afford to lose, and how every change will be tested before production.

AI Engineer vs Python Developer: Who Does Your AI Project Actually Need?

Paul-S — Wed, 12 Aug 2026 11:20:20 +0000

Artificial intelligence projects often begin with a promising idea: automate customer support, predict demand, analyze documents, personalize recommendations, or build an intelligent assistant. The first hiring decision, however, can create immediate confusion. Should you bring in an AI engineer or hire a Python developer?

Both professionals may use Python, work with APIs, and understand data. That overlap makes their roles appear interchangeable, but they solve different problems. A Python developer primarily builds reliable software systems, while an AI engineer creates and integrates systems that learn, predict, reason, or generate content.

Choosing the wrong role may lead to an application with strong backend engineering but weak AI capabilities, or an impressive model that cannot operate reliably in production. Understanding what your project actually requires can prevent wasted development time, unnecessary costs, and architectural problems.

What Does an AI Engineer Do?

An AI engineer designs, develops, integrates, and maintains artificial intelligence systems. Their work can involve machine learning models, large language models, natural language processing, computer vision, recommendation engines, forecasting, and intelligent automation.

Depending on the project, an AI engineer may:

Select suitable AI models and tools
Prepare and process training or retrieval data
Build machine learning pipelines
Fine-tune or evaluate models
Design prompts and structured outputs
Develop retrieval-augmented generation systems
Connect applications with commercial or open-source models
Measure accuracy, latency, safety, and cost
Monitor model performance after deployment

Modern AI engineering is not limited to training a model from the beginning. Many business applications use existing foundation models combined with company data, business rules, APIs, and carefully designed evaluation systems.

For example, an AI engineer building a customer support assistant may design how the system retrieves knowledge, chooses relevant documents, constructs prompts, checks the answer, handles uncertainty, and transfers difficult conversations to a human agent.

The engineer’s responsibility is not merely to make the AI produce an answer. It is to make that answer useful, measurable, secure, and dependable within a real business process.

What Does a Python Developer Do?

A Python developer builds applications, backend services, APIs, automation tools, and data-processing systems using the Python programming language. Python developers commonly work with frameworks such as Django, Flask, or FastAPI and connect applications to databases, third-party services, and cloud infrastructure.

Their responsibilities may include:

Developing backend application logic
Creating and maintaining APIs
Designing database structures
Building authentication and authorization
Integrating external platforms
Writing automated tests
Improving application performance
Managing background tasks
Supporting deployment and monitoring
Automating repetitive processes

A Python developer can integrate an AI API into an application. For a straightforward feature, such as sending text to a language model and displaying its response, an experienced Python developer may be entirely sufficient.

The difference becomes visible when the project requires more than a basic API connection. If the system must select models, control hallucinations, retrieve private knowledge, evaluate response quality, or improve predictions over time, specialized AI engineering becomes increasingly important.

The Main Difference: Software Logic vs Model Behaviour

Traditional software usually follows explicit rules. When a specific input is received, the program performs a defined operation and produces a predictable result. Python developers are trained to build and maintain this deterministic logic.

AI systems are probabilistic. The same or similar input can produce different results, and performance depends on the model, data, context, configuration, and evaluation criteria. AI engineers work with this uncertainty.

Consider an invoice-processing platform. A Python developer can create the upload service, user accounts, database, approval workflow, and accounting integration. An AI engineer can develop the system that identifies document types, extracts fields, assigns confidence scores, and detects unusual entries.

Both parts are necessary for a complete product, but they require different types of expertise.

When You Need an AI Engineer

An AI engineer is the stronger choice when artificial intelligence is a central part of the product rather than a small supporting feature.

You will likely need one when your project involves:

Custom Machine Learning

If the application must make predictions from historical business data, an AI engineer can select algorithms, prepare features, train models, and evaluate whether the results are genuinely useful.

Examples include demand forecasting, fraud detection, lead scoring, churn prediction, and predictive maintenance.

Generative AI With Private Data

Connecting a chatbot to internal documents requires more than uploading files. The system needs document processing, embeddings, retrieval, access controls, prompt construction, citation handling, and quality evaluation.

An AI engineer can design this retrieval pipeline and reduce the risk of incomplete or unsupported answers.

Computer Vision or Natural Language Processing

Projects involving image classification, object detection, speech processing, sentiment analysis, entity extraction, or document understanding usually require specialized model knowledge.

Model Evaluation and Improvement

An AI feature cannot be judged only by whether it works during a demonstration. Teams need representative test cases, quality metrics, failure analysis, and continuous evaluation.

AI engineers establish these systems and determine whether changes improve or damage performance.

AI Safety and Cost Control

Production AI systems must handle prompt injection, sensitive data, inappropriate outputs, response latency, token usage, and model-provider failures. An AI engineer can design safeguards and fallback strategies around these risks.

When a Python Developer Is Enough

Not every project marketed as “AI-powered” needs a dedicated AI specialist. A Python developer may be the practical choice when the intelligence already exists in a third-party service and the main challenge is building dependable software around it.

A Python developer may be enough if you need to:

Add a basic AI API to an existing product
Build a standard chatbot with limited scope
Automate a defined internal workflow
Create APIs and database-backed applications
Process data using established libraries
Connect an AI service with a CRM or business platform
Develop a proof of concept using a hosted model

Suppose a company wants to summarize customer calls using an existing transcription and language-model API. If no custom model, complex retrieval process, or advanced evaluation is required, a skilled Python developer can build the workflow successfully.

Hiring specialized AI talent for such a limited integration may increase costs without creating meaningful additional value.

When Your Project Needs Both

Many serious AI products require both an AI engineer and a Python developer. The AI engineer focuses on intelligence and model performance, while the Python developer builds the software foundation through which customers and internal systems use that intelligence.

*A typical division of responsibilities may look like this:
*
Project Area AI Engineer Python Developer
Model selection Primary responsibility Supports integration
Training and evaluation Primary responsibility Provides infrastructure
Prompt and retrieval design Primary responsibility Connects application services
Backend APIs Supports AI requirements Primary responsibility
Database and authentication Provides data requirements Primary responsibility
User workflows Advises on model limitations Implements business logic
Monitoring Tracks AI quality Tracks application reliability
Deployment Packages model components Manages application services

This collaboration becomes essential when an AI feature must serve real users at scale. A model may perform well in a notebook but fail under concurrent traffic, expose private information, or become too expensive in production. Similarly, a well-engineered application has little value if its AI results are consistently inaccurate.

Questions to Ask Before Hiring

Before choosing a role, define the actual business problem rather than beginning with a job title.

*Ask the following questions:
*
Is AI the core product or only one feature?
Are we using an existing model or developing a custom one?
Does the system need access to private business data?
How will we measure output quality?
What happens when the AI produces a wrong answer?
Do we require backend development, authentication, billing, or integrations?
Will the system need to support large numbers of users?
Are there privacy, security, or regulatory requirements?

If most of the complexity involves applications, APIs, databases, and workflows, prioritize a Python developer. If it involves model behaviour, data quality, retrieval, predictions, or evaluation, prioritize an AI engineer.

When both sides are complex, assemble a small cross-functional team instead of expecting one person to be an expert in every area.

Avoid Hiring Based Only on Tool Lists

Candidates often list Python, LangChain, PyTorch, TensorFlow, vector databases, and numerous model providers. These tools do not prove that someone can build a useful AI product.

A capable AI engineer should be able to explain how model quality will be tested, which failures are acceptable, how sensitive data will be protected, and when a simpler non-AI solution is better.

A strong Python developer should demonstrate clean architecture, testing, API design, database knowledge, security awareness, and production reliability.

If you plan to hire dedicated AI developers, evaluate them using a small version of your real business problem. Their decisions, questions, and evaluation approach will reveal more than a generic coding test.

Making the Right Decision

The right hire depends on where the project’s uncertainty lies.

Choose a Python developer when the AI capability is already available and your main challenge is turning it into a secure, scalable application. Choose an AI engineer when the value of the product depends on model quality, intelligent decision-making, specialized data, or reliable generative AI.

Choose both when you are building a complete AI product for production.

The most expensive mistake is not hiring the more costly professional. It is hiring for the wrong problem. Start with the business outcome, identify the project’s hardest technical risk, and select the expertise that directly addresses it. That approach will produce a stronger product than choosing a role simply because “AI” or “Python” appears in its title.

How Much Does It Cost to Hire an AI Engineer in 2026?

Paul-S — Tue, 28 Jul 2026 13:46:58 +0000

Artificial intelligence has moved from experimentation to practical business use. Companies now use AI for customer service, workflow automation, predictive analytics, fraud detection, recommendation systems, document processing, and intelligent software products.

However, building a reliable AI solution requires more than access to an AI model or API. Businesses need engineers who can prepare data, select models, create integrations, test performance, control costs, and maintain the system after deployment.

So, how much does it cost to hire an AI engineer in 2026?

The cost can range from approximately $35 per hour for freelance support to more than $200,000 per year for specialized, full-time talent. The final amount depends on the engineer’s experience, location, specialization, engagement model, and the complexity of the project.

Average AI Engineer Salary in 2026

AI engineering salaries vary significantly because the title can cover several different roles, including:

Machine learning engineers
Generative AI engineers
Natural language processing specialists
Computer vision engineers
Data scientists
MLOps engineers
AI architects
AI agent developers

According to Robert Half’s 2026 salary data, AI and machine learning engineers in the United States generally earn between $134,000 and $193,250 per year, with a midpoint of approximately $170,750.

Glassdoor reports average annual compensation of approximately $144,454 for an AI engineer in the United States, although pay can be considerably higher in industries such as information technology, consulting, media, and healthcare.

Indeed reports an even higher average of approximately $190,481 per year for machine learning engineers, based on salaries from job postings collected over the previous 36 months.

These numbers show why businesses should not rely on a single average. An engineer creating a basic AI chatbot is not priced the same as an expert building a real-time computer vision platform, autonomous agent system, or enterprise machine learning infrastructure.

AI Engineer Cost by Experience Level

Experience is one of the most important pricing factors.

Junior AI Engineer

A junior engineer usually has up to two years of professional experience. This person may support data preparation, API integration, prompt development, model testing, and basic machine learning tasks.

A junior AI engineer may cost approximately:

$70,000 to $110,000 per year as a full-time employee
$35 to $60 per hour as a freelancer or contractor
$3,000 to $7,000 per month through an offshore development team

Junior engineers are appropriate for clearly defined tasks, but they normally need technical supervision for complex architecture and production deployments.

Mid-Level AI Engineer

A mid-level engineer can independently develop AI features, integrate models, create data pipelines, evaluate outputs, and deploy solutions to production environments.

The typical cost may be:

$110,000 to $180,000 per year for full-time employment
$60 to $120 per hour for contract work
$5,000 to $10,000 per month through a dedicated offshore model

This level is often suitable for businesses building AI-powered SaaS products, recommendation engines, automation tools, internal assistants, or retrieval-augmented generation systems.

Senior AI Engineer or AI Architect

Senior professionals design the complete AI architecture, make model and infrastructure decisions, manage security risks, supervise engineering teams, and connect technical development with business objectives.

Their cost may range from:

$170,000 to more than $250,000 per year
$120 to $250 or more per hour for consulting
$8,000 to $18,000 per month through a specialized remote team

Senior specialists in agentic AI, computer vision, large-scale MLOps, or highly regulated industries may command even higher compensation.

Freelance AI Engineer Costs

Freelancers are useful for prototypes, technical consulting, audits, short integrations, and narrowly defined development work.

Upwork reports that artificial intelligence engineers on its platform typically charge between $35 and $60 per hour, with a median rate of approximately $50 per hour. Advanced AI development and strategic consulting can reach $100 per hour or more.

A small project requiring 100 hours could therefore cost between $3,500 and $10,000. A more advanced project requiring 500 hours might cost between $25,000 and $75,000 or more.

Freelance rates may appear affordable, but businesses must also consider availability, project management, documentation, testing, and long-term maintenance.

Full-Time Employee Costs

The salary is only part of the cost of employing an AI engineer.

Businesses must also budget for:

Recruitment and interview time
Payroll taxes
Health insurance and employee benefits
Bonuses or equity
Hardware and development tools
Cloud infrastructure
Training and certifications
Paid leave
Management and onboarding
Employee replacement risk

After these expenses are included, the total cost of a $150,000 employee may exceed $190,000 to $220,000 per year.

Full-time employment makes sense when AI is a permanent part of the company’s product, operations, or long-term strategy. It may be less practical when the business needs specialized knowledge for a limited project.

Offshore and Dedicated Hiring Costs

Companies can reduce development expenses by working with dedicated engineers from established offshore development companies.

Businesses that Hire AI Engineers through this model receive access to full-time talent without independently managing recruitment, employee benefits, infrastructure, and administrative responsibilities.

Depending on location and expertise, a dedicated offshore AI engineer may cost between $3,000 and $10,000 per month. The price may include development support, project management, quality assurance, and technical supervision.

The lowest quote is not always the best option. A poorly designed AI system can produce inaccurate responses, expose private information, create unexpectedly high API bills, or fail when real users begin using it.

Project-Based AI Development Costs

Some companies prefer to pay for an entire project rather than hire an individual engineer.

Approximate project costs may include:

Basic AI chatbot: $10,000 to $30,000
Custom knowledge assistant using RAG: $20,000 to $60,000
AI workflow automation platform: $30,000 to $100,000
Predictive analytics solution: $40,000 to $150,000
Computer vision application: $50,000 to $200,000 or more
Enterprise AI agent platform: $75,000 to $300,000 or more

These estimates depend on the number of integrations, data quality, security requirements, user volume, model complexity, and expected accuracy.

Hidden Costs Businesses Should Consider

Engineering fees are not the only expense involved in AI development.

Data Preparation

Business data may be incomplete, duplicated, poorly structured, or stored across several systems. Cleaning and organizing it can consume a significant portion of the project budget.

Model and API Usage

Applications built with commercial AI models normally generate recurring token, inference, embedding, or image-processing costs.

Cloud Infrastructure

Databases, vector storage, model hosting, monitoring, GPUs, and backup systems create monthly operating expenses.

Testing and Evaluation

AI outputs are probabilistic. Businesses need structured evaluation datasets, human review, security testing, hallucination checks, and performance monitoring.

Maintenance

Models, APIs, user expectations, and business data change over time. An AI product requires regular updates instead of one-time development.

How to Control the Cost

Before companies Hire AI Engineers, they should define the business problem, expected users, available data, required integrations, and measurable success criteria.

Start with a focused use case instead of building a large platform immediately. A controlled proof of concept can validate technical feasibility and business value before a major investment.

Businesses should also ask candidates or development partners about:

Relevant AI projects
Data security practices
Model evaluation methods
Infrastructure experience
Estimated API expenses
Source-code ownership
Documentation and maintenance
Communication and reporting processes

Conclusion

There is no universal price for hiring an AI engineer in 2026. Freelancers may charge $35 to $250 or more per hour, while experienced full-time professionals can cost between $134,000 and more than $250,000 annually. Dedicated offshore hiring can provide a more flexible option, especially for businesses that need specialized expertise without the overhead of permanent recruitment.

The right decision should not be based on the lowest hourly rate. It should be based on technical capability, relevant experience, communication, security, maintainability, and the engineer’s ability to turn AI into measurable business value.