DEV Community: snehaup1997

Your Queue Is Just Borrowed Time

snehaup1997 — Wed, 30 Sep 2026 15:37:12 +0000

When a system starts receiving more work than it can process, we usually reach for a queue. Instead of making the producer wait, we put the work somewhere safe and let consumers catch up. It feels like the obvious solution and, for a while, everything looks fine.

The API stays responsive. Consumers keep processing. It feels like success and I think this is where we sometimes misunderstand what a queue actually gives us.

A queue doesn't create processing capacity. It converts a rate mismatch into waiting time. And waiting time is not free. Sounds obvious, but it has interesting consequences once you look at what happens to that waiting work.

Consider a service that normally processes:

8,000 events / second

Suddenly, traffic increases to:

9,000 events / second

The queue starts growing. Nothing is obviously broken yet. The API is still responding. Consumers are running. The queue is accepting messages. The team notices the growing backlog and increases consumer concurrency.

The consumers depend on a database that is already close to its limit. More workers create more database contention. Database latency increases, consumer throughput falls. Retries start appearing and those retries create even more work. The system now looks like this:

Traffic increase
      ↓
Queue backlog
      ↓
Database pressure
      ↓
Consumer throughput ↓
      ↓
More backlog
      ↓
More workers
      ↓
More database contention
      ↓
More failures
      ↓
More retries
      ↓
Even more work

The queue wasn't the original problem. It was where the capacity problem became visible. This is why I don't think "the queue is growing" is enough information to decide what to do next.

The useful question is:

Why is the queue growing?

Consider a simpler system:

Producer
   ↓
 Queue
   ↓
Workers
   ↓
Database

Suppose producers send:

10,000 messages / second

while consumers can process:

7,000 messages / second

The deficit is:

10,000 - 7,000 = 3,000 messages / second

So the queue grows by roughly 3,000 messages every second. After five minutes:

3,000 × 300 = 900,000 messages

The queue has protected the consumers from immediate overload. But nothing has changed about the underlying capacity mismatch. If traffic falls below processing capacity, the backlog can drain. If it doesn't, the queue keeps accumulating work. This gives us two questions when designing a queue:

How much burst can we absorb?
How much waiting can the workload tolerate?

The second question is easy to miss.
A queue containing 100,000 messages doesn't tell us enough by itself. We also need to know how quickly the backlog is being processed and how old the waiting work has become.

Queue Depth Is Not the Same as Queue Health. Little's Law gives us a useful relationship:

L = λW

where:

L = average number of items in the system
λ = throughput
W = average time an item spends in the system

So:

W = L / λ

For example, if a system has approximately:

50,000 messages in the system
5,000 messages / second of throughput

then, under steady-state conditions:

50,000 / 5,000 = 10 seconds

That represents roughly 10 seconds of work at the current throughput. It is a useful mental model, not a precise prediction of when a queue will empty. If arrivals continue to exceed processing capacity, the backlog will keep growing. This is why I would monitor queue age alongside queue depth.

A large backlog that is draining quickly can be temporary. A smaller backlog that keeps getting older can be much more concerning.

For example:

P50 message age:    1 second
P95 message age:    4 seconds
P99 message age:   12 seconds
Max age:           10 minutes

is a very different situation from:

P50 message age:    8 seconds
P95 message age:   90 seconds
P99 message age:    5 minutes
Max age:            10 minutes

Both have a ten-minute-old message. Only the second suggests that a large portion of the workload is experiencing significant delay.

Suppose consumers can sustainably process:

8,000 messages / second

and normal traffic is:

7,600 messages / second

The system is already operating at:

7,600 / 8,000 = 95%

Technically, there is capacity remaining. Operationally, there isn't much room for reality. Traffic and processing time vary. Databases slow down, networks experience latency, retries appear, dependencies occasionally fail. So capacity isn't simply:

Arrival rate < Processing rate

It is also about how much headroom exists before normal variation starts creating backlog. A queue can absorb that variation. But only for a while.

Suppose:

Normal traffic:      5K/s
Burst traffic:      10K/s
Consumer capacity:   8K/s

During the burst, the queue absorbs:

10K - 8K = 2K/s

If the burst lasts 60 seconds:

2K × 60 = 120K messages

Now suppose traffic falls to 6K/s while consumers can process 10K/s. The backlog drains at:

10K - 6K = 4K/s

So recovery takes:

120K / 4K = 30 seconds

This gives us two different properties:

Burst tolerance — how much temporary excess work the queue can absorb.
Recovery capacity — how quickly the system can eliminate that backlog afterward.

Both belong in capacity planning. A system that survives a spike but takes ten minutes to recover has a very different operational profile from one that clears the backlog in thirty seconds.

More Consumers Are Not Always More Capacity. Suppose the queue is growing, so we increase workers:

50 workers
   ↓
100 workers
   ↓
200 workers

It sounds reasonable. But imagine the workers all depend on:

Database
   ↓
50 connections

At some point, the database becomes the bottleneck. Adding workers doesn't create more database capacity. It can actually make things worse through connection contention, lock contention, CPU pressure, or increased I/O. Real pipeline looks more like:

Queue
   ↓
Consumer Pool
   ↓
Database
   ↓
External API

The sustainable throughput is constrained by the bottleneck in that path. So when a queue grows, I would ask:

Are consumers under-provisioned?
Or are consumers waiting on something else?

That distinction matters. The queue may simply be the place where a downstream bottleneck becomes visible.

Retries can turn a problem into a feedback loop because they can increase load precisely when the system is already struggling.

Suppose a downstream service starts timing out:

Message
   ↓
Timeout
   ↓
Retry
   ↓
Timeout
   ↓
Retry

Now the system isn't just processing the original workload.

It is processing the original workload plus recovery work.

That can create:

Higher latency
      ↓
More timeouts
      ↓
More retries
      ↓
More work
      ↓
More pressure
      ↓
Higher latency

Retries therefore need their own controls:

exponential backoff;
jitter;
maximum attempts;
retry budgets;
dead-letter handling;
idempotent operations.

A retry mechanism should recover from transient failures without becoming another source of load.

This is another reason why queue depth alone isn't enough.

A queue might be growing because traffic increased.

Or because consumers slowed down.

Or because retries doubled the effective workload.

The metrics need to tell us which one.

Not All Queued Work Has the Same Value

There is another question I think queue designs should ask:

Does every message still have value by the time we process it?

Imagine a queue containing:

Update product 123 → $10
Update product 123 → $11
Update product 123 → $12
Update product 123 → $13

If the only thing that matters is the current price, processing all four events may be unnecessary.

We could potentially coalesce them:

$10
$11
$12
$13
 ↓
$13

Now the system has reduced four units of work to one.

We're not processing the backlog faster.

We're deciding that some of the backlog no longer needs to be processed.

The same idea can apply to:

repeated cache invalidations;
rapidly changing user preferences;
superseded state updates;
duplicate notifications;
refresh requests;
intermediate synchronization events.

The exact semantics depend on the application. Some events absolutely cannot be discarded or coalesced.

But where the business semantics allow it, eliminating stale work can be more effective than simply adding processing capacity.

Backpressure Is the Other Half

A queue becomes much more useful when it is paired with backpressure.

Instead of:

Producer
   ↓
Unlimited queue
   ↓
Workers

I prefer:

Producer
   ↓
Admission control
   ↓
Bounded queue
   ↓
Workers
   ↓
Downstream

When downstream capacity falls, that constraint should eventually reach the producer.

Depending on the system, that might mean:

rate limiting;
concurrency limits;
delayed retries;
HTTP 429 responses;
load shedding;
priority-based admission.

The implementation varies, but the principle is simple:

When the system cannot safely accept more work, it needs a mechanism to say no.

An unbounded queue can hide overload until the system runs out of memory, storage, or patience.

A bounded queue makes the constraint visible.

Not All Work Should Be Treated Equally

Once admission control exists, another question appears:

Which work should be protected when capacity runs out?

Imagine one queue contains:

Password reset
Payment processing
Analytics events
Search indexing
Report generation

During normal operation, treating them equally may be fine.

During severe overload, it may not be.

A more deliberate architecture could separate workloads:

                 Incoming Work
                       │
          ┌────────────┼────────────┐
          ↓            ↓            ↓
       Critical      Normal        Bulk
          │            │            │
       Queue A       Queue B      Queue C

Different workloads can then have different:

capacity limits;
priorities;
retry policies;
SLOs;
scaling rules.

This means overload behavior is designed rather than discovered in production.

Overload Should Be Deliberate

I don't want production systems to have only two states:

Healthy
   ↓
Broken

There should be intermediate behavior:

Normal
  ↓
Queue absorbs burst
  ↓
Queue approaching limit
  ↓
Throttle
  ↓
Reject low-priority work
  ↓
Protect critical traffic

An analytics event might be dropped during extreme overload.

A payment event should not necessarily be treated the same way.

The exact policy depends on the application, but the important thing is to decide what happens before the queue reaches its limit.

What Should We Measure?

If I were operating a queue-backed system, I'd monitor:

Queue depth
Queue capacity utilization

Arrival rate
Processing rate

P50 / P95 / P99 message age

Consumer concurrency
Consumer utilization
Processing latency

Retry rate
Dead-letter rate
Rejection rate

Backlog recovery rate

The important part isn't just collecting these numbers.

It's understanding their relationship.

For example:

Queue depth ↑
Message age ↑
Arrival rate > processing rate

suggests a capacity mismatch.

But:

Queue depth ↑
Processing rate ↓
Database latency ↑

points toward a downstream bottleneck.

And:

Queue depth ↑
Retry rate ↑
Downstream errors ↑

suggests retries may be amplifying the original failure.

So the useful question isn't simply:

"Is the queue growing?"

It's:

"Why is the queue growing?"

The Architecture I Would Build

Putting those ideas together, I'd want the queue surrounded by explicit controls:

                    Incoming Work
                         │
                         ↓
                ┌─────────────────┐
                │ Admission       │
                │ Rate / Priority │
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │ Bounded Queue   │
                │ Depth / Age     │
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │ Consumer Pool   │
                │ Concurrency     │
                └────────┬────────┘
                         ↓
                ┌─────────────────┐
                │ Downstream      │
                │ Capacity Limit  │
                └────────┬────────┘
                         │
              ┌──────────┴──────────┐
              ↓                     ↓
        Success / ACK          Failure / Retry
                                    │
                                    ↓
                           Backoff / Retry Budget
                                    │
                                    ↓
                              Dead Letter

Each part has a different responsibility:

Admission control limits incoming pressure.
The queue absorbs temporary rate differences.
Concurrency limits protect downstream dependencies.
Priority controls protect valuable work.
Retry budgets prevent failures from multiplying load.
Dead-letter handling isolates work that cannot be processed.
Queue age metrics expose the latency created by backlog.
Coalescing or expiration can eliminate work that no longer has value.

The queue is still valuable.

We're simply giving it a defined job.

The Tradeoff

There is no universally correct queue size.

A larger queue gives you more room to absorb bursts, but also allows more work to accumulate before the system pushes back.

A smaller queue creates earlier backpressure, but provides less burst tolerance.

So I wouldn't choose queue capacity simply because the underlying technology supports a particular number of messages.

I'd start with the workload.

Suppose:

Consumer capacity:      8,000/s
Burst arrival rate:    10,000/s
Burst duration:             60s

The excess work is:

(10,000 - 8,000) × 60
= 120,000 messages

That gives us an estimate of the buffer required to absorb that particular burst.

Then I'd ask:

How old can the work become?
How quickly can we recover?
What happens when the buffer fills?
Which work can be discarded?
Which work must be protected?

Those answers determine the actual queue design.

The Bigger Lesson

I like queues because they make systems tolerant of temporary differences in processing rates.

The problem begins when a temporary buffer becomes a permanent holding area for work the system cannot keep up with.

A queue can absorb a burst.

It can decouple producers from consumers.

It can protect downstream services from sudden traffic.

But it cannot change the fundamental relationship between incoming work and processing capacity.

And once work starts waiting, another resource appears:

time.

That work may become slower to process, more expensive to recover, or eventually not worth processing at all.

So I think about queues less as storage for work and more as a bounded buffer between systems operating at different rates.

Use them to absorb bursts.

Use backpressure when the buffer fills.

Protect important work.

Control retries.

Remove stale work when the business semantics allow it.

Measure age and recovery, not just depth.

And always know what happens when the queue reaches its limit.

Because ultimately:

A queue doesn't give you more capacity.

It gives you time to respond to a capacity problem.

And sometimes that time is exactly what you need.

Sometimes it is just borrowed time.

Prompt Size Is Becoming an Architecture Metric

snehaup1997 — Wed, 30 Sep 2026 09:19:19 +0000

When we build AI systems, one instinct shows up again and again : if the model can use more information, give it more information.

You have a coding agent with access to an entire repository, so why not give it more files? A RAG pipeline retrieved ten documents, so why throw any of them away? The agent returned a useful tool result, so you append it to the conversation and carry it into the next iteration.

I have found this pattern surprisingly easy to ignore because nothing breaks immediately. The model still responds. The answer may still look reasonable. The context window still has capacity. That is exactly what makes the problem interesting.

Over time, an AI system can accumulate conversation history, retrieved documents, source code, logs, tool output, and intermediate results. Eventually, prompt construction itself becomes part of the system's performance profile.

We already know how to deal with this class of problem elsewhere.

Databases use indexes. Caches use eviction. Operating systems manage working sets. Distributed systems put bounds around queues and buffers.

I think AI systems need the same discipline - context window is not a working set. A context window tells us how much information a model can accept. A working set tells us how much information the model actually needs for the current decision. Those are very different things.

Consider a coding agent working against:

2,400 files
180,000 lines of code

A persistence bug might involve:

OrderController
OrderService
OrderRepository
Order entity
One failing test
Relevant stack trace

Perhaps 6,000–8,000 tokens are enough.

The repository remains the source of truth. The model does not need the repository inside its context.

A better pipeline is:

Repository
    ↓
Search / Index
    ↓
Candidate files
    ↓
Symbol / Dependency analysis
    ↓
Relevant code slices
    ↓
Context builder
    ↓
LLM

This is fundamentally the same idea as database indexing. We don't scan an entire table into application memory just because the database contains the data. We narrow the working set before processing it.

Long-context research gives us another reason to be cautious.

The Lost in the Middle study found that model performance can degrade when relevant information is buried in the middle of a long context. The conclusion is not that long context is bad. It is that capacity is not relevance.

The problem becomes even more obvious when the model is called repeatedly. Suppose an agent performs 100 model calls and each call carries an average of 40,000 input tokens. If better retrieval and context construction reduce the working set to 12,000 tokens. That is a 70% reduction in input-token volume. The exact financial impact depends on the model, provider pricing, caching, and workload. The architectural point is :

Every unnecessary token can be multiplied by every model invocation that carries it.

This matters particularly in agentic workflows, where one user request can result in dozens of model and tool interactions.

There is another boundary I think AI architectures need to make explicit: the boundary between tool output and persistent context.

Suppose an agent executes:

run_tests()

and receives 15,000 tokens of output.

A naïve implementation does:

Tool
 ↓
15,000 tokens
 ↓
Conversation history
 ↓
LLM

A better design is:

Tool
 ↓
Raw output
 ↓
Filter / summarize
 ↓
Relevant findings
 ↓
Context builder
 ↓
LLM

The tool response is an intermediate result that may contribute to context. It should not become persistent context by default.

A test runner might produce thousands of lines containing successful tests, timestamps, warnings, stack traces, and environment details. The model may only need:

3 failing tests
2 relevant stack traces
5 changed files

The same principle applies to database queries, APIs, code search, logs, and browser results. Tool interfaces should therefore support filtering, pagination, field selection, structured responses, and bounded results wherever possible.

This is becoming a real design concern for agent platforms. Anthropic, for example, has documented tool-use patterns specifically aimed at avoiding the cost of loading large tool definitions and intermediate tool results into the model's context. Its Tool Search approach allows tools to be discovered and loaded on demand rather than putting every tool definition into context upfront.

A tool that routinely returns 50,000 tokens to an agent is not only a tool-design problem. It is a context-architecture problem.

Once context is treated as a working set, another question becomes important:

What is allowed into it?

I would treat context construction almost like a cache admission policy. For every candidate piece of information:

Is it relevant?
Is it authoritative?
Is it current?
Is it duplicated?
Will the model need it for the next decision?

That gives us:

Candidate information
        ↓
Relevance
        ↓
Deduplication
        ↓
Freshness
        ↓
Priority
        ↓
Token budget
        ↓
Model context

This also gives information a lifecycle.

A system constraint may remain relevant throughout a session. A retrieved document may matter for one decision. A raw database response might only be useful for one tool call. A 10,000-line build log might ultimately produce one useful fact. Treating all of these as permanent conversation history is effectively an eviction policy of never evict which is rarely good for any resource.

So How Do We Reduce Context?

Once the problem is framed this way, the controls become fairly mechanical.

Retrieve Less, but Retrieve Better

Don't retrieve 20 documents and ask the model to decide which three matter. Use metadata filters, hybrid search, reranking, namespace boundaries, and query-specific retrieval before constructing the prompt. The goal isn't simply a smaller top-k. It is a better signal-to-noise ratio.

Slice Code Instead of Retrieving Files

For coding agents, retrieving an entire 2,000-line class because one method is relevant is unnecessary. Use symbol-level retrieval, AST-aware slicing, dependency analysis, or line-range extraction where possible.

2,000-line file
      ↓
Relevant method
      ↓
Callers / dependencies
      ↓
Required context

The model gets the code it needs rather than the file that happens to contain it.

Summarize Tool Output Before It Becomes Context

Treat tool output as an intermediate representation.

Raw output
    ↓
Parse
    ↓
Filter
    ↓
Extract findings
    ↓
Context

Do not make the LLM perform work that deterministic software can perform first.

Compact Conversation State

Long-running agents accumulate history even after information has served its purpose. Instead of carrying every previous turn forward:

Raw conversation
      ↓
Decisions
Constraints
Open questions
Current state
      ↓
Compact state

The objective is to preserve state, not transcript. This is already becoming a standard technique in production agent architectures. OpenAI provides server-side and standalone context compaction for long-running conversations, while Anthropic describes compaction and structured note-taking as mechanisms for maintaining coherence across long-horizon tasks.

Put a Budget Around Context Sources

I would rather have explicit limits than discover context growth through production latency.

For example:

Conversation history:     4K
Retrieved documents:      8K
Tool output:              4K
Source code:              8K
System instructions:      2K
────────────────────────────
Input budget:            26K

The numbers are workload-specific. The principle isn't. Every source competing for context should have a budget and a priority.

The Tradeoff

There is an important tradeoff here. Reducing context is not automatically an optimization.mIf I aggressively trim a prompt and remove the one piece of information that explains a subtle business rule, the model may produce a faster answer that is simply wrong. So I don't think the goal should be:

Minimize context.

It should be:

Maximize useful information within a bounded context budget.

A retrieval system that returns only highly relevant documents but misses critical supporting information has poor coverage. A system that retrieves everything has poor selectivity. The engineering problem is finding the point where the model has enough information to make the decision without carrying everything that might be useful and that is why I would give different context sources different priorities:

System constraints        → mandatory
Current task              → mandatory
Relevant source code      → high
Authoritative documents   → high
Recent tool findings      → medium/high
Conversation history      → medium
Raw tool output           → low
Duplicate information     → discard

The exact ordering depends on the application but the important part is that the system has a policy rather than relying on the model to make the decision implicitly.

Once context has a budget, something has to enforce it. I would put context construction behind an explicit budget manager:

Candidate Context
       ↓
   Deduplicate
       ↓
Classify / Prioritize
       ↓
Per-source budgets
       ↓
Global budget
       ↓
┌──────────────────────────┐
│ Fit?                     │
│                          │
│ Yes → Build context      │
│ No  → Trim / Summarize   │
│       / Retrieve again   │
└──────────────────────────┘
       ↓
      LLM

There is another detail that is easy to miss - Input tokens and output tokens share the model's total context capacity.

If a model supports a 32K total window, allocating all 32K to input leaves no room for the response. So I would calculate:

output_reservation =
    expected_output + output_buffer

input_hard_limit =
    model_limit
    - output_reservation
    - safety_margin

For example:

Model limit:               32K
Expected response:          4K
Output buffer:              1K
Safety margin:              1K
──────────────────────────────
Hard input limit:          26K

The reservation should be dynamic.

A classification request might need only 500 output tokens, while a coding-agent step generating a patch may need 6,000. The same model can therefore have different input budgets for different operations. I would also distinguish between soft and hard limits:

0 ───────────── 20K ───────────── 26K ───── 32K
                │                  │
             Soft limit         Hard limit
                │                  │
          Optimize / trim       Never cross

The soft limit is an operating target. Crossing it should trigger more aggressive filtering or compaction. The hard limit is an enforcement boundary. The system should never send a request above it.

The hardest case is mandatory context that already exceeds the available input budget. Suppose:

Hard input limit:           26K

Mandatory context:
System constraints:         5K
Current task:               4K
Security policy:            3K
Required task state:       18K
──────────────────────────────
Total:                     30K

There is nothing meaningful to evict. The system should not silently truncate it and hope the model can recover. In such cases I would use an explicit escalation path:

Mandatory context > hard limit
             ↓
Try deterministic compression
             ↓
Compact structured state
             ↓
Split / stage the task
             ↓
Retrieve incrementally
             ↓
Still too large?
             ↓
Fail explicitly

For example, 18K of task state might contain verbose conversation history that can be converted into 7K of structured facts, decisions, constraints, and unresolved questions. If that makes the request fit, proceed. If the information is genuinely irreducible, stop. I would rather return:

CONTEXT_BUDGET_EXCEEDED

Required input:       30K
Available input:      26K

Suggested action:
- compact task state
- split the task
- retrieve information incrementally

than silently remove information classified as mandatory.

That gives us a useful rule:

Optional context can be evicted. Mandatory context must be compressed, partitioned, or cause the workflow to stop.

The policy itself can be implemented deterministically:

function buildContext(candidates, modelLimit, expectedOutput):

    outputReservation = expectedOutput + OUTPUT_BUFFER

    hardInputLimit =
        modelLimit - outputReservation - SAFETY_MARGIN

    candidates = deduplicate(candidates)

    mandatory = candidates.filter(isMandatory)
    optional  = candidates.filter(notMandatory)

    if tokenCount(mandatory) > hardInputLimit:

        mandatory = compact(mandatory)

        if tokenCount(mandatory) > hardInputLimit:
            mandatory = partitionForStagedExecution(mandatory)

        if tokenCount(mandatory) > hardInputLimit:
            return CONTEXT_OVERFLOW

    remaining =
        hardInputLimit - tokenCount(mandatory)

    optional = prioritize(optional)

    optional =
        admitUntilBudget(optional, remaining)

    context = mandatory + optional

    if tokenCount(context) > hardInputLimit:
        return CONTEXT_OVERFLOW

    return context

The important part is the order.

Mandatory context is protected first. Optional information competes for whatever budget remains.

And once prompt construction sits on the runtime path, I would measure it like any other resource. At minimum:

input_tokens / model_call
retrieved_tokens / request
tool_output_tokens / request
conversation_tokens / request
context_growth / agent step
P50 / P95 / P99 context size
context_truncation_rate
context_budget_utilization
context_overflow_rate

I would particularly watch P95 and P99 context size. An agent that normally operates around 12K tokens but occasionally reaches 120K has a very different architecture from one that consistently stays near 12K, even if their averages look similar. It is also useful to know where those tokens came from:

Context
├── Conversation history    18%
├── Retrieval               32%
├── Tool output             27%
├── Source code             19%
└── System instructions      4%

Now "the prompt is too large" becomes an engineering problem with a location. If tool output dominates, fix the tool boundary. If conversation history dominates, introduce compaction. If retrieval dominates, improve ranking and filtering. If source code dominates, make retrieval more code-aware. The value of these metrics is not in measuring context for its own sake. They tell us where the architecture needs to change and this leads to the architecture I would build: one where information can exist broadly across the system, but only a deliberately selected subset is allowed into the model's active context.

The model sits at the end of that pipeline, not at the center of the storage layer. Repository data can remain in the repository. Documents can remain in the retrieval system. Raw logs can remain in observability infrastructure. Tool results can remain transient until their findings are extracted. The context builder decides what crosses the boundary.

That separation also makes the system easier to reason about. If context grows unexpectedly, we can identify whether the problem is retrieval, tool design, conversation state, code selection, or budget enforcement instead of treating the entire prompt as one opaque string.

What I find interesting is that this isn't just a theoretical architecture anymore. The industry has started to treat context this way now.

OpenAI now exposes context compaction specifically for long-running interactions, reducing accumulated context while preserving state needed for subsequent turns.

Anthropic explicitly describes context engineering as the broader problem of curating the information available to a model, including system instructions, tools, external data, and message history. Its guidance covers just-in-time retrieval, compaction, structured note-taking, and sub-agent architectures.

Anthropic's Tool Search is another concrete example of the same idea. Instead of putting thousands of tool definitions into context, tools can be discovered and loaded when they are actually needed. Anthropic reports that this can substantially reduce context consumption in large tool environments.

Google takes a slightly different approach with long-context models and context caching. Gemini supports very large context windows, while its caching mechanisms allow repeatedly used context to be reused rather than repeatedly processed as ordinary input. Google's documentation also explicitly notes that longer contexts can increase latency and that unnecessary tokens should generally be avoided.

These are different mechanisms solving different problems.

Retrieval controls what enters the working set.
Compaction controls accumulated state.
Caching reduces repeated processing of stable context.
Tool discovery and filtering control what external systems contribute.

The implementation differs, but the architectural direction is remarkably similar - context is becoming something applications need to manage deliberately.

Key Takeaway

The model should not become the database, repository, cache, or log store. It should be the reasoning component operating over a deliberately constructed working set.

My simple rule is:

Store broadly. Retrieve selectively. Admit deliberately. Reason narrowly. Persist the result.

AI systems have given software architects another resource to manage, not because context windows are small, but because they are now large enough for poor context management to hide inside them. And that leads to the principle I keep coming back to:

A bigger context window gives you more capacity. It doesn't give you a reason to fill it.

References

Liu et al., Lost in the Middle: How Language Models Use Long Contexts — arXiv paper
OpenAI — Context Compaction
Anthropic — Effective Context Engineering for AI Agents
Anthropic — Advanced Tool Use
Google AI — Context Caching
Google AI — Long Context

Mutlimodal AI

snehaup1997 — Mon, 13 Oct 2025 14:21:00 +0000

Prompt Engineering vs Prompt Tuning

snehaup1997 — Thu, 18 Sep 2025 07:48:58 +0000

We’re living in the era of large language models (LLMs) — and the way we interact with AI has completely changed. Instead of writing algorithms line by line, we now talk to our machines. We guide them, not through code, but through prompts — simple lines of text that can unlock incredibly complex behaviors.

It’s a bit like having a conversation with intelligence itself. Whether you’re a machine learning engineer building custom tools or an experimenting data scientist, you’re already shaping how AI thinks and responds through your choice of words. This process, known as prompt design, is quickly becoming one of the most important skills in modern AI development.

As this space evolves, two main approaches to customizing LLMs have taken the spotlight: Prompt Engineering and Prompt Tuning. Both aim to get the best out of AI models — to make them faster, smarter, or more reliable — but they work in very different ways. Prompt Engineering is about crafting better instructions, while Prompt Tuning goes under the hood, adjusting how the model itself interprets those instructions.

So that brings us to the big question: where does the real power lie? Is it in the creativity of the prompt, or in the precision of the tuning?

Let’s dig a little deeper and break down the basics before we decide.

Prompt Engineering — the art of conversation; is all about how you talk to the model. Think of it as crafting the perfect question or instruction to get the answer you want. It’s less about code and more about communication. It's the art (and science) of writing effective instructions, examples, and contextual clues in natural language.

A good prompt can completely change the outcome. A vague prompt might leave the model confused, while a well-structured one can lead to precise, insightful, or even creative results. You can think of it as teaching by example — you show the model what kind of response you expect through tone, structure, and context. For instance,

✅ Prompt: Summarize this article in 2 sentences. Be concise but cover key facts.
❌ Prompt: Make this shorter.

It’s quick, flexible, and doesn’t require retraining the model. That’s why prompt engineering has become the go-to approach for most users — it’s accessible to anyone who can think clearly and ask good questions. You can experiment freely, iterate quickly, and often get surprisingly strong results without touching the underlying model. But, like any tool, it has its limitations. Prompt engineering can be brittle and inconsistent — even small changes in wording can lead to drastically different outputs. It can be challenging to scale or automate for large workloads, and when it comes to highly specialized tasks or adapting to specific domains, it sometimes struggles to produce reliable results.

In short, prompt engineering is a bit like “command-line AI.” It’s fast, lightweight, and perfect for prototyping, experimentation, or casual use — but it’s not always robust enough for complex, high-stakes, or large-scale applications. And that’s where Prompt Tuning enters the picture. When crafting the perfect prompt isn’t enough, prompt tuning lets you go a step deeper — shaping the model’s behavior from the inside out. Instead of just telling the AI what to do, you’re influencing how it thinks and responds, making it more consistent, reliable, and tailored to your specific needs.

Prompt Tuning - fine tuning under the hood; is a more technical, machine-learning–centric approach. Rather than writing natural language instructions, it works by creating task-specific “soft prompts” — essentially trainable embeddings that are prepended to the input tokens of an LLM. These embeddings guide the model toward the desired behavior without changing all of its parameters. In fact, unlike full fine-tuning (which updates every parameter in the model), prompt tuning often adjusts less than 1% of the model’s weights, making it lighter, faster, and easier to manage.

Think of it as giving the AI a kind of memory or personality. Even if you phrase your prompts differently, the model retains its trained behavior.
The trade-off is that prompt tuning requires a bit more technical know-how than prompt engineering. You need to understand how to train the soft prompts, select good example data, and integrate them effectively. But in return, you get an AI that is reliable, repeatable, and highly specialized, capable of handling complex tasks without depending solely on carefully worded prompts.

Suppose a company wants an AI that drafts customer support emails in a friendly yet professional tone. With prompt engineering, you’d have to carefully phrase every prompt to maintain the tone. With prompt tuning, you train the model on a few examples of the desired tone, and then it consistently produces emails in that style, even when the input varies.

So, when should we use prompt engineering — and when prompt tuning?

Prompt Engineering excels at experimentation and exploration. Prompt Tuning shines in precision and production. Both have their strengths — but their real value depends on the context. The table below summarizes how they stack up in practice:

Use Case	Preferred Approach	Reason
Rapid prototyping, exploration	Prompt Engineering	Faster iteration, no training needed
Domain-specific style enforcement, scaling	Prompt Tuning	Stable behavior, predictable output
Resource-constrained deployment	Prompt Tuning	Minimal parameter updates, lower memory cost
High-stakes tasks (legal, medical)	Prompt Tuning (or full fine-tuning)	Harder to “break” via prompt variation

Striking the Balance
Most powerful results rarely come from using just one approach. The magic often happens when both are combined as the hybrid approach allows creativity upfront and consistency downstream. By blending the two, you can move from a trial-and-error experimentation to a polished system that scales beautifully -_ prompt engineering is the conversation, the examples, the guidance you provide in real time and prompt tuning is the memory — the part that remembers and applies those lessons consistently. Together, they create an AI that is both *responsive and dependable.

Red Teaming for Responsible AI

snehaup1997 — Mon, 30 Sep 2024 15:19:37 +0000

As artificial intelligence (AI) technologies continue to evolve at an unprecedented pace, ensuring their responsible development and deployment becomes crucial. Along with AI's potential to bring about significant change comes the responsibility to confront different vulnerabilities and ethical considerations that may arise.

"Red Teaming", an effective strategy for ensuring AI systems are robust and ethically sound involves simulating potential threats and challenges to reveal weaknesses, providing a deeper understanding of how AI systems perform under adverse conditions. In this article, we will explore the concept of red teaming in detail, highlighting its significance within the broader framework of responsible AI.

What is Red Teaming?

Red teaming originated in military strategy as a method to test defenses by simulating an adversary's tactics. This concept has evolved and been adapted across various fields, most notably in cybersecurity, where red teams conduct simulated attacks to identify vulnerabilities in systems. In the context of artificial intelligence, red teaming involves assessing AI models & systems to uncover potential flaws, biases, thereby preventing unintended consequences. By simulating various scenarios that could challenge the integrity and functionality of AI, red teaming provides a rigorous framework for evaluating the reliability and ethical considerations of these technologies, ultimately contributing to their responsible development and deployment.

Red Teaming vs Penetration vs Vulnerability Assessment

Red teaming, penetration testing, and vulnerability assessment are distinct approaches to evaluating security, each serving specific purposes. Red teaming simulates real-world attacks to identify and exploit vulnerabilities, providing a comprehensive view of an organization's security posture and testing its defenses under realistic conditions. Penetration testing focuses on actively probing systems for weaknesses, often with defined scope and limitations, to assess the effectiveness of security measures. Vulnerability assessment, on the other hand, involves identifying and classifying security weaknesses within a system or network, usually through automated tools, without actively exploiting them.

In summary, red teaming provides a holistic and adversarial view of an organization's security, penetration testing focuses on targeted exploitation within defined boundaries and vulnerability assessment offers a broad overview of potential weaknesses without active testing.

Why is Red Teaming Important for AI?

There are several key reasons why red teaming is essential for AI; namely -

Identifying Vulnerabilities: AI systems can harbor hidden biases or vulnerabilities that may lead to unintended harm. Red teaming helps uncover these issues before the technology is deployed.

Example: While assessing an AI recruitment tool, red team finds that the model favours candidates from certain universities due to biases in historical data, risking unfair hiring practices. By identifying this issue, they recommend adjustments to the training data and algorithm, promoting fairness and inclusivity in hiring.

Enhancing Security: By simulating potential attacks, red teams can help organizations strengthen the security of their AI systems against malicious actors.

Example: A financial institution uses an AI algorithm to detect fraud. The red team simulates an attack with crafted transaction data and discovers the AI misses certain fraudulent patterns. With these insights, the organization updates the model to include diverse scenarios, enhancing its defenses and improving fraud detection.

Promoting Ethical Use: Red teaming can reveal ethical dilemmas or harmful implications of AI systems, ensuring that their deployment aligns with societal values and standards.

Example: A healthcare provider develops an AI tool to prioritize patient treatment. The red team finds that the algorithm favors younger patients, raising ethical concerns. By addressing this, the organization adjusts the model to prioritize treatment based on medical need, ensuring fairness and adherence to ethical standards in patient care.

Improving Trust: Demonstrating that an AI system has undergone thorough scrutiny can enhance public trust in AI technologies, leading to broader acceptance and use.

Example: A city implements an AI system for traffic management. After addressing the issues discovered by red team, the city publicly shares the results of the testing and the measures taken. This transparency demonstrates the system's reliability and fairness, leading to increased public confidence in the technology, encouraging its acceptance for use in urban planning and management.

How Does Red Teaming Work?

Red teaming consists of several key steps, starting with defining objectives that set clear goals, such as testing for biases or security vulnerabilities. Following this, red teams simulate scenarios that mimic potential attacks, challenging the AI with atypical data inputs. After conducting these tests, the outcomes are analysed and findings are compiled into a report, which include recommendations for mitigating identified risks. The assessment concludes with the red teaming collaborating with the AI development team to implement necessary changes based on their findings.

Types of Attacks in AI Red Teaming

AI red teams utilize various tactics to assess the robustness of AI systems. Common attack vectors include:

Prompt Attacks: Designing malicious prompts to manipulate AI models into producing harmful or inappropriate content.
Data Poisoning: Inserting adversarial data during the training phase to disrupt the model's behavior.
Model Extraction: Attempting to steal or replicate the AI model, which can lead to unauthorized use.
Backdoor Attacks: Modifying the model to respond in a specific manner when triggered by certain inputs.
Adversarial Examples: Crafting input data specifically to mislead the AI model.

Red Teaming Assessment Tools

The market for red teaming assessment tools is diverse, offering various solutions to enhance the security and ethical use of AI systems. Tools like Burp Suite and OWASP ZAP focus on web application security, while Google Cloud AutoML and IBM Watson OpenScale address biases and performance monitoring in AI models. Platforms like HackerOne enable organizations to crowdsource red teaming efforts, bringing in external expertise. By leveraging these tools, organizations can proactively identify vulnerabilities and biases, ensuring their AI applications are secure, reliable, and aligned with ethical standards.

Challenges and Considerations

Effective red teaming requires skilled personnel and resources, which can be challenging for smaller organizations. Organizations must balance transparency with security to safeguard against vulnerabilities.

Responsible AI 101

snehaup1997 — Mon, 30 Sep 2024 15:13:19 +0000

The conversation about Responsible AI has gained considerable momentum across different sectors, yet a universally accepted definition is still hard to pinpoint. Many people view RAI mainly as a tool for risk mitigation, but its reach goes much further. It involves not only managing risks and complexities but also the capacity to transform lives and improve experiences.

This article explores key principles that ensure AI technologies are developed and deployed ethically.

Core Principles of Responsible AI

Responsible AI practices focus on fairness, accountability, transparency, and privacy, ensuring that AI systems operate without bias, honor user rights, and are held accountable for their outcomes.

Let us understand these principles in the scenario of Hiring Algorithms.

a. Fairness:
AI systems must be designed to treat all individuals and groups fairly by identifying and addressing biases in training data to prevent discrimination based on any protected characteristics. Incase of a Hiring Algorithm the AI needs to be trained on a variety of datasets to prevent it from favoring any particular demographic, such as selecting candidates solely based on specific genders or backgrounds due to historical successes or other trends.

b. Transparency:
A hiring algorithm assesses candidates using specific criteria, but applicants are not informed about how these criteria are set. To improve transparency, the company could publish an internal report on the algorithm's operations and criteria, preparing the organization to address any applicant challenges regarding the decision-making process.

c. Accountability:
Organizations should be ready to address the effects of their AI decisions and have processes in place for recourse. If a candidate is unfairly rejected due to biased algorithmic decisions, there should be a clear grievance process.

d. Privacy:
Respecting user privacy is paramount. During the hiring process information like LinkedIn profiles maybe required. To safeguard privacy, the company should restrict the algorithm's data collection to what is essential and ensure that information is securely stored and used.

e. Inclusivity:
A hiring algorithm that focuses more on experience than potential might miss promising candidates. Designing algorithms that consider diverse candidate backgrounds and experiences, help create a more representative hiring process.

f. Robustness:
If an algorithm is meant to identify the best candidates but struggles with unconventional profiles, it may lead to poor results. To improve its robustness, the company could perform tests to see the algorithm's adaptability.

Git-it!

snehaup1997 — Mon, 09 Oct 2023 11:33:55 +0000

Git is a version control system that developers use all over the world. It helps you track different versions of your code and collaborate with other developers.

If you can read only one chapter to get going with Git, you've come to the right place. This blog includes all the fundamental commands that cover most of the tasks you will be doing with Git. Once you finish reading this, you will be capable of setting up and initializing a repository, tracking files, and making commits.

git clone [repo_url]:
Creates a copy of a remote Git repository on your local machine.
git init:
Creates an empty Git repository or reinitializes an existing one
git add [name_of_modified_file] or git add . :
The specified file or all files in case of a dot are stages for commit
git commit -m "your commit message":
Commits the staged changes with the mentioned message.
git checkout [your_branch_name_goes_here]:
Switches to the specified different branch.
git status:
Shows the status of your working directory. This including changes and untracked files.
git pull:
Merged latest changes from a remote repo into the current branch.
git push:
Pushes your local commits to a remote repository.
git log:
Displays a log of all commits in the current branch.
git branch:
Lists all branches in your repository and highlights the current one.
git merge [branch_name]:
Merges changes from one branch into another.
git stash:
It temporarily saves the changes that are not ready to be committed yet
git diff [file]:
Enlists differences between working directory & last commit.
git remote add [name] [repository_url]:
Adds a remote repository with a specified name.

Memory management in JAVA

snehaup1997 — Fri, 06 Oct 2023 11:47:43 +0000

Picture yourself with a bottle of water that you consume from throughout the day. It is inevitable that unless you replenish the bottle, there will come a point in the day when it will be completely devoid of water. This concept also applies to memory and anything else that has a limited lifespan.

JVM designates memory whenever we create new variables, objects, call a method. If we mindlessly just keep on using memory without freeing it, we are bound to encounter a java.lang.OutOfMemoryError exception. Usually, this error is raised when there is not enough space in the Java heap to allocate an object. It can also occur when there is a lack of native memory to support class loading. The solution - Memory management

Memory management refers to the process of assigning resources for new objects & removing unused objects to free up space for new allocations. In Java, memory management is handled automatically, eliminating the need for us to implement intricate logic in our application code.

Understanding the key features of Java like platform independence, object life cycle, concurrency, security, libraries, memory management etc. allow us to maximize its offerings and also write clean and efficient code. Additionally, as JVM is the foundation for other Java-based programming languages, acquiring knowledge about Java internals assists in working with those programming languages.

Having set the context, let's understand various blocks of memory in Java.

Stack Memory is responsible for static memory allocation & executing threads. Whenever a new method is called, a new block is added on top of the stack, containing specific values for that method. After the method finishes executing, its corresponding stack frame is cleared & the program returns to the calling method.

Advantages :

Each thread has its own stack area in stack memory, ensuring thread safety.
Memory allocation and deallocation processes are faster.
Accessing stack memory is quicker.

Disadvantages :

Stack memory is fixed and cannot be resized once created.
It follows a Last-In-First-Out (LIFO) approach, making random access impossible.
Lacks scalability and flexibility.

Heap in Java is a shared chunk of memory that is created when the JVM starts up. It can be fixed or variable & does not need to be contiguous. It is used for dynamically allocating memory for Java objects and JRE classes during the execution of a Java program & is divided in three generations: young, old & permanent generation.

Young generation is where newly created objects are allocated. It consists of three sub-parts: Eden, Survivor1, and Survivor2. Objects are initially allocated in Eden. When Eden becomes full, a minor garbage collection occurs and the live objects are moved to Survivor1, and then to Survivor2. Therefore, we can say that Survivor1 and Survivor2 hold objects that survived the minor garbage collection.

Objects allocated in the young generation are assigned an age, and when that age is reached, they are moved to the old generation. Typically, long-surviving objects are stored in the old generation. A major garbage collection is performed on the old generation to collect dead objects.

The permanent generation is used by the JVM to store metadata about classes and methods, including the Java standard libraries. This space is cleaned during a full garbage collection.

Advantages :

It is not fixed in size and can grow and shrink as needed.
Allows for random access.

Disadvantages :

It is shared across threads therefore not thread-safe.
Accessing heap memory is slower compared to other.
Process of allocating and deallocating memory in the heap is more complex compared to stack memory.

Having understood the java memory model let's now delve into the concept of garbage collection (gc), the feature that enables java to manage memory automatically and efficiently. The aim of gc is to find unused objects and then eliminate or delete them in order to free up memory. An object is considered eligible for Garbage Collection under the following conditions:

It is not being used by any program or thread.
It has no static references or the references are null.
The object is created within a block and once the control exits that block, the reference goes out of scope.

GC is controlled by a thread known as the Garbage Collector. Java provides two methods, System.gc() and Runtime.gc(), for sending requests to the JVM for garbage collection. It is important to remember that this is merely a request and it is not guaranteed that garbage collection will occur. When the garbage collector removes an object from memory, it first calls the finalize() method of that object before removing it.

Garbage collection in Java employs a mark-and-sweep algorithm.

When a Java object is created in the heap, it initially has a mark bit set to 0 (false). During the mark phase, the garbage collector traverses object trees starting from their roots. If an object is reachable from the root, its mark bit is set to 1 (true). Objects that are unreachable will have their mark bits remain unchanged. During the sweep phase, the garbage collector scans the heap, reclaiming memory from all items with a mark bit of 0 (false).

Software Architecture as I know [always a WIP ;)]

snehaup1997 — Thu, 27 Oct 2022 15:52:19 +0000

Software architecture serves as a blueprint for a system. It is the organization of a system which includes all components - their interaction, operation with each other, environment, design principles used and the decisions made.

Why is it important?
Software architecture is the foundation of a software system which has a profound effect on the quality of what is built on top of it. A proper foundation yields number of benefits whereas sub-optimal decisions may later cause development, security, performance, scalability & maintenance concerns.

Different types of architecture patterns
Some common examples of architecture styles include Monolithic application, Layer based, Client-server pattern, Event-driven, Microservices etc.

Each pattern pertains to specific characteristics and behavior - some contribute to scalability whereas others are more agile. Knowing these strengths & weaknesses of each architecture pattern is necessary to choose the one that meets your use case.

Layered Architecture

Layered architecture (also known as the N-tier architecture) is the de-facto standard for designing majority of software. Here the codebase is separated into layers - each with a specific role & independent of the other.

Challenges/Limitations:

Defending boundaries.
Tight coupling leading to complex interdependencies.
Without proper coordination between team members, source code can turn into a mess.

Microkernel Architecture

Microkernel pattern has two major components - a core system and plug-in modules. The core handles fundamental and minimal operations whereas the plug-in modules handle the extended functionalities. IDEs are the best example for this pattern.

Challenges/Limitations:

not highly scalable.
changes to core system will result in changes to plug-ins.

Event-driven Architecture

EDA enables an application to detect “events” like mouse hovers, button clicks and act on them. This asynchronous communication replaces the request/response architecture where services would have to wait for a reply before moving onto the next.

Challenges/Limitations:

Complexity of implementation
Anticipation of the unknown
Error-Handling

Microservices Architecture

In microservices architecture the application is developed as a collection of services which are loosely-coupled, fine-grained & communicate via lightweight protocols. This makes additions of features and modification of existing ones independent of others.

Challenges/Limitations:

Increased development time
Limited reuse of code

Pattern Analysis

A good software architecture should facilitate scalability, agility, addition of new features, integration with external APIs & easily maintainable.

Attached below is a table enlisting key attributes of each of the architecture pattern discussed that can help you determine which pattern to select based on your use case.

Common Gateway Interface (CGI)

snehaup1997 — Thu, 27 Oct 2022 14:29:27 +0000

Providing the middleware between servers, external databases and information sources, a gateway interface is the mode in which the web application interacts with the external world.

On typing https://dev.to/ your browser would redirect you to the homepage of DEV. But is that all you really use the web for?

As you traverse further in the world of World Wide Web, you'll come across documents that make you wonder, "How did they do this?"

In this example, there are two pieces of dynamic information: the alphanumeric address (IP name) of the remote user and the load average on the serving machine. This is a very simple example but how was it done?

A more common use case would be forms. when a user fills out a form on a Web page and sends it in, it usually needs to be processed by an application program. The Web server typically passes the form information to a small application program that processes the data and may send back a confirmation message. This method or convention for passing data back and forth between the server and the application is called the common gateway interface (CGI). It is part of the Web's Hypertext Transfer Protocol (HTTP).

CGI turns the Web from a simple collection of static hypermedia documents into a whole new interactive medium, in which users can ask questions and run applications