DEV Community: Adreet Gogoi

[Boost]

Adreet Gogoi — Mon, 21 Sep 2026 14:10:44 +0000

Remdore

Sep 21

How much traffic can a $6 server handle? I measured it: 9,737 requests a second

#performance #devops #webdev #nginx

6 min read

Capability, Constraint, and the Real Work of Building AI

Adreet Gogoi — Mon, 21 Sep 2026 08:14:20 +0000

Science fiction gave us the templates. Helpful machines that expand human possibility. Systems that turn calm competence into quiet catastrophe. Early researchers carried both visions—some projected rapid intellectual progress toward better outcomes for everyone; others began asking harder questions about control, goals, and what happens when the systems we build become harder to steer. Those questions never went away. They simply became more concrete.

In 2026 the conversation sharpened again when researchers close to the frontier started speaking more bluntly. Jacob Coxon, who had worked on pretraining at both OpenAI and Anthropic, resigned and said the leading labs were racing toward self-improving systems while “gambling with our lives.” He claimed many people inside the labs earnestly believe the technology could pose an existential threat within the decade. A colleague publicly put a personal estimate of greater than 10 percent on the chance of catastrophic outcomes in that window. These statements are not proof that any particular future is locked in. They are signals that the people closest to the work see real, non-zero chances of large-scale failure modes that current methods do not yet fully address.

At the same time, the systems themselves are already constrained by the physical world. Energy demand from data centers continues to climb. Firm, low-carbon power is scarce. Water for cooling, land, materials, and grid interconnection timelines all act as binding limits. Efficiency improvements in hardware and algorithms help, yet overall demand has so far grown faster. Tokens and subscriptions remain the visible economic layer, but underneath them sit harder parameters: the quality and reliability of the electricity used, the carbon and water intensity of each unit of useful output, the utilization of existing hardware, and the true cost of scaling. Human physical activity as a way to “earn” compute remains a marginal idea—interesting as a cultural or health incentive, negligible as an energy source at the relevant scale.

The picture that is emerging is neither pure utopia nor inevitable doom. It is a system under pressure, where capability keeps rising while multiple constraints tighten. One plausible shape looks something like this: frontier systems concentrate where firm, relatively clean power can be secured; most everyday AI runs on smaller, more efficient models closer to the user; training and inference become more carefully scheduled against grid conditions and resource intensity; pricing begins to reflect more of the real costs. Residual risk remains because alignment and control problems are not fully solved, and competitive dynamics still reward speed.

The real job

In that environment the actual work of AI and systems design looks less like pure research and more like operating a high-stakes production system. The habits that matter most are familiar to anyone who has kept complex software alive under load:

Understanding the blast radius. What breaks if this model, this training run, or this deployment goes wrong—and how far does the damage travel?
Making tradeoffs under pressure. Speed versus safety, capability versus energy cost, openness versus control. Every choice has costs that someone will eventually pay.
Knowing what can fail. Not just the optimistic path, but the realistic failure modes: misaligned goals, unexpected agency, resource competition, cascading effects on infrastructure or society.
Rolling back safely. Having the technical and organizational ability to slow, contain, or reverse course when evidence shows the system is behaving in ways that were not intended.
Taking ownership when production breaks. The people closest to the systems carry a responsibility that cannot be fully outsourced to regulation, public statements, or future research.

These are procedural disciplines. They do not guarantee any particular outcome. They raise the chance that capability can be expanded without quietly accumulating irreversible risk. They also force honesty about the limits of current methods. When insiders say the stakes feel civilizational, the productive response is not dismissal or panic. It is to treat the claim as a serious hypothesis that needs better evidence, better measurement, and better coordination—while continuing to ship useful systems under real constraints.

Parameters that actually move the needle

Energy quality matters more than raw quantity. Dispatchable low-carbon sources—nuclear restarts, small modular reactors, geothermal, hybrids of renewables and storage—will shape where the most capable systems can be trained and run. Efficiency across layers (specialized hardware, model design, operational scheduling, utilization) multiplies the value of every watt. Water, land, and materials act as co-equal constraints. Economics that begin to price intensity into the cost of tokens and access will change incentives. Measurement systems that track energy or carbon per unit of useful work, rather than tokens alone, create feedback that design teams can actually use.

None of these parameters eliminates risk. Together they define the feasible region in which capability can grow. The organizations that treat them as first-class design inputs—rather than externalities to be managed later—are more likely to scale without exhausting the physical and social license they depend on.

A working posture

The future of AI will probably look stratified and constrained rather than unlimited. Some systems will be extraordinarily capable and tightly coupled to large energy and compute resources. Many others will be smaller, more specialized, and more widely distributed. Progress on alignment and control will continue, but so will the competitive pressure that makes deliberate slowdowns difficult. Insider warnings about large-scale failure modes will keep surfacing because the underlying technical and incentive problems have not been fully resolved.

The useful stance is procedural rather than prophetic. Understand the blast radius of what is being built. Make the tradeoffs explicit. Catalog the ways things can fail. Design for safe rollback. Take ownership when the system behaves outside its intended envelope. Treat energy, water, efficiency, measurement, and control as co-equal design parameters alongside accuracy and speed.

Science fiction prepared the imagination for both the helpful partner and the uncontrolled optimizer. The present is teaching that both possibilities remain open, and that the difference between them will be determined less by grand declarations than by the daily discipline of systems that can fail—and of the people who choose to own that failure mode before it becomes irreversible.

[Boost]

Adreet Gogoi — Mon, 21 Sep 2026 07:51:39 +0000

Jem for The DEV Team

Sep 18

Join the Sanity Challenge: $2,500 in prizes for FIVE winners!

#sanitychallenge #devchallenge #agents #webdev

150

7 min read

The old SRE is dead.

Adreet Gogoi — Thu, 17 Sep 2026 10:58:00 +0000

If you’re in SRE, DevOps, Platform, or Cloud Engineering and still thinking in the old loop of

Monitor → Alert → Investigate → Fix,

you’re already behind the systems you’re supposed to keep reliable.

The traditional model assumed systems were mostly deterministic.

Something broke → you got an alert → you found the root cause → you fixed it.

That world is disappearing.

Here’s the new operating model that is quietly taking over:

Observe → Understand → Predict → Decide → Automate → Verify

This isn’t just a nicer flowchart. It’s a fundamental shift in how reliability is achieved.

1. AI workloads broke the old assumptions

LLMs, agents, and inference pipelines are not “experiments” anymore. They are production systems with non-deterministic behavior.

Traditional monitoring still works for CPU, memory, and latency.

It completely fails when the failure mode is:

The model started hallucinating more after a quiet dependency update
Token cost exploded because of a prompt change no one noticed
Output quality drifted even though every service is “green”
An agent started looping or making irrational decisions

You can no longer treat model accuracy, hallucination rate, output consistency, and cost-per-request as second-class metrics. They are now first-class reliability signals — the same way availability and error rate used to be.

2. Observability is no longer just visibility. It’s becoming the control plane

Classic monitoring answers: “CPU is at 90%.”

Modern observability must answer:

“Why did this start happening 17 minutes ago?

Which service or model version is involved?

Did a deployment, data drift, or external API change cause it?

What is the business impact right now?”

The future stack is not just Logs + Metrics + Traces.

It is:

Logs + Metrics + Traces + Topology + AI signals + Business context → one intelligence layer

When observability can reason across all of these, it stops being a passive dashboard and starts becoming the system that decides when and how to act.

3. SLOs must expand or they become irrelevant

A 99.95% availability SLO still matters.

But for AI systems it is no longer sufficient.

You now need multi-dimensional SLOs that include:

Accuracy / quality
Latency (including inference latency)
Reliability of the decision itself
Cost efficiency
Output consistency over time

Because AI systems can change their behavior after deployment without any code change, continuous verification becomes mandatory. You don’t just ship and monitor. You continuously test whether the system is still behaving the way it was supposed to.

The old SRE mindset optimized for keeping things up.

The new SRE mindset optimizes for keeping things correct, predictable, and economically sane in a world where the systems themselves are learning and changing.

This is not optional knowledge for people who want to stay relevant in reliability engineering.

This is the new baseline.

If you’re still only watching infrastructure metrics, you’re watching the wrong dashboard.

From Physical Racks to Intelligent Modules: How the Meaning of Infrastructure Has Fundamentally Shifted

Adreet Gogoi — Sun, 13 Sep 2026 08:54:41 +0000

Infrastructure used to be something you could walk up to, touch, and hear humming in a raised-floor room. Capacity planning dominated the conversation: How many servers? How much power and cooling headroom? How many years until the next major refresh? Today, for many architects, infrastructure is less about forecasting static capacity and more about composing modular building blocks intelligently—and extracting far more value from every single one of them.

That shift is not merely technological. It is a redefinition of what “infrastructure” even means.

Then: Infrastructure as Place, Asset, and Capacity Plan

In the early days of enterprise computing, infrastructure was concrete. Mainframes occupied climate-controlled rooms. Later, dedicated data centers became the physical manifestation of IT capability. Power, cooling, cabling, racks, switches, and servers were the primary concerns. Network design meant physical topology and latency budgets measured in meters of copper or fiber. Capacity planning meant ordering hardware months (sometimes years) in advance and living with those decisions for a long time.

Cost efficiency lived in procurement negotiations, power usage effectiveness (PUE), and the art of densifying racks without tripping thermal or structural limits. Reliability was engineered through redundancy at every layer. The architect’s job was largely to design a resilient place that could host applications and to size it correctly so it neither ran out of capacity nor sat wastefully oversized.

Ownership was clear. You bought or leased the hardware, controlled the physical security perimeter, and absorbed the constraints of that physical plant.

The Intermediate Layer: Virtualization and the First Real Modularity

Virtualization (and later containers and orchestration) began the great decoupling. The same physical server could host many logical machines. Storage became pools. Networks acquired overlays. Suddenly the unit of work moved upward from the individual server or switch to more flexible, shareable resources.

This period introduced the first meaningful tension in definition: was infrastructure the hardware, or the virtual resources carved from it? Most organizations answered “both.” Hybrid operating models emerged. The physical layer remained critical for latency-sensitive, regulated, or high-throughput workloads, while the logical layer accelerated delivery and improved utilization. Capacity planning was still important, but it started to operate at multiple levels—physical headroom underneath, and more elastic logical capacity on top.

Now: Infrastructure as Composable Modules and Intelligent System Design

Today the dominant mental model is infrastructure as a set of modular, programmable building blocks. Cloud platforms, software-defined everything, Infrastructure as Code, and platform engineering have turned what used to be long-lived assets into composable capabilities. The conversation has moved from “Do we have enough capacity?” toward “How do we assemble the right modules—with the right isolation, locality, performance, cost, and policy characteristics—for this workload?”

Yet the market conversation still often defaults to compute—more cores, more GPUs, bigger clusters. That focus is incomplete. The more interesting shift is happening inside the box and across the system: recent technologies and engineering practices are focused on extracting dramatically more useful work from the same GPU, the same server, or the same node. Better scheduling, smarter resource sharing, finer-grained isolation, improved memory and interconnect utilization, and careful co-design of software with hardware allow teams to balance and run more applications—or significantly heavier ones—on the same underlying compute.

This is fundamentally a system-design problem, not just a capacity problem:

From capacity forecasting to modular composition and density. Classic capacity planning still matters at the physical substrate (especially for AI clusters, high-density compute, or regulated environments). But for most teams the daily work is assembling and governing modules while maximizing what each module can deliver. Understanding the blocks deeply—their performance envelopes, coupling points, operational behaviors, and true cost drivers—matters more than simply adding more of them.
From raw compute to intelligent utilization. The same GPU or server can now support higher concurrent workloads, better multi-tenancy, or more demanding applications because the surrounding system (orchestration, networking, storage paths, scheduling, and application design) has become more sophisticated. Efficiency gains come from system-level thinking, not just faster silicon.
From static topology to intent, constraints, and balance. Modern designs declare desired outcomes (latency envelopes, data residency, blast-radius limits, cost ceilings, fairness across workloads) and rely on control planes and careful architecture to realize them. Balancing multiple heavy applications on shared infrastructure requires deliberate system design: isolation boundaries, priority and preemption policies, observability that surfaces contention, and architectures that avoid hidden coupling.
From pure technical sizing to multi-dimensional intelligence. Choosing and assembling modules now requires understanding not just raw capacity but utilization efficiency, carbon intensity, data-transfer economics, operational complexity, and the cognitive load on the teams that will run the system. Intelligence at the modular and system level means knowing which combinations create fragile contention and which create resilient, high-utilization platforms.

Physical infrastructure has not disappeared. Hyperscale data centers, specialized AI training clusters, and carefully engineered on-premises environments still demand deep hardware, power, cooling, and interconnect expertise. The difference is that these physical realities increasingly sit underneath a modular abstraction layer, and the highest leverage often comes from designing the system so that each physical or virtual block is used more completely and more intelligently.

What Has Not Changed

Some fundamentals remain stubbornly physical. Electrons need paths. Heat must be removed. Light travels at finite speed. Regulatory boundaries still force locality decisions. Large-scale AI infrastructure continues to push power delivery, cooling, and fabric design in ways that feel very much like classic data-center problems—only denser and more expensive.

The best practitioners keep one foot in each world. They can still design a resilient physical fabric when required, and they can also reason clearly about modular boundaries, interfaces, composition, and the system-level techniques that extract more useful work from every GPU and every server. They treat capacity not as the primary question but as one constraint among many that intelligent system design must satisfy.

A Working Definition for Today

Infrastructure is the set of modular capabilities—compute, storage, networking, identity, observability, and policy—required to run workloads reliably, securely, and economically. These capabilities are delivered through whatever combination of physical assets, virtual resources, and managed services best meets the constraints of the moment. The architect’s core responsibility is no longer primarily capacity planning or simply adding more compute. It is understanding the blocks deeply enough to compose them intelligently, designing the system so that the same compute power can productively support more (and heavier) applications, keeping failure domains contained, and evolving the platform without constant reinvention.

It is no longer primarily a place or a fixed capacity plan. It is a set of well-understood modules and the intelligent system design that assembles and balances them under real-world constraints.

The definition of infrastructure has expanded and become more abstract. The need for clear thinking about modularity, utilization, system design, and the physical realities that still underpin everything has not.

What does “infrastructure” mean in your organization today—and how much of the real leverage now comes from smarter system design rather than simply more compute?

AI Makes Us Faster. Systems Thinking Makes Us Better.

Adreet Gogoi — Fri, 11 Sep 2026 12:38:54 +0000

AI is changing the way we work. But I think the bigger change is happening somewhere else:

It is changing the way we need to think about technology.

For years, many of us learned technology from the top down.

Application → Database → Operating System → Virtual Machine → Infrastructure → Network.

You write code, deploy an application, connect a database, and only look deeper when something breaks. AI is slowly forcing us to look in the opposite direction. Because an AI workload is not just an application.

It can be:

AI model → GPU → Server → Network → Storage → Power → Cooling → Data Center → Physical Infrastructure

Suddenly, the layers underneath the application matter a lot more and this isn't only an infrastructure problem. It is becoming a technology problem for everyone.AI doesn't remove the system.

Imagine an application suddenly becomes slow.

The first thought might be:

"Something is wrong with the application."

But what if the database is waiting on storage? What if storage latency is caused by an overloaded host? What if the host is competing for resources? What if network congestion is involved?
What if the workload has simply outgrown the infrastructure?

AI can help investigate all of these possibilities incredibly quickly.

It can read logs. It can correlate symptoms. It can generate commands.It can suggest hypotheses.It can compare architectures.
It can even automate parts of the investigation.

But there is still one important question:

Does the person using AI understand what they are looking at?

That is where systems thinking becomes extremely valuable.

The new skill isn't just prompting. There is a lot of discussion around "prompt engineering."

But in technical work, I think something deeper matters.

System understanding.

A good engineer can tell AI:

what the architecture looks like
what changed
what is expected
what is actually happening
which metrics matter
which logs are relevant
what constraints exist
what has already been tested
what cannot be changed
and what the risk of a wrong decision is

That creates a very different interaction with AI.

Instead of:

"My server is slow. Fix it."

You can say:

"The application latency increased after deployment. CPU is normal, memory is stable, but database queries are showing increased I/O wait. Storage latency has also increased on the underlying VM host. Here are the relevant metrics and logs. What are the most likely failure paths, and what evidence would distinguish them?"

Now AI becomes much more useful.

Because context is intelligence.

For systems, network and DevOps engineers, this doesn't mean everyone needs to become an expert in everything.

A systems engineer doesn't need to become a database specialist.

A network engineer doesn't need to become a kernel developer.

A software engineer doesn't need to become a data-center engineer.

But understanding how the pieces interact is becoming increasingly important.

For example, engineers working around infrastructure should increasingly be comfortable thinking about:

TCP/IP and network behaviour
DNS, routing and load balancing
Linux internals and resource limits
CPU, memory and I/O behaviour
virtualization and containers
storage performance and failure modes
databases and replication
observability and telemetry
distributed systems
cloud architecture
security boundaries
automation and IaC
CI/CD pipelines
power, cooling and physical capacity
and increasingly, GPUs and AI infrastructure

You don't need mastery of every layer.

You need to understand where your layer ends and another one begins.

AI is also creating infrastructure

There is an interesting contradiction happening.

AI can automate parts of existing jobs.

At the same time, AI is creating completely new technical requirements.

More models mean more GPUs.

More GPUs mean more servers.

More servers mean more networking, storage, power and cooling.

That creates demand for new infrastructure, new platforms, new tooling, new observability systems, new security models and new engineering roles.

So the question isn't simply:

"Will AI take jobs?"

The harder question is:

"Can new human demand and new economic activity grow fast enough to balance the work that automation removes?"

That's not purely a technology question.

It's an economic one.

Productivity can increase dramatically, but people still need income and purchasing power to create demand for the products and services that new technology enables.

Automation changes supply.
Human purchasing power creates demand.

The balance between the two matters.

And this is where the conversation becomes much bigger than "AI versus humans."

So where does the human fit?

Maybe the future isn't about humans doing everything manually.

And it probably isn't about AI doing everything either.

It may be about human interpretation + machine acceleration.

AI can generate ten possible explanations in seconds.

The engineer has to determine which one actually makes sense.

AI can generate a command.

The engineer has to decide whether running it in production is safe.

AI can propose an architecture.

The engineer has to understand whether it fits the real system, budget, constraints and failure scenarios.

AI can analyse thousands of lines of logs.

The engineer still needs to know what question to ask.

That's the gap.

Not necessarily intelligence.

Context. Responsibility. Judgment.

"Jitna tez AI, utni tez galti bhi ho sakti hai."

There is a simple idea here:

If you don't understand the system, AI can simply help you make mistakes faster.

That's why technical fundamentals aren't becoming obsolete.

They may actually become more valuable.

The engineer who understands systems can use AI as a force multiplier.

The engineer who doesn't may simply generate more things they don't understand.

The direction I see

Perhaps we should stop thinking about technology as isolated layers.

Instead, think about it as one connected system:

Physical → Power & Cooling → Hardware → Network → Storage → OS → Virtualization → Platform → Database → Application → AI

You don't have to become an expert at every level.

But knowing that these levels exist—and understanding how failures travel between them—is becoming increasingly important.

Because a slow application might not be an application problem.

A failed AI workload might not be a model problem.

A network issue might not be a network problem.

And an infrastructure problem might eventually turn out to be a physical capacity problem.

What you see is often just the surface.

The deeper you understand the system, the better questions you can ask.

And the better questions you ask, the more useful AI becomes.

Maybe that's the real shift.

AI makes us faster.
Systems thinking makes us better.

The future may not belong to people who know every technology.

It may belong to people who understand how the technologies connect—and know when to let the machine help, and when to question it.

When a Tableau TSM HTTP 500 Wasn't Really an HTTP Problem

Adreet Gogoi — Thu, 10 Sep 2026 17:33:20 +0000

A real-world troubleshooting journey through Tableau Server, ZooKeeper, systemd, and a misleading No space left on device error.

Sometimes infrastructure incidents are easy.

A service is down.
A port is closed.
The logs tell you exactly what happened.

And then there are incidents where everything looks almost healthy.

That was the case with a Tableau Server environment I was troubleshooting.

The symptom was straightforward:

Tableau Services Manager (TSM) was returning HTTP 500 on port 8850.

The interesting part was discovering that the HTTP 500 wasn't really the problem.

The Architecture

At a simplified level, the dependency chain looked like this:

              Administrator
                   │
                   ▼
          Tableau TSM :8850
                   │
                   ▼
          TabadminController
                   │
                   ▼
         Tableau coordination
                   │
                   ▼
              ZooKeeper
                :8866
                   │
                   ▼
         Persistent storage

So rather than immediately restarting Tableau, I started at the symptom and worked down the dependency chain.

Step 1: Is TSM Actually Down?

The TSM CLI was reporting:

Creating ServerApi object - host:ubuntu, port:8850

Checking server health at https://ubuntu:8850/health

Client request:
GET https://ubuntu:8850/api/0.5/status

TabadminController did not return a TsmResponse.

500 - Server Error

The first question was simple:

Is anything actually listening on 8850?

sudo ss -lntp | grep 8850

It showed Tableau's tabadmincontrol process listening on the port.

So this wasn't a simple:

Port closed → service down

Something was alive.

Step 2: Does the HTTPS Service Respond?

I tested the endpoint locally:

curl -k -s https://localhost:8850/health

It returned:

{
"status": 403,
"error": "Forbidden"
}

A 403 might initially look like another failure.

But it actually told us something important:

The TabadminController was reachable and responding.

So the investigation moved further down the stack.

This is one of the most useful troubleshooting habits:

Don't just look at the error code. Ask what the response proves.

In this case, 403 proved that the HTTPS service was alive.

Step 3: Follow the Dependency Chain

Tableau Server depends on several internal services, including its coordination layer.

The Tableau-managed ZooKeeper instance was:

appzookeeper_0

I checked its Tableau service status:

sudo /var/opt/tableau/tableau_server/data/tabsvc/services/appzookeeper_0.20253.26.0206.0336/status.sh

The important part of the output was:

"details" : {
"message" : "Connection refused"
},
"processStatus" : "DOWN"

That was our first strong indication that the problem was deeper than TSM.

sudo ss -lntp | grep 8866

Nothing was listening.

So we now had:

TSM :8850
│
▼
TabadminController
│
▼
ZooKeeper
│
✕
:8866

The HTTP 500 was beginning to look like a downstream dependency failure.

Step 4: The Logs Finally Told the Story

This was the turning point.

Instead of continuing to restart services, I went to the ZooKeeper logs.

There it was:

Severe unrecoverable error, from thread : SyncThread:0

java.io.IOException: No space left on device

The stack trace pointed into ZooKeeper's transaction-log handling:

org.apache.zookeeper.server.persistence.FileTxnLog.append(...)

Immediately afterwards:

SyncThread:0 ... Thread exits, error code 1

Then:

SingleNodeZookeeperWrapper - Attempting to shutdown zookeeper instance.

And finally:

NettyServerCnxnFactory - shutdown called ... :8866

Now the failure chain was clear:

ZooKeeper tries to write transaction log
│
▼
Filesystem returns ENOSPC
│
▼
ZooKeeper critical thread exits
│
▼
ZooKeeper shuts down
│
▼
Port 8866 disappears
│
▼
Tableau coordination becomes unavailable
│
▼
TSM API request fails
│
▼
HTTP 500

That was the real story.

The Strange Part: The Disk Wasn't Full

Naturally, the next command was:

df -h

And the result was surprising:

/dev/vda1 991G 181G 811G 19% /

There was 811 GB free.

So what exactly did No space left on device mean?

This is where the investigation needs to be precise.

ENOSPC does not necessarily mean the entire filesystem has reached 100% capacity.

Possible explanations include:

inode exhaustion
filesystem quotas
a full temporary filesystem
application-specific storage limits
deleted-but-open files
filesystem or storage errors

So I checked additional evidence rather than simply declaring:

"The disk was full."

For example:

df -i

checks inode usage.

And:

sudo lsof +L1

can reveal deleted files that processes still have open.

In this environment, Tableau processes were holding deleted files open, including a large deleted Hyper temporary file.

However, that alone was not enough to prove that those files directly caused ZooKeeper's ENOSPC event.

That distinction matters.

The evidence proves:

ZooKeeper's transaction-log write received ENOSPC.

It does not prove exactly which underlying storage condition produced that response.

Good troubleshooting means knowing the difference between evidence and hypothesis.

Another Trap: systemd Said ZooKeeper Was Running

There was another confusing clue.

Checking the Tableau user-level systemd service:

sudo -u tableau \
XDG_RUNTIME_DIR=/run/user/999 \
systemctl --user status appzookeeper_0

reported:

Active: active (running)
Main PID: 881

But:

sudo ss -lntp | grep 8866

showed nothing.

And Tableau's own status script said:

processStatus : DOWN

So which one was correct?

The answer is: they were measuring different things.

systemd knew that the process existed.

Tableau knew that the application wasn't functioning.

The port check knew that nothing was accepting connections.

The logs explained why.

This is why a single:

systemctl status

should never be treated as the complete definition of application health.

I like to think of it as four independent signals:

      Process
         +
       Port
         +
  Application health
         +
        Logs

When those disagree, investigate the disagreement.

One More Lesson: start Is Not Always restart

Because systemd believed the service was already running, simply doing:

systemctl --user start appzookeeper_0

didn't necessarily give us a fresh application process.

The service manager already believed:

Active: active (running)

So the appropriate operational action was a restart, executed in the Tableau user's systemd context:

sudo -u tableau \
XDG_RUNTIME_DIR=/run/user/999 \
systemctl --user restart appzookeeper_0

The important part here isn't the command itself.

It's understanding who owns the service.

Using:

sudo systemctl ...

and:

sudo -u tableau systemctl --user ...

are not equivalent.

A user-level systemd service belongs to its user session, so operating it in the wrong context can produce errors such as:

Failed to connect to bus
What I Took Away From the Incident

The actual fix was only one part of the exercise.

The bigger lesson was the troubleshooting methodology.

When TSM reported:

HTTP 500

I could have immediately restarted Tableau.

Instead, I asked:

Is 8850 listening?
↓
Yes.

Does the HTTPS service respond?
↓
Yes.

Is ZooKeeper healthy?
↓
No.

Is 8866 listening?
↓
No.

Why did ZooKeeper stop?
↓
Read the logs.

What happened?
↓
ENOSPC while writing the transaction log.

Every step reduced the search space.

What I Would Monitor in Production

After an incident like this, monitoring should go beyond CPU, memory and overall disk usage.

I'd monitor:

Filesystem capacity
df -h
Inodes
df -i
Deleted-but-open files
sudo lsof +L1
Critical Tableau ports
8850 → TSM
8866 → ZooKeeper
Application health

A process being alive isn't enough.

A port being open isn't enough.

A service showing active (running) isn't enough.

The application must actually be functional.

The Bigger Lesson: Troubleshooting Is an Art

This incident started with:

"Tableau TSM is returning HTTP 500."

But the real failure was several layers below it.

That's what makes infrastructure troubleshooting interesting.

The goal isn't to memorize every possible command.

It's to know what question to ask next.

Don't assume an HTTP error is an HTTP problem.

Don't assume a running process is a healthy application.

Don't assume df -h showing free space means ENOSPC is impossible.

And don't confuse a possible explanation with a proven root cause.

Instead:

Follow the evidence.

Check the port.

Check the process.

Check the dependency.

Read the logs.

Challenge your assumptions.

Then fix the layer that's actually broken.

Because sometimes the most misleading part of an incident is the very first error you see.

In this case:

HTTP 500

was merely the messenger.

The real story was:

ZooKeeper
↓
Transaction log write
↓
ENOSPC
↓
ZooKeeper shutdown
↓
Coordination failure
↓
TSM HTTP 500

And that, to me, is the essence of everyday systems troubleshooting:

The art isn't knowing every answer.
The art is knowing where to look next.