When a Tableau TSM HTTP 500 Wasn't Really an HTTP Problem

Adreet Gogoi — Thu, 10 Sep 2026 17:33:20 +0000

A real-world troubleshooting journey through Tableau Server, ZooKeeper, systemd, and a misleading No space left on device error.

Sometimes infrastructure incidents are easy.

A service is down.
A port is closed.
The logs tell you exactly what happened.

And then there are incidents where everything looks almost healthy.

That was the case with a Tableau Server environment I was troubleshooting.

The symptom was straightforward:

Tableau Services Manager (TSM) was returning HTTP 500 on port 8850.

The interesting part was discovering that the HTTP 500 wasn't really the problem.

The Architecture

At a simplified level, the dependency chain looked like this:

              Administrator
                   │
                   ▼
          Tableau TSM :8850
                   │
                   ▼
          TabadminController
                   │
                   ▼
         Tableau coordination
                   │
                   ▼
              ZooKeeper
                :8866
                   │
                   ▼
         Persistent storage

So rather than immediately restarting Tableau, I started at the symptom and worked down the dependency chain.

Step 1: Is TSM Actually Down?

The TSM CLI was reporting:

Creating ServerApi object - host:ubuntu, port:8850

Checking server health at https://ubuntu:8850/health

Client request:
GET https://ubuntu:8850/api/0.5/status

TabadminController did not return a TsmResponse.

500 - Server Error

The first question was simple:

Is anything actually listening on 8850?

sudo ss -lntp | grep 8850

It showed Tableau's tabadmincontrol process listening on the port.

So this wasn't a simple:

Port closed → service down

Something was alive.

Step 2: Does the HTTPS Service Respond?

I tested the endpoint locally:

curl -k -s https://localhost:8850/health

It returned:

{
"status": 403,
"error": "Forbidden"
}

A 403 might initially look like another failure.

But it actually told us something important:

The TabadminController was reachable and responding.

So the investigation moved further down the stack.

This is one of the most useful troubleshooting habits:

Don't just look at the error code. Ask what the response proves.

In this case, 403 proved that the HTTPS service was alive.

Step 3: Follow the Dependency Chain

Tableau Server depends on several internal services, including its coordination layer.

The Tableau-managed ZooKeeper instance was:

appzookeeper_0

I checked its Tableau service status:

sudo /var/opt/tableau/tableau_server/data/tabsvc/services/appzookeeper_0.20253.26.0206.0336/status.sh

The important part of the output was:

"details" : {
"message" : "Connection refused"
},
"processStatus" : "DOWN"

That was our first strong indication that the problem was deeper than TSM.

sudo ss -lntp | grep 8866

Nothing was listening.

So we now had:

TSM :8850
│
▼
TabadminController
│
▼
ZooKeeper
│
✕
:8866

The HTTP 500 was beginning to look like a downstream dependency failure.

Step 4: The Logs Finally Told the Story

This was the turning point.

Instead of continuing to restart services, I went to the ZooKeeper logs.

There it was:

Severe unrecoverable error, from thread : SyncThread:0

java.io.IOException: No space left on device

The stack trace pointed into ZooKeeper's transaction-log handling:

org.apache.zookeeper.server.persistence.FileTxnLog.append(...)

Immediately afterwards:

SyncThread:0 ... Thread exits, error code 1

Then:

SingleNodeZookeeperWrapper - Attempting to shutdown zookeeper instance.

And finally:

NettyServerCnxnFactory - shutdown called ... :8866

Now the failure chain was clear:

ZooKeeper tries to write transaction log
│
▼
Filesystem returns ENOSPC
│
▼
ZooKeeper critical thread exits
│
▼
ZooKeeper shuts down
│
▼
Port 8866 disappears
│
▼
Tableau coordination becomes unavailable
│
▼
TSM API request fails
│
▼
HTTP 500

That was the real story.

The Strange Part: The Disk Wasn't Full

Naturally, the next command was:

df -h

And the result was surprising:

/dev/vda1 991G 181G 811G 19% /

There was 811 GB free.

So what exactly did No space left on device mean?

This is where the investigation needs to be precise.

ENOSPC does not necessarily mean the entire filesystem has reached 100% capacity.

Possible explanations include:

inode exhaustion
filesystem quotas
a full temporary filesystem
application-specific storage limits
deleted-but-open files
filesystem or storage errors

So I checked additional evidence rather than simply declaring:

"The disk was full."

For example:

df -i

checks inode usage.

And:

sudo lsof +L1

can reveal deleted files that processes still have open.

In this environment, Tableau processes were holding deleted files open, including a large deleted Hyper temporary file.

However, that alone was not enough to prove that those files directly caused ZooKeeper's ENOSPC event.

That distinction matters.

The evidence proves:

ZooKeeper's transaction-log write received ENOSPC.

It does not prove exactly which underlying storage condition produced that response.

Good troubleshooting means knowing the difference between evidence and hypothesis.

Another Trap: systemd Said ZooKeeper Was Running

There was another confusing clue.

Checking the Tableau user-level systemd service:

sudo -u tableau \
XDG_RUNTIME_DIR=/run/user/999 \
systemctl --user status appzookeeper_0

reported:

Active: active (running)
Main PID: 881

But:

sudo ss -lntp | grep 8866

showed nothing.

And Tableau's own status script said:

processStatus : DOWN

So which one was correct?

The answer is: they were measuring different things.

systemd knew that the process existed.

Tableau knew that the application wasn't functioning.

The port check knew that nothing was accepting connections.

The logs explained why.

This is why a single:

systemctl status

should never be treated as the complete definition of application health.

I like to think of it as four independent signals:

      Process
         +
       Port
         +
  Application health
         +
        Logs

When those disagree, investigate the disagreement.

One More Lesson: start Is Not Always restart

Because systemd believed the service was already running, simply doing:

systemctl --user start appzookeeper_0

didn't necessarily give us a fresh application process.

The service manager already believed:

Active: active (running)

So the appropriate operational action was a restart, executed in the Tableau user's systemd context:

sudo -u tableau \
XDG_RUNTIME_DIR=/run/user/999 \
systemctl --user restart appzookeeper_0

The important part here isn't the command itself.

It's understanding who owns the service.

Using:

sudo systemctl ...

and:

sudo -u tableau systemctl --user ...

are not equivalent.

A user-level systemd service belongs to its user session, so operating it in the wrong context can produce errors such as:

Failed to connect to bus
What I Took Away From the Incident

The actual fix was only one part of the exercise.

The bigger lesson was the troubleshooting methodology.

When TSM reported:

HTTP 500

I could have immediately restarted Tableau.

Instead, I asked:

Is 8850 listening?
↓
Yes.

Does the HTTPS service respond?
↓
Yes.

Is ZooKeeper healthy?
↓
No.

Is 8866 listening?
↓
No.

Why did ZooKeeper stop?
↓
Read the logs.

What happened?
↓
ENOSPC while writing the transaction log.

Every step reduced the search space.

What I Would Monitor in Production

After an incident like this, monitoring should go beyond CPU, memory and overall disk usage.

I'd monitor:

Filesystem capacity
df -h
Inodes
df -i
Deleted-but-open files
sudo lsof +L1
Critical Tableau ports
8850 → TSM
8866 → ZooKeeper
Application health

A process being alive isn't enough.

A port being open isn't enough.

A service showing active (running) isn't enough.

The application must actually be functional.

The Bigger Lesson: Troubleshooting Is an Art

This incident started with:

"Tableau TSM is returning HTTP 500."

But the real failure was several layers below it.

That's what makes infrastructure troubleshooting interesting.

The goal isn't to memorize every possible command.

It's to know what question to ask next.

Don't assume an HTTP error is an HTTP problem.

Don't assume a running process is a healthy application.

Don't assume df -h showing free space means ENOSPC is impossible.

And don't confuse a possible explanation with a proven root cause.

Instead:

Follow the evidence.

Check the port.

Check the process.

Check the dependency.

Read the logs.

Challenge your assumptions.

Then fix the layer that's actually broken.

Because sometimes the most misleading part of an incident is the very first error you see.

In this case:

HTTP 500

was merely the messenger.

The real story was:

ZooKeeper
↓
Transaction log write
↓
ENOSPC
↓
ZooKeeper shutdown
↓
Coordination failure
↓
TSM HTTP 500

And that, to me, is the essence of everyday systems troubleshooting:

The art isn't knowing every answer.
The art is knowing where to look next.

DEV Community: Adreet Gogoi

When a Tableau TSM HTTP 500 Wasn't Really an HTTP Problem