DEV Community

Cover image for The Server Is Reachable, but the Application Is Down: Troubleshooting with JC SRE NetProbe
JacivaldoCarvalho
JacivaldoCarvalho

Posted on Edited on

The Server Is Reachable, but the Application Is Down: Troubleshooting with JC SRE NetProbe

#python #networking #devops #sre

A practical guide to troubleshooting DNS, TCP, TLS, and HTTP failures using a single Python CLI for network and service diagnostics.

Introduction

Imagine receiving an alert indicating that a web application is unavailable.

The server is still running. The infrastructure appears to be operational. However, users cannot access the service.

The first assumption might be a connectivity problem. But what if the TCP connection is working correctly?

Does that mean the application is available?

Not necessarily.

A web service depends on several components working together. DNS resolution must identify the destination, a TCP connection must be established, TLS must meet the required security conditions, and the application must respond correctly to HTTP requests.

A failure at any of these stages can affect the user experience.

In my previous article, Building JC SRE NetProbe: A Python CLI for Network Health Checks, DevOps, and SRE, I introduced JC SRE NetProbe, an open-source Python tool designed for IP network diagnostics and service health checks.

That article focused on the project itself, its capabilities, and its applications in Networking, DevOps, and Site Reliability Engineering (SRE).

Now, let's explore a different scenario.

How can we use JC SRE NetProbe to investigate an application that appears reachable over the network but is not functioning correctly?

In this tutorial, we will use the tool to examine different stages of communication and interpret the results of an operational troubleshooting scenario.

1. The Problem: Connectivity Does Not Mean Availability

Consider an HTTPS application hosted in a corporate infrastructure environment.

A user attempts to access:

https://api.example.com

The browser displays an error or reports that the service is temporarily unavailable.

The infrastructure team needs to determine where the problem originates.

A simplified communication flow looks like this:

HTTPS Communication Flow

Figure 1. Simplified stages involved in establishing an HTTPS connection and receiving an application response.

Each stage represents a different condition.

Successful DNS resolution does not guarantee that the TCP port is accessible.

An established TCP connection does not guarantee that the TLS certificate is valid.

And a successful TLS check does not guarantee that the application is responding correctly.

This separation is what makes layered troubleshooting so useful.

2. Introducing JC SRE NetProbe

JC SRE NetProbe is a Python command-line tool designed for IP connectivity diagnostics and service health checks.

The project combines several capabilities into a single interface:

  • DNS resolution checks.
  • TCP connectivity tests with timing measurements.
  • HTTP and HTTPS endpoint checks.
  • TLS certificate validation and expiration information.
  • IPv4 and IPv6 network information.
  • Structured JSON output.
  • Exit codes for automation.
  • Multiple checks using YAML configuration files.

The tool is designed to support troubleshooting, automation, and basic availability verification across networking, telecommunications, DevOps, and SRE environments.

The source code is available on GitHub:

github.com/jacivaldocarvalho/jc-sre-netprobe

3. Installing JC SRE NetProbe on Ubuntu 24.04 LTS

Before starting the troubleshooting scenario, we need to prepare the environment.

JC SRE NetProbe is distributed as a Python command-line application.

On modern Linux distributions, including Ubuntu 24.04 LTS, installing Python packages directly into the operating system's managed Python environment may be restricted by PEP 668.

This specification helps protect Python environments managed by the operating system's package manager, reducing potential conflicts between system-managed and user-installed dependencies.

For standalone Python CLI applications, a recommended approach is to use pipx.

pipx installs Python applications in isolated virtual environments while making their commands available from the terminal without modifying Ubuntu's system-managed Python environment.

3.1. Installing pipx

Update the package index and install the required packages:

sudo apt update
sudo apt install -y pipx python3-venv
Enter fullscreen mode Exit fullscreen mode

Configure your PATH to include executables installed by pipx:

pipx ensurepath
Enter fullscreen mode Exit fullscreen mode

Refresh your terminal session:

exec $SHELL -l
Enter fullscreen mode Exit fullscreen mode

Verify that pipx is available:

pipx --version
Enter fullscreen mode Exit fullscreen mode

3.2. Installing JC SRE NetProbe

Once pipx is configured, run:

pipx install "jc-sre-netprobe>=0.3.0"
Enter fullscreen mode Exit fullscreen mode

This command performs three main operations:

  1. Creates an isolated virtual environment for the application.
  2. Installs the package and its dependencies inside that environment.
  3. Makes the application's published executables available in the user's command directory, typically ~/.local/bin/.

This allows you to run the tool directly from the terminal without manually activating a virtual environment.

The version constraint >=0.3.0 allows installation of a compatible release equal to or newer than version 0.3.0. For strictly reproducible environments, pin a specific tested version.

3.3. Verifying the Installation

First, list the applications managed by pipx:

pipx list
Enter fullscreen mode Exit fullscreen mode

Next, verify the executable location:

command -v jc-sre-netprobe
Enter fullscreen mode Exit fullscreen mode

On a typical installation, the output should resemble:

/home/username/.local/bin/jc-sre-netprobe
Enter fullscreen mode Exit fullscreen mode

Finally, display the CLI help:

jc-sre-netprobe --help
Enter fullscreen mode Exit fullscreen mode

If the tool displays its help information, the environment is ready for the examples in this tutorial.

If your terminal reports command not found, verify that ~/.local/bin is included in your PATH and that the package installed the expected executable:

pipx list
Enter fullscreen mode Exit fullscreen mode 
ls -l ~/.local/bin/ | grep netprobe
Enter fullscreen mode Exit fullscreen mode

3.4. Alternative: Installing with venv

For development environments or project-specific testing, you can also use a dedicated Python virtual environment.

mkdir -p ~/projects/netprobe
cd ~/projects/netprobe

python3 -m venv .venv
source .venv/bin/activate

python -m pip install "jc-sre-netprobe>=0.3.0"
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

jc-sre-netprobe --help
Enter fullscreen mode Exit fullscreen mode

Deactivate the virtual environment when finished:

deactivate
Enter fullscreen mode Exit fullscreen mode

Engineering note: Avoid using --break-system-packages to bypass Ubuntu's Python environment protections. For standalone CLI applications, pipx provides a more appropriate installation approach.

4. Practical Scenario: Investigating an Unavailable API

Let's consider a hypothetical scenario.

An API is hosted at:

api.example.com

The infrastructure team has received reports of service unavailability, but it is not yet clear whether the problem involves DNS, TCP connectivity, TLS, or the application itself.

Our objective is to perform independent checks to narrow down the investigation.

The domain api.example.com is used for demonstration purposes. Replace it with a real, authorized endpoint when reproducing this troubleshooting workflow.

Step 1 — Checking DNS Resolution

The first step is to determine whether the service hostname can be resolved.

Run:

jc-sre-netprobe dns api.example.com
Enter fullscreen mode Exit fullscreen mode

This check helps determine whether DNS resolution is functioning from the environment where NetProbe is running.

If resolution fails, possible causes include:

  • DNS resolver problems.
  • Incorrect domain configuration.
  • Missing or inconsistent DNS records.
  • Connectivity issues affecting DNS infrastructure.

A successful resolution indicates that this stage of the diagnostic process worked.

However, we still do not know whether the destination is running an accessible service.

Key takeaway: Working DNS resolution does not mean the application is available.

Step 2 — Checking TCP Connectivity

After checking DNS, we need to determine whether the service's TCP port accepts connections.

For an HTTPS application, we will use port 443:

jc-sre-netprobe tcp api.example.com 443
Enter fullscreen mode Exit fullscreen mode

We can also configure a timeout:

jc-sre-netprobe tcp api.example.com 443 --timeout 3
Enter fullscreen mode Exit fullscreen mode

NetProbe attempts to establish the connection and reports the result.

Three situations are particularly important.

Successful connection

The TCP connection was established.

This indicates that the port is reachable from the environment where the check was performed.

Connection refused

The destination or an intermediate component rejected the connection attempt.

Possible causes include the absence of a listening service or a network policy that explicitly rejects connections.

Timeout

The connection attempt did not complete within the configured timeout.

This may indicate traffic filtering, connectivity problems, packet loss, or destination unavailability.

The TCP check also provides timing information, but this measurement does not necessarily represent the application's complete response time.

Key takeaway: A successful TCP connection confirms the tested transport connectivity, not application availability.

Step 3 — Checking TLS

A successful TCP connection does not guarantee that HTTPS communication is functioning correctly.

The next step is to inspect the TLS certificate presented by the service.

Run:

jc-sre-netprobe tls api.example.com
Enter fullscreen mode Exit fullscreen mode

You can also specify the port and timeout:

jc-sre-netprobe tls api.example.com \
  --port 443 \
  --timeout 5
Enter fullscreen mode Exit fullscreen mode

This check helps investigate TLS certificate conditions, including validation and expiration information.

Potential problems include:

  • Expired certificates.
  • Certificate validation failures.
  • TLS negotiation issues.
  • Unavailable TLS services.

It is important to distinguish certificate validity from overall application functionality.

A valid certificate does not guarantee that the API can process requests correctly.

Likewise, a certificate check is not a substitute for a complete security assessment of the service's TLS configuration.

Key takeaway: TLS checks provide evidence about the secure communication layer, but they do not establish application availability.

Step 4 — Checking HTTP

We have now reached the application layer.

The next step is to determine whether the HTTPS endpoint responds correctly.

Run:

jc-sre-netprobe http https://api.example.com
Enter fullscreen mode Exit fullscreen mode

Or configure a specific timeout:

jc-sre-netprobe http https://api.example.com \
  --timeout 5
Enter fullscreen mode Exit fullscreen mode

NetProbe performs the HTTP/HTTPS check and reports the endpoint's health status.

During a real investigation, we may encounter different conditions:

Observed condition Possible interpretation
Healthy HTTP response The endpoint passed the configured health check
HTTP 500 The application returned an internal server error
HTTP 502 A gateway or proxy received an invalid response from an upstream service
HTTP 503 The service reported that it was unavailable
Timeout The request did not complete within the expected interval
Connection failure The required connection could not be established

The exact interpretation depends on the service's expected behavior and the health-check rules implemented by the tool.

An endpoint that responds successfully may still have failures in functionality not covered by the check.

Key takeaway: HTTP checks bring the investigation closer to application behavior, but their results must be interpreted in the context of the endpoint being tested.

5. Interpreting the Results Together

Now imagine that our checks produced the following results:

Check Hypothetical result
DNS Healthy
TCP 443 Healthy
TLS Healthy
HTTP Failed

What can we conclude?

First, DNS resolution worked.

Second, the TCP connection was established.

Third, the TLS check did not identify a failure condition.

However, the HTTP check was not considered healthy.

This suggests that the investigation should focus on HTTP behavior, the application, or components involved in processing the request.

It does not mean that the root cause has already been identified.

For example, an HTTP 503 response might be related to an unavailable dependency, an overloaded service, or an operational configuration issue.

NetProbe provides evidence to guide the investigation. It does not replace internal logs, application metrics, or other information required to identify the exact root cause.

This distinction is fundamental to effective troubleshooting and SRE.

6. Running Multiple Checks with YAML

So far, we have executed each check individually.

This approach is useful during manual troubleshooting, but it can become repetitive when multiple components need to be verified.

Starting with version 0.3.0, JC SRE NetProbe supports bulk execution using YAML configuration files.

Create a file named api-checks.yaml:

max_workers: 4

checks:
  - name: api-dns
    type: dns
    host: api.example.com

  - name: api-tcp
    type: tcp
    host: api.example.com
    port: 443
    timeout: 3

  - name: api-tls
    type: tls
    host: api.example.com
    port: 443
    timeout: 5

  - name: api-http
    type: http
    url: https://api.example.com
    timeout: 5
Enter fullscreen mode Exit fullscreen mode

This configuration defines four independent checks.

The max_workers parameter controls the maximum number of checks that can run concurrently.

Execute the configuration:

jc-sre-netprobe run api-checks.yaml
Enter fullscreen mode Exit fullscreen mode

NetProbe processes the checks and provides individual results along with an aggregated summary.

An illustrative output might look like this:

Bulk Check Results
------------------
[OK] api-dns (dns)
[OK] api-tcp (tcp)
[OK] api-tls (tls)
[FAIL] api-http (http)

Total:   4
Healthy: 3
Failed:  1
Enter fullscreen mode Exit fullscreen mode

This is a hypothetical result demonstrating an HTTP failure scenario. It does not represent an actual execution against api.example.com.

The operational benefit is clear: multiple diagnostic checks can be organized into a single procedure.

It is also important to note that the checks are independent. Bulk execution does not enforce a sequential troubleshooting process or automatically identify the root cause.

7. Using JSON Output for Automation

In addition to human-readable output, JC SRE NetProbe supports structured JSON results.

Run:

jc-sre-netprobe run api-checks.yaml --json
Enter fullscreen mode Exit fullscreen mode

The output structure contains aggregated information and individual check results.

A simplified example:

{
  "total": 4,
  "healthy": 3,
  "failed": 1,
  "results": [
    {
      "name": "api-http",
      "type": "http",
      "healthy": false,
      "details": {},
      "error": "Illustrative HTTP check failure"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

This example shows only one item from the results array to keep the output concise.

The details field may contain information specific to each check type.

This format allows other systems to process diagnostic results programmatically.

For example, these checks can support automation workflows, infrastructure validation processes, or integrations with external monitoring systems.

NetProbe also uses operational exit codes:

Exit code Meaning
0 All checks are healthy
1 One or more checks failed
2 Invalid input or configuration

These exit codes are particularly useful in operational scripts and CI/CD pipelines.

8. How Does This Relate to Site Reliability Engineering?

In SRE, the objective is not simply to verify whether a server is running.

What matters is understanding whether a service meets its expected operating conditions.

An API may be reachable over the network while still failing to serve users correctly.

That is why an effective troubleshooting strategy must consider multiple signals.

JC SRE NetProbe can contribute as an external diagnostic tool, using an approach commonly associated with black-box probing.

It allows engineers to observe selected service behaviors without requiring direct access to the application's source code.

This approach can support:

  • Initial incident diagnosis.
  • Basic service availability checks.
  • Post-deployment service validation.
  • Automated operational checks.
  • Identification of communication stages requiring further investigation.

However, these checks do not replace internal observability, business metrics, distributed tracing, or a complete SLI and SLO strategy.

Their main contribution is transforming individual network and service tests into a structured, repeatable diagnostic process.

Final Thoughts

A server can be reachable while its application remains unavailable.

This situation demonstrates why infrastructure troubleshooting must go beyond a single connectivity check.

By investigating DNS, TCP, TLS, and HTTP separately, we can identify which communication stages are functioning and which require additional attention.

JC SRE NetProbe brings these checks together in a single Python CLI, allowing engineers to execute individual tests or groups of checks defined in YAML.

In this article, we moved beyond the project's initial introduction. Instead of focusing only on its capabilities, we explored how to apply them to a practical troubleshooting scenario.

For professionals working in networking, telecommunications, DevOps, and SRE, understanding the difference between connectivity and availability is essential for effective incident investigation.

When an application fails, which layer do you usually investigate first: DNS, TCP, TLS, or HTTP?

Share your approach in the comments.

References

Top comments (0)