DEV Community

Muhammad Dhiyaul Atha
Muhammad Dhiyaul Atha

Posted on

UnvibeComplex Code to Real Workflows: Finding and Fixing a Secret-Handling Boundary Bug in UnvibeCode

#cicd #debugging #python #security

From CI/CD Secrets to External APIs: Finding a Boundary Bug in UnvibeCode

When evaluating AI-powered developer tools, we often focus on the big picture: how well a repository is mapped, how useful the generated architecture diagrams are, or how accurate the code summaries can be.

However, the reliability of an open-source project is also determined by small details that are easy to overlook—CI/CD scripts, environment variables, and requests to external APIs.

While analyzing UnvibeCode, I found an interesting example of a bug at the boundary between configuration and runtime behavior: a secret was correctly passed by GitHub Actions and successfully read by the Python script, but it was never actually used in the outgoing HTTP request.

This article explains:

  • how the bug was discovered;
  • why the existing test suite did not catch it;
  • how the issue can be fixed;
  • and what this teaches us about testing pipelines that rely on secrets.

Note: This analysis is based on source-code inspection and the repository's quality suite. No production credentials or tokens were accessed, displayed, or used.

Tracing the Secret Flow

UnvibeCode includes a GitHub Actions workflow for archiving repository traffic statistics:

.github/workflows/archive-github-traffic.yml
        |
        v
.github/scripts/archive_github_traffic.py
        |
        v
GitHub Traffic API
Enter fullscreen mode Exit fullscreen mode

The workflow requires a secret named TRAFFIC_TOKEN.

1. The Secret Is Passed by the Workflow

The workflow maps the GitHub Actions secret to an environment variable:

env:
  TRAFFIC_TOKEN: ${{ secrets.TRAFFIC_TOKEN }}
  GITHUB_REPOSITORY: ${{ github.repository }}
Enter fullscreen mode Exit fullscreen mode

At this stage, everything appears correct. GitHub Actions makes the secret available to the environment in which the Python script runs.

2. The Python Script Reads the Secret

The script retrieves the environment variable and exits if it is missing:

TOKEN = os.environ.get("TRAFFIC_TOKEN", "").strip()

if not TOKEN:
    fail(
        "TRAFFIC_TOKEN is not set. Add a repository secret named "
        "TRAFFIC_TOKEN with read access to repository traffic."
    )
Enter fullscreen mode Exit fullscreen mode

Again, the flow appears correct: the secret is available and the application successfully reads it.

3. The Secret Is Lost When the Request Is Built

The problem appears in the api_get() function:

headers = {
    "Accept": "application/vnd.github+json",
    "Authorization": f"*****",
    "User-Agent": "unvibecode-traffic-archiver",
}
Enter fullscreen mode Exit fullscreen mode

The f"*****" value is not an f-string that incorporates the TOKEN variable. It is simply a string literal.

As a result, the TOKEN value that was successfully read earlier never reaches the Authorization header.

The request is then sent to several GitHub Traffic API endpoints, including:

traffic/views?per=day
traffic/clones?per=day
traffic/popular/referrers
traffic/popular/paths
Enter fullscreen mode Exit fullscreen mode

If the API rejects the resulting authorization header, the workflow fails before the traffic data can be written to the CSV output.

Why Is This Bug Easy to Miss?

This finding highlights three distinct stages that should be considered when reviewing secrets in CI/CD pipelines:

  1. Availability — Is the secret available to the workflow?
  2. Readability — Does the application read the correct environment variable?
  3. Usage — Does the actual runtime value reach the external API boundary?

In this case, the first two stages appeared correct. The bug only became visible at the third stage.

This is why simply reviewing configuration files is not enough. We need to trace the data flow all the way to the outgoing request.

Why Didn't the Existing Tests Catch It?

I ran the repository's quality suite, which completed successfully with:

906 passed
Enter fullscreen mode Exit fullscreen mode

The test suite validates several important aspects of the repository, including:

  • workflow-pack structure;
  • scenario counts;
  • skill metadata;
  • local Markdown links;
  • required automation files.

However, there was no test that invoked api_get() with urlopen mocked and then inspected the resulting request headers.

In other words, the existing tests demonstrated that the repository structure and related metadata were valid, but they did not verify that the secret was actually used when the HTTP request was constructed.

This is an important distinction:

A test suite can provide strong structural coverage while still missing a critical runtime integration bug.

Recommended Fix

The authorization header should use the runtime token:

headers = {
    "Accept": "application/vnd.github+json",
    "Authorization": f"Bearer {TOKEN}",
    "User-Agent": "unvibecode-traffic-archiver",
}
Enter fullscreen mode Exit fullscreen mode

The change itself is small, but it should be accompanied by a regression test.

A suitable test should:

  1. provide a dummy token through the environment;
  2. mock urlopen;
  3. call api_get();
  4. inspect the generated request;
  5. verify that the Authorization header contains the runtime token;
  6. ensure that the token is never printed to stdout or stderr;
  7. continue to verify that a missing token produces a clear error.

The core assertion could be:

assert request.headers["Authorization"] == "Bearer test-token"
Enter fullscreen mode Exit fullscreen mode

The actual test should use a dummy value such as test-token, never a real credential.

Engineering Lessons

1. A Secret Being Available Does Not Mean It Is Being Used

An environment variable can be correctly configured and successfully read without ever reaching the component that needs it.

For external integrations, the final boundary—such as an HTTP header, SDK argument, or subprocess invocation—must be explicitly verified.

2. Test the External Boundary, Not Just Internal Logic

For code that communicates with an API, mock the network client or network function at the point where the request leaves the application.

Verify at least:

  • HTTP method;
  • URL;
  • authorization headers;
  • relevant request parameters;
  • and, where appropriate, request payloads.

This approach avoids making real API calls while keeping the test fast, deterministic, and safe.

3. Never Print Secrets for Debugging

When inspecting authentication behavior, assert against the request object or sanitized metadata.

Do not print the raw token into CI logs—even temporarily.

A debugging statement such as:

print(TOKEN)
Enter fullscreen mode Exit fullscreen mode

can turn a functional debugging session into a credential-exposure incident.

4. Analysis Tools Help Trace the Path, but They Do Not Replace Verification

UnvibeCode can make it easier to connect workflows, files, and dependencies and identify the path a secret takes through a repository.

However, the final conclusion still needs to be validated against the actual source code and, where possible, a test at the runtime boundary.

Static analysis can show that a variable is present.

A boundary test can show that its value is actually transmitted.

Those are different guarantees.

Conclusion

This is not an example of a secret being leaked into the repository. The problem is almost the opposite: the configured secret was never actually used.

That distinction matters.

  • The secret was not exposed.
  • The API request could nevertheless fail.
  • The workflow appeared to be correctly configured.
  • The existing tests could still pass because they did not inspect the outgoing request.

When auditing CI/CD pipelines, don't stop at the question:

"Is the secret configured?"

Ask the more important question:

"Does the secret actually reach the external boundary where it is required?"

That boundary is often where a seemingly correct configuration turns into a runtime failure.

References

Disclosure: This article was written with AI assistance for language editing and presentation. The technical finding, repository inspection, and final conclusions are based on analysis of the source code and the repository's test results.

Top comments (0)