DEV Community: Abdulswamad Rama

New Relic's Python agent rejected the key its own infra agent was using

Abdulswamad Rama — Wed, 03 Jun 2026 08:32:20 +0000

The infra agent worked immediately. The Python APM agent spent an hour rejecting the same license key that was already authenticating the infra agent. Same account. Same VPS. Same key. One worked, one didn't.

That's the story.

Getting New Relic free via GitHub Student Dev Pack

If you have a .edu email or a GitHub Student account, the GitHub Student Developer Pack includes a New Relic Pro subscription — the full observability platform, not a trial.

To claim it:

Go to education.github.com/pack and verify your student status if you haven't already.
Search for "New Relic" in the pack offers.
Click the offer — it redirects to New Relic's partner page. Or simply sign up directly if you're logged in to your existing GitHub account
Create a New Relic account via GitHub.
The Pro plan activates automatically. Your account is provisioned in one of two regions: US or EU. If you're outside North America, you'll likely get EU (one.eu.newrelic.com).

Region matters later.

What I'm monitoring

The target: a Flask app running in Docker on a single VPS. Stack:

Flask + Gunicorn (2 workers, 2 threads)
PostgreSQL 16 in a sidecar container
Nginx in front, TLS via Let's Encrypt
docker-compose.yml as the orchestrator

The goal:

Infrastructure metrics — CPU, memory, disk on the host
Log forwarding — Flask app logs into New Relic Logs
Python APM — request traces, error rates, slow transaction data

Step 1: Install the infra agent

New Relic's guided install gives you a one-liner. For an EU account:

curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && \
  sudo NEW_RELIC_API_KEY=<your-api-key> \
       NEW_RELIC_ACCOUNT_ID=<your-account-id> \
       NEW_RELIC_REGION=EU \
       /usr/local/bin/newrelic install

The installer is interactive. It detects your OS, offers integrations (PostgreSQL, nginx, etc.), and installs newrelic-infra as a systemd service, Fluent Bit for log forwarding, and Golden Signal alert policies.

When it asked for PostgreSQL credentials, I skipped — PostgreSQL runs inside Docker with no port exposed to the host. The infra agent runs on the host and can't reach it. Skip it.

After install, sudo systemctl status newrelic-infra shows active. The license key is stored at /etc/newrelic-infra.yml — you'll need it later.

Step 2: Add the Python APM agent to Flask

The plan:

Add newrelic>=9.0 to pyproject.toml
Drop a newrelic.ini config file in the repo
Change the Gunicorn CMD to newrelic-admin run-program gunicorn ...
Pass the license key via environment variable

The config file

The New Relic setup wizard generates a newrelic.ini. Strip the license key out — don't commit it to git — and keep the meaningful settings:

[newrelic]
app_name = your-app-name

monitor_mode = true
log_file = stdout
log_level = info

distributed_tracing.enabled = true
otlp_host = otlp.eu01.nr-data.net

transaction_tracer.enabled = true
transaction_tracer.record_sql = obfuscated

error_collector.enabled = true
application_logging.enabled = true
application_logging.forwarding.enabled = true

[newrelic:production]
monitor_mode = true

Do not put license_key = in this file — not even empty. An explicit empty value is ambiguous. The NEW_RELIC_LICENSE_KEY env var handles it cleanly.

Dockerfile change

CMD ["newrelic-admin", "run-program", "gunicorn", \
     "-w", "2", "--threads", "2", "--timeout", "120", \
     "-b", "0.0.0.0:5000", "server:app"]

docker-compose.yml

environment:
  NEW_RELIC_CONFIG_FILE: /home/appuser/app/newrelic.ini
  NEW_RELIC_ENVIRONMENT: production
  NEW_RELIC_HOST: collector.eu01.nr-data.net

The license key lives in a gitignored .env on the server, loaded via env_file::

NEW_RELIC_LICENSE_KEY=eu01x<your-key>NRAL

Step 3: The license key that didn't work

Deployed. Checked logs:

newrelic.core.agent INFO - New Relic Python Agent (13.1.0)
newrelic.core.agent_protocol ERROR - Data collector is indicating that
an incorrect license key has been supplied by the agent.

I tried three keys:

The one generated by the New Relic Python setup wizard — rejected
The "Original account license" key from the API Keys admin page — rejected
A fresh INGEST - LICENSE key created manually in the UI — rejected

The infra agent was running fine on the same host the whole time.

A note on copying keys from New Relic: the table masks all key values as eu01×xxx*****. The ... row menu has "Copy key ID" (the ID, not the value) and "Edit" shows the value masked in a form. The only way to get the full value is to create a new key and copy it at creation — shown once, then masked permanently.

The actual problem

The infra agent was installed with NEW_RELIC_REGION=EU explicitly — it knew to send data to the EU collector. The Python agent had no such directive. It's supposed to auto-detect the EU region from the eu01 license key prefix, but version 13.1.0 wasn't doing it reliably.

The Python agent was connecting to the US APM collector (collector.newrelic.com) with an EU-region key. The US collector correctly said: this key doesn't belong here.

Fix: one env var.

NEW_RELIC_HOST: collector.eu01.nr-data.net

Redeployed. Logs:

newrelic.core.agent INFO - New Relic Python Agent (13.1.0)
newrelic.core.agent_protocol INFO - Reporting to:
  https://one.eu.newrelic.com/redirect/entity/...

Connected.

What you end up with

Infrastructure tab — CPU, memory, disk, network I/O on the host. The newrelic-infra service uses about 48MB RSS.

Logs tab — Flask app logs, Gunicorn worker logs, all searchable. Filtering by level:error across everything in one query is useful.

APM & Services — request throughput per route, response time distribution, error rate, slow transaction traces, and Python exception capture with full stack traces.

Golden Signal Alerts — pre-configured policies that email you if CPU spikes, error rate jumps, throughput drops, or response time degrades.

EU account checklist

If your New Relic account is EU (one.eu.newrelic.com):

Where	Setting	Value
`.env`	`NEW_RELIC_LICENSE_KEY`	Your INGEST - LICENSE key
`docker-compose.yml` env	`NEW_RELIC_HOST`	`collector.eu01.nr-data.net`
`newrelic.ini`	`otlp_host`	`otlp.eu01.nr-data.net`
Install CLI flag	`NEW_RELIC_REGION`	`EU`

Miss NEW_RELIC_HOST on the Python agent and you'll spend an hour on a key rejection that is a region mismatch.

Gemini called it a public API. Careerjet's registration portal disagreed.

Abdulswamad Rama — Mon, 01 Jun 2026 03:02:14 +0000

I was building something that needed job listings. Gemini flagged Careerjet as a solid option:

"Careerjet has a public API — structured data, no scraping, Kenya support."

Four requirements it didn't mention.

What I built first

The old Careerjet affiliate endpoint (public.api.careerjet.net/search) responds to unauthenticated requests. You pass an affid parameter — originally used so Careerjet could track clicks and split revenue with partner sites. I passed a placeholder and moved on.

params = urlencode({
    "keywords": "software engineer",
    "location": "Kenya",
    "locale_code": "en_KE",
    "affid": "your-affid",       # placeholder — I planned to "sort this out later"
    "pagesize": 100,
})
url = f"http://public.api.careerjet.net/search?{params}"

It returned JSON. I wrote a parser around it. I shipped the file.

This would have worked right up until it didn't — either Careerjet deprecates the old endpoint, tightens auth requirements, or (more likely) nothing gets properly attributed and whatever publisher agreement exists is quietly violated. But the response came back clean, so I moved on.

Then I actually registered

Out of curiosity, I went to Careerjet's publisher portal. What I found:

An API key tied to my registered domain
A mandatory IP whitelist — I had to declare which server IPs were allowed to call the API before anything worked
HTTP Basic auth on every request: API key as the username, empty string as the password

The real endpoint is https://search.api.careerjet.net/v4/query. Note the v4, the https, the different subdomain, and the complete absence of an affid param.

"Public" in Careerjet's vocabulary means available to any publisher who registers, without an approval process. Not unauthenticated. The AI had collapsed two different meanings into one confident sentence.

The corrected implementation

import base64
import urllib.request
import json
from urllib.parse import urlencode

API_URL = "https://search.api.careerjet.net/v4/query"

def fetch_jobs(keywords: str, location: str, api_key: str, user_ip: str) -> dict:
    params = urlencode({
        "locale_code": "en_KE",
        "keywords": keywords,
        "location": location,
        "sort": "date",
        "page_size": 100,
        "user_ip": user_ip,
        "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    })

    # Basic auth: base64(api_key + ":")  — password is empty string
    credentials = base64.b64encode(f"{api_key}:".encode()).decode()

    req = urllib.request.Request(f"{API_URL}?{params}")
    req.add_header("Authorization", f"Basic {credentials}")
    req.add_header("Referer", "https://your-site.com/jobs/")

    with urllib.request.urlopen(req, timeout=20) as resp:
        return json.loads(resp.read())

Three things the AI's description had entirely omitted:

Basic auth with an empty password. The format is base64("your-api-key-here" + ":"). Username is the key, password is blank. Standard HTTP Basic auth — but you'd only know to look for it in the actual docs.

user_ip and user_agent are required fields. This one is the most interesting. The v4 API was designed for publishers embedding job search in their website — a real user types a query, clicks a button, the API fires. Careerjet wants the user's IP and browser string for analytics.

My use case: a cron job, no browser, no user, runs at 23:00 UTC. There is no user IP to pass. The spec says "the IP of the user whose action triggered the API call." The nightly cron is the trigger. I pass the server's outbound IP (203.0.113.1 in these examples). The API accepts it.

IP whitelisting is mandatory before anything works. In the publisher dashboard there's a text box — up to 8 IP addresses. Your server's outbound IP must be there or every request returns 403 regardless of correct auth headers. Register it first, then deploy.

Handle the response type

The API returns either "JOBS" or "LOCATIONS". The LOCATIONS case happens when your location param is ambiguous:

data = fetch_jobs(...)

if data.get("type") != "JOBS":
    # LOCATIONS response: Careerjet couldn't resolve the location
    # log data.get("message") and bail
    return []

jobs = data.get("jobs", [])

"Kenya" and "Nairobi" both resolve cleanly. Vaguer strings may not.

Each job object has title, company, locations, description (excerpt), salary, and url. The salary field comes back as a human-readable string like "KES 80,000 – 120,000 per month" — more useful than trying to parse salary out of a scraped description.

What the AI got wrong

It wasn't lying. The old affiliate endpoint at public.api.careerjet.net is real, responds to requests, and technically works. The description "public API, no scraping" isn't false — it's just about a different version of the API than the one publishers are supposed to use.

The failure mode is: the AI described behavior that was accurate at some point, for some use case, and I didn't verify which use case it was describing. "Public API" is a phrase that means something specific to the developer who wrote the docs and something looser to a model that's synthesizing from across the web.

The fix took twenty minutes — register, whitelist the IP, swap the endpoint, add the auth header. The registration is instant, no approval needed. The documentation is clear.

The cost of not checking: an integration that works in development, silently misattributes traffic, and breaks when the old endpoint eventually goes away.