For about a month I've had a nightly job that installs every community-maintained plugin listed in Pipecat's docs into a fresh environment, imports it, and checks that every service class it defines can still be constructed. It publishes a table: works, broken, pinned to an old version, can't install.
It installs from PyPI, because that's what a user gets.
In late September I opened a pull request against one of the red ones — pipecat-boson, failing on an import that pipecat moved in 1.8. It sat for a fortnight, and by the time anyone looked at it the maintainer had landed their own fix, so mine was closed. Fair enough; theirs is better than mine was:
try:
from pipecat.utils.types import assert_given, is_given
except ImportError: # pipecat-ai < 1.8.0
from pipecat.services.settings import assert_given, is_given
That landed on 30 September. PyPI still serves the version published before it. So for more than two weeks my dashboard printed a red row, with their project's name on it, for code that was already correct.
The check wasn't buggy. It answered exactly the question I wrote it to answer. The question was wrong.
"Broken" was doing two jobs
There are two different sentences hiding in one red row:
- Your users are getting a package that doesn't work. True for Boson. Still true today.
- Your code is broken. False for Boson, and false for two weeks.
The first is the user's question and the wheel answers it. The second is the maintainer's question, and the wheel cannot answer it at all — a wheel is a snapshot of a repository at some past moment, and nothing says how far behind that moment is.
This matters more than tidiness, because the two sentences imply different actions. "Your code is broken" asks for a patch. "Your users are getting a broken package" asks for a release. If you only ever send the first one, you're asking people to redo work they've already done, and the thing that would actually help their users never gets mentioned.
So the checker now asks both. When the released version fails, it installs the plugin's own source from its repository, pinned to the commit under test, and probes that too. A plugin whose source imports and constructs gets a new status — fixed, not released — which keeps the released failure alongside it and names the commit that passed.
Finding the repository is the hard part
I assumed this would be a lookup. PyPI has project_urls. You read Repository, you clone it, done.
pipecat-boson declares nothing. No Homepage, no Repository, no project_urls at all. Its repository appears in exactly one place in the entire package metadata: a git clone line in the README body.
And then there's this one, which is worse:
Homepage https://github.com/your-username/pipecat-firered-vad
Repository https://github.com/your-username/pipecat-firered-vad
Bug Tracker https://github.com/your-username/pipecat-firered-vad/issues
Changelog https://github.com/your-username/pipecat-firered-vad/releases
All four published URLs are a template placeholder that was never filled in. Anyone who installs that package and hits a problem follows the Bug Tracker link to a 404. Their bug report doesn't go to the wrong place — it goes nowhere, and the maintainer never learns the report existed. There's no route from the package page back to the repository either. I only found the real one because the install error happened to name the dependency.
So resolution reads the metadata first and falls back to scanning the README. That gets 58 of 67, and every plugin that's currently failing.
If you maintain a published package: go and look at your own project_urls right now. It takes ten seconds and the failure mode is silent.
The guard matters more than the feature
Here's the part I'd want to read if someone else wrote this.
pipecat-supertonic resolves to supertone-inc/supertonic — the vendor's own text-to-speech model repository, thirteen thousand stars — because the two names share a word. The plugin actually lives somewhere else entirely.
That particular one fails to install, so it reports unknown either way and I got lucky. But a wrong repository that happened to be pip-installable and to import cleanly would have promoted the plugin to "fixed, not released" on the strength of a completely unrelated codebase, and published that claim under a maintainer's name.
The fix is that the source build has to produce the distribution the plugin actually is, normalised per PEP 503, or the answer is unknown:
expected = {_canonical(project_name(spec)) for spec in plugin.specs if project_name(spec)}
built = {_canonical(dist.split("==")[0]) for dist in probe.get("distributions") or []}
if expected and not expected & built:
return {"source": "unknown", "source_reason": "that repository does not build " + ...}
Measuring something in public means you can be wrong about it in public. The guards are the work, not the measurement.
What it found, versus what I expected
I expected a pile of these. My assumption going in was that several vendors had quietly merged fixes without cutting releases, which would have given me a stack of genuinely welcome messages to send — your fix is merged, your users still get the broken wheel, a release closes it. Nobody minds receiving that.
Thirteen plugins currently fail. Rebuilt from their own source:
| count | |
|---|---|
| Still broken when built from source | 8 |
| Fixed in source, stale on PyPI | 1 |
Couldn't be read, reported unknown
|
4 |
One. Not a stack.
And the harder number: zero of the nine vendors has fixed their own plugin. I read the HEAD of each repository to check. Two of them looked fixed and were my own false positives — one still imports the removed name from the old path, and the other still uses a frame class that was renamed, which my first grep matched as a substring of the new name.
So the thing I built to find a pile of easy wins found one, and mostly confirmed that the plugins are broken in source too. That's a worse story and a more useful tool.
The one that gained something real
One plugin isn't on PyPI at all under the name its documentation publishes. For a month its row said can't install, with no diagnosis, because there was nothing to install and therefore nothing to diagnose.
Built from source, it produces a real one — the same import failure as the others.
That's the part I'd keep if I could only keep one thing. For a package that was never published, the source isn't a second opinion. It's the only place its state exists at all.
The checker, the full table and the nightly results:
https://github.com/Rehansanjay/pipecat-plugin-health
If you maintain a plugin for a fast-moving framework, I'd genuinely like to know whether you run CI against its main — and if not, whether it's because nobody asked.
Top comments (0)