"A win, but on my features, not a thrashing" is the most honest line in any benchmark post I've read this month. The tree models all landing within a couple of points of each other says the features did most of the work. Curious how the split was done: with samples every 5 seconds, neighbouring rows are near-duplicates, so a random split can quietly leak. Did you hold out whole time windows or whole anomaly events? If the numbers survive that, it's a seriously strong result.
Thank you, and that's exactly the right question. Two parts to the answer:
The model works on whole segments, not individual samples. Each row is one telemetry fragment summarised into features, so points from the same fragment can't straddle the split.
But your concern still applies one level up. The official OPSSAT-AD test split is stratified random sampling of segments, so neighbouring segments from the same channel can sit on opposite sides. I used it because every published result uses it, which keeps the comparison like-for-like. It doesn't rule out neighbour leakage, though.
For my own model selection I used blocked CV: contiguous runs of 10 segments per channel stay together in one fold. That gave AUPRC 0.978, lower than the 0.996 on the official split, which is at least consistent with the random split being a bit easier.
I haven't yet held out whole time windows or whole anomaly events on the test side. That's the honest next experiment, and if the numbers drop, I'll post them.
The 'known edges (measured, not hidden)' section is the most impressive part of this whole project. Admitting that you missed your own 95% target, that fine-tuning made one task worse, and that the offline adapter is a sibling configuration — that's the kind of honesty that makes the rest of the results trustworthy.
Thank you, that section was the hardest one to publish. It's tempting to bury the 95% miss and the task that got worse, but if I'm asking people to trust a model that says "unknown," I have to say "I missed" in the same voice. Glad it read that way.
The 'if I'm asking people to trust a model that says unknown, I have to say I missed in the same voice' is the best argument for transparency I've read. You can't ask for trust while hiding your own failures. The consistency is what makes it work.
Thank you, that really means a lot. Honestly, the model forced it on me: after a whole night teaching it to say "not known" instead of guessing, writing the post any other way would have felt like cheating.
The 'the model forced it on me' is a really interesting way to put it. After a whole night of teaching it to say 'not known,' you'd built a relationship with that honesty. Writing the post any other way would have felt like betraying the work.
The four demo cards falling back to deterministic text stood out to me. How does that fit with the 100% pass rate for investigation notes in the evaluation? I'd be interested in what differed between those inputs, since the fallback is catching failures the eval didn't expose.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The 83-cent part is what got me, honestly — not the price, the fact that the fix was "stop inventing and say not documented" rather than a bigger model. TabPFN beating the published bar with no training loop is the quieter flex. Did you ever get tempted to let it guess just a little, on the cases where the archive is silent but the pattern is obvious? That's where I'd have caved.
The step nobody talks about here is that you reproduced the paper first before trying to beat it. most "beat SOTA" posts compare against numbers they never ran themselves
Did your reproduction actually land on their published scores or was there a gap?
PS.
Did the same when benchmarking my disk tool against du clones, ran every one myself instead of trusting their readmes
Highlighting the zero point zero nine eight false negative instead of sweeping it under aggregate AUCPR is the honest way to report telemetry evaluation. In operational monitoring, aggregate area under the curve hides the asymmetric loss function. A false alarm burns diagnostic hours, but an unflagged blackout risks losing the asset entirely. Showing the raw classification boundary where prior-fitted tabular models struggle with abrupt dropouts gives practitioners an actual risk profile rather than benchmark marketing. Forcing the retrieval adapter to say the archive is silent also treats absence of evidence as a hard bound rather than an invitation to hallucinate explanations.
Thanks, that asymmetry is exactly why I didn't want to stop at one aggregate number. A false alarm costs an engineer an hour; a missed dropout can cost the spacecraft. And yes, the adapter treats a silent archive as a hard limit, not an invitation to fill in the gap. If the documents don't say why, the answer is "not known," full stop.
Every committed test is an annuity of maintenance debt. When an agent writes fifty exploratory assertions to pin down an edge case, committing all fifty forces the team to pay that CI compute bill and triage flakes indefinitely. Keeping generative checks in an ephemeral sandbox while requiring a strict threshold for the permanent suite treats test volume as a liability rather than an asset.
Fair point in general, though I didn't touch on test suites here. The guards in this project are run-time validators and a sealed test split, not a CI suite. Agreed that test volume is a cost, not a trophy.
The strongest safeguard here is refusing to convert an anomaly score into an invented cause. I’d also freeze the queue threshold policy and track post-review precision from Confirm/Reject feedback over time; excellent offline AUCPR does not by itself define an operational investigation queue.
Completely agree, and that's the gap between a benchmark and an ops tool. Right now the threshold is chosen on training data only and then frozen, but nothing yet learns from Confirm/Reject. Logging those decisions and tracking precision at the queue cutoff over time is the obvious next step. A 0.996 AUCPR tells you the ranking is good. It doesn't tell you whether an engineer trusts the top 10 on a Tuesday. Thanks for this.
Thank you! That's exactly the part I cared most about. The base model kept inventing channel IDs and quoting things that weren't in the source; after 83¢ of Tinker training it made zero of those errors on held-out documents, and every "why" without a source comes back as "not documented in the archive." The dataset is real and public if you want to explore it: OPSSAT-AD on Zenodo (zenodo.org/records/12588359)
Love the house rule — 'nobody speaks for the dead' is a genuinely principled constraint for model grounding, and the honest output ('root cause not documented in the archive') is the right failure mode for uncertain inference. The TabPFN zero-training result is striking too: beating a published FCNN on the paper's own 18 features suggests the benchmark may be more about feature structure than learned dynamics. Keeping the deliberate miss in the demo is a great credibility touch.
The method benchmarking on these signals is perfectly placed. I didn't even imagine a dead satelite would still be passing signals, you pinched my curiosity on this one🤗❤️🏆
"A win, but on my features, not a thrashing" is the most honest line in any benchmark post I've read this month. The tree models all landing within a couple of points of each other says the features did most of the work. Curious how the split was done: with samples every 5 seconds, neighbouring rows are near-duplicates, so a random split can quietly leak. Did you hold out whole time windows or whole anomaly events? If the numbers survive that, it's a seriously strong result.
Thank you, and that's exactly the right question. Two parts to the answer:
The model works on whole segments, not individual samples. Each row is one telemetry fragment summarised into features, so points from the same fragment can't straddle the split.
But your concern still applies one level up. The official OPSSAT-AD test split is stratified random sampling of segments, so neighbouring segments from the same channel can sit on opposite sides. I used it because every published result uses it, which keeps the comparison like-for-like. It doesn't rule out neighbour leakage, though.
For my own model selection I used blocked CV: contiguous runs of 10 segments per channel stay together in one fold. That gave AUPRC 0.978, lower than the 0.996 on the official split, which is at least consistent with the random split being a bit easier.
I haven't yet held out whole time windows or whole anomaly events on the test side. That's the honest next experiment, and if the numbers drop, I'll post them.
The 'known edges (measured, not hidden)' section is the most impressive part of this whole project. Admitting that you missed your own 95% target, that fine-tuning made one task worse, and that the offline adapter is a sibling configuration — that's the kind of honesty that makes the rest of the results trustworthy.
Thank you, that section was the hardest one to publish. It's tempting to bury the 95% miss and the task that got worse, but if I'm asking people to trust a model that says "unknown," I have to say "I missed" in the same voice. Glad it read that way.
The 'if I'm asking people to trust a model that says unknown, I have to say I missed in the same voice' is the best argument for transparency I've read. You can't ask for trust while hiding your own failures. The consistency is what makes it work.
Thank you, that really means a lot. Honestly, the model forced it on me: after a whole night teaching it to say "not known" instead of guessing, writing the post any other way would have felt like cheating.
The 'the model forced it on me' is a really interesting way to put it. After a whole night of teaching it to say 'not known,' you'd built a relationship with that honesty. Writing the post any other way would have felt like betraying the work.
The four demo cards falling back to deterministic text stood out to me. How does that fit with the 100% pass rate for investigation notes in the evaluation? I'd be interested in what differed between those inputs, since the fallback is catching failures the eval didn't expose.
The 83-cent part is what got me, honestly — not the price, the fact that the fix was "stop inventing and say not documented" rather than a bigger model. TabPFN beating the published bar with no training loop is the quieter flex. Did you ever get tempted to let it guess just a little, on the cases where the archive is silent but the pattern is obvious? That's where I'd have caved.
The step nobody talks about here is that you reproduced the paper first before trying to beat it. most "beat SOTA" posts compare against numbers they never ran themselves
Did your reproduction actually land on their published scores or was there a gap?
PS.
Did the same when benchmarking my disk tool against du clones, ran every one myself instead of trusting their readmes
Highlighting the zero point zero nine eight false negative instead of sweeping it under aggregate AUCPR is the honest way to report telemetry evaluation. In operational monitoring, aggregate area under the curve hides the asymmetric loss function. A false alarm burns diagnostic hours, but an unflagged blackout risks losing the asset entirely. Showing the raw classification boundary where prior-fitted tabular models struggle with abrupt dropouts gives practitioners an actual risk profile rather than benchmark marketing. Forcing the retrieval adapter to say the archive is silent also treats absence of evidence as a hard bound rather than an invitation to hallucinate explanations.
Thanks, that asymmetry is exactly why I didn't want to stop at one aggregate number. A false alarm costs an engineer an hour; a missed dropout can cost the spacecraft. And yes, the adapter treats a silent archive as a hard limit, not an invitation to fill in the gap. If the documents don't say why, the answer is "not known," full stop.
Every committed test is an annuity of maintenance debt. When an agent writes fifty exploratory assertions to pin down an edge case, committing all fifty forces the team to pay that CI compute bill and triage flakes indefinitely. Keeping generative checks in an ephemeral sandbox while requiring a strict threshold for the permanent suite treats test volume as a liability rather than an asset.
Fair point in general, though I didn't touch on test suites here. The guards in this project are run-time validators and a sealed test split, not a CI suite. Agreed that test volume is a cost, not a trophy.
The strongest safeguard here is refusing to convert an anomaly score into an invented cause. I’d also freeze the queue threshold policy and track post-review precision from Confirm/Reject feedback over time; excellent offline AUCPR does not by itself define an operational investigation queue.
Completely agree, and that's the gap between a benchmark and an ops tool. Right now the threshold is chosen on training data only and then frozen, but nothing yet learns from Confirm/Reject. Logging those decisions and tracking precision at the queue cutoff over time is the obvious next step. A 0.996 AUCPR tells you the ranking is good. It doesn't tell you whether an engineer trusts the top 10 on a Tuesday. Thanks for this.
Links (if anyone wants to explore more about it or dead satellites in general)
Dataset
Paper TabPFN beat (best published AUCPR 0.979)
The satellite (re-entered 22 May 2024)
2,123 signals from a dead satellite is a wild dataset. The 83-cent fix stopping the LLM from speaking for it is the detail that stuck with me.
Thank you! That's exactly the part I cared most about. The base model kept inventing channel IDs and quoting things that weren't in the source; after 83¢ of Tinker training it made zero of those errors on held-out documents, and every "why" without a source comes back as "not documented in the archive." The dataset is real and public if you want to explore it: OPSSAT-AD on Zenodo (zenodo.org/records/12588359)
zero invented IDs on held-out docs is the number that matters. 'not documented in the archive' beats a confident guess every time.
Love the house rule — 'nobody speaks for the dead' is a genuinely principled constraint for model grounding, and the honest output ('root cause not documented in the archive') is the right failure mode for uncertain inference. The TabPFN zero-training result is striking too: beating a published FCNN on the paper's own 18 features suggests the benchmark may be more about feature structure than learned dynamics. Keeping the deliberate miss in the demo is a great credibility touch.
Solid work!! All the best for week 4👍
Great work Advik👍
Thank you so much☺️
Exceptionally presented and will definitely research into this topic. Thank you for sharing
Thankss!! It's a good topic to research into. Dead satellites do have some information left in them to be received.
The method benchmarking on these signals is perfectly placed. I didn't even imagine a dead satelite would still be passing signals, you pinched my curiosity on this one🤗❤️🏆
Glad to know that. Thankyouu!!☺️
Really exceptional work!! The results are just amazing🙌🏻👏🏻
Crazyyy techie!!!! I’ve been really enjoying all your posts atp, great topic for last week submission. All the best to you, hope you win!🏅
Glad to know that you liked my work☺️ Thank you so much