"I fixed a retrieval bug from part 1 with a noise filter and reranking, then found something far more interesting hiding underneath it: a real prom...
For further actions, you may consider blocking this person and/or reporting abuse
The accident is the useful warning here. A lot of RAG demos treat retrieved text like inert evidence. The model still reads the chunk as instructions unless the harness puts a hard boundary around source text. I like the filter, and I would keep one test where a known hostile chunk has to stay quoted and powerless.
you're right that the accidental part is the real signal here. I wasn't stress testing for this, it just happened on the first real book i threw at the pipeline. That's a much stronger argument than if I'd gone looking for it. I like the regression test idea a lot. Right now I only have one before/after example, not an ongoing check. I'll add a fixed "hostile chunk" test to the pipeline (something more deliberately aggressive than the accidental "return 0" one) and verify the wrapper holds up as I keep changing other parts of the system.
Good call ,that's going in the next post.
The rank 2 example is a good catch. A lot of RAG demos treat retrieval quality as the whole game. The boundary around retrieved text is part of the system too. I like that you tested the prompt wrapper after reranking instead of declaring the pipeline fixed.
Appreciate this and you're right. i almost stopped at “reranking improved retrieval” and moved on. It wasn't until i ran the exact same (still imperfect) retrieved chunks through the new prompt wrapper that I was able to clearly separate the injection fix from the retrieval fix.
In hindsight, the distinction seems obvious, but when a fix makes the overall output look better, it's really easy to conflate the two. Retrieval quality and prompt safety are two different failure surfaces, and I'm going to keep that distinction much more explicit going forward.
Really interesting findings! I especially liked how fixing the retrieval issue led you to uncover the prompt injection problem, and the question-phrasing result was equally fascinating. Looking forward to the next part!
Thankyou ! It was such a wild twist finding that prompt injection bug while just trying to clean up the retrieval noise. The phrasing sensitivity was a big eye-opener for me too! So glad you enjoyed reading it working on part 3 now, can't wait to share it!
horw
The question phrasing finding is the one that got me. We hit something similar on a knowledge graph pipeline where "what companies does X own" vs "subsidiaries of X" pulled completely different node sets, same question in human terms, zero vocabulary overlap with the document structure. The injection part is what happens when instruction-following training makes models respond to imperative-tense text regardless of where it sits in the prompt, so your fix has to be structural, you can't out-clever it at the retrieval stage. I've also seen it show up silently where retrieved text just happens to be phrased in a commanding tone and shifts the model's output format without anything obvious triggering.
the reason this one was catchable is that it was loud. output was a single digit. couldn't miss it.
hannune's format-shift cases are the ones i'd worry about and they don't announce themselves at all. answer is still a paragraph, still plausible, just quietly restructured by something in a chunk.
cheap detector for those: run the query twice, once real, once with every retrieved chunk swapped for neutral filler of roughly the same length. diff the two outputs on shape only, not content. length, prose vs list, whether it opens with a hedge. content should differ, that's the whole point of retrieval. structure mostly shouldn't. when structure moves and content barely does, something in the chunks was talking to the model rather than being read by it.
no second model needed and it runs against the 917 chunks you already have. wouldn't catch everything, but it turns the silent class into the loud class, and the loud class is the only version you can put in a test.
on kartik's question, i'd guess most people's honest answer is prod, and the reason is that nobody puts genuinely hostile real documents in an eval set. a book about prompting is a better adversary than anything you'd write on purpose.
wow