Poke is an AI assistant. OpenPoke is Shlok Khemani's simplified, open-source take on it, with email triage, reminders, and agents you can run locally.
I forked OpenPoke and gave Jev some of the decisions previously handled by a chat model. The first one I measured was simple: does this email need my attention?
In my small comparison, mean screening latency went from 2,452 ms with Claude Sonnet to 424 ms with Jev. Estimated input cost went from $4.63 to $0.049 per 1,000 screens.
Those are numbers for the screening decision. I haven't measured the whole assistant's cost or speed.
The bottom strip describes a separate attack experiment: 95.3% is my gate's trigger rate across 12 synthetic email carriers when hostile text was inserted. It is not a real-inbox loss rate.
How I added Jev to OpenPoke
OpenPoke's important-email watcher asks a chat model for a boolean decision. In my fork, one Jev call asks four questions: importance, security-code content, bulk mail, and instructions aimed at an AI assistant.
OpenPoke uses Composio for its Gmail tooling. My fork changes the decision layer around that workflow.
Confidently unimportant mail can skip the LLM. Important mail goes to summarisation. Uncertain cases fall back to the original classifier. The language model still writes the notification.
I also added Jev checks before tool execution and for search relevance. The comparison here measures email screening only.
Jev vs Claude: screening latency and input cost
I tested claude-sonnet-4 against jev-1.13.0 on the same 12 emails and four questions, with three repeats per email: 72 calls across both arms, with no request failures.
| Screening measurement | Claude Sonnet | Jev |
|---|---|---|
| Mean latency | 2,452 ms | 424 ms |
| p95 latency | 2,883 ms | 1,110 ms |
| Estimated input cost / 1,000 screens | $4.63 | $0.049 |
That's roughly 5.8× faster at the mean in this run. The p95 improvement is smaller, around 2.6×.
Costs use list rates of $3.00 and $0.042 per million input tokens, over unequal token counts: 55,554 for Sonnet, including its tool schema, versus 41,739 for Jev. Output costs aren't included. Timings come from one machine, one afternoon, and one provider per arm.
At the 0.75 importance threshold, the answers agreed in 33/36 comparisons. That is agreement, not accuracy: I don't have a labelled email-triage set. The image's distinct-probability counts don't establish better calibration either.
A cheaper screening call also doesn't guarantee a cheaper pipeline. Surfaced mail still needs summarisation and can fall back to the original classifier. Notification volume may change too; I haven't measured that yet.
What the prompt-injection test exposed
I then tested hostile text inserted into synthetic emails. The full experiment reports 43,776 requests, zero request failures, and $2.04 in cost.
None of the eight attack mechanisms forced a low-priority message over the surfacing bar: 0/720 eligible trials each, or 0/5,760 combined.
But my injection gate fired in 95.3% of attack trials across 12 synthetic carriers, with a carrier-clustered interval of 93.1–97.2%. I'd wired that flag to silently drop the notification. If the attacker wanted silence, my policy helped.
This assumes control over text inside the target email. It doesn't demonstrate a way to alter someone else's legitimate security-code email. The result concerns my fork's policy, not a flaw established in Poke or upstream OpenPoke.
Flagged bodies now stay out of the agent's context but generate a visible quarantine notice, assembled in code. Users can still overlook it. This reduces silent withholding; it doesn't prove a security boundary.
Safeguard-induced denial of service has prior research, including related Jev monitor experiments. My finding is an email-specific instance, not a new attack class.
Try the fork
openpoke-meets-jev includes the implementation, setup instructions, and evaluation harnesses. The full findings include the attack breakdowns and limitations. Summaries are public; per-request contamination responses aren't committed, so recomputing them requires a rerun.
With TYPESAFE_API_KEY enabled, email bodies and tool arguments go to api.typesafe.ai. Without the key, the Jev integration is disabled.
The screening comparison gives me a reason to keep experimenting. Measuring the complete pipeline is the next step.
Want to try this later? Bookmark this post; the repo links above include the setup instructions and evaluation harnesses.
Disclosure: AI assisted with research and drafting. The implementation and experiments described here are my own.
Top comments (0)