DEV Community: bossandboss

👻 GhostEnv

bossandboss — Sat, 18 Jul 2026 13:12:31 +0000

Secure your development secrets inside your native OS Keychain without ever polluting your source code or exposing secrets on Git.

GhostEnv is a revolutionary, polyglot, and 100% language-agnostic tool designed to completely eradicate accidental API key and authentication secret leaks on GitHub.

Moving away from obsolete background auto-rewriting and language-specific bootstrap injections (like node -r), GhostEnv introduces an Interactive CLI Terminal UX Flow during Git pre-commits and a Universal Transient Environment Wrapper (ghostenv run).

🚀 Key Features

1. Interactive Git Pre-Commit Hook

No more silent, background file rewrites. When GhostEnv's AST and regex scanner detects secrets in staged files (git diff --cached):

Instant Terminal Interaction: The commit pauses and displays an interactive CLI prompt directly in your console.
Visual Confirmation (Simulated Git Diff): If you select Option [1] to secure your keys, GhostEnv simulates and displays the exact code changes via a clean, colorized terminal git diff representation.
Final [Y/n] Confirmation: Your files are modified and re-staged (git add) only after you give explicit validation.

2. Universal Command Execution Wrapper (`ghostenv run`)

GhostEnv eliminates the need for language-specific runtime hooks or custom library imports:

100% Language-Agnostic: Works with any language, framework, or process runner.

ghostenv run -- npm start
ghostenv run -- python main.py
ghostenv run -- docker-compose up

In-Memory Enriched Injections: Right before your target application starts, the wrapper extracts the project's secrets from your OS native credentials manager (macOS Keychain, Windows Credential Manager, Linux Secret Service) and injects them strictly into the transient process memory environment variables ($ENV).
Zero Local Disk Footprint: Credentials never touch local storage, and they completely evaporate from memory as soon as the child process exits.

3. Premium Performance & Logging

Ultra-fast execution path (<300ms before displaying the terminal interactive prompt).
Premium terminal layouts featuring intuitive emojis, clean spacing, and readable status indicators.
Full backward compatibility with native operating system keychains.

https://github.com/bossandboss/GhosteEnv

Building EdgeSync-LLM: The Final Architecture for Decentralized, Offline-First Local AI 🚀

bossandboss — Sun, 12 Jul 2026 10:32:26 +0000

Published: true

Description: A deep dive into the final version of EdgeSync-LLM—bringing fast, secure, synchronized Large Language Models straight to edge hardware.

Tags: ai, open source, architecture, edgecomputing, webdev

The cloud dependency trap in modern AI is real. Relying on remote servers means paying hefty subscription fees, coping with unpredictable network latency, and hoping your sensitive data doesn’t leak.

But what if you could run a Large Language Model completely offline on consumer-grade edge hardware, and have multiple local devices intelligently synchronize their states and contexts without a single drop of internet?

Meet EdgeSync-LLM — a local-first, decentralized AI synchronization framework designed for the modern edge. In this post, I will break down the final architecture, key optimizations, and how you can get started with it.

💡 The Vision: Why "Edge" + "Sync"?

Running local LLMs (using tools like llama.cpp or quantized models) isn't entirely new. However, most local setups treat the device as an isolated island.

If you have a network of edge devices (like smart home nodes, local workplace computers, or off-grid field tablets), they have no native way to share learned context, active system prompts, or conversational memory with each other safely.

EdgeSync-LLM solves the synchronization gap by establishing a lightweight, decentralized communication layer right on top of optimized local inference.

🏗️ The Core Architecture

The final version of EdgeSync-LLM splits the workload into three distinct, highly integrated layers:

1. The Optimized Inference Engine

To run an LLM efficiently on hardware with constrained RAM, we rely heavily on advanced quantization formats (like GGUF). The framework is optimized to run 1B to 8B parameter models, allowing it to achieve smooth token generation speeds even on everyday laptops or high-end single-board computers.

2. The Peer-to-Peer Sync Layer

Instead of routing context histories back to a centralized cloud database, EdgeSync-LLM establishes a secure, local peer-to-peer (P2P) network mesh. When one device gains a specific contextual update or systemic state change, it broadcasts a lightweight cryptographic delta patch across the local network.

📊 Final Version Performance Metrics

After rigorous testing and optimization for this final release, the framework has hit some massive performance milestones:

Blazing Fast Generation: Achieved stable 12-15+ tokens per second on standard, unaccelerated consumer edge CPUs.
Aggressive Memory Optimization: Cut active RAM usage down to under 3.5 GB, meaning it runs quietly in the background without hogging host resources.
Instant Local Syncing: Sub-second state synchronization latencies across local nodes.

🛠️ Getting Started (Quick Setup)

Want to deploy it locally? The framework is designed to be developer-friendly. You can spin up a node with just a few terminal commands.

Prerequisites

Make sure you have your target quantized model ready (.gguf format) and node packages/Python dependencies installed according to your system environment.

Installation

# Clone the repository
git clone https://github.com/bossandboss/EdgeSync-LLM.git
cd EdgeSync-LLM

# Install dependencies and initialize the engine
npm install

Spin up a synchronized node

# Start the EdgeSync engine and bind it to a local mesh port
npm run start -- --model ./models/llama-3-8b-q4.gguf --port 8080 --sync-port 9090

Once multiple instances are live on your local network, they automatically detect peer broadcast beacons and begin securely sharing context deltas.

🌐 Real-World Impact

Strict Privacy Sectors: Medical environments can deploy decentralized triage AI assistants across floor tablets without risking patient data compliance.
Zero-Connectivity Fieldwork: Disaster relief or research groups operating entirely off-grid can maintain shared AI synchronization using local Wi-Fi or radio mesh.
Next-Gen Smart Homes: Private automation where your home data stays localized to the physical property.

⭐ Open Source & Next Steps

EdgeSync-LLM is built entirely out in the open. The code, full architectural diagrams, and comprehensive deployment documentation are live on GitHub right now.

We are currently looking into hardware-native NPU acceleration and zero-knowledge proof integrations for even tighter syncing security. If this sounds like something you want to build or experiment with, we'd love your feedback, issues, and pull requests!

👉 Check out the repo here: https://github.com/bossandboss/EdgeSync-LLM

What are your thoughts on running synchronized AI on the edge? Drop your questions or optimization ideas in the comments below!

I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State

bossandboss — Sat, 11 Jul 2026 21:17:57 +0000

Local LLM inference has an expensive habit:
It recomputes prefixes it has already seen.
A system prompt.
A reused RAG document.
A few-shot block.
A long static context.
If the prefix is identical, why pay the prefill cost again?
That's the problem I explored with EdgeSync-LLM.
The idea
The mechanism is simple:
Prompt = shared prefix + new suffix
On the first request, EdgeSync prefills the prefix and captures its KV cache state.
On the next request sharing that exact prefix, it restores the state and decodes only the new suffix.
No llama.cpp fork.
No patch.
The current validated path uses the public:
llama_state_seq_get_data
and
llama_state_seq_set_data
APIs.
Measured on a real Android ARM64 phone
Model:
Qwen2.5-0.5B-Instruct Q4_K_M
Shared prefix:
123 tokens
40 requests. 4 threads. Release build.
Path
Mean TTFT
p50
p95
Cold
4828 ms
4752 ms
5297 ms
KV state reuse
486 ms
476 ms
569 ms
9.9× lower TTFT on cache hits.
The warm path was approximately:
363 ms to decode the 10-token suffix
123 ms to restore the state blob
Fragment size: 1.64 MB
I also measured the same mechanism on x86-64.
Cold mean TTFT: 1395 ms
Warm mean TTFT: 185 ms
That's 7.5× on cache hits.
But I almost published a fake 8.8× speedup
This was the most important part of the project.
My first implementation directly copied raw K/V tensors.
It was fast.
Very fast.
The benchmark reported an 8.8× speedup.
There was one problem.
It was wrong.
llama.cpp tracks more than the K/V tensor values. Cache cells also have position and sequence metadata used to construct the attention mask.
Copying tensor values without restoring that bookkeeping produced an inert fragment.
The model skipped prefix computation...
...but attention could not actually see the restored prefix.
14 of 24 cache hits reproduced, token for token, the output of a generation with no prefix at all.
The “speedup” was dropped context.
So I discarded it.
Timing is not enough
A broken cache can be fast.
That's why EdgeSync now runs two correctness checks.
First:
Warm output must match cold output for the same prompt.
Results:
ARM64: 28/28
x86-64: 24/24
Second:
The warm output is compared with a generation from the suffix alone, without the prefix.
If both outputs match, the injected fragment may be doing nothing.
Results:
ARM64: 0/28 matches
x86-64: 0/24 matches
This second control is what caught the broken raw-tensor implementation.
A latency benchmark alone would not have caught it.
What I learned
There is a broader lesson here for LLM infrastructure benchmarking:
Skipping computation is not proof that state reuse works.
If a cache implementation silently drops context, it can look extremely fast.
Correctness must be part of the benchmark.
Not a separate test.
Not an assumption.
Part of the measurement itself.
What EdgeSync does not claim
I deliberately marked several parts of the project as unvalidated or unsound.
Semantic approximate prefix reuse is not proven.
Reusing KV state between merely “similar” prefixes is incorrect with the current mechanism.
Cross-engine reuse does not work with opaque llama.cpp sequence-state blobs.
Per-layer striding cannot preserve output.
The Android JNI bridge and several cache management components still need end-to-end validation.
The on-device ARM64 sampling is also thin and needs to be rerun with more timed repeats.
I would rather publish a smaller result that can be falsified than a larger claim that cannot survive reproduction.
What's next
The next steps are:
Rerun ARM64 with multiple timed repeats.
Compare complete generated strings instead of one token.
Benchmark HNSW against an exact radix-tree lookup.
Measure state restoration cost as prefix length increases.
Remove or redefine the current PARTIAL-hit mechanism.
The code, benchmark methodology and measured manifests are public.
EdgeSync-LLM:
https://github.com/bossandboss/EdgeSync-LLM
If you work on llama.cpp, local LLM inference, KV cache management or edge AI, I would especially value attempts to reproduce or break the benchmark.
Finding a flaw is more useful to me than a GitHub star.