DEV Community: Manikant Kella

Hermes Agent Has Four Memories — And That's Why It Doesn't Forget You

Manikant Kella — Sat, 23 May 2026 17:37:26 +0000

This is a submission for the Hermes Agent Challenge: Write About Hermes Agent

A cognitive-science tour of the only open-source agent that gets smarter the longer you run it. We map Hermes' file layout onto the four classical memory systems of the human brain — and find an architecture that quietly solves the problem every other agent fakes.

I want to argue something that sounds like marketing but isn't.

Hermes Agent isn't impressive because it's "self-improving." That phrase has been on a thousand landing pages. It's impressive because — possibly by accident, possibly on purpose — the architecture matches how a human brain stores memory. Once you see the mapping, the whole project stops looking like a clever Python wrapper and starts looking like a thesis: if you want an agent that compounds, you have to give it the same memory systems we use.

I'm an ML engineer who builds recommendation and intent systems for a living, and I spend a lot of time thinking about the gap between an agent that's interesting and an agent that holds up after week three. Almost few article write-up of Hermes I've read covers the surface — "it remembers things, it has skills, it has crons." All true. None of them ask the more useful question:

Why this particular decomposition? Why these specific files? Why does a system designed in 2026 look so much like a 1970s cognitive-psychology paper?

That's the post. Grab a coffee.

The goldfish problem (and why "context window" isn't the answer)

Every agent in production today has the same failure mode, and we've all felt it.

Monday: you carefully explain your project, your stack, the three constraints that always trip up the model. The agent crushes the task. You're delighted.

Tuesday: you open a fresh session. The agent has no idea who you are. You re-explain. You re-correct. You re-paste the same context blob. This is what people mean when they say agents "feel like demos." They aren't building anything that lasts; they're paying the same context tax every single morning.

The standard responses to this problem are bad:

Bigger context windows — expensive, doesn't scale with how long you actually want the agent to live (months, not turns), and still amnesiac the moment the conversation ends.
Vector databases — useful for retrieval, terrible for identity. A RAG store doesn't know who you are; it knows which 8 chunks resemble your last question.
Fine-tuning — slow, expensive, and updates an agent's beliefs at the speed of GPU procurement, not the speed of Tuesday.

None of these is memory. They're all variations on "search more text faster." The reason your brain doesn't work this way is that your brain has at least four distinct memory systems, and each one solves a different problem. When cognitive psychologists separated them in the 1970s and 80s — names like Tulving, Squire, Cohen — they did it because patients with brain damage would lose one and keep the others. The systems are genuinely independent.

So here's the simple claim: the reason Hermes Agent works is that it has all four, and they're separate, and they're stored on disk where you can read them.

Let me show you.

A 90-second cognitive-science primer

You don't need a degree to follow this. Four words, four jobs:

System	Holds	Example
Procedural memory	How to do things	Riding a bike. Touch-typing. Driving a stick shift.
Semantic memory	Facts about the world	Paris is the capital of France. Mitochondria are the powerhouse of the cell.
Episodic memory	Things that happened to you	What you had for breakfast yesterday. The argument you had in 2019.
Working memory	What you're holding right now	The phone number someone just said out loud.

There's a fifth thing that isn't a memory system but is the substrate the others run inside: your identity — the sense of who you are that makes all four useful instead of just being a pile of disconnected data. Without a self, your "memories" are just other people's footage.

Now look at this directory listing from a fresh Hermes Agent install:

~/.hermes/
├── SOUL.md           # identity (the substrate)
├── memories/
│   ├── USER.md       # who you are (the agent's model of me)
│   └── MEMORY.md     # what's true in my world right now
├── skills/           # how to do things  (one folder per SKILL.md)
├── sessions/         # SQLite + FTS5: every conversation, searchable
└── cron/             # scheduled jobs that fire in fresh sessions

I'm going to take you through these one at a time and show you which cognitive system each one is doing, why the boundaries are drawn where they are, and what each one prevents from going wrong. We'll end with the part nobody on dev.to has written about yet: how a feature called the Curator (added in v0.12) functions as the agent's equivalent of sleep.

SOUL.md — the substrate (identity)

SOUL.md is the first file loaded into the system prompt at every turn. The Hermes architecture docs call it "slot #1." It contains the agent's personality, communication style, role, and constraints. It's about 1,000–3,000 characters in most setups.

"You are a senior research analyst. You write in short paragraphs. You don't apologize. You never use the word 'leverage.'"

If you stop reading here you'll think this is just a system prompt. It isn't. The trick is what SOUL.md doesn't contain: facts about your projects, your name, anything that changes week-to-week. Those go in different files. SOUL.md is the slow-moving part — the part that makes a Telegram bot reading your code reviews behave consistently different from one that drafts your customer support replies.

You can run multiple Hermes profiles on the same machine, each with its own SOUL.md. A finance-ops agent and a research agent share the same underlying model but read as different colleagues. This is exactly how human personality works: same neural hardware, different self-concepts shaping the output.

What this prevents: tone drift. The slow, frustrating phenomenon where an agent that was crisp on day 1 has become a verbose, hedging, em-dash-spamming mess by day 30 because nothing was holding its character in place.

USER.md and MEMORY.md — semantic memory (facts about your world)

Open ~/.hermes/memories/USER.md after a few weeks of use and you'll find something like this (this is illustrative, not from any specific install):

- Name: Manikant
- Role: ML/AI engineer
- Communication: prefers terse responses, no apology preambles
- Working on: prompt-recommendation pipelines, intent taxonomy
- Avoid: marketing language, "leverage", em-dashes in code blocks

And MEMORY.md:

- Active project: CCx personalization pipeline
- Stack: Databricks, FastAPI on AKS, Redis, OneLLM
- p95 target: sub-1s on warm path
- Recent issue: Kong/Okta 403 — fixed by adding personalization_services scope

Two files. One is who you are (slow-changing). One is what's currently true in your world (medium-changing). Both get injected into the system prompt at session start as a frozen snapshot.

This is semantic memory — declarative facts that the agent can state without re-deriving them. The reason it's split into two files instead of one is the same reason your brain separates "I am a software engineer" from "the build is currently broken." One is identity-level. One is operational. Mixing them is how you end up with an agent that introduces itself in every reply.

The character limits matter. The published numbers I've seen are roughly USER.md ≤ 1,375 chars, MEMORY.md ≤ 2,200 chars. That's tiny on purpose. Memory in Hermes is a pointer index, not a data lake. If something needs more space, it doesn't belong here — it belongs in a skill, or in the episodic store (the session DB). This is the single most important design choice in the system and the one most newcomers fight against.

What this prevents: the context-amnesia tax. You stop re-explaining yourself every Monday. The agent walks into the session already knowing the constraints.

The non-obvious failure mode: stale memory. If MEMORY.md still says "the build is broken" three weeks after you fixed it, the agent will keep working around a phantom. Treat these files like sticky notes on your monitor, not like a database. If your agent starts behaving oddly, this is almost always the first place to look.

SKILL.md — procedural memory (how to do things)

This is the part of Hermes that most reviewers fixate on, but they usually describe what it is without explaining why this particular shape.

A skill is a markdown file with YAML front matter:

---
name: deploy-fastapi-aks
description: Deploy a FastAPI service to Azure Kubernetes Service with our standard manifests
version: 1.2.0
metadata:
  hermes:
    tags: [devops, azure, aks]
---

# Deploy FastAPI to AKS

## When to use
The user wants to ship a FastAPI service to our AKS cluster.

## Procedure
1. Confirm the image tag exists in ACR
2. Run `kubectl apply -f manifests/`
3. Watch the rollout with `kubectl rollout status`
4. Hit /healthz from a pod in the cluster to confirm

## Pitfalls
- If the rollout stalls > 60s, check the readiness probe path
- Token expiry: rotate via `az acr login` before applying

This is procedural memory. It's not facts about your stack; it's a recipe for executing a workflow that survives the agent forgetting your conversation. You can install skills from URLs (hermes skills install https://example.com/SKILL.md), browse a community hub at agentskills.io, or write your own — but the most interesting case is the one where the agent writes them itself.

After Hermes solves a complex task, it can call a tool called skill_manage and propose saving the solution as a new skill. You confirm, it writes the file, and from then on the workflow is one slash command away. The next time you ask for the same kind of thing, the agent doesn't re-derive the solution from first principles — it loads the recipe.

Here's the part that took me three readings of the Hermes architecture page to appreciate. Skills don't cost tokens until they're used. The mechanism is called progressive disclosure:

At session start: skills_list() loads a compact list of names and descriptions only. Roughly 3k tokens for ~100 skills.
When the agent decides one is relevant: skill_view(name) reads the full SKILL.md.
If the skill references a deep file: skill_view(name, "references/api-docs.md") reads that one file.

That's a three-tier lazy-load. The agent never pays the full cost of all its procedural knowledge — only the tip of the iceberg you're using right now. This is the unlock that makes the "more skills makes the agent better, not slower" property actually hold. Without progressive disclosure, every skill you add is a tax on every conversation. With it, the cost is bounded by what you actually invoke.

You don't have to take my word for it. The official skills docs (here) lay out the three-tier load explicitly.

What this prevents: workflow drift. The same task done differently every time because the agent is improvising rather than following a known-good procedure. Skills are the difference between "the agent figured it out again" and "the agent did it the way we always do it."

sessions/ + FTS5 — episodic memory (things that happened)

Open ~/.hermes/sessions/ and you'll find a SQLite database. Inside, FTS5 — SQLite's full-text search — indexes every turn, every tool call, every result the agent has ever produced.

This is episodic memory. Not facts ("I prefer dark mode"), not skills ("how to deploy"), but the actual diary of what happened. When does this matter? When you say something like: "that bug we hit two weeks ago with the Redis TTL — what did we end up doing?" Hermes runs a search across the session history, summarizes the relevant turns with the LLM, and brings the answer back.

This is the layer that most people who shrug off "the agent has memory" never actually use. They put facts in MEMORY.md and call it done. The episodic layer is what makes the agent feel like a colleague who was there with you, not just a search engine that remembers your settings.

The reason this is implemented as SQLite + FTS5 instead of a vector database is worth pausing on. Embeddings are great for "find me something semantically similar." But episodic memory is dominated by literal recall: specific error messages, file paths, dates, function names. FTS5 is faster, cheaper, has no extra service to run, and handles literal queries better than embeddings. The trade-off is that fuzzy queries work less well — but that's what the LLM-on-top-of-search layer compensates for. The agent reads the raw matches and synthesizes.

What this prevents: the "did we already try that?" tax. You stop re-running experiments you've already run. The agent can tell you what was tried, what worked, and what was abandoned without you remembering the original session.

Working memory — the running conversation

The current session — your active conversation, the in-flight tool calls, the scratchpad of what the agent has done in the last 20 turns — is working memory. It's bounded by the model's context window, and Hermes has a dedicated component called context_compressor.py that summarizes older turns once the context crosses about 50% of the available window, preserving recent messages and grouping related tool calls together.

This isn't fancy. But it's the unglamorous plumbing that makes long sessions possible without the agent suddenly forgetting what it was just doing. Working memory in humans has the same property: about 7±2 items, and the brain dumps older items into longer-term stores constantly. Hermes does it explicitly because LLMs don't.

The part nobody writes about: the Curator (v0.12+)

If you've read other Hermes write-ups, you've seen the four-pillar story (memory, skills, soul, crons) and the five-pillar story (add the self-improving loop). Almost nobody is talking about what shipped in version 0.12: the autonomous Curator.

The Curator is a cron job — running on a 7-day cycle by default — whose only job is to grade, consolidate, and prune the agent's own skill library. It walks through every skill in ~/.hermes/skills/, scores it, merges near-duplicates, archives skills that haven't been invoked in N weeks, and rewrites descriptions for clarity. The agent does this on itself, while you sleep.

Why does this matter? Because every learning system has the same problem: growth alone isn't intelligence. If your agent adds a skill every time you do something complex, in six months you have 800 skills, half of them redundant, a third of them stale. The skill index gets noisy. The agent picks the wrong skill for a task because the descriptions overlap. The compounding starts to compound backwards.

The Curator is consolidation. In humans, the analogous process is sleep — specifically slow-wave sleep, when the hippocampus replays the day's episodic memories and the cortex selectively keeps, merges, and discards them. You wake up and the things that didn't matter are gone, the things that did are integrated. Memory researchers have been writing about this since the 1990s. Hermes is, as far as I can tell, the first widely-used open-source agent to ship it as a first-class feature.

When I realized this is what the Curator was doing, the architecture story clicked all the way: Hermes isn't a smart loop on top of an LLM. It's an attempt to give a stateless model the same memory-management apparatus a biological system uses. Identity, semantic, procedural, episodic, working — plus consolidation. That's the whole stack.

A diagram, because diagrams help

Here is the mental model I've been building toward:

                    ┌───────────────────────────────────────────┐
                    │            SOUL.md  (identity)            │
                    │  the substrate that makes the rest useful │
                    └────────────────────┬──────────────────────┘
                                         │
        ┌────────────────────┬───────────┼────────────────────────┐
        │                    │           │                        │
   ┌────▼─────┐    ┌─────────▼────┐  ┌───▼───────┐    ┌──────────▼─────────┐
   │ USER.md  │    │  MEMORY.md   │  │ SKILL.md  │    │  sessions/ (FTS5)  │
   │ + USER MODEL    │              │  │           │    │                    │
   │ semantic  │   │  semantic    │  │ procedural│    │      episodic      │
   │ (who I am)│    │ (my world)   │  │ (how-to)  │    │  (what happened)   │
   └───────────┘    └──────────────┘  └─────┬─────┘    └──────────┬─────────┘
                                            │                     │
                                       progressive            FTS5 + LLM
                                        disclosure              recall
                                            │                     │
                                       ┌────▼─────────────────────▼────┐
                                       │   working memory (the turn)   │
                                       │   + context_compressor.py     │
                                       └────────────────┬──────────────┘
                                                        │
                                                ┌───────▼────────┐
                                                │  the Curator   │
                                                │ (weekly cron)  │
                                                │  = "sleep"     │
                                                └────────────────┘

If you've ever stared at a Tulving diagram of memory systems and then stared at this layout, the resemblance is uncanny.

The bonus layer most people miss: trajectory export

There's one more thing I want to flag, because it's the part that makes Hermes interesting if you're an ML engineer rather than just an end user.

Every conversation Hermes has is, by design, exportable in ShareGPT format — the de-facto schema for instruction-tuning data. The repo includes an Atropos integration for reinforcement-learning environments. The agent's own sessions become training data for the next version of the model that runs the agent.

This closes a flywheel that almost no other open-source agent has closed:

You use Hermes. It logs what worked.
The Curator consolidates the useful skills.
The trajectories get exported.
Someone (Nous Research, your team, you) fine-tunes a model on those trajectories.
The fine-tuned model becomes a better Hermes.

This is also why Hermes is described as "built by model trainers." Nous Research isn't a UX shop; they ship the Hermes, Nomos, and Psyche model families. The agent is also their data-collection apparatus. Once you know this, the design choices stop looking arbitrary. Of course they separated procedural from semantic from episodic — they need clean labels to train on.

I'm not making a claim about what fine-tuning runs they're doing internally. The mechanism is there in the codebase; what they do with it is their business. What I'm pointing out is that the architecture is shaped by the data it generates, which is a property almost no other open-source agent has.

What this means if you're building something

If you've read this far you probably want practical takeaways. Here are five.

1. Don't treat MEMORY.md as a dumping ground. It's a sticky note, not a database. If you put everything in it, the agent will get confused about what's currently true. Aggressively prune.

2. Let the agent write skills, then edit them. Hand-written skills from scratch tend to be too generic. Agent-written skills are concrete because they were extracted from a real workflow you just completed. Your job is editorial — tighten the description, add the pitfall you noticed, ship it.

3. The "when to use" section in a SKILL.md is the most important sentence in the file. That's what the agent searches against when deciding to invoke the skill. Treat it like ad copy: specific, keyword-rich, unambiguous.

4. Multiple SOUL.md profiles beat one mega-agent. A finance agent and a research agent with different SOUL.md files and different credentials are cleaner, safer, and easier to debug than one agent with all the skills and all the API keys.

5. The cron + skill combo is the actual moat. A single sentence — "every night at 1am, pull the latest commits and summarize them on Slack" — produces a skill, a scheduled job, and an output destination, all at once. This is what shifts an agent from reactive to proactive. Most "agent" demos can't do this without 200 lines of glue code.

The honest caveats

I'd rather not write a love letter, so:

Stale memory is the #1 reason Hermes "starts acting weird." If your agent is misbehaving, open MEMORY.md before anything else.
Skill name collisions are real. A skill in your home directory shadows the same name in an external repo, which means "it works on my machine" is a known failure pattern when a team shares skills.
The cost story depends on your model choice. Progressive disclosure keeps the agent lean. The model behind it can still be expensive if you point it at a frontier API.
"Self-improving" doesn't mean automatic. The official docs say it themselves: the loop works best when you actively correct mistakes, save things to memory, and update skills. Passive use produces some improvement; active use compounds.
The 40% research-task speedup number that floats around is from a third-party benchmark I haven't personally re-run. Treat it as directional.

Closing

I started this post with a claim: that Hermes Agent works because its architecture matches how a brain stores memory. I think the mapping is real, and I think it explains why this particular agent has 164,000+ GitHub stars while a dozen frameworks with louder marketing are stuck at 5k.

Identity that doesn't drift. Facts that update. Procedures that get reused. Episodes that can be searched. Consolidation that happens while you sleep.

None of those features is novel on its own. Vector DBs do semantic-ish search. Prompt frameworks do identity-ish anchoring. RAG does episodic-ish retrieval. What's new is putting them all in one place, with clean boundaries, in plain markdown files you can cat. That's the architecture. And once you see it through this lens, the next time you build an agent, you'll find yourself reaching for the same five buckets — because they're the buckets that work.

If you've made it this far, I'd love to hear which of the five layers you'd want to extend first. The Curator is the one I'm most curious about — there's a whole research direction in "how should an agent forget" that nobody's seriously explored yet, and it's sitting right there in ~/.hermes/cron/.

If you found this useful, the conversation is open — drop your own mental model in the comments. And if you spot something I got wrong about the internals, please correct me. Hermes is moving fast and I want to keep this honest.

Gemma 4 26B Is Not the MoE You Think It Is

Manikant Kella — Sat, 23 May 2026 17:02:16 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

"Everyone describes Gemma 4 26B as a standard MoE. The architecture says otherwise. Here's the design choice nobody's unpacking."

When Google released Gemma 4 on April 2, 2026, the headlines about the 26B model were predictable: "It's an MoE!" — "Activates only 3.8B parameters!" — "26B quality at 4B cost!"

All of those are technically true. None of them describe what Google actually built.

Read any of the great write-ups on this topic — and there are a lot of them — and you'll find the same explanation of Gemma 4's MoE: "A router picks a small subset of experts for each token." That's the description of Mixtral. Of DeepSeek. Of Qwen. It's a fine description of Mixture-of-Experts in general.

It is not what Gemma 4 26B does.

Google built something quietly different. If you're planning to fine-tune Gemma 4, serve it at scale, or just want to know what's inside the binary you're downloading, this is the architecture detail that's worth your time.

The standard MoE pattern (Mixtral, DeepSeek, Qwen)

Pretend you've never seen MoE before. A transformer is a stack of blocks. Each block has two main components:

Attention — figures out which other tokens matter for the current one.
Feed-forward network (FFN), also called the MLP — applies a learned non-linear transformation. This is where most of the parameters live.

The standard MoE trick is: replace the FFN with multiple FFNs (called "experts") and add a small "router" that picks which ones to run for each token.

Standard MoE block (Mixtral / DeepSeek / Qwen style):

         Input
           │
           ▼
      [Attention]
           │
           ▼
       [Router]  ── decides top-K of N experts
           │
           ▼
  ┌────────┼─────────┬─────────┐
  ▼        ▼         ▼         ▼
[Exp 1] [Exp 2]  [Exp 3] ... [Exp N]
  │        │         │         │
  └────────┴─────────┴─────────┘
           │ (only top-K fire)
           ▼
         Output

Mixtral 8x7B: 8 experts, 2 active per token. DeepSeek V3: 256 experts, 8 active. The router gives you sparse compute — all parameters are stored in VRAM, but only a small fraction does math for any given token.

When people say "MoE," this is what they mean. The dense FFN is replaced by sparse experts. That word matters: replaced.

What Gemma 4 26B actually does

Gemma 4's MoE block keeps the dense FFN. It doesn't replace it. The sparse experts run as a parallel path alongside the dense one, and the outputs are summed. There's also a shared expert that fires on every token regardless of what the router decides.

Three pathways. Two of them always on. One sparse.

Gemma 4 26B MoE block (the actual design):

                 Input
                   │
                   ▼
              [Attention]
                   │
       ┌───────────┼───────────┐
       ▼           ▼           ▼
  [Dense FFN] [Shared Exp]  [Router]
  (always on)  (always on)     │
                               ▼
                    ┌──────────┼──────────┐
                    ▼          ▼          ▼
                 [Exp 1]   [Exp 2]  ...  [Exp 128]
                    │          │           │
                    └──────────┴───────────┘
                       │ (only 8 fire)
                       ▼
       ┌───────────┴───────────┐
       ▼
   [Sum all three pathways]
       │
       ▼
     Output

The numbers, with the caveat that Google hasn't released a full technical report at time of writing: roughly 128 routed experts, 8 active per token, one always-on shared expert, plus the always-on dense FFN. Total parameters ≈ 25.2B, active per token ≈ 3.8B.

That third always-on pathway is what makes this design unusual. Standard MoE is a substitution. Gemma 4's MoE is an addition.

Why Google built it this way

Here's the part nobody is unpacking. Why bother keeping a dense path when you've already built an expert system?

The cleanest answer is robustness.

In a standard MoE, the output quality of each block depends entirely on the router making the right choice. If the router picks the wrong experts for a token, the representation degrades, and there's no fallback. Routers are small networks, easy to break during training, and prone to collapse — a failure mode where the router gets stuck picking the same few experts and the rest go to waste.

In Gemma 4, the dense FFN provides a stable, always-on representational backbone. The shared expert adds a second always-on signal. The routed experts add specialization on top. If the router whiffs on a token, two coherent signals still carry it forward.

That buys you several real things:

Easier training. Router collapse is less catastrophic when the dense path is doing useful work in parallel — your loss still goes down even when the routing is bad.
Cleaner distillation from a dense teacher. Gemma 4 was trained alongside a 31B dense sibling. A student that already has a dense path can absorb a dense teacher's signal naturally; a pure MoE student has to learn to route at the same time it's learning to imitate.
Graceful degradation under quantization. When you Q3 or Q2 the routed experts to save memory, the dense path stays at full precision and keeps the model coherent.

The tradeoff is efficiency. A pure MoE wastes nothing — every FLOP serves the network. Gemma 4 spends FLOPs on a dense path that runs even when the experts could have handled a token alone. Google deliberately traded peak MoE efficiency for training robustness and distillation friendliness.

That choice has downstream consequences. Let's get into them.

The catch: memory ≠ compute

Here is the most expensive mistake you can make with Gemma 4 26B: assuming "3.8B active" means it'll run like a 4B model on your hardware.

It computes like a 4B model. It does not fit like one.

The router doesn't know in advance which experts will be needed for the next token, so all 128 of them must be loaded in VRAM simultaneously. You save nothing on memory.

Real numbers (a recent controlled benchmark on Gemma 4, Phi-4, and Qwen3 measured peak VRAM under realistic inference loads):

Model	Active params / token	Total in VRAM	Measured peak VRAM
Gemma 4 E4B (dense)	4.5B	4.5B	14.9 GB
Gemma 4 26B A4B	3.8B	25.2B	48.1 GB

Same active parameter count. Three-times the memory footprint. People running the 26B MoE on Macs with 24 GB have reported swap thrashing and ~2 tok/s. The model is fast in terms of compute per token; it's slow at fitting on consumer hardware.

The 31B Dense, by comparison, is more honest about its appetite. You see 31B, you allocate for 31B. The MoE label hides the fact that you still need workstation- or server-class memory to use it.

Rule of thumb: if you can't comfortably hold ≥48 GB of model + KV cache + activation buffers, skip the 26B MoE. Use E4B for an edge build, or 31B with quantization-and-offloading for a quality build.

What this means for production serving

Standard playbook for serving a 26B-ish model: one beefy GPU, one request at a time, cap concurrency to control latency.

That's exactly the wrong setup for Gemma 4 26B.

The parallel-dense-plus-experts design rewards concurrency. Different tokens from different users activate different sets of experts. With a sane batcher (vLLM, SGLang, TGI), you can:

Send token A from user 1 through experts {3, 17, 42, 88, 91, 102, 110, 124}
Send token B from user 2 through experts {1, 5, 9, 26, 33, 57, 71, 119}
…in the same forward pass.

This is expert parallelism, and the flag that turns it on is the one that matters in vLLM:

python -m vllm.entrypoints.openai.api_server \
    --model google/gemma-4-26B-A4B-it \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --enable-expert-parallel \
    --served-model-name gemma4-moe

Without --enable-expert-parallel, you're paying the memory cost of an MoE and getting dense-model throughput. With it on, the MoE earns its keep.

The decision tree, in one line:

Single user, laptop, predictable latency → Gemma 4 31B Dense.
Multi-user API with 8+ concurrent requests → Gemma 4 26B MoE with expert parallelism enabled.

Pick one based on your traffic shape, not just the parameter counts on the model card.

What this means for fine-tuning

This is where the parallel-dense design becomes really interesting, and where most tooling hasn't caught up yet.

In Mixtral, DeepSeek, or Qwen, fine-tuning an MoE means choosing what to update:

Update the router → risks expert collapse.
Update the experts → safe but slow and gradient-sparse.
LoRA on the experts → the standard pattern.

In Gemma 4 26B you have a third surface to fine-tune: the dense FFN that runs on every token. That changes the strategy.

A LoRA on the dense FFN gives you a steady, broadly-applied signal that doesn't require any router decision to take effect. The gradients are clean — they fire on every single token in your dataset, no sparse activation problem. Your domain knowledge "lifts the floor" across the whole model.

This suggests a layered approach:

First pass: LoRA on the dense FFN paths only. Cheap, fast, broadly effective. Best for general domain adaptation (legal tone, medical terminology, your company's writing style).
Second pass: LoRA on the shared expert. Similar broad coverage, smaller capacity, slightly more specialized.
Third pass (only if needed): LoRA on selected routed experts. Targeted, expensive, useful when you need specialization the dense path can't absorb (a rare schema, an unusual task format).

I haven't seen this layered strategy spelled out anywhere yet for Gemma 4 — it falls naturally out of how the architecture works, but standard MoE fine-tuning guides don't anticipate having a dense path to lean on.

Practical heads-up: as of launch, full QLoRA tooling for Gemma 4 26B was not ready in the major libraries. Unsloth and the TRL team are working on it. If you're planning a serious fine-tune, watch their repos before committing to a stack.

What this tells us about Google's research direction

Gemma 4 26B's hybrid design isn't an accident. It's a deliberate bet on a few principles:

Distillation matters more than peak MoE efficiency. A model that distills cleanly from a dense teacher is more useful than one that wrings every FLOP out of routing.
Robustness beats benchmark peaks. Quality you can count on across diverse inputs is worth more than a leaderboard score that depends on the router being perfectly trained.
Open weights have to fine-tune well. A model the community can't reliably fine-tune won't grow the ecosystem Google needs to compete with Llama.

The other open-model labs are running a different play. DeepSeek and Qwen are pushing pure MoE further — more experts, finer sparsity, more peak efficiency. Google is going the opposite direction: more pathways in parallel, more redundancy, more graceful failure.

It's too early to say which approach wins long-term. But the next time someone tells you "Gemma 4 26B is just Google's MoE," you can correct them. It's something else — a hybrid that quietly rewrites the rules of how a Mixture-of-Experts model should look.

What to do with this

Three concrete takeaways:

Read architecture diagrams before benchmarks. "MoE" is not one design. The label hides differences that show up in production — in your memory bill, your throughput, your fine-tuning recipe.
Size your hardware for total parameters, not active parameters. Active is compute. Total is memory. The 26B MoE eats ~48 GB at BF16; if you can't hold that, use E4B or 31B with offloading.
If you're fine-tuning, target the dense FFN first. It's the most stable surface in Gemma 4 26B, and most MoE models don't even have one. Use that.

Architecture details aren't trivia. They're the gap between a model that fits your workload and one that quietly disappoints.

Sources: Gemma 4 model card, Welcome Gemma 4 on Hugging Face, Louis Wang's Gemma 4 architecture analysis (cited for the parallel-dense observation), and a controlled multi-model benchmark on Gemma 4 / Phi-4 / Qwen3 (cited for VRAM numbers). Exact expert counts (8 of 128) come from third-party architecture write-ups; Google hasn't released a full Gemma 4 technical report at time of writing. If a paper drops and numbers shift, this post will be updated.

Built on a workstation; nothing here required cloud access. If you want to verify the memory figures, run nvidia-smi during a vLLM serve of google/gemma-4-26B-A4B-it at BF16. The numbers are reproducible.

Google I/O 2026 quietly ended a 20-year-old web problem — meet the HTML-in-Canvas API

Manikant Kella — Sat, 23 May 2026 16:40:23 +0000

This is a submission for the Google I/O Writing Challenge

TL;DR

Among the Gemini 3.5, Antigravity 2.0, and Android XR headlines from Google I/O 2026, Chrome's engineering team quietly shipped an origin trial for an API that resolves a tradeoff every web developer has lived with for two decades: DOM for rich, semantic, accessible UI, or Canvas for fast 3D and pixel-level graphics — pick one.

The new HTML-in-Canvas API lets you draw live DOM elements directly into a 2D canvas, a WebGL texture, or a WebGPU texture, while preserving accessibility, find-in-page, translation, dark mode, autofill, browser extensions, and DevTools inspection. The DOM elements remain real DOM elements — they just render inside the canvas.

That sentence sounds simple. It isn't. This is one of the most consequential changes to the web platform in years, and it shipped behind a one-paragraph bullet point on the dev keynote recap.

This post is the deep-dive that bullet point deserved.

The choice nobody wanted to make

If you've ever built anything ambitious in the browser, you've hit this wall.

Want a beautiful 3D product configurator with real text labels users can copy-paste? Either you live in the DOM and give up on 3D, or you live in WebGL and re-implement font layout, text selection, and accessibility from scratch.

Want a Figma-like canvas with rich form controls inside? You either go full DOM (slow once you have thousands of nodes) or you go canvas and accept that screen readers see a black hole where your UI used to be.

Want a WebXR scene that includes a tooltip the browser can actually translate to the user's language? Tough luck. Canvas is a pixel grid. The browser's translation engine doesn't speak in pixels.

This is the choice the Chrome team's official I/O 2026 blog post names directly:

"For years, web developers have had to make a tough architectural choice when building complex, highly-interactive visual applications on the web: do you lean on the DOM for its rich semantic features, or do you render directly to the <canvas> element for low-level graphics performance?"

The honest answer until I/O 2026 was: both choices were wrong, you picked the one that was less wrong for your app, and you spent the rest of the project apologizing for it.

Look at how the industry coped:

html2canvas / dom-to-image: thousands of GitHub stars, built around taking screenshots of the DOM and dumping them as static images. CORS-restricted. Video and SVG render blank. Slow on complex pages. The output is dead — no interactivity, no accessibility, no text selection.
Figma, Google Docs, Miro: all built bespoke text-layout engines in JavaScript on top of canvas because the DOM couldn't keep up at scale. Beautiful products. Multi-megabyte bundle sizes. Accessibility implementations that had to mirror everything into hidden DOM trees for screen readers.
WebGL frameworks like Three.js and PlayCanvas: an entire cottage industry of HUD libraries, in-scene UI systems, billboard text renderers — none of which inherited any of the DOM's accessibility, find-in-page, translation, or autofill behavior.

Every "solution" was a workaround for the same architectural sin: the browser had two rendering pipelines that didn't speak to each other.

What HTML-in-Canvas actually does

The pitch in one sentence: the <canvas> element can now have HTML children, and the canvas can paint them anywhere it wants, while the browser keeps treating them as real DOM elements.

You don't snapshot the DOM. You don't proxy events. The element is the element — selectable, focusable, accessible, find-in-page-able, extension-friendly, autofilled-by-Chrome — and your canvas is just drawing it at the coordinates you want, with the transform you want, in the texture you want.

Three rendering paths ship at once:

2D Canvas — ctx.drawElementImage(element, x, y) returns a CSS transform you apply back to the DOM element so events line up.
WebGL — gl.texElementImage2D(target, level, internalFormat, format, type, element) works like texImage2D but accepts a DOM element as the source. The element ends up as a real WebGL texture you can map onto any geometry.
WebGPU — device.queue.copyElementImageToTexture(element, { texture }) does the equivalent for WebGPU.

All three keep the DOM element alive underneath the rendered output. Click in the right spot and the real <button> gets the click. Press Cmd+F and your text gets highlighted. Switch Chrome to translate the page and the labels in your 3D scene change language. Enable a dark-mode extension and your in-canvas form follows along.

The smallest possible demo

Set the layoutsubtree attribute on a <canvas> and put real HTML inside it:

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
  <div id="form_element">
    <label for="name">Name:</label>
    <input id="name" type="text" />
  </div>
</canvas>

Then on every paint, draw the element and re-sync its transform so click/focus targets stay correct:

const ctx = document.getElementById("canvas").getContext("2d");
const form_element = document.getElementById("form_element");
const canvas = document.getElementById("canvas");

canvas.onpaint = () => {
  ctx.reset();

  // Draw the form element at (0, 0) inside the canvas
  const transform = ctx.drawElementImage(form_element, 0, 0);

  // Keep the underlying DOM hit-test region aligned with the painted output
  form_element.style.transform = transform.toString();
};

That's the entire API surface for 2D mode. A real <input> lives inside a real <canvas>. Autofill works. Tab order works. Screen readers find it. Find-in-page finds it. None of those sentences were true on the web a month ago.

The layoutsubtree attribute is the key signal — it tells the browser "this canvas has DOM children that are alive; lay them out, expose them to the a11y tree, but let me decide where their pixels go."

The WebGL path — where it gets fun

The really interesting use case isn't 2D — it's mapping live DOM onto 3D meshes. Here's the WebGL primitive:

canvas.onpaint = () => {
  if (gl.texElementImage2D) {
    gl.texElementImage2D(
      gl.TEXTURE_2D,
      0,
      gl.RGBA,
      gl.RGBA,
      gl.UNSIGNED_BYTE,
      form_element  // <-- this is a DOM element, not an HTMLImageElement
    );
  }
};

If you've written WebGL before, look at that line again. The texture source is a <div>. You can put it on a cube, a sphere, a 3D book page, a curved billboard. The text stays selectable. Find-in-page still highlights it on the 3D mesh. The browser's translation engine can localize it in place.

There's a price: getting events to line up in 3D space is harder than in 2D. The on-screen position of a textured element depends on your shader's model-view-projection matrix, and the browser can't deduce it from your draw call. So when you need hit-testing to follow a 3D-transformed element, you compute a DOM matrix that maps clip-space back to pixel-space and hand it to canvas.getElementTransform(element, screenSpaceTransform):

if (canvas.getElementTransform) {
  // 1. WebGL MVP matrix → DOM matrix
  const mvpDOM = new DOMMatrix(Array.from(htmlElementMVP));

  // 2. Normalize the HTML element (pixel size → 1x1 unit square)
  const width = targetHTMLElement.offsetWidth;
  const height = targetHTMLElement.offsetHeight;
  const cssToUnitSpace = new DOMMatrix()
    .scale(1 / width, -1 / height, 1)        // shrink + flip Y
    .translate(-width / 2, -height / 2);     // center

  // 3. Map clip space back to the actual canvas viewport in pixels
  const clipToCanvasViewport = new DOMMatrix()
    .translate(canvas.width / 2, canvas.height / 2)
    .scale(canvas.width / 2, -canvas.height / 2, 1);

  // 4. Compose: viewport · MVP · normalize
  const screenSpaceTransform =
    clipToCanvasViewport.multiply(mvpDOM).multiply(cssToUnitSpace);

  // 5. Tell the browser where the element actually sits on screen
  const computedTransform = canvas.getElementTransform(
    targetHTMLElement,
    screenSpaceTransform
  );
  if (computedTransform) {
    targetHTMLElement.style.transform = computedTransform.toString();
  }
}

The first time I read that snippet I thought "that's a lot of matrix math for one element." Then I realized — this is the math you were doing implicitly anyway to position your fake HUD over your WebGL scene. The difference is now it's blessed by the browser, so the DOM element actually lives at that 3D location and hit-testing works.

If you don't want to write the math yourself, Three.js and PlayCanvas have already shipped wrappers.

Three.js in one line

The Three.js team merged experimental support with a new HTMLTexture class. Mapping a DOM element onto a cube becomes this:

import * as THREE from "three";

const material = new THREE.MeshBasicMaterial();
material.map = new THREE.HTMLTexture(uiElement);  // pass any DOM element

const geometry = new THREE.BoxGeometry(1, 1, 1);
const mesh = new THREE.Mesh(geometry, material);
scene.add(mesh);

If you've used Three.js, you know how striking that is. A texture that's a live, accessible, find-in-page-able, browser-translatable DOM element. PlayCanvas has the equivalent. The framework boilerplate is gone.

What this unlocks that wasn't possible

Let me try to be concrete instead of hand-wavy. Here are five things you genuinely could not build before this API, broken down by what specifically changes:

1. WebXR / 3D product configurators with real, accessible text.
The 3D Pottery Barn sofa demo on a phone could finally have a label that screen readers can read, that Chrome's auto-translate localizes for German users, and that find-in-page highlights when someone searches "leather." Today, all three of those features silently fail inside WebGL textures.

2. Figma-class apps that don't ship a 4MB text engine.
Anything in the canvas-app category — Figma, Miro, Whimsical, Lucid, Photopea — built its own text layout, IME handling, font fallback, copy-paste, and accessibility shadow tree on top of canvas. With HTML-in-Canvas, the layout engine is the browser's. Bundle size goes down. CJK and bidirectional text become free. Accessibility stops being a parallel mirror you have to maintain.

3. WebGL games where the in-world terminal is a real <textarea>.
Every time I've seen a "computer terminal" inside a 3D game on the web, it's been faked with canvas text drawing and a hidden DOM element absorbing keypresses. Now the terminal can be a <textarea>, mapped onto the in-game CRT, with autofill, undo/redo, IME, and clipboard all just working.

4. AI-agent-ready 3D scenes.
This is the part most coverage misses. The Chrome team explicitly calls out indexability and AI agents as a use case: web crawlers and AI agents can now read the text rendered into 2D and 3D scenes because it's still in the DOM. When you combine HTML-in-Canvas with the other underrated I/O 2026 announcement (WebMCP), suddenly canvas-driven web apps become first-class agent surfaces. They were second-class for 20 years.

5. Translatable, dark-mode-aware, extension-friendly 3D experiences.
A user with a Chrome dictionary extension installed gets word definitions when they highlight text inside your 3D scene. A user with prefers-color-scheme: dark gets a dark UI inside your WebXR app. Nobody had to do anything to enable these. They just inherit them now, because the DOM was always responsible for them and the DOM never left.

What I'd build first

Two ideas I can't stop thinking about:

The translatable 3D book. Chrome has a demo where a WebGL-rendered book has pages that are real DOM. Users can change the font with CSS. The browser's translation feature works on the actual page content. This isn't a tech demo — this is the future of edtech, immersive journalism, and museum web experiences. Build one for a specific use case (kids' anatomy book, historical document, recipe book) and you have a portfolio piece that didn't exist a week ago.

A "real form" inside a WebGPU jelly slider. Chrome's WebGPU jelly slider demo shows an <input type="range"> refracting through a 3D jelly material while still responding to step and keyboard arrows. That's the killer pattern: take a piece of HTML you'd find on any boring form and put it inside a WebGPU effect that previously would have required you to give up form semantics entirely. Replace "jelly" with "frosted glass," "paper," "liquid metal" — every brand-marketing site shipping with WebGL right now becomes a candidate.

What I'm skeptical about

This is genuinely exciting. I still have three concerns I'd want to see addressed before I ship anything load-bearing on this API:

1. Chrome-only, behind a flag in Canary, and the origin trial spans only M148–M151.
The Intent to Experiment trail confirms a four-version window. That's three months. After that the API either becomes stable, gets pushed for another OT, or — historically — gets reshaped enough that early adopters have rewrites coming. There's no Firefox or Safari signal yet. The WICG explainer is healthy, but "WICG" doesn't mean "standardized." Plan for change.

2. Main-thread scrolling inside the canvas isn't a free win.
This is buried in the docs, but it matters: HTML-in-Canvas content is drawn from JavaScript, so scrolling and animations inside the canvas can't be off-the-main-thread the way ordinary DOM scrolling can. If you put a long scrollable list inside a canvas, every scroll event walks through your paint handler. Sometimes that's fine. Sometimes it's a 60fps → 24fps cliff you didn't see coming. Profile early.

3. Cross-origin content is blocked.
For security reasons, the API doesn't work with cross-origin iframes. That's the right call — letting one origin paint another origin's content into its own canvas would be an obvious info-leak vector. But it does mean every "embed a third-party widget into my WebXR scene" idea is dead on arrival. Plan to ship same-origin or proxy through your own server.

How to actually try it this weekend

If you want to play with this today:

Install Chrome Canary (you want at least 149).
Navigate to chrome://flags/#canvas-draw-element and enable it. Relaunch.
Clone the WICG html-in-canvas examples and open one of them.
Open the Chrome demos page to see what's possible end-to-end — the 3D book, the animated billboard, the refractive fluid prism text.
Sign up for the origin trial if you want to expose it to real users on your origin during M148–M151.

If you're a Three.js or PlayCanvas dev: jump straight to THREE.HTMLTexture or PlayCanvas's HTML texture support. The framework wrappers turn a 50-line matrix exercise into one line of code.

The big picture

Almost every I/O 2026 recap I've read leads with Gemini 3.5 and Antigravity 2.0. Those are the right headlines for an AI conference. But the announcements that quietly change what the web is capable of tend to come from the platform team, not the model team. WebMCP got coverage because it has "MCP" in the name. HTML-in-Canvas got one bullet because "DOM in canvas, but better" doesn't fit on a slide.

Here's what's actually happening, in one sentence: the browser is no longer forcing you to choose between semantic web and graphical web.

That's a 20-year-old constraint dissolving in real time. The first generation of apps built on this stack — the WebGL stores that read like text, the 3D book your kid can actually use a screen reader on, the canvas dashboards that finally have native accessibility — won't ship this month. They'll ship over the next 18 months as the origin trial matures and Three.js / PlayCanvas integrations stabilize.

But the work to be one of the people who shipped them starts now. Today. With a Chrome Canary flag and one weird layoutsubtree attribute on a <canvas>.

That's the announcement I think actually matters from Google I/O 2026.

If you build anything with HTML-in-Canvas — especially in WebGL or WebGPU — I'd love to see it. Drop a link in the comments.

Find me on GitHub or LinkedIn.

Firewall and governance for your LLM usage

Manikant Kella — Wed, 17 Dec 2025 06:37:46 +0000

Xano AI-Powered Backend Challenge: Full-Stack Submission

Manikant Kella

Dec 14 '25

PromptShield AI – An AI Cost & Risk Firewall Built with Xano

#devchallenge #xanochallenge #ai #backend

Comments 8

5 min read

Guardrail your LLMs

Manikant Kella — Mon, 15 Dec 2025 18:06:39 +0000

Xano AI-Powered Backend Challenge: Full-Stack Submission

Manikant Kella

Dec 14 '25

PromptShield AI – An AI Cost & Risk Firewall Built with Xano

#devchallenge #xanochallenge #ai #backend

Comments 8

5 min read

As teams build AI-powered apps, one issue shows up fast 👉 runaway LLM costs with no guardrails. Thats where PromptShield AI comes into picture

Manikant Kella — Mon, 15 Dec 2025 05:59:38 +0000

Xano AI-Powered Backend Challenge: Full-Stack Submission

Manikant Kella

Dec 14 '25

PromptShield AI – An AI Cost & Risk Firewall Built with Xano

#devchallenge #xanochallenge #ai #backend

Comments 8

5 min read

PromptShield AI – An AI Cost & Risk Firewall Built with Xano

Manikant Kella — Sun, 14 Dec 2025 16:51:17 +0000

This is a submission for the Xano AI-Powered Backend Challenge: Full-Stack, AI-First Application

PromptShield AI – An AI Cost & Risk Firewall for LLM Applications

As teams rapidly build agentic apps and AI-powered features, one problem shows up almost immediately:

LLM costs explode, usage becomes opaque, and there are no guardrails.

Developers lack:

Per-user and per-feature budgets
Visibility into token and cost usage
Protection against risky prompts (PII, secrets)
Smart routing to cheaper models when budgets are exceeded

PromptShield AI solves this by acting as an intelligent backend control plane that sits between applications and LLM providers.

It enforces:

Cost budgets (tenant / user / feature)
Usage analytics and spend visibility
Safety and routing policies
Multi-provider cost control

The result is a production-ready AI infrastructure backend, not just a wrapper around LLM APIs.

🧱 Architecture Overview

Backend: Xano (Postgres, APIs, background jobs, AI workflows)
Frontend: Lovable.dev (low-code SaaS dashboard)
AI-first approach: Backend generated with AI, refined by hand
Public API + Admin UI: Production-ready by design

🎬 Demo

🔗 Live Application:

https://promptshield.lovable.app/

💻 Source Code (GitHub):

https://github.com/Manikant92/promptshield_ai

🎥 Demo Walkthrough Video:

📸 Product Screenshots

🔎 Swagger / Public API:

https://x8ki-letl-twmt.n7.xano.io/api:q5xLch4v

The dashboard shows real API keys, budgets, policies, providers, and usage analytics powered entirely by Xano.

🧠 The AI Prompt I Used (Backend Generation)

All backend workflows, API definitions, and schema refinements are tracked in the GitHub repository below for transparency and reproducibility:

👉 https://github.com/Manikant92/promptshield_ai

I used XanoScript with an AI-first workflow to generate the initial backend.

Below is the original prompt used to bootstrap the system:

You are an expert backend architect building a production-ready, multi-tenant AI infrastructure backend using Xano.

Build a backend called "PromptShield AI" — an AI Cost & Risk Firewall that sits between applications and multiple LLM providers (OpenAI, Anthropic, etc.) to enforce budgets, rate limits, and safety policies before requests reach the LLM.

The backend must be secure, scalable, and suitable for public API consumption.


Create the initial backend for PromptShield AI with the following requirements:

1. Core Concept
PromptShield AI acts as a proxy API for LLM calls. Applications send standard chat/completion payloads to PromptShield, which enforces usage policies, budgets, and risk checks before forwarding requests to LLM providers.

2. Database Schema (Postgres)
Design tables for:
- tenants (org_id, name, plan, created_at)
- api_keys (key, tenant_id, status, last_used_at)
- users (user_id, tenant_id, role)
- llm_providers (provider, model, cost_per_1k_tokens)
- usage_logs (tenant_id, user_id, feature, provider, model, tokens_in, tokens_out, cost, timestamp)
- budgets (tenant_id, scope_type [tenant/user/feature], scope_id, daily_limit, monthly_limit)
- policies (tenant_id, preferred_models, fallback_model, blocked_categories)

3. API Endpoints
Create the following APIs:

POST /llm/proxy
- Accepts OpenAI-compatible chat/completion payload
- Authenticates using API key
- Identifies tenant, user, and feature
- Performs budget checks and policy enforcement
- Routes request to the selected LLM provider
- Logs token usage and cost

POST /limits/configure
- Allows tenants to define per-user, per-feature, or per-tenant budgets
- Supports daily and monthly limits

GET /usage/summary
- Returns aggregated usage by tenant, user, feature, and model
- Optimized for dashboards

4. AI Logic
Use AI workflows to:
- Classify prompts for risky categories (PII, secrets, unsafe content)
- Block or redact requests that violate policy
- Automatically downgrade to cheaper models when nearing budget limits
- Detect anomalous usage spikes (e.g., sudden 10x increase)

5. Background Jobs
- Aggregate daily and monthly usage
- Recalculate remaining budgets
- Run anomaly detection periodically

6. Security & Scalability
- Multi-tenant isolation
- Rate limiting per API key
- Clean error responses
- Extensible provider abstraction

7. Output
Generate:
- Database tables
- API endpoint logic
- AI workflows
- Background jobs
Use clean, maintainable naming and comments suitable for a production backend.

Do NOT generate frontend code.
Focus entirely on the backend implementation in Xano.

This prompt allowed AI to quickly generate a solid baseline backend, which I then refined heavily inside Xano.

## 🛠️ How I Refined the AI-Generated Backend in Xano

AI gave me a starting point — **human refinement made it production-ready**.

### Key Improvements I Made in Xano

#### 🔐 Security & Multi-Tenancy
- Introduced tenant isolation across all tables
- Added API key lifecycle management (create, revoke, rotate)
- Hardened error handling and rate limits

#### 💰 Cost & Budget Enforcement
- Added scoped budgets (tenant / user / feature)
- Implemented background aggregation jobs for daily & monthly usage
- Enabled budget thresholds and warning states

#### 🧠 AI Logic Enhancements
- Added prompt classification for risky categories (PII, secrets)
- Implemented policy-based model fallback when budgets are exceeded
- Designed provider abstraction for future expansion

#### 📊 Observability & Analytics
- Normalized usage logs for dashboards
- Enabled cost-by-model and cost-by-feature views
- Optimized APIs for frontend consumption

Before: AI-generated CRUD-style endpoints

After: A scalable, secure AI infrastructure backend suitable for real-world use

🎨 Frontend: Turning APIs into a Product

I connected the Xano backend to Lovable.dev to build a clean, enterprise-style dashboard.

The UI allows users to:

Manage API keys securely
Define and monitor budgets
Configure routing and safety policies
Analyze token and cost usage with filters and charts

This step demonstrated how Xano’s backend capabilities translate directly into product value.

🚀 My Experience with Xano

What I Loved

AI + Human workflow: AI for speed, Xano for control
Background jobs: Perfect for cost aggregation and analytics
Clean API design: Easy to connect to any frontend
Production mindset: Xano encourages scalable patterns by default

Challenges

Thinking through multi-tenant isolation correctly (worth the effort)
Designing APIs that balance flexibility and simplicity

Overall, Xano made it incredibly easy to go from idea → AI-generated backend → production-grade system in a very short time.

🏁 Final Thoughts

PromptShield AI is not just a demo — it’s a realistic example of how AI-assisted backend development, combined with thoughtful human refinement, can produce scalable, secure, and maintainable systems.

Xano was the perfect platform to bring this idea to life.

Thanks for checking it out! 🚀

How I Built an AI-Powered Website Modernizer in 3 Days with Kiro

Manikant Kella — Fri, 05 Dec 2025 18:38:32 +0000

I built ReVitalize - an AI-powered platform that transforms outdated websites into modern, accessible designs - in just 3 days using Kiro. What would typically take 2-3 weeks became a weekend project thanks to Kiro's context-aware vibe coding.

The Result:

60+ files of production-ready code
Complete AI transformation pipeline
Beautiful UI with animations
Side-by-side preview system
Chat refinement, component extraction, accessibility auditing
Export to ZIP and GitHub

The Secret: Natural conversations with an AI that actually remembers context and evolves with you.

The Problem: Outdated Websites Everywhere

Have you ever visited a website that looks like it's from 1995? Plain HTML, no styling, broken layouts? Companies like Berkshire Hathaway still run websites that look ancient!

I thought: "What if AI could instantly modernize any outdated website?"

That's how ReVitalize was born - an AI-powered tool that transforms any website in seconds, not weeks.

The Kiro Difference: Context-Aware Development

Before Kiro: The Traditional Approach

1. Plan everything upfront (2-3 days)
2. Write detailed specs (1 day)
3. Set up project structure (1 day)
4. Build features one by one (1-2 weeks)
5. Debug and refine (3-4 days)
6. Write documentation (1-2 days)

Total: 2-3 weeks

With Kiro: Iterative Conversations

Day 1: "Build a full production-grade application called ReVitalize"
       → 60+ files generated with complete architecture

Day 2: "Make UI much better, add example sites"
       → Beautiful landing page with animations

Day 3: "Preview isn't showing" → Fixed with debugging
       "Token limit exceeded" → Switched to better model
       "Secure API keys" → Complete security audit

Total: 3 days

What Makes Kiro Special?

1. Context Awareness

Traditional AI coding tools forget context. You have to re-explain everything.

Kiro remembers:

Previous conversations
Project structure
Design decisions
Code patterns

Example:

Me: "Add example sites with auto-fill"
Kiro: *Remembers the URLInputForm component*
      *Updates LandingPage to pass URL*
      *Modifies TransformationWorkspace*
      *Updates all connected components*

No need to specify which files to modify. Kiro knows.

2. Intelligent Problem-Solving

When we hit a token limit error with GPT-4 Turbo:

Me: "Token limit exceeded"
Kiro: "Let me analyze the OpenAI pricing table..."
      "GPT-4o Mini has 200K TPM vs 30K for GPT-4 Turbo"
      "It's also 98% cheaper!"
      *Updates all model references*
      *Updates UI branding*
      *Creates documentation*

Kiro didn't just fix the error - it optimized our entire approach.

3. Vibe Coding

Instead of writing detailed specs, I had natural conversations:

// Traditional approach:
"Create a React component called LandingPage with:
- Animated gradient background using Framer Motion
- Three floating orbs with different animation delays
- Hero section with gradient text
- Feature grid with 9 cards
- Hover effects on cards
- Stats section with 4 metrics
- CTA section
- Footer with links"

// With Kiro:
"Make UI much better than this reference website"

// Result: 500+ lines of beautiful code in 5 minutes

The Build Process: A Day-by-Day Breakdown

Day 1: Architecture & Core Features

Initial Prompt:

Build a full production-grade application called ReVitalize,
an AI-powered platform that modernizes any outdated website.

What Kiro Generated:

Complete Next.js 14 project structure
TypeScript configuration
API routes for scraping, transformation, refinement
React components (URLInput, Preview, Chat, etc.)
AI integrations (OpenAI, Google Gemini)
Puppeteer scraping system
Type definitions
Documentation

Files Created: 60+
Time: ~2 hours of conversation

Day 2: UI/UX & Refinement

Conversations:

Me: "Make UI much better than reference website"
Kiro: *Creates stunning landing page*
      - Animated gradient background
      - Floating orbs animation
      - Feature cards with hover effects
      - Professional typography

Me: "Add example sites that auto-populate"
Kiro: *Adds 4 example cards*
      *Implements click-to-fill*
      *Updates all connected components*

Me: "Text size too large"
Kiro: *Adjusts all typography*
      *Maintains hierarchy*
      *Updates responsive breakpoints*

Time: ~4 hours of iterative refinement

Day 3: Debugging & Polish

The Preview Problem:

Me: "Preview isn't showing"
Kiro: *Adds comprehensive logging*
      *Identifies issue: React imports instead of HTML*
      *Creates React-to-HTML converter*
      *Implements three-level fallback system*
      *Generates beautiful fallback HTML*

The Token Limit Issue:

Me: "Token limit exceeded"
Kiro: *Analyzes OpenAI pricing*
      *Suggests GPT-4o Mini*
      *Updates all references*
      *Documents the change*

Security Audit:

Me: "Check for hardcoded API keys"
Kiro: *Scans entire codebase*
      *Finds exposed keys*
      *Updates .gitignore*
      *Clears sensitive data*
      *Creates SECURITY.md*

Time: ~6 hours of problem-solving

The Most Impressive Moments

1. Complete AI Transformation System

One conversation generated:

// src/lib/ai/openai.ts
- Smart prompt engineering
- Structured JSON output
- React-to-HTML conversion
- Three-level fallback system
- Error handling
- Comprehensive logging
- Theme-specific guidelines

// 500+ lines of production-ready code
// Time: 5 minutes

2. Beautiful Landing Page

// src/components/LandingPage.tsx
- Animated gradient background
- Three floating orbs with physics
- Hero section with triple gradient text
- 9 feature cards with hover-to-expand
- Stats section with animated counters
- Premium CTA section
- Professional footer

// 800+ lines with Framer Motion
// Time: 10 minutes

3. Security Audit & Fix

# One prompt: "Check for hardcoded API keys"

# Kiro:
1. Scanned entire codebase
2. Found exposed keys in .env.local
3. Updated .gitignore with comprehensive rules
4. Cleared all sensitive data
5. Created SECURITY.md with guidelines
6. Updated .env.example with documentation

# Files modified: 4
# Time: 3 minutes

Key Features That Made the Difference

1. Steering Documents

Created transformation-rules.md in .kiro/steering/:

## LLM Output Format

Always return JSON matching this exact structure:
{
  "summary": "...",
  "components": [...],
  "final_build": { "files": [...] }
}

Impact: Consistent AI outputs, no parsing errors, predictable structure.

2. Spec-Driven Development

Created .kiro/spec.yaml:

features:
  - url_input
  - scraping
  - ai_transformation
  - preview
  - chat_refinement
  - export

api_endpoints:
  - POST /api/scrape
  - POST /api/transform
  - POST /api/refine

Impact: Clear architecture, consistent APIs, complete features.

3. Iterative Conversations

Broad → Specific → Refine → Polish

"Build ReVitalize" 
  → "Add example sites"
    → "Make text smaller"
      → "Fix preview display"

Each conversation built on the previous context.

The Technical Stack

Frontend:

Next.js 14 (App Router)
TypeScript
Tailwind CSS
Framer Motion
React Syntax Highlighter

Backend:

Next.js API Routes
Puppeteer (scraping)
Cheerio (HTML parsing)

AI:

OpenAI GPT-4o Mini (primary)
Google Gemini 1.5 Flash (fallback)

All generated and integrated by Kiro through conversations.

Challenges & How Kiro Helped

Challenge 1: Model Selection

Problem: Started with GPT-5.1 (doesn't exist), then GPT-4 Turbo (token limits)

Kiro's Solution:

"Let me analyze the OpenAI pricing table...
GPT-4o Mini has:
- 200K TPM (vs 30K)
- 98% cheaper
- Optimized for code generation
- Perfect for your use case"

Result: No more token errors, 98% cost reduction

Challenge 2: Empty Preview

Problem: Preview showed "No preview available"

Kiro's Solution:

1. Added comprehensive logging
2. Identified issue: AI generating React imports
3. Created React-to-HTML converter
4. Implemented three-level fallback
5. Generated beautiful fallback HTML

Result: Preview always works, even if AI fails

Challenge 3: Security

Problem: API keys potentially exposed

Kiro's Solution:

1. Scanned entire codebase
2. Found all sensitive data
3. Updated .gitignore
4. Created security guidelines
5. Documented best practices

Result: Production-ready security

What I Learned

1. Context is Everything

Traditional AI tools lose context. Kiro maintains it across conversations, making iterative development natural.

2. Hybrid Approach Works Best

Spec-driven for initial architecture
Vibe coding for refinement and debugging
Steering docs for consistency

3. AI as Partner, Not Tool

Kiro doesn't just generate code - it:

Debugs with you
Suggests optimizations
Maintains consistency
Writes documentation
Solves problems

4. Speed Without Sacrificing Quality

3 days of development with Kiro produced:

Production-ready code
Comprehensive error handling
Security best practices
Complete documentation
Beautiful UI/UX

The Results

ReVitalize Features:

✅ Instant website transformation
✅ Side-by-side comparison
✅ 5 theme options
✅ Chat refinement
✅ Component extraction
✅ Accessibility audit
✅ Export to ZIP/GitHub
✅ Beautiful animations

Development Stats:

Time: 3 days (vs 2-3 weeks)
Files: 60+ production-ready
Lines of Code: ~5,000+
Cost per transformation: $0.003
Performance: 15-20 seconds per transformation

Try It Yourself

Want to experience Kiro's power?

ReVitalize Demo: [https://youtu.be/-qBKbOdIa7U]
GitHub: [https://github.com/Manikant92/ReVitalize]

Get Kiro: [https://kiro.dev/]

Final Thoughts

Kiro changed how I approach development. Instead of:

Planning everything upfront
Writing detailed specs
Building in isolation
Debugging alone

I now:

Start building immediately
Refine through conversation
Iterate with an AI partner
Solve problems together

The future of development isn't about replacing developers - it's about augmenting our capabilities with AI that truly understands context.

If you're still writing boilerplate by hand or spending hours on repetitive tasks, you're missing out on what's possible with context-aware AI development.

Questions?

Drop a comment below! I'd love to hear about your experiences with AI-assisted development.

Built with ❤️ using Kiro for the Kiroween Hackathon

AICHA: AI-Powered Healthcare Assistant with Permit.io Authorization

Manikant Kella — Sun, 04 May 2025 10:40:42 +0000

This is a submission for the Permit.io Authorization Challenge: AI Access Control

What is AICHA?

AICHA (AI-powered Healthcare Assistant) is a modern healthcare management platform that uses artificial intelligence to help doctors, nurses, and administrators deliver better care. With features like AI-driven medical analysis, treatment suggestions, and secure patient data management, AICHA is designed to make healthcare smarter and safer.

But with great power comes great responsibility—especially when it comes to sensitive medical data and AI features. That's why AICHA uses Permit.io for fine-grained, externalized authorization, ensuring that only the right people can access the right features at the right time.

How AICHA Leverages Permit.io, Gemini AI, and Supabase.

Permit.io Integration:

Permit.io is the backbone of AICHA's access control and security.
Every time a user logs in or tries to use a feature (like running Gemini AI analysis or viewing patient data), AICHA checks with Permit.io to see if they're allowed.
All access policies (who can do what, when, and where) are managed centrally in Permit.io, not scattered in the code.
User roles and permissions are synced from Supabase to Permit.io, so changes are reflected instantly.
Permit.io logs every permission check and action for auditing and compliance.
This means:
- Only doctors in the right department can run Gemini AI analysis on their patients
- Nurses can view results but not run or approve AI features
- Admins can manage everything, but all actions are tracked
Permit.io makes it easy to update policies (like restricting AI use to working hours) without changing any code—just update the policy in the dashboard.

Gemini AI Integration:

AICHA uses Google Gemini AI to power its medical analysis and treatment suggestion features.
Doctors can submit patient data for analysis, and Gemini provides:
- Disease risk predictions
- AI-generated treatment recommendations
- Natural language explanations for results
Gemini's advanced language and reasoning capabilities help make AI suggestions more accurate and explainable for medical staff.
All AI-powered actions are protected by Permit.io policies, so only authorized users can access Gemini features.

Supabase Integration:

Supabase is used as the backend database for AICHA.
It stores:
- User accounts and roles
- Patient records
- Audit logs of all actions (including Permit.io permission checks)
Supabase's real-time features help keep user data and permissions in sync with Permit.io.
When users are created or updated in Supabase, they are automatically synced to Permit.io for up-to-date access control.

Key Features of AICHA (and Who Gets to Use Them)

AICHA is built for different types of users, each with their own set of features:

Administrators
- Full access to all patient records and system settings
- Manage users and roles
- Approve or audit AI analysis and treatments
Doctors
- Run AI-powered medical analysis on patient data
- Suggest and approve treatments
- View and update patient records
Nurses
- View patient records and AI analysis results
- Monitor patient vitals
Patients (optional/future)
- View their own health records and AI-generated reports

How does AICHA make sure only the right people can do these things?

That's where Permit.io comes in! Every action—like running an AI analysis or viewing a patient record—is checked against Permit.io's policies before it happens. This means:

Doctors can't approve treatments for patients outside their department
Nurses can't run AI analysis, but can view results
Admins can do everything, but every action is logged for compliance

Why Permit.io? (vs. Traditional Authorization)

Traditionally, access control is hard-coded into the app. This means:

Authorization logic is scattered everywhere
Changing permissions means changing code (and redeploying)
It's easy to miss a check, leading to security holes
Auditing who did what is a pain

With Permit.io (externalized authorization):

All policies are managed in one place (the Permit.io dashboard or config)
You can update permissions instantly—no code changes needed
Every access check is logged automatically
It's easy to create complex, context-aware rules (like department-based or time-based access)

Example:

Want to let doctors run AI analysis only during working hours? Just update the policy in Permit.io—no code changes required!

How AICHA Leverages Permit.io (In Simple Terms)

Role-Based Access: Each user is assigned a role (admin, doctor, nurse) and only sees features they're allowed to use.
Context-Aware Policies: Access can depend on department, time of day, or even the type of AI analysis.
User Management: Users are synced from our database to Permit.io, so roles and permissions are always up to date.
Real-Time Checks: Every time a user tries to do something important, AICHA checks with Permit.io to see if it's allowed.
Audit Logging: Every action is logged for compliance and security.

Setting Up Permit.io for AICHA (For First-Time Users)

Option 1: Manual Setup via Permit.io Dashboard

Create a Permit.io Account:
- Go to Permit.io and sign up
- Create a new project called "AICHA"
Configure Resources:
Navigate to the Resources tab and create three key resources:

a. Patient Records:

   Key: patients
   Name: Patient Records
   Description: Medical patient information
   Actions:
     - create (Create new patient records)
     - read (View patient information)
     - update (Modify patient details)
     - delete (Remove patient records)
   Attributes:
     - department_id: string
     - record_type: string

b. AI Analysis:

   Key: ai_analysis
   Name: AI Analysis
   Description: AI-powered medical analysis
   Actions:
     - run (Execute AI analysis)
     - approve (Approve analysis results)
     - view (View analysis reports)
   Attributes:
     - analysis_type: string
     - department_id: string

c. Treatment Plans:

   Key: treatment
   Name: Treatment Plans
   Description: Patient treatment recommendations
   Actions:
     - suggest (Create treatment suggestions)
     - approve (Approve treatment plans)
     - view (View treatment details)
   Attributes:
     - department_id: string
     - treatment_type: string

Set Up Roles and Permissions: Create the following roles with specific permissions:

a. Administrator:

   Key: admin
   Permissions:
     - create:patients
     - read:patients
     - update:patients
     - delete:patients
     - run:ai_analysis
     - approve:ai_analysis
     - view:ai_analysis
     - suggest:treatment
     - approve:treatment
     - view:treatment

b. Doctor:

   Key: doctor
   Permissions:
     - read:patients
     - update:patients
     - run:ai_analysis
     - view:ai_analysis
     - suggest:treatment
     - approve:treatment
     - view:treatment

c. Nurse:

   Key: nurse
   Permissions:
     - read:patients
     - view:ai_analysis
     - view:treatment

Get API Credentials:
- Go to Settings > API Keys
- Copy your API key, project ID, and environment
- Add them to your .env file:
```
 PERMIT_API_KEY=your_api_key
 PERMIT_PROJECT_ID=your_project_id
 PERMIT_ENVIRONMENT=dev
```

Option 2: Automated Setup (Using Our Scripts)

If you want to set up everything automatically, just run our setup script!

Configure your .env file (see above)
Run the setup script:

   cd backend
   npm install
   node scripts/permit-setup.js

This will:

Create all resources and roles in Permit.io
Set up permissions and policies
Sync all users from your database

How Permissions Are Checked in AICHA

Whenever a user logs in or tries to use a feature, AICHA:

Checks their role and department
Asks Permit.io if they're allowed to do the action (like run AI analysis)
If Permit.io says yes, the action happens; if not, it's blocked
Every check and action is logged for auditing

Example:

// Check if a doctor can run AI analysis
const canRunAI = await permit.checkAIAccess(currentUser, 'diagnosis', patientDepartment);
if (!canRunAI) {
  return res.status(403).json({ error: 'Not allowed to run AI analysis' });
}

Core Implementation Files (How We Use Permit.io)

1. permit.js (Core Permit.io Integration)

const { Permit } = require('@permit.io/permit-node');
const dotenv = require('dotenv');
dotenv.config();

class PermitService {
  constructor() {
    this.permit = new Permit({
      token: process.env.PERMIT_API_KEY,
      pdp: "https://cloudpdp.api.permit.io",
      projectId: process.env.PERMIT_PROJECT_ID,
      environment: process.env.PERMIT_ENVIRONMENT
    });
    this.initializeSyncService();
  }
  async initializeSyncService() {
    const { syncUsers } = require('./permit-sync-users');
    await syncUsers();
    setInterval(syncUsers, 5 * 60 * 1000);
  }
  async checkPermission(user, action, resource) {
    return await this.permit.check(user.id, action, resource);
  }
  async checkContextualPermission(user, action, resource, context) {
    const baseContext = {
      user: { role: user.role, department_id: user.department_id },
      resource: { type: resource, department_id: context.department_id },
      time: new Date().toLocaleTimeString('en-US', { hour12: false })
    };
    return await this.permit.check(user.id, action, resource, { ...baseContext, ...context });
  }
  async checkAIAccess(user, analysisType, patientDepartment) {
    const context = { analysis_type: analysisType, department_id: patientDepartment, time: new Date().toLocaleTimeString('en-US', { hour12: false }) };
    return await this.checkContextualPermission(user, 'run', 'ai_analysis', context);
  }
  async checkTreatmentAccess(user, treatmentType, patientDepartment) {
    const context = { treatment_type: treatmentType, department_id: patientDepartment };
    return await this.checkContextualPermission(user, 'suggest', 'treatment', context);
  }
  async logAudit(userId, action, resource, details) {
    await this.permit.audit.log({ user: userId, action, resource, details });
  }
}
module.exports = new PermitService();

What does this do?

Connects to Permit.io
Syncs users
Checks permissions (including context-aware checks)
Logs every important action

2. permit-utils.js (Permission Helpers)

async function checkAccess(permit, user, action, resource) {
  const permitted = await permit.check(user.id, action, resource);
  if (!permitted) return false;
  const context = {
    user: { role: user.role, department_id: user.department_id },
    resource: { type: resource, department_id: user.department_id }
  };
  return await permit.checkWithContext(user.id, action, resource, context);
}
module.exports = { checkAccess };

What does this do?

Checks if a user can do something, with extra context (like department)

3. permit-sync-users.js (User Sync)

const { Pool } = require('pg');
const permit = require('./permit');
const dotenv = require('dotenv');
dotenv.config();
const pool = new Pool({
  user: process.env.DB_USER,
  host: process.env.DB_HOST,
  database: process.env.DB_NAME,
  password: process.env.DB_PASSWORD,
  port: process.env.DB_PORT
});
async function syncUsers() {
  try {
    const result = await pool.query(`
      SELECT u.id, u.email, u.role, u.department_id, u.first_name, u.last_name
      FROM users u
      WHERE u.active = true
    `);
    for (const user of result.rows) {
      await permit.users.sync({
        key: user.id,
        email: user.email,
        firstName: user.first_name,
        lastName: user.last_name,
        attributes: { role: user.role, department_id: user.department_id }
      });
    }
    console.log(`✅ Successfully synced ${result.rows.length} users with Permit.io`);
  } catch (error) {
    console.error('❌ Error syncing users:', error);
  } finally {
    await pool.end();
  }
}
module.exports = { syncUsers };

What does this do?

Syncs all active users from our database to Permit.io, so permissions are always up to date

4. permit.json (Resource & Role Config)

{
  "resources": {
    "patients": {
      "name": "Patient Records",
      "actions": ["create", "read", "update", "delete"]
    },
    "ai_analysis": {
      "name": "AI Analysis",
      "actions": ["run", "approve", "view"]
    },
    "treatment": {
      "name": "Treatment Plans",
      "actions": ["suggest", "approve", "view"]
    }
  },
  "roles": {
    "admin": {
      "permissions": ["create:patients", "read:patients", "update:patients", "delete:patients", "run:ai_analysis", "approve:ai_analysis", "view:ai_analysis", "suggest:treatment", "approve:treatment", "view:treatment"]
    },
    "doctor": {
      "permissions": ["read:patients", "update:patients", "run:ai_analysis", "view:ai_analysis", "suggest:treatment", "approve:treatment", "view:treatment"]
    },
    "nurse": {
      "permissions": ["read:patients", "view:ai_analysis", "view:treatment"]
    }
  }
}

What does this do?

Defines all resources, actions, and roles for Permit.io

5. permit-setup.js (Automated Setup Script)

const { Permit } = require('@permit.io/permit-node');
const dotenv = require('dotenv');
const { syncUsers } = require('../utils/permit-sync-users');
dotenv.config();
const permit = new Permit({
  token: process.env.PERMIT_API_KEY,
  pdp: "https://cloudpdp.api.permit.io",
  projectId: process.env.PERMIT_PROJECT_ID,
  environment: process.env.PERMIT_ENVIRONMENT
});
const resources = {
  patients: {
    name: "Patient Records",
    description: "Medical patient information",
    actions: ["create", "read", "update", "delete"],
    attributes: { department_id: "string", record_type: "string" }
  },
  ai_analysis: {
    name: "AI Analysis",
    description: "AI-powered medical analysis",
    actions: ["run", "approve", "view"],
    attributes: { analysis_type: "string", department_id: "string" }
  },
  treatment: {
    name: "Treatment Plans",
    description: "Patient treatment recommendations",
    actions: ["suggest", "approve", "view"],
    attributes: { department_id: "string", treatment_type: "string" }
  }
};
const roles = {
  admin: {
    name: "Administrator",
    description: "Full system access",
    permissions: ["create:patients", "read:patients", "update:patients", "delete:patients", "run:ai_analysis", "approve:ai_analysis", "view:ai_analysis", "suggest:treatment", "approve:treatment", "view:treatment"]
  },
  doctor: {
    name: "Doctor",
    description: "Medical staff with AI access",
    permissions: ["read:patients", "update:patients", "run:ai_analysis", "view:ai_analysis", "suggest:treatment", "approve:treatment", "view:treatment"]
  },
  nurse: {
    name: "Nurse",
    description: "Basic medical staff access",
    permissions: ["read:patients", "view:ai_analysis", "view:treatment"]
  }
};
async function setupPermit() {
  try {
    console.log('🚀 Starting Permit.io setup...');
    for (const [key, resource] of Object.entries(resources)) {
      console.log(`Creating resource: ${resource.name}`);
      await permit.api.resources.create({ key, name: resource.name, description: resource.description, actions: resource.actions, attributes: resource.attributes });
    }
    for (const [key, role] of Object.entries(roles)) {
      console.log(`Creating role: ${role.name}`);
      await permit.api.roles.create({ key, name: role.name, description: role.description, permissions: role.permissions });
    }
    console.log('Syncing users...');
    await syncUsers();
    console.log('✅ Setup completed successfully!');
  } catch (error) {
    console.error('❌ Setup failed:', error);
    process.exit(1);
  }
}
setupPermit();

What does this do?

Creates all resources and roles in Permit.io
Sets up permissions and attributes
Syncs users from the database

Validating Permit.io Functionality at Runtime

Every time a user logs in or tries to use a feature, AICHA checks with Permit.io to see if they're allowed
All permission checks are context-aware (role, department, time, etc.)
All actions are logged for auditing
User roles and permissions are kept in sync automatically

Conclusion

AICHA shows how externalized authorization with Permit.io makes it easy to build secure, flexible, and auditable healthcare apps. With Permit.io, you can:

Control who can access AI features and sensitive data
Update policies instantly without code changes
Keep a full audit trail for compliance
Make your app safer and easier to manage

Project Repo

https://github.com/Manikant92/AICHA

Demo

Subtitles.ai - There is an ability in every disability

Manikant Kella — Mon, 11 Apr 2022 17:22:36 +0000

Innovative Ideas Challenge

Introduction

The Innovative Ideas category is an ideal choice where I want to leverage Deepgram STT to solve the problem for Hard of Hearing Community(Deaf). I had experience with Deepgram as part of this hackathon where I have submitted one project leveraging Deepgram STT. Due to time constraints, I couldn't be able to submit the solution as a project, but want to bring to notice that with DG STT, real world problems can be solved.

My Deepgram Use-Case

We all know we are in a Sound tech era and there is always a problem for deaf and people with some or other hearing problems (Hard of Hearing Community) where they face many challenges what the others are speaking it can be during phone call interactions, Zoom/Teams/Slack meetings, any YouTube video, any movie watching channel/video or even to listening to audio channels like Spotify etc.,.

With Deepgram STT, there comes the power where subtitles can be generated as generic way without integration with any apps like mentioned above.

An Windows App can be built using Python Tkinter GUI and Deepgram STT and via websocket programming, everything that is getting played/listened can be heard via Windows mic and generate the subtitles so that Hard of Hearing Community can enjoy keeping aside all their disabilities.

Dive into Details

In order to solve the problem for Hard of Hearing people, we can build an Windows (since its the OS majority of the users use) application which converts all sound apps with Subtitles under the bar which makes an effective solution for Hearing problem users.
It converts any application sound that is getting played or listened (Youtube, Netflix, Prime, Spotify, Zoom/Teams call, any Video, any Audio etc.,.) to on Windows OS into Subtitles of any language preference the user have.
Now using the app, without disturbing their other apps, it displays Subtitles (leveraging Deepgram STT) near taskbar of Windows for any sound getting listened to or played to.
It can also have the capability of displaying subtitles in any language the user prefers to.

Conclusion

With Deepgrams STT tech, there is an ability for every disability.
Impacting the Hard of Hearing community with a generic solution where they can leverage the app to full extent with DG STT.
Expand the same solution for Mac, Linux, Android users.

VProfressor.ai - A 24x7 Virtual Professor for Students

Manikant Kella — Mon, 11 Apr 2022 16:56:41 +0000

Overview of My Submission

Education is everything. Knowledge empowers anything.

That said, Covid-19 forced students to virtual learning where students are struggling to interact and get timely feedback, new learnings, clarifications with the professor or friends due to various reasons.

To solve the problem, we came up with web application where it helps the students and increase their speed, efficiency of learning and also get their doubts clarified with additional learnings.

VProfessor.ai contains different features designed to facilitate online learning. Users can upload a wav/mp3/ text files. Then, a Deepgram STT transcript of the audio is returned, along with a summary of the data. This includes key words and main topics, Wikipedia page links, current events from NewsAPI, and recommended YouTube videos. From here, users can either read the summary report on the website or download it as a pdf for their personal studying. Additionally, students may "ask the professor" a question, and get a quick short answer for themselves which leverages wolframalpha api's.

Submission Category

Wacky Wildcards

Link to Code on GitHub

GitHub Link: https://github.com/Manikant92/DG_VProfressor.ai

Additional Resources / Info

Demo Video: https://youtu.be/hPWR2ED0NK4

With just this piece of code comes the entire power for application.

  async function transcribe(file){
    const streamSource = {
      stream: fs.createReadStream(file),
      mimetype: mime.getType(file),
    };

    const response = await deepgram.transcription.preRecorded(streamSource, {
      punctuate: false,
      utterances: true,
    });
    console.log("DG Response" + response);

    var srtTranscript = response.toSRT(); // toWebVTT() //toSRT()
    srtTranscript = srtTranscript.replace(/\d+/g, '');
    srtTranscript = srtTranscript.replace(/:/g,'');
    srtTranscript = srtTranscript.replace(/-/g,'');
    srtTranscript = srtTranscript.replace(/>/g,'');
    srtTranscript = srtTranscript.replace(/,/g,'');
    console.log("DG SRT Transcript" + srtTranscript);
    return srtTranscript;
  }

Dive into Details

We are leveraging Deepgram Speech-to-Text API and use it to convert audio/video files to a written transcript.
With Azure Text Analytics, we will generate an analysis report containing transcript, summary, and keywords.
With that keywords output, we will use the Wikipedia API, NewsAPI to generate links based on the keywords. To provide more information, we will also create a system to search recommended YouTube videos based on a search query, which used the YouTube-Data API.
We are using Azure Blob Storage to store video/audio files.
An additional feature of "ask the professor" is provided to users, where users can search any questions or doubts, it fetches the answer from wolframalpha api and displays to user.

Impact

It creates a wide impact and provides huge benefits to all Students due to Virtual mode of learning during Covid-19.
It saves students time and effort where they can get all information at one place.
It increases efficiency of students/users.
It improves speed of learning.
A one stop app which integrates with multiples api's and makes it easy for students for learning.

Conclusion

Deepgram STT is so accurate and fast which makes it more reliable for students education.
With Deepgram STT comes the whole power for entire application.