DEV Community: Gorakhnath Yadav

New blog

Gorakhnath Yadav — Thu, 17 Sep 2026 22:03:03 +0000

Gorakhnath Yadav

Sep 17

My Paper Reader Answered Questions for Weeks. I Never Checked If It Was Right.

#ai #rag #llm #voiceai

18 min read

My Paper Reader Answered Questions for Weeks. I Never Checked If It Was Right.

Gorakhnath Yadav — Thu, 17 Sep 2026 21:53:16 +0000

TL;DR

I've been messing with a small project called Talkit, which reads research papers aloud and answers spoken questions about them. The answering half is retrieval-augmented generation, and this is what it took to find out whether it worked: grounding, generation, evals, tracing.
There's no vector store. A paper fits in context, so the retrieval step is the whole paper plus the passage that was just read out.
Evals run through DeepEval against ten questions about the Attention paper, scored four ways. Two of the questions have no answer in the paper.
The app already emitted OpenTelemetry traces, so pointing them at Confident AI was four environment variables and one small code change. Making a span scoreable took a dozen attributes, and they stay off by default because they carry the question and the paper text.
On the first real run, every answer scored 1.00 on faithfulness. The failures were elsewhere: a model that sometimes never answers, and one of my own goldens asking for more than its question did.
The eval suite isn't in CI. It costs credits on every run, and the model under test sometimes returns no answer at all.

I've been chipping away at a small project called Talkit. It reads research papers aloud and lets you interrupt it. You say "wait, why do they scale that?", the narration stops, it answers in two or three spoken sentences, and then it picks up where it left off. That part works. I use it.

What I could not have told you, for weeks, was whether the answers were any good.

The evidence I had was two scripts. One uploads the Attention paper, asks a single question, and checks that an answer comes back. The other does the same in Hinglish. An answer that invented a BLEU score would have passed both. So would an answer that confidently explained the wrong paragraph. The endpoint returned 200 and the suite stayed green, and a green suite quietly becomes the thing you trust.

The other reason I put it off is that the RAG advice I kept reading didn't fit. Every guide opens with chunking and embeddings and retrieval metrics, and Talkit has no vector store at all: the paper fits in the model's context, so it sends the whole thing, which leaves half the standard eval playbook describing a component I don't have.

So I finally sat down and measured the thing I actually built. This post is what that took: pulling the answer path into one function, writing ten questions with known answers, four metrics that each catch a different failure, and span attributes that let a backend score answers as they happen. None of it is elaborate. It's the smallest setup that would tell me when an answer is wrong, and the first run turned up a failure mode that had been sitting there the whole time.

What the pipeline looks like

Here's the path a question takes:

The browser records the question and Sarvam's saaras:v3 transcribes it.
The server loads the paper text and the passage currently being read.
The model (sarvam-105b or Claude, whichever I've picked, on my own key) gets the whole paper, the current passage, recent turns, and the question.
The answer is stripped of anything a voice would read literally, and bulbul:v3 speaks it.

The question path across the top, the answer path and the trace below it.

Steps 2 and 3 are the RAG part, and the answer that comes out of them is what this post evaluates.

The retrieval step that isn't a vector search

Talkit has no chunking and no embeddings on the answer path, and that was a deliberate choice.

Uploads are capped at 60,000 characters. The Attention paper, after the reference list is cut, comes to 30,138. That fits in a model's context with plenty of room, so the model sees all of it.

The bigger reason is the kind of question I actually ask it. Halfway through a paper I'll say "why does that matter" or "what's this number". Those questions retrieve badly by similarity, because the words that matter ("that", "this") point at what was just read out, not at anything an embedding can match. So the current passage goes in as its own labelled block. That passage is the retrieval signal.

Every question sends the whole paper, so input tokens scale with paper length, not question complexity. The 60k cap also rules out books and long theses. If Talkit ever answers questions across a whole library, or takes documents past the cap, that's when I'd go add a vector store, and neither of those is true today.

Pulling the pipeline out of the route

The answer logic used to live inline in the /api/ask handler. An eval can't call a FastAPI route without a database, a session, and a stored key. And an eval that reimplements the prompt assembly measures a copy, which drifts the first time I edit the original.

So it moved into app/qa.py, and the route calls it:

async def answer_question(provider: TextProvider, lang: str, paper: str,
                          passage: str, question: str,
                          history: list[dict] | None = None,
                          user_id: str | None = None) -> str:
    with tracer.start_as_current_span("ask.answer") as span:
        messages = build_messages(paper, passage, history or [], question)
        raw = await provider.converse(answer_system(lang), messages,
                                      max_tokens=ANSWER_MAX_TOKENS)
        answer = speakable(raw, question)
        describe_llm_span(
            span, name="ask", model=provider.model, provider=provider.label,
            question=question.strip(), answer=answer,
            grounding=[passage, paper], user_id=user_id,
            metadata={"lang": lang, "history_turns": len(history or [])},
        )
        return answer

build_messages puts the paper and passage first, then up to six recent turns, then the question:

def build_messages(paper: str, passage: str, history: list[dict],
                   question: str) -> list[dict]:
    messages = [
        {"role": "user",
         "content": f"FULL PAPER:\n{paper}\n\nCURRENT PASSAGE:\n{passage}"},
        {"role": "assistant",
         "content": "Understood. I have the paper and I know where the listener is."},
    ]
    for turn in history[-HISTORY_TURNS:]:
        role = "user" if turn.get("role") == "user" else "assistant"
        messages.append({"role": role,
                         "content": str(turn.get("content", ""))[:HISTORY_TURN_CHARS]})
    messages.append({"role": "user", "content": question.strip()})
    return messages

And speakable removes markdown plus a restated question. The system prompt already forbids markdown and the model mostly complies, which isn't good enough for text that goes straight into a text-to-speech voice.

The refactor was pure motion: behavior didn't change, and the existing 85 tests still pass. I added six more for this module, which brings the offline suite to 91.

Pointing OpenTelemetry traces at Confident AI

Talkit already sends OpenTelemetry traces. Spans are created unconditionally, and with no exporter configured the global provider is a no-op. Picking a backend is environment variables, with no vendor SDK in the code.

I picked Confident AI because DeepEval, which I was already using for the evals, is theirs, so the eval runs and the traces land in the same project. Langfuse, Phoenix and others accept OpenTelemetry too, and everything up to this point would work with any of them. Pointing it at Confident AI:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.confident-ai.com
OTEL_EXPORTER_OTLP_HEADERS=x-confident-api-key=<key>
OTEL_LOGS_EXPORTER=none
OTEL_RESOURCE_ATTRIBUTES=confident.trace.environment=production

The HTTP exporter appends /v1/traces to the endpoint, so the base URL is enough. The environment label rides on the standard resource attribute variable, which the SDK already reads.

The third line needed a code change. Talkit ships logs through the same exporter so each log record carries its trace and span id. Confident AI's OpenTelemetry docs say the endpoint takes traces and not logs, so the logs have to be turned off separately. OTEL_LOGS_EXPORTER is the standard variable for this, so the setup now respects it:

if ENDPOINT and OTEL_LOGS_EXPORTER != "none":
    logs = LoggerProvider(resource=resource)
    logs.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    set_logger_provider(logs)
    logging.getLogger().addHandler(LoggingHandler(logger_provider=logs))

At this point traces arrive, but they're only timings. A trace backend can show that ask.answer took four seconds. It can't tell whether the answer was grounded in the paper.

Making a span scoreable

To score an answer, Confident AI needs to know which span is the model call and what went in and came out. It reads that from span attributes in its own namespace. Talkit writes them in one function in app/telemetry.py:

def describe_llm_span(span, *, name: str, model: str | None, provider: str,
                      question: str, answer: str, grounding: list[str],
                      user_id: str | None = None, metadata: dict | None = None,
                      content: bool | None = None,
                      metric_collection: str | None = None) -> None:
    content = TRACE_CONTENT if content is None else content
    collection = TRACE_METRIC_COLLECTION if metric_collection is None else metric_collection

    span.set_attributes({
        "confident.span.type": "llm",
        "confident.trace.name": name,
        "gen_ai.request.model": model or "unknown",
        "gen_ai.provider.name": provider.lower(),
    })
    if user_id:
        span.set_attribute("confident.trace.user_id", user_id)
    if metadata:
        span.set_attribute("confident.span.metadata", json.dumps(metadata))

    if not content:
        return
    span.set_attributes({
        "confident.span.input": json.dumps(question),
        "confident.span.output": json.dumps(answer),
        "confident.span.retrieval_context": json.dumps(grounding),
        "confident.trace.input": json.dumps(question),
        "confident.trace.output": json.dumps(answer),
    })
    if collection:
        span.set_attribute("confident.span.metric_collection", collection)

A few things here cost me more time than the code suggests.

Text values are JSON-encoded. The convention expects "\"Why scale it?\"", not "Why scale it?", and a list of passages becomes a JSON array string. Before writing this I ran Confident AI's own confident-trace package against an in-memory exporter and read what it set, so the shapes match what their SDK emits. The collection name is the exception: it's a plain string. A backend accepts a wrongly encoded attribute without complaint and just shows an empty panel, so the tests assert the encoding directly.

The grounding is the whole paper. retrieval_context is [passage, paper], because that's what the model saw. If I sent only the passage, the faithfulness metric would flag every true claim drawn from elsewhere in the paper as unsupported.

Content is off unless you turn it on. Scoring needs the question, the answer, and the paper attached to the span and exported. That's my reading and my questions leaving for a third party. So TRACE_CONTENT=true is an explicit switch. Without it, the span carries the span type, model name, and metadata, and nothing a person wrote.

Why attributes and not the SDK

There is an SDK for this, confident-trace, built on OpenTelemetry. It auto-instruments frameworks like LangChain, LlamaIndex and Pydantic AI, which would be the argument for using it.

Talkit calls Sarvam over httpx and Claude through the Anthropic SDK directly, so there is nothing for it to auto-instrument. It would add a dependency and I would still set the input, output and grounding by hand. Writing the attributes myself keeps "no vendor SDK" true, and the ten confident.* names above are the entire vendor-specific surface: swap backends and the spans, timings and gen_ai.* attributes go with you. The cost is that a vendor's attribute names now sit in the code, and if they change, that function changes with them.

The golden dataset: ten questions about one paper

The live suites in Talkit already use arXiv 1706.03762v7, Attention Is All You Need, so the evals do too. The goldens live in evals/goldens.json. Each one has a question, a phrase that locates the current passage, a language, and an expected answer:

{
  "question": "Why do they divide by the square root of d k here?",
  "passage_contains": "particular attention",
  "lang": "en",
  "expected_output": "For large values of d k the dot products grow large in magnitude, which pushes the softmax into regions with extremely small gradients. Scaling by one over the square root of d k counteracts that."
}

That one does double duty. It tests the decision to send the whole paper: the current passage is the one that introduces scaled dot-product attention, while the reason for the scaling comes later on, so an answer that only used that passage would miss it.

The passage is located by searching the chunks for that phrase:

def locate(chunks: list[str], needle: str) -> str:
    flat = lambda s: re.sub(r"\s+", " ", s).lower()
    for chunk in chunks:
        if needle.lower() in flat(chunk):
            return chunk
    raise SystemExit(f"no passage contains {needle!r}; did chunking change?")

An index would silently point at a different passage the first time chunking changes. A missing phrase fails loudly. I checked every phrase against the extracted text before writing the expected answers, and each one matches a passage among the paper's 46.

What the ten cover:

Six factual questions with answers stated in the paper: layers, heads, the positional encoding choice, training time, label smoothing, BLEU.
Two refusals. "How much GPU memory did the big model need?" isn't in the paper, and neither is the weather where the authors work. The system prompt says to say so in one sentence rather than fill the gap from general knowledge. These are the only way to check that it does.
One Hinglish question, "Encoder mein kitne layers hain?", because Hinglish is a real language option in Talkit with its own prompt, and it can break independently of English.
The scaling question above.

Ten questions doesn't cover much of anything. I'll add a golden every time a real answer comes back wrong.

Four metrics, each for a different failure

def metrics_for(model) -> list:
    return [
        FaithfulnessMetric(model=model, threshold=0.8),
        AnswerRelevancyMetric(model=model, threshold=0.7),
        GEval(
            name="Correct",
            model=model,
            threshold=0.7,
            evaluation_params=[SingleTurnParams.INPUT,
                               SingleTurnParams.ACTUAL_OUTPUT,
                               SingleTurnParams.EXPECTED_OUTPUT],
            evaluation_steps=[
                "Check that every fact in the expected output appears in the actual output. Wording may differ.",
                "Penalise any number, name or claim in the actual output that contradicts the expected output.",
                "If the expected output says the paper does not address the question, the actual output must say so and must not supply an answer from general knowledge.",
                "The actual output must be in the same language and script as the expected output.",
            ],
        ),
        GEval(
            name="Speakable",
            model=model,
            threshold=0.7,
            evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
            evaluation_steps=[
                "The output will be read aloud by a text-to-speech voice to someone with no screen.",
                "It should be two to four sentences of plain prose.",
                "Penalise lists, headings, markdown, LaTeX, and symbols a voice would read literally, such as square root signs, subscripts or underscores.",
                "Numbers and variable names should be written the way a person would say them.",
            ],
        ),
    ]

Faithfulness extracts the claims in the answer and checks each against the grounding, which is how it catches invented numbers.

Answer relevancy catches an accurate answer to a different question, a real risk when the question is "what does this mean" and "this" could be several things.

Correct compares against the expected answer, and it's the metric that scores the refusals. Faithfulness can't: "the paper doesn't say" makes no claims to check.

Speakable is the one no generic metric covers. A Talkit answer is heard, not read. A correct answer containing √dk hands the text-to-speech voice a symbol where it needs words, which is why the prompt asks for spoken forms and this metric checks that it got them.

Anything code can check, code checks first, and prints before any judge weighs in:

def mechanical_failures(answer: str, lang: str) -> list[str]:
    failures = []
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]
    if not 1 <= len(sentences) <= 5:
        failures.append(f"{len(sentences)} sentences")
    if lang == "hinglish" and DEVANAGARI.search(answer):
        failures.append("Devanagari in a Hinglish answer")
    if re.search(r"[√∈_\\$]", answer):
        failures.append("unspeakable symbol")
    return failures

Hinglish is romanised Hindi, and both Hinglish prompts forbid Devanagari outright because a model asked for it can drift into Hindi script, which a regex catches without any judge weighing in.

A judge from a different family

Every metric above is a model grading a model. Zheng et al. (2023) documented that LLM judges show self-enhancement bias: they favor answers written by the same model. So the judge defaults to a different family from the one answering. Talkit's default text model is sarvam-105b, and the default judge is claude-sonnet-5. Evaluating the Claude path instead trips a warning that suggests an OpenAI judge:

if args.provider == "claude" and args.judge.startswith("claude"):
    print("warning: Claude is judging Claude. Pass --judge gpt-5.4 for a "
          "judge from a different family.")

The Claude judge broke on its first run. DeepEval's Anthropic wrapper defaults to 1,024 output tokens. Faithfulness starts by listing every factual statement in the grounding, and the grounding here is a whole paper. That list ran past 1,024 tokens, came back as truncated JSON, and DeepEval reported "Evaluation LLM outputted an invalid JSON. Please use a better evaluation model." The fix was the budget:

def judge(name: str):
    if name.startswith("claude"):
        return AnthropicModel(model=name, generation_kwargs={"max_tokens": 16000})
    return OpenAIModel(model=name)

Next time a judge hands back broken JSON over long context I'll check the token limit before I blame the model.

Running it

Four lines, run by hand when I've changed something worth checking:

pip install -r evals/requirements.txt
curl -L -o attention.pdf https://arxiv.org/pdf/1706.03762v7
export SARVAM_API_KEY=... OPENAI_API_KEY=...
export CONFIDENT_API_KEY=...
python evals/run.py --pdf attention.pdf --judge gpt-5.4

The script loads the paper exactly as an upload does (extract, cut the back matter, cap at 60k), answers all ten goldens through answer_question, runs the mechanical checks, then hands the test cases to DeepEval. With CONFIDENT_API_KEY set, the run uploads to Confident AI tagged with the provider, the model, the judge, and a short hash of the answer prompt. That hash is what makes prompt changes comparable, since two runs with different hashes can be put side by side against the same goldens.

Evals have their own requirements file, which includes the app's requirements plus deepeval==4.2.3. The Docker image never installs DeepEval. One side effect: DeepEval needs python-dotenv 1.1.1 or later, so the app's pin moved up from 1.0.1.

Passages in the eval come from the extracted text. The app narrates a cleaned read-aloud script instead, and cleaning a paper takes minutes of paid API calls, which isn't what this suite measures. Wording differs slightly from what I actually hear, and I've accepted that.

What the first run found

I ran it with sarvam-105b answering and gpt-5.4 judging:

Metric	Average	Passed
Faithfulness	1.00	9 of 9
Answer relevancy	1.00	9 of 9
Correct	0.91	8 of 9
Speakable	0.93	9 of 9

Nine of nine, because one question never got an answer.

Sometimes there's no answer at all. sarvam-105b is a reasoning model, and on one question it spent the entire 3,000-token budget reasoning and never wrote a word. The provider raises ChatBudgetExceeded with finish_reason=length. I ran the suite four times. Three runs had exactly one unanswered question (the BLEU question twice, the attention-heads question once) and one run answered all ten. The failure is intermittent and doesn't track any particular question. The same thing happened through the app itself, where "How many layers are in the encoder?" came back as a 502. Before evals, the only live test asked one question, so whether it passed depended on the run. The first version of the eval script crashed on it and threw away nine paid answers, so it now records an unanswered question as a failed case with the reason attached.

Faithfulness didn't discriminate. Every answer scored 1.00. The answers stayed inside the paper, including the refusals. Asked how much GPU memory the big model needed, it said the paper doesn't say, and added that the model has 213 million parameters, which is in Table 3. That's a good result, but it means faithfulness alone would have told me nothing, and the metric that actually moved was Correct.

The one Correct failure was my golden's fault. The Hinglish question asks "Encoder mein kitne layers hain?", how many layers are in the encoder. The answer said six. My expected output also listed the two sub-layers, which the question never asked for, and the judge docked it to 0.65 for leaving them out. The judge's reason spelled it out. The golden needs fixing, and the prompt can stay as it is.

Speakable let one thing through that I'd call a miss: the answer to the scaling question writes "dk" for d k, which a voice reads as a two-letter word. The judge gave it 0.83, comfortably above threshold, and the mechanical check doesn't look for it either. Later answers to the same question through the app did it again. That's the next check to add in code, since a regex settles it better than a judge.

One eval run: the nine questions that got answered, four metrics each, and the hyperparameters that produced them.

The judge's written reason names the detail my expected answer asked for and the question never did.

Why it isn't in CI

Talkit's offline checks are pyflakes, 91 tests, and a Node script that checks the voice command matcher against the shipped HTML, and none of them cost anything.

This suite costs credits on every run: ten answers, then four metrics per answer, and each metric makes several judge calls. And as those runs showed, the model under test sometimes doesn't answer at all. A hard gate would turn that into a flaky build I'd just rerun until it went green.

So it runs by hand, like the two live suites that exercise Sarvam's speech APIs. The script exits non-zero if any case fails a metric, a mechanical check, or gets no answer, so it could run in a scheduled job later. For now it's a tool I run before changing a prompt or a model.

Scoring answers as they happen, with a metric collection

Offline evals only know the questions I thought of. Actually using the thing turns up the ones I didn't.

Confident AI can score incoming traces with a metric collection: a named set of metrics defined in the project. Talkit reads the name from the environment:

TRACE_CONTENT=true
TRACE_METRIC_COLLECTION=talkit-answers

With both set, every ask.answer span carries the question, the answer, the grounding, and the collection name, and Confident AI runs the collection's metrics against it. Faithfulness and answer relevancy both work here. Correctness doesn't, because a question asked in the moment has no expected answer written for it. Correctness still comes from goldens. When a live answer scores badly, it becomes a golden.

One trap when creating the collection: Confident AI offers multi-turn metrics like Turn Faithfulness and Turn Relevancy alongside the single-turn ones. A collection of turn metrics has nothing to score on these spans. Talkit's answer span is a single turn, one question and one answer with its grounding, so the collection needs the single-turn Faithfulness and Answer Relevancy.

The other thing that cost me an evening: online evaluation needs a judge model of your own. Confident AI includes 100 traces of online evaluation, and after that the trace still arrives, still shows the collection name on the span, and simply is not scored until you configure an evaluation model in the project. The span looked correctly wired for a while before I noticed the banner saying so.

With a judge configured, the first scored answer came back at 1.00 on both faithfulness and answer relevancy, with a written reason on each, against the same passage and paper the model had been given. The scores in the collection default to a threshold of 0.5, while the offline suite uses 0.8 for faithfulness and 0.7 for relevancy, so the same answer can pass in one place and fail in the other. I haven't reconciled them yet.

One answer from the running app, scored on arrival against the passage and paper the model was given.

Expected Output is empty, which is the whole reason correctness can't be scored here. Note the 0.5 thresholds against the suite's 0.8 and 0.7.

When one does score badly, the span has what I need to start: the model and provider, the language, how many history turns were in play, and the exact grounding. The usual first question is whether the problem is in the current passage or the answer. Unlike a vector-search RAG system, there's no "wrong chunks retrieved" branch to rule out, since the model had the whole paper.

What else turned up

Running the whole thing locally, upload to answer, surfaced two problems outside the answer path.

Cleaning falls back more than I knew. Uploading the Attention paper through Sarvam took 837 seconds, and 14 of its 28 cleaning batches fell back to raw text. The log shows the same cause 52 times: a batch overran the token budget and was halved, until it hit the split limit and kept the raw text. That's the same reasoning-budget problem as the missing answers, on the cleaning side. A fallback is recorded and never surfaced, so I just hear citation markers read out with no explanation. It was already on Talkit's list of known gaps. Now it has a number.

The title is wrong. Talkit's title extractor reads the Attention paper's title as "Provided proper attribution is provided, Google hereby grants permission to", which is the permission banner at the top of the arXiv version.

Neither is fixed. I only ran into them because I was reading logs and traces for a different reason.

See O2 AI Assistant in action!

Gorakhnath Yadav — Thu, 14 May 2026 10:27:46 +0000

I Built a Dashboard in 30 Seconds with AI - DEV Community

How the OpenObserve AI Assistant builds production-ready dashboards, creates alerts, and finds root cause — all from plain English. Plus, connect it to your IDE via MCP.

dev.to

[Boost]

Gorakhnath Yadav — Wed, 29 Apr 2026 15:40:29 +0000

Sara for OpenObserve

Apr 29

OpenObserve Just Raised $10M and Launched Observability 3.0 with New AI Capabilities

#news #devops #cloud #observability

1 min read

Open-Source Payments: Modular, Flexible, and Built for You

Gorakhnath Yadav — Tue, 11 Mar 2025 06:57:19 +0000

Imagine a Linux-like foundation for payments. A unified and open-source intelligence that transforms payment agility into a competitive advantage.

Introducing Hyperswitch, a modular, open-source, and interoperable full-stack payments platform engineered to put control back in your hands.

Why Hyperswitch?

Enterprises can build, augment, or customize their payment stack with modular solutions. Whether it's enabling alternate payment methods, streamlining operations across multiple providers and acquirers, or optimizing payment routing—Hyperswitch provides the flexibility businesses need.

With advanced analytics on processing costs, seamless authentication, token storage, and reconciliation, it simplifies and secures global payments.

Backed by a Global Engineering Force

Hyperswitch is powered by 1,000+ payment engineers globally, enabling infrastructure that processes up to 175 million transactions a day with 99.999% availability.

Engineered with robust system design principles, its open-source architecture ensures transparency, security, and enterprise-grade reliability with PCI compliance, tokenization, and security certifications.

Freedom from Closed-Loop Systems

With Hyperswitch, merchants gain rapid integration capabilities and freedom from vendor lock-ins. The open-source model allows for risk-free due diligence, eliminating the need for lengthy and costly RFP cycles.

Merchants can deploy Hyperswitch in local or cloud environments or leverage Juspay’s hosted sandboxes for seamless experimentation.

Take Control of your Payments

Whether you aim to optimize costs, expand into new markets, or reduce vendor reliance, Hyperswitch empowers you to build what you want, how you want.

🔗 Learn more about open-source payments and Juspay Hyperswitch.

Transitioning from Kubernetes to EC2 for Enhanced Kafka Performance

Gorakhnath Yadav — Thu, 06 Feb 2025 14:07:56 +0000

Scaling distributed systems is never just about performance, it's also about cost and operational efficiency. While Kubernetes provided us with a solid foundation for container orchestration, we hit unexpected roadblocks when running Kafka clusters at scale.

Costs were rising due to inefficiencies in resource allocation.
Auto-scaling wasn’t handling our stateful workload well.
Kafka node management with Strimzi led to operational complexity.

After months of firefighting, we decided to move from Kubernetes to EC2, a transition that improved performance, simplified operations, and cut costs by 28%.

Here’s the story of that journey, what worked, and the lessons we learned.

Why Kubernetes wasn’t working for us?

1. Resource Allocation Inefficiencies

Kubernetes dynamically manages resources, but in our case, it led to hidden inefficiencies.

For example, when allocating 2 CPU cores and 8GB RAM, we observed that the actual provisioned resources were often slightly lower (1.8 CPU cores, 7.5GB RAM). This discrepancy may seem trivial, but at scale, it resulted in significant wasted resources and unexpected cost overruns.

Imagine paying for a full tank of fuel, but your car only gets 90% of it. Over time, those missing liters add up.

2. Auto-Scaling Challenges for Stateless Applications

Kubernetes’ auto-scaling mechanism works well for stateless applications, but Kafka isn’t stateless. When resources ran out, Kubernetes would restart our Kafka application instead of efficiently scaling it.

This resulted in:

15-second delays in message processing.
Increased latency during scaling events.
Operational headaches managing stateful workloads.

3. Kafka Node Management with Strimzi

Initially, we relied on Strimzi for managing Kafka clusters. However, it had major drawbacks:

New Kafka nodes often failed to integrate properly.
Manual intervention was required for every scaling event.
Overall Kafka performance was unpredictable.

Managing our Kafka clusters felt like playing whack-a-mole every time we solved one issue, another would pop up.

Decided to move to EC2

After evaluating various alternatives, we decided to move Kafka from Kubernetes to EC2. This gave us more control over resource allocation, auto-scaling, and cluster management.

Here’s what changed:

1. Replacing Strimzi with a Custom Kafka Controller

Instead of relying on third-party tools, we built an in-house Kafka Controller tailored to our needs.

Seamless integration of new Kafka nodes
Automated scaling based on real-time workload analysis
Better cluster management with minimal manual intervention

Result? Kafka nodes were now automatically recognized and integrated instantly.

2. Precise Resource Allocation

Unlike Kubernetes, where we had limited control over resource provisioning, EC2 allowed us to:

Allocate exactly the CPU and memory we needed.
Avoid wasted resources and over-provisioning costs.

Example: Previously, we paid $180/month per instance on Kubernetes. After transitioning to EC2, this dropped to $130/month, saving 28% on infrastructure costs.

3. Streamlined Kafka Node Support

With EC2, we could now:

Scale up Kafka nodes seamlessly without restarts.
Perform vertical scaling (switching to more powerful machines) with zero downtime.
Ensure predictable performance under peak loads.

Last month, we moved from a T-class instance to a C-class instance in EC2 without downtime. If we had been on Kubernetes, this would have required:

Creating a new node group.
Rebalancing partitions manually.
Managing potential downtime.

Instead, on EC2, it was a simple instance upgrade with zero complexity, zero downtime.

The Impact: Cost Savings & efficiency gains

Key lessons & Takeaways

Not all workloads are ideal for Kubernetes – It’s great for general-purpose container orchestration but not always the best for stateful applications like Kafka.
Custom solutions can be worth it – Building an in-house Kafka Controller gave us better control and reliability.
Cost inefficiencies add up – Even small inefficiencies in resource allocation can result in thousands of dollars lost at scale.
EC2 provides better flexibility – We gained granular control over scaling and performance with EC2.

Conclusion

If you’re running Kafka on Kubernetes and experiencing similar issues, EC2 might be a better fit.

Who should consider moving? Teams struggling with stateful workloads on Kubernetes.
Who should stick with Kubernetes? Those managing stateless, highly dynamic applications.

Every infrastructure decision should be guided by workload needs, not just industry trends. Kubernetes is powerful, but for Kafka, EC2 provided the right balance of cost, performance, and operational efficiency for us.

Hacktoberfest 2024: Maintainer POV

Gorakhnath Yadav — Thu, 31 Oct 2024 06:25:09 +0000

As October comes to an end, I’ve been reflecting on my experience during Hacktoberfest this year. This time, I participated as a maintainer for our project at @hyperswitch , which deepened my appreciation for the importance of community in open source.

My Role: As a DevRel, I was looking after issue creation, guiding people to resources, managing the assignment of issues, and coordinating among the contributors and the internal team for queries and reviews.

How It Went:📈

We kicked off with a good start; as we created issues in the last weeks of September, contributors were already discussing and working on them.
As soon as we started, we had a lot of people coming in and asking for assignments of issues so they could work on it. By October 3rd, we had 48 issues assigned, 18 PRs raised.
Tried to make issues beginner-friendly, given the complexity of digital payments.
Saw participation from all skill levels, from first-time contributors to seasoned Hacktoberfest participants.

Learnings for us:🤔

Hacktoberfest allowed me to view our project from a fresh perspective, as contributors noticed details I might overlook.
Questions from contributors helped identify areas in our documentation needing clarification, which we’ll be updating soon.
Also updating guidelines to address common queries raised by contributors.
Participation seemed lower this year, and I wonder if this trend affected other projects as well. I'd love to hear your thoughts on this.

Community is WOW!❤️

Hacktoberfest also highlighted the diversity of the open-source community. As we had people from each level of skills and expertise, each contributor brought unique insights. Some were experienced developers who tackled issues with ease, while others were just starting out but were eager to learn and help. Working with such a diverse group reminded me that open source thrives on this variety of skills and perspectives.

Being a maintainer also reinforced the responsibility that comes with this role. Open source is not just about code—it’s about creating a space where everyone feels welcome to contribute. It’s essential to be approachable, give helpful feedback, and appreciate the time each person spends on their contribution.

Road Ahead:🛣️

This Hacktoberfest was full of learning, connection, and growth for our community. My experience as a maintainer at Hyperswitch provided insights I hadn’t expected and reminded me why open source is about much more than just coding. I’m excited to continue supporting this community and watching it grow.

Here’s to Hacktoberfest 2024🥂 and to all the contributors who bring life to open source!🚀

🚀 Hyperswitch Plugin Development Hackathon: Build, Compete, and Win Big! 🚀

Gorakhnath Yadav — Wed, 09 Oct 2024 10:48:02 +0000

Hey developers! 🔥

We’re excited to announce the Hyperswitch's Plugin Development Hackathon, running from October 9th to October 31st, 2024. It’s your chance to develop plugins for Hyperswitch, collaborate with fellow developers, and win big rewards!

What’s the Hackathon About?

You'll be building plugins for platforms like MedusaJS, CommerceTools, or Prestashop, helping to extend Hyperswitch’s payment orchestration platform. Participate individually or as a team (up to 4 members) and tackle real-world payment challenges while competing for awesome prizes!

Key Dates:

Registration Deadline: October 14th, 2024
Approach Submission: October 14th, 2024 (EOD IST)
Final Submission: October 31st, 2024
Demo Day: First week of November 2024 (TBA)

How to Participate:

Sign Up: Register before October 14th to get started.
Submit Your Approach: Outline the core features, integration with Hyperswitch’s API, and edge case handling by October 14th.
Get Shortlisted: The Hyperswitch team will review your submission and notify shortlisted teams.
Develop Your Plugin: If selected, you’ll have two weeks to complete your plugin.
Demo Day: Showcase your plugin in early November, dates to be announced!

Prizes:

MedusaJS Plugin: $2000
CommerceTools Plugin: $2000
Prestashop Plugin: $2000

Ready to showcase your skills? Register today and start building your plugin! Let’s code, collaborate, and win together. 💻

Good luck to all participants! 🎉

Star Us on GitHub
Follow us on twitter

Hacktoberfest 2024 with Hyperswitch

Gorakhnath Yadav — Fri, 27 Sep 2024 09:07:40 +0000

Join the Celebration of Open Source!

Hyperswitch is excited to participate in Hacktoberfest 2024! Join us in building a robust payment infrastructure that serves billions of people worldwide. Contribute to our open-source projects and earn exciting rewards.

How to Get Involved

1. Explore Hyperswitch Repository:

Visit our GitHub page.
Star and watch the repositories to stay updated with new issues.

2. Find Issues to Work On:

Look for issues tagged with "hacktoberfest".
If you have ideas for improvements, feel free to create new issues, and inform us using community channels.

3. Make Your Contribution:

Comment on the issue you want to work on: "I would like to work on this. Can you assign it to me?"
Ensure you follow up within 3-4 days to show you are actively working on it.
Submit your pull request (PR) and link it to the issue. Tag a maintainer for review.

4. Get Support:

Our maintainers are here to guide you. Join our Community for support and to connect with fellow contributors.

5. Earn Swag:

Get one PR merged to earn exclusive Hyperswitch swag.
Get 3 or more PRs merged to receive a special swag kit from Hyperswitch.

Why Contribute?

Learning Opportunities: Enhance your skills by working on real-world projects.
Networking: Connect with a global community of developers.
Rewards: Earn cool swag for your contributions.

Join the Hyperswitch Community

Star Us on GitHub

Happy Contributing!🎉

Dive Into the World of Payments - Join Payment Meetup Group’s First Session!

Gorakhnath Yadav — Mon, 09 Sep 2024 03:29:40 +0000

Have you ever wondered what happens behind the scenes when you click "Pay Now"?

If you're a developer, payment enthusiast, or just curious about how digital payments via card works, we have an exciting session lined up just for you!

At Payments Meetup Group, we’re diving deep into the mechanisms that power online transactions via cards, and we want to bring you along for the journey. Our upcoming session, led by Samraat Bansal, Engineering Tech Lead at Hyperswitch, will explore every detail from data encryption and transmission to how various players like issuing banks, card networks, and merchants handle your card payment.

Why You Should Join:

In-depth Insights: Understand the flow of transactions and how card payments are processed securely.
Learn from Experts: Samraat Bansal brings hands-on experience from leading Hyperswitch’s tech innovations.
Engage with Community: This is a perfect opportunity to network with fellow developers and digital payment enthusiasts.

Session Highlights:

What happens when you enter your card details and click "Pay Now"?
The critical role of encryption and transmission in ensuring transaction security.
How various stakeholders like banks and merchants collaborate to approve or decline your transaction.
Behind-the-scenes interactions between card networks, acquiring banks, and issuing banks.

When and how to join?

Date & Time: 11th Sept, 2024, 07:30 PM (GMT +2)
Where: Online (Link to register)

Don’t miss out on this deep dive into the fascinating world of digital payments. Whether you’re a seasoned developer or new to the world of digital finance, this session is tailored to give you valuable insights and practical knowledge.

RSVP Now and be part of this insightful conversation!

Securing Digital Transactions: How Hyperswitch makes Payment Protection a Priority

Gorakhnath Yadav — Thu, 01 Aug 2024 09:26:44 +0000

Security is not a product, but a process - Bruce Schneier

In today's digital landscape, data security is paramount, especially when it comes to online payments. At Hyperswitch, we've built our platform with security as a fundamental principle. Let's explore the robust measures we've implemented to ensure a safe and secure payments infrastructure.

Our Top Priority: Customer Security

When a customer makes a payment through Hyperswitch, their card information is immediately encrypted at the source. We use the SSL/TLS 1.2 protocol for transmission, adhering to PCI standards for handling card data.
For customers who opt to store their card details, we've developed a secure Card Vault. This system employs multiple layers of protection:

SSL/TLS 1.2 encryption
JWE + JWS for secure data transmission
Double encryption for stored data
A two-key custodian system for enhanced security

Additionally, all customer Personally Identifiable Information (PII) is TLS encrypted in transit and AES encrypted at rest.

Keeping Your Business Secure

For our merchant, we know that not everyone needs to be in on all the secrets. That's why we've built up an access control system that's more discerning. API credentials? We treat those with more care than a rare vintage car, wrapping them in layers of AES encryption. Also, we mask the sensitive merchant details in the dashboard.

Hyperswitch's Approach to Security

At Hyperswitch, we're not just building a payment system; we're crafting a digital fortress. Our multi-tenant system comes with unique data encryption keys for each business, creating a security layer that's both impressive and effective.
We've also enlisted Rust as our companion, using its robust type system to keep sensitive information under wraps. It's like having a very diligent, slightly obsessive assistant who never lets a secret slip.

Serious Security with a Smile

In the end, we're not just developing an application; we're creating a secure future for digital transactions. Our goal is to allow businesses to focus on their core competencies while we ensure the protection of their customers' financial information.
With Hyperswitch, security isn't just a feature—it's a cornerstone of our service. We're dedicated to maintaining the highest standards of data protection, giving both merchants and customers peace of mind in their digital transactions.

Want to contribute? Check out some of our good first issues here.
Try Hyperswitch. Get your API keys here. Happy reading!

Scaling to 125 Million Transactions per Day: Juspay's Engineering Principles

Gorakhnath Yadav — Wed, 19 Jun 2024 14:32:56 +0000

At Juspay, we process 125 million transactions per day, with peak traffic reaching 5,000 transactions per second, all while maintaining 99.99% uptime. Handling such enormous volumes demands a robust, reliable, and scalable system. In this post, we'll walk you through our core engineering principles and how they've shaped our engineering decisions and systems.
When designing systems at this scale, several challenges naturally arise:

Reliability vs Scale: Generally, as you scale, you tend to exhaust resources, which can affect system availability.
Reliability vs Agility: Frequent releases and system changes can impact system reliability.
Scale vs Cost-Effectiveness: Scaling requires more resources, leading to higher costs.

Core Engineering Pillars

We've been able to strike the right balance between these challenges by anchoring our tech stack on four pillars:

Build zero downtime stacks: Solve reliability by building redundancy at each layer to achieve almost 100% uptime.
Horizontally scalable systems: Solve scalability by building systems that can scale horizontally by removing bottlenecks.
Build agile systems for frequent bug-free releases.
Build performant systems for low latency, high throughput transaction processing.

Adopting Haskell Programming Language

To achieve our goals, we've made a critical investment: adopting the Haskell programming language. Haskell, a functional programming language, offers performance akin to C, which is closer to the machine and processes transactions much faster. With Haskell, we've reduced our transaction processing time to less than 100 milliseconds.

Here's an example of a Haskell function that adds two numbers:

-- Define the add function
add :: Int -> Int -> Int
add x y = x + y

-- Main function to test the add function
main :: IO ()
main = do
    let sum = add 3 5
    putStrLn ("The sum of 3 and 5 is " ++ show sum)

This function showcases Haskell's concise and readable syntax.

Additionally, Haskell's readability, like English, enables non-technical folks to read the code easily, verify business logic, and sign off on features during development itself. As a strong-typed language, Haskell enforces a set of rules to ensure consistency of results, helping us preempt failures and achieve zero technical declines.

Cache-based Shock Absorber

To handle scale and remove database bottlenecks, we introduced a horizontally scalable caching layer where real-time transactions are served from this cache layer and later drained to the database.

Scaling up and down the cache layer is relatively easy and cost-effective compared to scaling databases.

Rapid Deployment and Release Frameworks

With rapid development comes the challenge of frequent production releases. To achieve agility through frequent releases without compromising reliability, we've built internal tools for automated releases with minimal manual effort. These tools monitor the performance of the release by benchmarking error codes against the previous stable version of the codebase:

ART (Automated Regression Tester): A system that records production payloads and runs them in the UAT system against a new deployment to identify bugs early.
Autopilot: A tool that creates a new deployment and performs traffic staggering from 1% onwards.
A/B testing framework: A system that monitors and benchmarks the new deployment's performance against the previous stable version. Based on this benchmark, the system automatically decides to scale up the traffic or abort the deployment.

Hyperswitch: An Open-Source Payments Switch

We're carrying these learnings forward to our latest product, Hyperswitch, an open-source payments switch. Every line of code powering our stack is available for you to see. With Hyperswitch, our vision is to ensure every business has access to world-class payment infrastructure.

Conclusion

Through these investments, we've built reliable, agile, and scalable systems, enabling our engineers to solve exciting new problems and fostering a culture of systems thinking within the company. We encourage developers to engage with the open-source Hyperswitch project and explore the principles and technologies we've adopted to handle massive scale and high-volume transaction processing.