DEV Community: Faishal Wahiduddin

The Best Interface Was One I Didn't Build: WhatsApp on a Construction Site

Faishal Wahiduddin — Sun, 18 Oct 2026 13:17:47 +0000

The first version of any B2B platform has a login page. Then a dashboard. Then, if the product touches anyone outside the office, a long quiet argument about why nobody is using it.

AutoKon verifies construction progress with photo evidence and uses that evidence to gate contractor disbursements across thousands of housing units. The evidence has to come from the site, from people standing in the mud with a phone, often on a patchy connection, often not the same people from one week to the next.

I did not build them a portal. They submit over WhatsApp.

Count the real cost of an account

A portal looks free when you are the one building it. The cost lands somewhere else:

Someone has to create an account for every field worker, and deactivate it when they rotate off.
Someone has to train them, in person, on a screen they will use a handful of times.
The app has to go on a phone that may already be near full.
Each of those steps is a place where the evidence does not get submitted, which means a disbursement does not get processed, which means somebody's payment is late.

None of that appears in a sprint estimate. It appears in adoption.

Meanwhile every one of those workers already had WhatsApp open, already knew how to send a photo, and had used it that morning. The interface was installed and understood. The only question was whether I was willing to meet it.

What the architecture actually looks like

Field submissions come in through a WhatsApp-first flow built on n8n and Botpress. The conversation collects what the record needs (which unit, which stage, the photo) and hands a structured submission to the platform. From there the evidence enters the vision pipeline, gets graded by the evaluation stage, and lands in a five-role approval workflow.

Approvers work through guided chat too. Someone with four items to approve does not open a dashboard, find the queue and filter it. The items come to them in order, photo attached, decision reduced to a reply.

The dashboards still exist: S-curve and earned-value views for the people who plan and audit. Planning and auditing happen at a desk with a large screen. Capturing evidence and approving it do not. Two audiences, two surfaces, and only one of them had to be built from scratch.

Conversational does not mean unstructured

The failure mode of chat interfaces is that they accept anything and produce garbage, and a financial verification pipeline cannot afford that.

So the flow is guided rather than open. It asks one thing at a time. It rejects a submission with no unit reference instead of guessing. It never takes a free-text status in place of a photo. What reaches the platform is as structured as anything a form would have produced. The structure just lives in the conversation design instead of in a set of input fields.

Choosing WhatsApp did not remove the work of designing a data-entry flow. It moved that work, and made it harder, because you cannot lean on the layout to explain the model.

The leverage

I built and ran AutoKon's platform solo, which made the channel decision unusually clear: every surface I built was a surface I would maintain, support and onboard people onto, by myself, indefinitely.

Field workers across three timezones now submit evidence without a single account being provisioned, and the system that gates real money stays fed.

Before you build another screen

Ask who is on the other side and where they are standing, because a muddy site at 7am is not a desk. Add up what an account really costs across provisioning, training, installation and rotation, and work out who absorbs that cost. Check whether the task is recurring and complex or occasional and simple, since dashboards earn their keep on the first and conversations win the second. And if the structure is not going into a form, be able to say where it is going instead.

The most useful interface decision I made that year was noticing that the right one was already on everybody's phone.

Twelve Hours Apart: Leading DoCheck's Engineering Team From the Other Side of the Planet

Faishal Wahiduddin — Sun, 18 Oct 2026 13:17:05 +0000

From 2021 to 2023 I was CTO of DoCheck, a productivity SaaS with a 20+ engineering team in Indonesia. I was in Orlando, finishing an M.S. in Computer Science at the University of Central Florida on a Fulbright scholarship.

Indonesia runs roughly twelve hours ahead of Florida. My team's afternoon was my middle of the night; their standup was my bedtime. No scheduling trick makes that go away. With a twelve-hour offset the only overlap you get is the overlap you deliberately buy, and you pay for it with somebody's evening.

Velocity went up 30% over that period. Not because of the distance, but because the distance made it impossible to keep getting away with several things.

The time zone audits your process

A co-located team runs on ambient information. Someone overhears a decision. Someone taps a shoulder. A ticket says "fix the thing we discussed" and everyone knows which thing.

Move the lead twelve hours away and each of those shortcuts becomes a twenty-four-hour round trip. A ticket that needs one clarifying question does not lose ten minutes. It loses a day.

That is a harsh audit, and a useful one. Everything vague in our process now produced a delay we could measure, so we fixed those things in self-defense:

Decisions written down with the reasoning attached, not just the outcome, because the person reading could not ask me why.
Tickets that could be started without me. If an engineer had to wait for an answer before beginning, the ticket was not ready.
Agreed definitions of done, so "finished" did not need adjudicating across a day boundary.

None of that is novel. What was new was having no way to avoid it.

Spend the overlap on what only humans can do

We had a narrow band of shared waking hours, and the instinct is to fill it with status. Status is the worst possible use of overlap. It is exactly the information that survives being written down.

So status moved to writing, and live time went to the things that genuinely degrade in text: disagreements about architecture, someone stuck and not saying so, career conversations, feedback whose tone lands wrong in a message. I rebuilt our agile rituals around that split. Rituals that existed to broadcast information became documents. Rituals that existed to build shared judgment kept their synchronous slot.

Automation is a leadership tool

When you cannot be present, every manual step between commit and production becomes a step waiting on someone who might be asleep.

That is why we invested in CI/CD upgrades. The goal was not developer comfort. It was removing the human bottlenecks that a twelve-hour offset turns into day-long stalls. A pipeline that checks quality on its own does not need me awake.

The same logic drove the infrastructure work. We migrated key workloads to AWS serverless with disciplined resource governance and cut infrastructure costs by 20%. Some of that was straightforward FinOps. Some of it was serverless removing a class of operational babysitting that someone would otherwise have done at an inconvenient hour.

What the team gained that I did not plan for

Autonomy. When the lead is unreachable twelve hours a day, decisions get made without the lead. That is uncomfortable exactly once: the first time someone makes a call you would have made differently and you have to choose between reversing it and backing it.

Backing it, then fixing the guidance that made the call ambiguous, is how a team stops needing you for that class of decision. Distance forced that on me faster than proximity ever would have. By the end the team made most calls without me and escalated a much smaller, much better-chosen set.

What I kept

I have led co-located teams since. These stayed.

Write the decision and the reasoning. The reasoning is what lets someone extend the decision to a case you did not anticipate.
Never spend synchronous time on what a document does better.
Treat every manual gate as a future stall and automate accordingly.
A ticket nobody can start without asking you is not ready.
Back the judgment calls your team makes in your absence, then fix the ambiguity that made them hard.

The gap did not make us faster. It removed the option of postponing the changes that did.

Before AI Moves Money: The Eval Gate I Built for AutoKon

Faishal Wahiduddin — Sun, 18 Oct 2026 13:16:03 +0000

Most AI features fail quietly. A summary is a little off, a label is wrong, a suggestion gets ignored. Nobody files a ticket and the product keeps working.

AutoKon is not that kind of product. It is a financial verification pipeline for Indonesian property developers: a field worker photographs a housing unit, a vision-language model reads the photo, and that reading is part of what releases a contractor's payment. A wrong description does not degrade the experience. It moves money to the wrong place.

That one property changed how I built everything else.

The pipeline does not end at the model

The obvious architecture is photo in, VLM out, decision made. Claude Opus Vision is very good at describing a construction site. It will tell you the walls are rendered, the roof trusses are up, the floor screed is done. It will occasionally tell you all of that, with the same confidence, about a photo showing something else. Rarely, but often enough to matter.

So every AI-generated description passes through an LLM-as-judge evaluation stage before it goes anywhere near a disbursement decision. A second model grades the first one against the image and the claim, and the item proceeds only if it clears the bar.

The judge is not a quality dashboard bolted on after launch. It sits in the payment path. If the eval stage is down, disbursements wait. That is the right failure mode for this domain, and settling it early is what made the rest of the design tractable.

What the judge is actually measuring

Most eval work I see optimizes for whether the answer is good. For a payment gate that is the wrong question.

The question is whether the description is grounded: whether every claim in it is supported by what is in the photograph. A description can be well written, plausible and useful, and still assert a completed stage that is nowhere in the frame. That is the failure that must never reach the disbursement engine.

So the judge gets a narrow prompt. Is each claim here supported by this image? It is never asked whether the description reads well. Ungrounded claims fail the item even when the writing is excellent, and most of the value sits in that narrowing. A broad quality rubric produces a judge that agrees with the generator far too often, because both models share the same optimistic priors.

The number that makes a judge worth having

An automated judge is only worth having if it agrees with a human reviewer, and you have to be able to say by how much.

The pipeline targets Cohen's κ ≥ 0.80 against human labels, which is substantial agreement on any reading of the scale. That is an operating threshold, not a figure for a slide. Below it the judge is a source of false confidence rather than a control, and the honest response is to widen human review until the rubric or the labels improve.

Three things I learned getting there. Human labeling time is scarce, so spend it where the generator and the judge disagree rather than on the cases that were never in doubt. Ambiguous rubrics show up as low κ well before they show up as bad outcomes; when agreement dropped, the cause was almost always a rubric that let two reasonable reviewers score the same photo differently, not a model that got worse. And the input distribution keeps moving. New site types, new phone cameras and the rainy season all change what the camera sees, so an eval suite frozen at launch decays.

Where the humans sit

The eval gate does not replace approval. AutoKon runs a five-role photo-evidence approval workflow, and field submissions arrive over WhatsApp through an n8n and Botpress flow rather than through a portal that every field worker would need an account for and training on.

The AI approves nothing. It clears the obviously complete, obviously grounded evidence out of a reviewer's queue and flags what needs a person. Reviewers get the ambiguous cases with their context attached, which is where human judgment earns its cost. That is what let one engineer run a platform across thousands of housing units and three timezones without the review burden growing in step with the number of units.

What transfers

Not every product moves money, but the questions carry. What is the worst thing a confident wrong answer can do here, and does your design start from that or from what the model can do? Is your metric measuring correctness against the source, or plausibility? Can you say what your agreement with a human actually is? Are the people still in the loop getting the hard cases or the easy ones?

The model was never the hard part of this build. The hard part was the structure around it, and being willing to let that structure stop the happy path when it has to.

Calibrating an LLM-as-Judge for a Vision Check to κ 0.80

Faishal Wahiduddin — Thu, 15 Oct 2026 12:50:21 +0000

At AutoKon, a construction platform where I was CTO and the only engineer, a contractor's payment was approved only after a human approval chain reviewed the field photos, with an AI check as evidence. Claude Opus Vision, with Pinecone retrieval for context, described what each photo showed. A separate LLM-as-judge pipeline in Arize AX graded those descriptions for groundedness and completeness, and the judges had to reach an agreement score of κ ≥ 0.80 against human review before their results counted toward payment. The evals lived in a version-controlled repo and ran in CI with phoenix-evals and GitHub Actions.

A note on scope before the method. I am not sharing dataset sizes, prompt text, label counts or per-round scores here. Where I describe a step below, treat it as the general method I use and recommend, written so you can apply it to your own vision task. I am not reproducing the exact AutoKon configuration.

Why a judge needs calibrating at all

The generator model writes descriptions. A judge model grades them, which lets you score every output instead of a hand-picked sample. The catch is obvious once you say it out loud: the judge is also a model, and you have no idea how good it is until you compare it with people.

Calibration means measuring how often the judge agrees with careful human reviewers, adjusting it until that agreement is high enough, and then holding it there. Without that step, a dashboard full of judge scores is a number nobody has checked.

Define the rubric before you write the judge prompt

Most of the work is here. If two humans cannot agree on what "good" means, no judge will either.

For photo descriptions, I split quality into two separate dimensions:

Groundedness. Is every claim in the description visible in the photo? A description that says "plastering complete on the east wall" when the east wall is out of frame fails, even if the claim happens to be true.
Completeness. Does the description cover the items that matter for this stage of work? A grounded but vague description ("a wall is visible") can still be useless for a payment decision.

Rubric habits that help agreement:

Prefer binary labels at first. Pass or fail is easier for humans to agree on than a 1 to 5 scale. Move to an ordinal scale only if binary loses information you actually need.
One judge prompt per dimension. Asking one prompt to score both lets a strong score on one bleed into the other.
Write each criterion as a question that can be answered from the image and description alone. If the reviewer needs outside knowledge, the judge needs it too, so put it in the prompt or drop the criterion.
Include worked examples of pass and fail, especially borderline ones. Borderline cases are where raters disagree.
Give the judge the image. A text-only judge grading a vision model's output can only check whether the text sounds plausible. A vision judge can check the text against the pixels.

Build a human-labeled set

The general method:

Sample for coverage. Pull photos across lighting conditions, angles, stages of work, blurry shots, and cases where the generator is known to struggle.
Oversample failures. If 95% of descriptions are fine, a random sample gives you very few failures to calibrate against, and kappa becomes unstable.
Have two people label independently using the written rubric, without seeing each other's labels.
Measure human to human agreement first. That number is your practical ceiling. If two humans only reach 0.70, a judge reaching 0.80 against either one is more likely a fluke than a triumph, and the rubric needs work.
Adjudicate disagreements into a single gold label, and update the rubric wording whenever a disagreement reveals ambiguity.
Split the set into a calibration portion you iterate against and a held-out portion you touch only to confirm.

Why kappa and not accuracy

Accuracy rewards a judge for agreeing on the easy majority class. Cohen's kappa corrects for the agreement you would expect by chance:

κ = (p_o - p_e) / (1 - p_e)

p_o is observed agreement and p_e is the agreement expected if both raters labeled at random with their own label frequencies. For two raters and categorical labels, Cohen's kappa is the standard choice. For ordinal scales, weighted kappa gives partial credit for near misses.

On the common Landis and Koch scale, 0.61 to 0.80 counts as substantial agreement and 0.81 to 1.00 as almost perfect, so a 0.80 bar sits right at that boundary.

Here is a small, dependency-free implementation that shows why accuracy misleads:

def cohens_kappa(rater_a, rater_b):
    """Cohen's kappa for two raters labeling the same items."""
    if len(rater_a) != len(rater_b) or not rater_a:
        raise ValueError("need two equal-length, non-empty label lists")
    n = len(rater_a)
    labels = set(rater_a) | set(rater_b)
    p_o = sum(a == b for a, b in zip(rater_a, rater_b)) / n
    p_e = sum((rater_a.count(k) / n) * (rater_b.count(k) / n) for k in labels)
    if p_e == 1:
        return float("nan")  # both raters used one identical label: undefined
    return (p_o - p_e) / (1 - p_e)


human = ["pass"] * 18 + ["fail"] * 2
lazy_judge = ["pass"] * 20
better_judge = ["pass"] * 17 + ["fail"] + ["fail"] * 2

for name, judge in [("lazy", lazy_judge), ("better", better_judge)]:
    accuracy = sum(h == j for h, j in zip(human, judge)) / len(human)
    print(f"{name}: accuracy={accuracy:.2f} kappa={cohens_kappa(human, judge):.2f}")

Output:

lazy: accuracy=0.90 kappa=0.00
better: accuracy=0.95 kappa=0.77

The lazy judge passes everything and still scores 90% accuracy, with a kappa of zero. The better judge catches both real failures and makes one false rejection. It reaches 95% accuracy and still falls short of 0.80. That gap between a flattering accuracy number and an honest kappa is the whole reason to use kappa on imbalanced data. (These are toy labels for illustration, not AutoKon data.)

The calibration loop

Run the judge on the calibration split.
Compute kappa per dimension, separately. A combined score hides which dimension is weak.
Read every disagreement. Build a confusion matrix and look at the actual photos. Is the judge too strict on blurry images? Too lenient when the description is long?
Change one thing at a time: rubric wording, examples, the instruction about uncertain cases, or the judge model.
Re-run and re-score.
Once the calibration split clears 0.80, confirm on the held-out split. If held-out drops noticeably, you tuned the prompt to the examples you were staring at.
Freeze the judge: pin the prompt, model version and rubric in version control, and have CI re-run the agreement check whenever any of them change.

Pitfalls

Overfitting the prompt to the calibration set. The held-out split exists for this.
Shared blind spots. If the judge sees exactly the same context as the generator, they can fail in the same way. Giving the judge the raw image, and asking it to check claims against it, reduces this.
Verbosity bias. Judges tend to rate longer descriptions as more complete. Include short complete examples and long incomplete ones in your few-shot set.
Silent drift. A model version change or a small prompt edit can move agreement. Treat the judge like code.
Low failure prevalence. With very few failures, kappa swings a lot between runs. Oversample failures and report the case counts behind each score.
Label drift in humans. Reviewers' standards shift over time. Re-measure human to human agreement when you refresh labels.
Treating kappa as the only signal. Kappa measures agreement with people, so keep a domain expert involved in what the rubric asks.

Where the judge sits

At AutoKon the AI check was one input to a five-role approval flow (field worker, site manager, project manager, finance, owner), with two rounds of photo evidence, Opname and BASTP, before payment. A calibrated judge made the AI's descriptions measurable. Accountable people still approved the payment, and I would design any money-touching system the same way.

If you are building evals for a vision or LLM feature and want to compare notes, you can find me at https://faishalwahiduddin.com/?utm_source=devto&utm_medium=article&utm_campaign=blog-network-2026-10