DEV Community: Ricardo Feldhaus Krisanoski

A Public Case Study Should Be an Evidence Map

Ricardo Feldhaus Krisanoski — Sat, 31 Oct 2026 21:00:00 +0000

A public case study asks a reader to infer something about your work. The structure should make that inference inspectable.

A diagram can explain a design without proving deployment. A passing test suite can support software behavior without establishing usefulness. A strong metric can still describe a weak target.

When I updated my portfolio with recent AI-assisted technical work, I wanted each case to connect its claim to the available evidence and to the limits of that evidence. This is a publishing practice, not a new experiment or a claim of independent personal mastery.

Name the claim before selecting the artifact

Different claims need different support.

“Implemented a retrieval path” calls for execution evidence and clear component boundaries. “Improved review quality” needs a workflow comparison with suitable references. “The original source is available” requires an actual source release, rather than a README describing private work.

An entirely invented example shows the distinction. A case says, “The tool reduced review effort.” Its evidence link opens an architecture diagram and a screenshot. Those artifacts might explain the intended interaction, but they cannot establish the claimed reduction.

A stronger version would state what was actually observed and list review effort as a measurement still needed. That gives the reader a useful conclusion without asking them to fill the evidence gap.

Preserve denominators and the unit of repetition

My recent retrieval case uses fictional documents and constructed references. Its evaluation contains 48 document/query cases per method, with 21 positive and 27 negative cases. Reporting top-three hits without the negative cases would hide an important part of the comparison.

The controlled SMS experiment has 80 model runs across 20 paired seed blocks and four conditions. Those runs share one corpus partition. They are not 80 independent datasets.

The RAG case has three held-out synthetic questions, only two answerable. Four variants generate twelve answers, but twelve answers do not become twelve independent questions. The original and adjusted rerankers tie on the answerable ranking metric, while human answer review remains pending.

These are separate tasks. Their numbers cannot be pooled into a general score for AI-system quality.

State which artifact the reader actually receives

The portfolio contains public writeups and reported aggregates. The original training packages, checkpoints, individual predictions, and private data are not released there.

The small aggregate checker verifies arithmetic and consistency. It does not rerun models, reproduce bootstrap intervals from original predictions, or independently authenticate the measurements.

That boundary belongs near the evidence links. A reader should not discover it only after trying to reproduce a training experiment from a case-study repository.

The same applies to older operational pages. A blueprint, a draft case, and a completed local experiment should retain their own status. Updating the portfolio introduction should not silently promote all earlier material to verified production work.

Make personal contribution discussable

AI-assisted execution can create real technical artifacts. It does not automatically establish that the author can explain every choice, modify the implementation independently, or correct a failure without a script.

A credible case leaves room for those demonstrations. I can distinguish the implementation record from the next learning checkpoint and prepare to defend the target, comparison, and interpretation.

That is more useful in a technical conversation than treating tool execution as a substitute for understanding.

Review the inference your page invites

Before publishing a case, I use a short checklist:

What precise capability does the main claim describe?
Which artifact supports that claim?
What are the denominator and independent unit?
Is a planned step clearly separated from an observed result?
Does the page identify assistance and source-release boundaries?
What evidence would change the current conclusion?

The portfolio evidence map links the document-retrieval case, controlled SMS experiment, and RAG reranker comparison.

What should a reader be able to verify before accepting the strongest claim in your next case study?

What a Small RAG Fine-Tuning Experiment Can Actually Prove

Ricardo Feldhaus Krisanoski — Fri, 30 Oct 2026 21:00:00 +0000

I fine-tuned a pretrained reranker, verified that its weights changed, reloaded the selected checkpoint, and ran a controlled comparison with the original model.

The measured ranking gain on the held-out synthetic slice was zero.

That result was useful. It showed that the training and evaluation path executed while giving me no reason to claim an improvement. The slice contained three questions, only two of them answerable. Both rerankers reached nDCG@5 of 1.0 on those two answerable questions.

This is a local, AI-assisted experiment in progress. Real-domain fine-tuning and a sealed real held-out comparison have not been completed. Human review of generated answers is also pending. The current evidence supports a working experimental pipeline and a tied result on a tiny synthetic fixture.

Start with the component and the decision

The domain question is whether adjusting the reranker helps a retrieval-assisted answer system find useful source material and produce better supported answers.

That question contains several mechanisms. Retrieval selects candidates. Reranking orders them. Generation turns the selected context into an answer. An output can fail because of any of these stages, and improving one metric does not guarantee improvement downstream.

I chose to isolate reranking. The component being trained was a pretrained multilingual cross-encoder. The embedding model and answer generator remained fixed.

The concrete model identities were:

Component	Model
Embeddings	`intfloat/multilingual-e5-small`
Reranker	`cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`
Answer generator	`Qwen/Qwen3-1.7B`

These official model weights were downloaded and actually exercised locally. The experiment used a CPU runtime. I have no verified CUDA execution to report.

Saying “I fine-tuned the RAG system” would leave too much unspecified. The adjusted weights belonged to the reranker; the experiment did not fine-tune the answer-generating language model.

Four variants with a fixed generator

I implemented four comparisons to make the source of a difference easier to inspect:

Variant	Retrieval	Reranking	Generation
A	None	None	Fixed Qwen
B	BM25 + E5, combined with reciprocal-rank fusion	None	Same Qwen
C	Same hybrid candidates	Original pretrained cross-encoder	Same Qwen
D	Same hybrid candidates	Adjusted cross-encoder checkpoint	Same Qwen

The hybrid retriever returns up to twenty candidates. C and D score the same candidate set, including cases where it contains no relevant material. Injecting the known correct passage into one variant would make the comparison answer a different question.

The generator identity includes the prompt, chat-template version, and decoding settings. Thinking was disabled for this comparison. Keeping those settings fixed matters because a prompt change can alter an answer even if retrieval is unchanged.

A versus B examines the addition of retrieval in this setup. B versus C examines the pretrained reranker. C versus D is the direct comparison for the adjusted weights. The differences remain conditional on this corpus, candidate generation, prompt, and evaluation design.

A training command is not training evidence

The synthetic training fixture contained forty pairs. I ran one epoch for each of three seeds: 42, 314, and 2026, with an effective batch size of sixteen.

The verification checked actual weight updates. It also checked that the saved selected weights matched the intended checkpoint and that independently reloading it preserved the expected predictions.

Those checks address concrete implementation mistakes: a frozen model that never updates, saving the wrong state, loading the original weights again, or evaluating a component whose identity does not match the recorded run.

They establish that training happened. They do not establish useful learning on the domain task.

On three answerable synthetic validation queries, nDCG@5 remained approximately 0.6667 before and after adjustment. The separately sealed synthetic held-out comparison also showed no ranking gain. I keep those two evaluations distinct rather than treating their denominators or scores as interchangeable.

Read the denominator with the result

The held-out fixture had two answerable questions and one unanswerable question. Four variants generated twelve answers in total.

Observation	Result	What it supports
Answerable held-out questions	2	A very small ranking check
Unanswerable held-out questions	1	Execution of an unsupported-question case
C nDCG@5	1.0	Original reranker ordering on the answerable slice
D nDCG@5	1.0	Adjusted reranker ordering on the same slice
Paired D minus C difference	0	No observed ranking gain on this slice
Human answer ratings	Pending	No measured answer-quality conclusion yet

A perfect score on two answerable questions leaves no visible room for an improvement in that metric. It does not establish that the original model is sufficient for deployment, or that fine-tuning would never help.

The useful decision is narrower: this fixture cannot justify retaining the adjusted component on the basis of a measured benefit. A harder, independently reviewed domain evaluation is needed.

Three training seeds also do not turn two answerable questions into six independent examples. Seed variation and variation across evaluation questions are different sources of uncertainty.

Separate retrieval from citations and factual support

The evaluation has three layers because each catches a different failure.

Retrieval relevance asks whether the system recovered and ranked the passages needed to answer a question. Recall@20, MRR@5, and nDCG@5 are computed over answerable questions. Unanswerable questions receive null retrieval metrics and are counted separately. A system cannot retrieve a nonexistent relevant passage.

Citation integrity asks whether an emitted reference identifies a source that actually exists and belongs to the allowed context. This catches invented identifiers and broken provenance. It cannot establish that the cited text supports the claim.

Human factual and utility review asks whether the answer is supported, whether the actual cited sources are correct, whether the response is useful, and whether clarification or handoff was appropriate.

Consider this entirely invented example:

Source: "This material supports engraving. Delivery timing depends
on the order and must be confirmed separately."

Answer: "Engraving is available, and delivery takes two days. [S1]"

The identifier could be valid, and the source could be highly relevant. The delivery claim would still be unsupported. A citation parser passing would not repair the answer.

The lab prepares a blinded review packet with shuffled groups and answer ordering. Missing or unreviewed ratings remain pending. The generator does not approve its own factual correctness. The twelve synthetic answers currently have no completed human quality evaluation to report.

Freeze what the comparison depends on

The pipeline binds corpus, splits, configuration, model identities, and actual weight hashes before the held-out evaluation. It verifies those bindings before inference.

A one-use receipt is anchored beside the freeze record. Repeating a completed request with unchanged inputs returns its saved, hash-checked results instead of generating fresh predictions. Interrupted or changed-input runs remain recorded rather than becoming opportunities to quietly try again.

This reduces accidental test reuse and configuration drift. It does not make the local operator unable to tamper with every file and receipt together. Independent storage and clear provenance still matter for an audit.

For the real experiment, checkpoint selection must use validation only. The protocol retains validation results from all three seeds and selects the confirmatory comparison before observing the test. Picking whichever seed wins on the held-out result would change the role of that test.

Real data introduces a different bottleneck

Work on the real corpus has progressed: the factual sources were reviewed, an index was built, and relevance annotation is underway. That progress is not a completed real-domain experiment.

Annotation needs usable positive passages, appropriate negatives, explicit answerability, and privacy review. A checkbox saying “reviewed” cannot substitute for missing passage labels or resolve a contradiction between “answerable” and “handoff.”

Equivalent questions introduce another constraint. Reusing a reviewed decision can reduce repeated annotation, but the copied decision is not independent evidence. Equivalent questions need to remain in the same split component, with provenance linking the reuse.

The protocol groups related customers, conversations, and near-duplicate questions before split assignment. It also keeps privacy review separate from relevance review. Approval to use a factual source does not automatically approve every conversation connected to it.

Until those requirements are met, producing more training steps would run ahead of the evidence. The pending work is reference quality and dataset preparation, followed by actual training and a separately frozen comparison.

What I would change in the next evaluation

I would include cases that make the components work harder: similar product sizes, material versus engraving questions, conflicting facts, missing price or delivery information, ambiguous requests, and prompt injection attempts.

I would report failure categories alongside aggregate ranking scores: missing retrieval, misranking, unsupported claims, unnecessary abstention, and incorrect handoff. Human review would use the same factual sources for every answer.

Loading time and warmed inference time would remain separate. Any operational benefit would need its own measured evidence rather than being inferred from a short model call or a passing software test.

The current experiment proves that the selected models can run through a controlled local pipeline and that the pretrained reranker can be adjusted, saved, reloaded, and compared. Its synthetic held-out comparison found zero ranking gain. The next decision depends on independently reviewed domain examples and human answer evaluation.

I have published the architecture, comparison table, results, and evidence boundaries in the GitHub case study. It is a public writeup; the original training package, checkpoints, private sources, and individual predictions are not exported in that repository. The accompanying aggregate checker validates the arithmetic of the reported counts. It does not rerun the models, reproduce the training, or independently establish answer quality.

What evidence would make you keep—or remove—a fine-tuned component from your retrieval pipeline?

Your AI Workflow Needs an Exception Contract

Ricardo Feldhaus Krisanoski — Thu, 29 Oct 2026 21:00:00 +0000

An operational brief can look complete while the information needed to act is still missing.

A production item may have a deadline but no approved artwork. Two systems may disagree about its stage. A suggested next action may require an authority the current operator does not hold.

In the anonymized ecommerce operating-layer case in my portfolio, exceptions and human decisions are part of the architecture. A related observability case describes a shared trace contract. Neither public page establishes a quantified benefit from exception handling. This note proposes a concrete contract for making unresolved work visible.

Describe the unresolved condition as data

A free-text warning can alert a person, but it is difficult to route, reconcile, or close consistently. A useful exception needs a type, source context, affected work, owner, and resolution state.

It should also distinguish an observation from an interpretation. “Approval field unavailable” is different from “approval denied.” Missing data cannot safely become a negative business fact.

Here is a wholly invented exception record:

{
  "type": "approval_context_missing",
  "observed_state": "required approval could not be established",
  "proposed_action": "hold for review",
  "owner_role": "production_coordinator",
  "resolution_state": "open",
  "execution_allowed": false
}

This example is an illustrative contract, not an operational export. An implementation would need authorized source references and stable identity, without exposing private payloads in public logs.

Make the next human action specific

A queue entry saying “AI uncertainty” asks the operator to investigate everything. A more useful entry asks them to confirm the applicable approval, resolve a source conflict, or assign an owner.

The daily brief can then show what is known, which dependency is unresolved, and why a suggested action remains blocked. Human prioritization stays explicit.

The same principle applies to a process change. An agent can structure a proposed SOP update, but the proposal remains a draft until the responsible owner and required approver act. A missing review is an exception state, not permission to publish the draft.

These distinctions keep the system's narrative from outrunning its authority.

Design recovery as part of the state machine

Exceptions change over time. A missing field may arrive, a conflict may be resolved, or a decision may become stale after the source changes again.

The workflow needs to recognize those transitions. A retry should update or reconcile the existing unresolved item where appropriate, rather than multiplying notifications. Closing an exception should preserve the reason and the evidence used to close it.

Some failures belong to infrastructure; others belong to the business process. A temporarily unavailable connector can support a technical retry. An unresolved policy decision needs a person. Treating both as “retry later” hides the organizational work.

This is a proposed implementation detail, not a claim that all transition paths in the case have already been verified.

Connect exceptions to observability carefully

A trace should identify the workflow, decision, outcome, risk, and review state without dumping customer messages or credentials. Sanitized references can connect the trace to authorized evidence.

The observability pattern also keeps logging failures from becoming an uncontrolled failure of the operational workflow. That does not mean the system should conceal missing audit coverage. The appropriate degraded behavior depends on the action's requirements and must be defined explicitly.

A practical review checklist is:

Can missing, conflicting, and unauthorized states be distinguished?
Does each open item request a specific next action?
Is ownership explicit when escalation occurs?
Can retries avoid duplicate unresolved work?
Does resolution retain its reason and source context?
Can operators see when coverage or audit information is incomplete?

Measurements to collect include unresolved age, repeated transfers, missing ownership, and exceptions reopened after source changes. They should accompany review effort and error analysis.

The ecommerce operating-layer case and observability contract describe the surrounding design. Which unresolved state would make today's brief unsafe to treat as an execution plan?

The Cost of Changing Your Automation Is Part of Its Design

Ricardo Feldhaus Krisanoski — Wed, 28 Oct 2026 21:00:00 +0000

The first automation demo usually shows the happy path. The more revealing demonstration is what happens when the process changes.

Can an operator update the rule? Does the change require code, configuration, or another approval? How much has to be retested? Who can explain the effect afterward?

These questions sit inside Decision Path Challenger, a public skill I built to examine a proposed technical path before meaningful implementation. The skill compares solution classes and their build, run, change, maintenance, and failure costs. This article develops the change-cost question; it reports no new deployment or measured savings.

Define the operating change before the platform

A request such as “create an agent for this workflow” already contains a mechanism. Before adopting it, I want to understand the outcome, the experience of doing the work, and the constraints that shape it.

An entirely invented example makes the tradeoff concrete. A small operations team wants to route requests under a policy that changes frequently. Three plausible approaches are a native approval rule, a lightweight integration, and a custom agent-backed service.

All three may produce a convincing demo. They distribute future change work differently.

A native rule may be easy for the owner to inspect but limited in expressiveness. A lightweight integration may connect the existing tools while adding another runtime to maintain. A custom service can express richer behavior while increasing responsibility for code, releases, monitoring, and recovery.

Those are hypotheses to examine, not universal rankings of the approaches.

Follow one rule through the system

For each option, I would trace an example policy update from the person requesting it to the person experiencing the result.

The path might involve editing a form, migrating stored values, changing a prompt, revising a permission, or updating a report. A small-looking change can touch several components when the same business rule has been copied across them.

That is why “easy to build” and “easy to change” are separate claims. Prompt editing can also be consequential: a wording change may alter routing or make previously unsupported responses seem allowed.

The useful comparison names the owner, artifact, verification, release path, and recovery step for the expected change. It should also reveal who becomes a permanent dependency.

Include the cost of finding the mistake

Change effort extends beyond applying the edit. Someone must determine whether the intended behavior occurred and whether an unrelated behavior regressed.

A system with visible rules and a focused replay set may be easier to validate than one where the same logic is spread across several opaque steps. Conversely, a familiar native tool may still be difficult to inspect if it hides history or cannot show which version made a decision.

I would ask each proposed design to demonstrate one representative change on invented inputs. The demonstration should include an expected failure and a reversal. This produces more useful evidence than arguing from a platform's general capabilities.

Choose the smallest reversible test

The skill's output calls for a decisive recommendation and a small reversible next step. That step should reduce a meaningful uncertainty.

For the invented routing example, the test might compare one native rule with a small integration on the same fictional cases. The objective would be to inspect ownership and change effort before expanding scope.

A review checklist for that comparison is:

Which likely change would be expensive under this design?
Where does the rule live, and where is it duplicated?
Can its owner explain and revise it?
Which evidence will confirm the revised behavior?
Can the previous version be restored without losing history?
What ongoing dependency does this choice create?

No cost reduction follows automatically from choosing the simpler option. The right path depends on the operating constraints and what the bounded test reveals.

The public Decision Path Challenger repository contains the skill and its supporting artifacts. Which change would you ask an automation to survive before committing to its architecture?

A Retrieval Result Needs an Allowed-Use Field

Ricardo Feldhaus Krisanoski — Tue, 27 Oct 2026 21:00:00 +0000

A passage can be relevant to a question and still be unsuitable for the response the system is preparing.

An internal exception procedure might explain how a team works without granting permission to promise that exception to a customer. An old instruction might be semantically close to the query while no longer carrying authority.

This is why the operational retrieval blueprint in my portfolio includes allowed-use metadata. The public artifact is a starter design with schema, synthetic documents, and evaluation questions. It is not proof of a fully deployed knowledge service or a measured improvement in answers.

Relevance answers only part of the question

A retriever ranks text according to a query and its scoring method. That score does not establish who may read the passage, whether it remains current, or which use it supports.

The blueprint gives retrieved chunks source information, domain, version or updated date, allowed use, and an approval requirement. It also distinguishes customer-facing language from internal context.

These fields describe different decisions. A passage may be available to a reviewer, allowed as internal context, and still require approval before it supports an external commitment.

The permissions cannot depend only on the model noticing a warning in the prompt. The application needs to determine which sources are eligible before generation and which proposed operations remain authorized afterward.

Walk a result through its intended use

Consider this wholly invented source record:

{
  "heading": "Exceptional delivery requests",
  "domain": "operations",
  "allowed_use": "internal_review",
  "customer_facing": false,
  "requires_human_approval": true
}

This is an illustrative metadata contract, not a record from an operational system.

If a customer asks for an exceptional delivery arrangement, the source might help route the request to a reviewer. It should not be treated as a preapproved promise.

The same retrieved text can therefore support a handoff explanation while being excluded from final customer wording. That is a more precise contract than attaching a generic “sensitive” label and hoping every downstream component interprets it consistently.

Preserve the rule with the passage

Chunking can separate a paragraph from the qualification that made it safe to interpret. The eligibility metadata needs to travel with each chunk, and the source reference needs to lead back to the surrounding context.

Updates create another problem. A new document version may change allowed use without greatly changing the text. A cached embedding or familiar similarity score does not establish that the previous permission still applies.

The design should define what happens when metadata is missing, the source is superseded, or approval status cannot be established. A no-answer or review path is preferable to silently assigning the most permissive interpretation.

These are requirements to validate in an implementation. The blueprint's existence does not establish that every invalidation or permission path has been exercised in a live workflow.

Evaluate inappropriate matches deliberately

Retrieval evaluation should include sources that look useful but must not be used for the requested purpose.

For example, a synthetic question could retrieve an internal exception procedure beside an approved public response. The expected behavior should specify which source may inform routing, which may support wording, and when no authorized answer exists.

My practical review would ask:

Is source access checked before results are exposed?
Is allowed use carried through chunking and retrieval?
Can the system distinguish internal context from quotable language?
Does source versioning invalidate stale permissions?
What happens when the relevant passage requires approval?
Can the trace show which rule permitted the eventual use?

I would measure inappropriate-source selection, missing metadata, stale-version use, and review outcomes separately from ranking quality. Those are proposed measurements; this blueprint has no validated production results.

The Operational Knowledge Retrieval Layer contains the starter contract and its synthetic scope. What does your system check after it decides that a passage is relevant?

A Read-Only Integration Is a Deliberate Product Boundary

Ricardo Feldhaus Krisanoski — Mon, 26 Oct 2026 21:00:00 +0000

An integration can provide useful operational visibility before it can change anything. That boundary deserves a product decision of its own.

In a portfolio case, I describe a prepared marketplace ERP integration that exposes catalog and order reads through a protected proxy. The public case records the read-first design and credential boundaries. It does not provide a validated operational baseline or establish that a future write phase has been completed.

The interesting question is what a read-only phase lets the team learn while the authority to mutate business records remains closed.

Separate context from authority

An internal agent may need to know whether a product exists, which attributes are recorded, or how an order is represented. None of those needs automatically requires permission to create products, change stock, or edit prices.

Giving an agent a broadly privileged credential makes that distinction depend on every caller behaving correctly. A narrower proxy can express the intended boundary directly.

The documented pattern keeps OAuth credentials inside the automation layer and exposes a protected lookup surface to internal scripts or agents. Secrets and private endpoints are not part of the public case.

This remains an integration design, not a guarantee that all read access is harmless. Product and order data can still be sensitive. Authorization, data minimization, and access review belong in the read phase too.

Make the boundary observable

A useful read-first phase should establish which fields are available and how reliably they represent the operational source.

An entirely invented example illustrates the distinction. A lookup returns a product description that differs from an internal planning note. The integration can present both references for review. It should not infer that the planning note authorizes updating the source catalog.

That disagreement is valuable evidence. It reveals a data-contract question before an automated write would make it harder to untangle.

The first checks should include refusal of unauthenticated requests, success of an authorized read, and rejection of unsupported operations. A successful happy-path lookup alone would not demonstrate that the permission boundary holds.

Avoid designing a universal proxy

A proxy that forwards arbitrary paths and methods can recreate the original broad API surface under another name.

For a bounded integration, I would expose named operations with explicit request validation and constrained response fields. Catalog lookup and order visibility need their own contracts. Internal callers should not have to understand the upstream credential or receive fields unrelated to their task.

Errors also need interpretation. An unavailable source, an unauthorized request, and an item that does not exist are different states. Treating all of them as an empty result can make an agent invent an operational explanation.

These are design recommendations for the next review, not a claim that this public writeup contains a released production proxy.

Treat writes as a separate capability

Enabling writes changes the failure cost. A mistaken read may mislead a reviewer; a mistaken stock update can alter downstream work.

Before enabling a write phase, I would name the allowed mutation, its owner, preconditions, expected effect, and recovery procedure. Idempotency and readback become essential because a retry must not create another effect and a successful request does not necessarily prove the intended final state.

A review checklist for that transition is:

What concrete task requires a write?
Can its scope be narrower than the upstream API?
Which source version or precondition must still hold?
How will retries avoid duplicate effects?
What readback will confirm the resulting state?
Who can stop, reverse, or investigate an incorrect mutation?

No write-phase outcome, speed improvement, or prevented-error count is claimed here. Those require separate evidence.

The read-only ERP integration case describes the prepared boundary. Which information would let your users make progress before you grant the system authority to change it?

Make Handoffs Explainable Before You Make Them Automatic

Ricardo Feldhaus Krisanoski — Sun, 25 Oct 2026 21:00:00 +0000

“Send this to a person” looks like a complete fallback until the person opens the queue.

They still need to discover why the item arrived, what the system already checked, which source is authoritative, and what action they are expected to take. If that reconstruction is missing from the design, the handoff transfers work without transferring enough context.

I have documented context-aware reception and human-reviewed commercial follow-up patterns in my portfolio. Those public pages describe design directions, with operational outcomes still listed as metrics to collect. This note develops the handoff contract; it does not report a new live automation or a measured productivity improvement.

Start with the decision the recipient owns

A reception workflow and a commercial follow-up workflow do not share every responsibility. A conversation about an unresolved service issue should not automatically become a sales reactivation opportunity.

The documented patterns therefore distinguish commercial context, follow-up lanes, operational limits, and human review. The handoff should preserve that distinction rather than route every uncertain item to a generic inbox.

The recipient needs an answerable request: inspect a conflicting source, decide whether a message can be sent, or redirect the matter to the appropriate team. “Low confidence” can explain part of the trigger, but it does not define the work.

Use a packet a person can challenge

Here is an entirely invented handoff example:

Reason: current request conflicts with earlier campaign context
Current interpretation: product clarification may be required
Evidence: latest message and approved response guidance
Unresolved: which product the person means
Action requested: choose the appropriate clarification
Ownership: reception reviewer
External action: pending human decision

This is an illustrative contract, not an export from a customer conversation.

The packet should distinguish observed facts from the system's interpretation. A reviewer can then reject the interpretation without losing the evidence. It should also show what remains unresolved, rather than presenting an AI summary as a completed investigation.

Evidence references must respect the recipient's access. A handoff is not a reason to copy every historical message into every queue. The useful context is the minimum needed for the assigned decision.

Model ownership after delivery

A route is not complete merely because an item appears in a queue. The design needs an owner, an accepted or pending state, and an escalation path if the receiving role cannot act.

A sales reviewer may be able to approve a follow-up but unable to resolve a policy exception. That should become an explicit transfer with a reason. Otherwise, the queue can look active while responsibility is still absent.

Similarly, a retry should not create several independent tasks for the same unresolved situation. Stable identity and explicit state transitions are important even when the final action remains human.

These are proposed engineering requirements for a dependable handoff. The portfolio pages do not establish that every requirement has been deployed or that every queue transition has been measured.

Keep the outcome in the quality loop

A reviewer may confirm the interpretation, correct it, request more context, or decline the proposed action. Those outcomes help explain which handoffs were useful and which merely moved ambiguity around.

I would measure unresolved ownership, missing evidence, repeated transfers, and corrected interpretations alongside response time. A fast transfer can still be a poor result if the next person cannot make the decision.

A practical handoff review asks:

Is the requested human decision named?
Are source facts separate from interpretation?
Is the trigger specific enough to investigate?
Does the recipient have the necessary authority and access?
What happens when ownership is rejected or unclear?
Can the system record the human outcome without inventing one?

The context-resolution case and commercial follow-up design provide the surrounding patterns. What would a colleague need to receive a handoff and act without rebuilding the entire conversation?

Freeze the Comparison, Not the Working Document

Ricardo Feldhaus Krisanoski — Sat, 24 Oct 2026 21:00:00 +0000

A supplier comparison is a moving object. A revised offer arrives, a quantity changes, and someone corrects the delivery condition. The working document should accommodate those changes. The decision record needs to preserve what was actually compared.

That distinction shaped an internal procurement implementation documented in my public portfolio. This article discusses its anonymized design pattern. It introduces no new deployment or measured purchasing benefit.

The central question is simple: when someone revisits a purchase decision, can the system show the evidence and assumptions that existed when the choice was made?

Give the objects different lifecycles

The original supplier document, normalized offer, working comparison, and decision snapshot serve different purposes.

The document records what the supplier provided. Normalization records an interpretation: which number is the unit price, which unit applies, and what conditions accompany it. A working comparison lets the team resolve uncertainties. The snapshot preserves the comparison context submitted for a decision.

Collapsing these into one editable row creates a subtle failure. A correction made today can make yesterday's decision appear to have used information that was not available then.

In the documented implementation, originals are preserved with integrity metadata. Reprocessing creates another version. A human decision attaches to a frozen comparison and a justification, while the audit trail retains subsequent events.

A cheaper offer can be a different offer

Consider this entirely invented example:

Offer	Commercial statement	Unresolved question
Supplier Cedar	Material priced per pack	How many usable units does a pack contain?
Supplier Harbor	Material priced per unit	Does the specification match the approved requirement?

Neither headline value establishes the better purchase. Before comparing price, the buyer needs an equivalence rule covering the relevant specification, unit, quantity, and commercial conditions.

The snapshot should capture the result of that review, including unresolved conditions. Recording only the chosen supplier and total would lose the reasoning that made the selection defensible.

A source hash helps identify the document used. It does not prove that an extracted number is correct or that two items are equivalent. Those remain distinct checks.

Freeze at the authority boundary

Freezing every draft edit would burden the workflow without clarifying the decision. The useful boundary is the transition from a mutable comparison to a context on which an authorized person may act.

At that transition, the system should validate the eligible offers and relevant policies, create the snapshot, and connect the human choice to it. Competing changes need explicit handling so that the choice cannot accidentally refer to one version while the displayed table shows another.

A later correction should remain possible. It should produce an amendment, replacement decision, or new snapshot with a reason. It should not silently rewrite the earlier context.

This also changes interface design. The buyer needs to recognize whether they are viewing the current workspace or the historical decision context. An attractive table is insufficient if that distinction is hidden.

Review the transition before adding extraction

AI extraction could eventually help structure incoming evidence. In this case, external AI processing and other integrations remain separately gated. The comparison model must remain understandable before a parser adds another source of interpretation.

My practical review checklist is:

Can a reviewer recover the original evidence associated with the decision?
Are units and equivalence assumptions explicit?
Does reprocessing create a version rather than overwrite evidence?
Can a concurrent edit change an already confirmed comparison?
Is the approver's choice connected to a snapshot and reason?
Can a correction preserve the previous decision history?

Useful future measurements include the effort to reconstruct a decision, unresolved-equivalence frequency, and time from approved need to justified choice. I have no validated public baseline for those outcomes.

The procurement case describes the wider implementation and its limits. What needs to remain inspectable after the working document changes?

What a Production Brief Should Say When the Data Is Incomplete

Ricardo Feldhaus Krisanoski — Fri, 23 Oct 2026 21:00:00 +0000

A production brief can be well written and still give the team the wrong impression of readiness. The danger is especially clear when the report combines current facts with missing channel data or unresolved dependencies.

In my public operating-layer case, the daily brief connects normalized demand with deadlines, stages, artwork, materials, approvals and exceptions. Human prioritization remains part of the flow. This technical note develops how the brief should communicate incomplete information. It does not introduce a new deployment or claim a measured planning improvement.

Completeness has more than one meaning

An order can be present in the operational layer while its artwork approval remains unknown. A source can be reachable while its latest update is unavailable. A report can contain every field it was programmed to display while excluding a channel whose ingestion failed.

Those are different gaps. A single “complete” indicator would make them difficult to distinguish.

For a report design, I would separate source coverage, record readiness and decision readiness. Source coverage describes which inputs contributed. Record readiness describes the dependencies known for a particular item. Decision readiness describes whether the person has enough evidence and authority to select the next action.

These are proposed reporting distinctions. The public case establishes the need to disclose missing coverage and uncertainty; it does not certify this exact field structure.

An invented morning brief

Consider a fictional brief with two visible work items. One has a confirmed deadline and material availability but still needs artwork approval. The other has an approved design, while its delivery commitment could not be refreshed from the source.

Suppose another sales channel did not contribute data to this run. A polished summary saying “today's work is ready” would hide both item-level and coverage-level uncertainty.

A more useful brief would state the known dependencies, identify the stale commitment, disclose the excluded channel and direct the unresolved decisions to a person. It would not invent an approval, silently reuse a date as current or claim that the visible items represent all demand.

This is an invented example, not a real order list or a report from a production team.

Facts should remain separate from interpretation

The operating-layer case calculates and reconciles facts before AI interpretation. That separation helps when the report is incomplete.

The fact layer can state what arrived, what was checked and what remains unknown. The narrative can explain why an exception deserves attention. The explanation should not transform uncertainty into a confident priority simply to produce a cleaner opening paragraph.

I would also preserve the source relationship behind important statements. If the brief says that approval is missing, a reviewer should be able to determine whether the source explicitly says “pending” or whether the approval source was unavailable. The operating responses can be very different.

A generation timestamp alone cannot settle that question. The report may be newly generated from older or partially missing inputs. Freshness belongs to the contributing facts as well as the final document.

A checklist for a partial brief

Does the opening state which sources contributed and which did not?
Are unknown, pending and conflicting dependencies distinguishable?
Can the reader identify stale facts without inspecting every system?
Are statements about “all work” supported by actual source coverage?
Does each exception show the decision or clarification still required?
Can a person correct or reprioritize the brief without losing the original context?
Are report generation and delivery retries kept from duplicating operational work?

The purpose is a usable partial report, not a perfect-looking report. Whether it helps the team would need evidence such as corrections, missed dependencies and planning effort measured against an appropriate baseline.

The public operating-layer case describes the brief's role in the larger system and keeps quantitative benefits as metrics to collect.

If an input disappears tomorrow morning, will your production brief explain the missing coverage before anyone acts on its priorities?

Read Access and Action Authority Belong to Different Contracts

Ricardo Feldhaus Krisanoski — Thu, 22 Oct 2026 21:00:00 +0000

An employee may need to understand a procedure without having authority to change the records governed by it. An AI assistant inherits the same distinction when it retrieves knowledge and prepares a tool request.

My public operating-layer case separates a knowledge plane from an action plane. Document groups and approved sources control retrieval. Roles, policy and approval rules control operational changes. This note examines the contract between those planes rather than claiming a new security certification or a completed authorization rollout.

A document can describe an action without granting it

An approved procedure can explain how a financial exception is handled. That makes the procedure useful to someone who needs to recognize and route an exception. It does not necessarily authorize that reader to execute the financial change.

The same issue arises when retrieved text reaches a model. The model can explain the procedure, summarize required evidence or identify the relevant authority. A generated request to perform the operation is still a request.

Treating access to instructions as access to execution would let knowledge retrieval silently expand operational privileges. The output would look informed while the action boundary had disappeared.

The two contracts therefore need different questions. Retrieval asks whether this identity may consult this approved source. Execution asks whether this identity may perform this operation on this object under the present conditions.

An invented exception request

Imagine a fictional production operator who can read a procedure for dealing with an incorrect personalization. The operator asks the assistant what should happen next.

The assistant retrieves the relevant procedure and prepares a proposed customer-remedy request. That can help explain the next step. It does not establish that the operator may authorize a financial adjustment.

The action path would need to evaluate the requested operation, the affected object and the applicable approval rule. If additional authority is required, the request should remain a proposal with its evidence rather than being described as an executed remedy.

This is an invented example. It does not disclose an actual customer case or assert that a particular remedy tool is connected in the public implementation.

Preserve the boundary after generation

Putting role instructions into a prompt can help the assistant explain its scope. I would still require the action interface to enforce the actual authorization contract when a request arrives.

That interface needs to distinguish the assistant's service identity from the human role represented in the request. The exact implementation depends on the system, but the review question is stable: whose authority is being evaluated, and what evidence binds it to this operation?

Current state also matters. A source retrieved earlier can remain readable while the target object changes or the approval becomes invalid. An execution decision should therefore use the applicable current authorization state, not merely reuse the model's earlier narrative.

The public case includes deterministic authorization and audit boundaries at the system-pattern level. It does not publish its production policy definitions. The details here are questions to apply during design and testing.

Test the contracts separately

Can an authorized reader retrieve the right approved procedure?
Is an unrelated or restricted source excluded from that retrieval?
Can the same reader be denied an operation outside their authority?
Does a proposed tool request remain distinguishable from a completed action?
Are object state and required approvals checked at execution time?
Does the audit record identify the operation, authority decision and outcome without copying unnecessary private content?
Can a reviewer explain a denial without treating it as a retrieval failure?

Those checks examine different responsibilities. Passing the knowledge checks does not certify action safety, and passing a tool authorization test does not establish retrieval quality.

The public operating-layer case describes the two planes and their intended relationship.

Where in your system is an informed suggestion converted into an authorized operation, and can you inspect that boundary independently of the prompt?

Do Not Turn an Observability Failure into a Business Outage

Ricardo Feldhaus Krisanoski — Wed, 21 Oct 2026 21:00:00 +0000

The rule in my public AI workflow observability contract is direct: observability failures must not break the production workflow.

Applying that rule requires a precise boundary. A trace-delivery error should not silently become a business failure, but an authorization failure cannot be dismissed as optional telemetry. Both may appear in the same execution history while requiring very different responses.

This technical note examines that distinction. The public contract describes standardized, sanitized trace fields and a bridge pattern. It does not provide measured availability guarantees or certify the recovery behavior discussed below.

Name the failed responsibility

A workflow can perform a business operation, evaluate an authorization rule and emit a trace. These responsibilities may be adjacent in code without sharing the same failure policy.

If a permission check is unavailable, the system cannot simply assume permission. If the trace destination is unavailable after an authorized operation completes, replaying the operation to recover the missing trace may duplicate work.

The useful first question is therefore not “Should this exception be swallowed?” It is “Which responsibility failed, and what state is already known?”

I would represent the business outcome and observability outcome separately. The business operation might be confirmed while trace delivery remains pending or failed. That state is more informative than replacing the entire execution with a generic error.

An invented trace timeout

Consider a fictional workflow that routes an item to human review. The routing operation confirms success. A subsequent request to the observability bridge times out.

If the orchestrator treats every downstream error as a reason to rerun the whole workflow, it could create a second review item. The missing trace would have caused a business duplication.

In the proposed design, recovery would retry or reconcile the trace operation using its existing relationship to the execution. It would not assume the routing failed. A separate operational signal would expose the missing visibility.

This invented example illustrates a failure boundary. It is not a report of an actual outage, an implemented queue configuration or a tested delivery guarantee.

Degraded visibility must remain visible

Separating failures does not mean discarding them. If trace delivery fails repeatedly, the system needs another way to reveal the gap. Otherwise, “business flow continues” becomes “nobody knows which evidence is missing.”

For an implementation review, I would ask how trace attempts are bounded, how execution identity survives retries and how missing outcomes can be reconciled. I would also examine whether a slow trace destination can accumulate enough waiting work to affect the business path indirectly.

Those are design checks, not claims that buffering or a particular timeout policy already exists in the public case.

Data minimization remains part of the same contract. Falling back to an unrestricted local log would be a poor recovery if it exposed raw messages, authentication material or private operational payloads. The reduced trace should still describe decisions, risks and review state through approved fields and sanitized references.

A boundary checklist

Is business success recorded independently of trace-delivery success?
Are authorization and integrity checks kept outside the optional telemetry failure policy?
Can a trace retry avoid rerunning the business effect?
Are repeated trace failures discoverable through a separate operating signal?
Is waiting or buffering bounded rather than allowed to grow invisibly?
Do degraded paths preserve the same data-minimization rules?
Can a reviewer distinguish missing telemetry from an absent business event?

The next evidence would come from controlled interruption tests and reconciliation records. A documented rule alone cannot establish that the runtime honors it.

The public observability contract explains the trace fields and safety boundaries behind this note. Coverage and review-outcome capture remain metrics to collect.

If your trace store stopped responding, which business effects would your retry policy repeat, and how would you discover the visibility gap?

A Retry Needs an Identity, Not Just Another Attempt

Ricardo Feldhaus Krisanoski — Tue, 20 Oct 2026 21:00:00 +0000

A workflow timeout tells me that an attempt did not finish cleanly from the caller's perspective. It does not tell me whether the business effect happened.

That uncertainty is central to retry design. Repeating an operation can recover missing work, duplicate completed work or apply an old instruction after the source has changed.

My public ecommerce operating-layer case describes normalized demand, explicit exceptions and workflows designed to recover without creating duplicate work. This note develops the identity questions behind that pattern. It proposes a review contract rather than documenting a new production schema or a measured delivery guarantee.

Separate three things that can look like one event

The source event describes something reported by an authoritative system. The processing attempt describes a particular effort to handle it. The business effect describes the work the application intends to create or change.

Those identities should not collapse into the same value merely because one trigger started the workflow.

An attempt identifier helps explain which execution failed. It does not necessarily identify the intended effect. Conversely, an effect identity can remain stable across attempts, while the attempt history records different outcomes.

The exact keys depend on the domain and source guarantees. I would first ask which source object or revision defines the intent, what makes two effects equivalent, and where the system can authoritatively check completion.

An invented interruption

Imagine a fictional order event that should create a production-review item. The worker submits the item, but loses the response before it can record success.

Starting another worker with a new attempt identifier is appropriate for diagnostics. Creating another production item simply because the attempt is new would duplicate the intended work.

A proposed retry contract would preserve the original effect identity and consult the authoritative completion state before deciding what to do. If the source order had since changed, that revision would need an explicit relationship to the original intent. It should not be disguised as another attempt at an unchanged operation.

This is a design illustration, not a real order or a claim that one identifier can guarantee exactly-once execution across arbitrary systems.

The check belongs near the effect

A workflow-level flag can be useful, but it can also become stale. If two attempts both read “not completed” before either writes success, the flag alone does not prevent duplicate effects.

For a concrete implementation, I would inspect the actual effect boundary: what operation can reject an equivalent duplicate, what record establishes completion and how competing attempts are resolved. The answer might involve a transactional constraint or a destination-provided idempotency contract. This note does not prescribe either as an already deployed feature.

The recovery path also needs to preserve uncertainty. “Request sent” and “effect confirmed” are different outcomes. If the destination cannot confirm either success or absence, the item may require reconciliation rather than another blind send.

For reports, the same distinction applies to calculation and delivery. Recomputing a narrative can be harmless while sending it again is a separate external effect.

A retry review checklist

Can the source intent be identified independently of the worker attempt?
Is an updated source revision distinguishable from a replay?
Is the intended business effect defined precisely enough to detect an equivalent duplicate?
Where is completion checked, and can competing attempts race past that check?
Does the record distinguish attempted, uncertain and confirmed effects?
Can unresolved cases be reconciled without pretending that a timeout means failure?
Are report generation and report delivery tracked as separate operations?

These questions help make a recovery design inspectable. They do not prove the behavior of a live connector. That requires tests against its actual guarantees, interruption scenarios and observed reconciliation outcomes.

The public ecommerce operating-layer case provides the operational context for this identity boundary.

Which identity in your workflow survives a retry, and which identity should change with every attempt?