DEV Community: Ricardo Feldhaus Krisanoski

A Quote Comparison Needs an Equivalence Policy

Ricardo Feldhaus Krisanoski — Sun, 18 Oct 2026 21:00:00 +0000

Sorting supplier offers by price is easy. Establishing what those prices refer to is the part that needs a policy.

An offer may use a different unit, specification, delivery condition or validity period. A clean comparison table can conceal those differences if normalization turns them into apparently equivalent rows without recording a judgment.

My public procurement case separates original supplier evidence, normalized working data, a frozen comparison snapshot and a justified human decision. This technical note focuses on the boundary between normalization and equivalence. It does not report a new deployment or a measured purchasing benefit.

Normalization does not grant eligibility

Normalization makes information easier to compare. It can standardize a unit name, preserve the quoted quantity and expose a delivery term in a consistent field.

Equivalence asks a different question: does this offer satisfy the approved need under the comparison policy?

A unit conversion can be mathematically valid while the product remains unsuitable. An apparently matching description may omit a required specification. A delivery date can be acceptable for one requisition and disqualifying for another.

The data model should let a reviewer inspect those distinctions. A normalized price should retain its relationship to the original offer, and an unresolved equivalence check should remain unresolved. It should not become eligible because a parser successfully populated the numeric fields.

An invented comparison

Imagine a fictional requisition for mounting kits. One offer describes a pack of compatible kits with delivery within the required window. Another describes individual brackets with a lower headline price, but leaves the fasteners and delivery condition unclear.

Converting both offers into a price per component would not establish that either component meets the requested kit specification. The missing fasteners could change the required purchase, and uncertain delivery could change whether the offer is usable.

Under a proposed category policy, the second offer would need clarification before entering an eligible-price comparison. The point is not that it must lose. It is that its eligibility cannot be inferred from the lower number. This is an invented example, not a published supplier quote.

Make the policy reviewable

I would describe an equivalence policy in terms of required attributes, permitted transformations and unresolved conditions.

Required attributes establish what must be known. Permitted transformations describe how quantities or units can be reconciled. Review conditions identify judgments that cannot safely be automated, such as whether a substitute material satisfies the operational need.

Policy versioning matters because a justified decision belongs to the rules and evidence available at the time. If a category policy changes later, the system should preserve the original decision context rather than silently reinterpreting it.

The procurement case uses frozen comparison snapshots for that reason. New evidence can support a new comparison. It should not rewrite what an earlier buyer saw or the reason they recorded.

A checklist before sorting eligible prices

Is the approved need explicit enough to define a comparable item?
Are unit, quantity and specification checks recorded separately?
Are commercial and delivery conditions available for inspection?
Is every transformed value linked to its original evidence?
Can an offer remain pending without receiving a fabricated equivalent value?
Is the policy version bound to the frozen comparison?
Does an authorized person record the final choice and justification?

This checklist is a design aid. A completed checklist does not independently prove that a particular product is suitable or that the buying decision saved money.

The next useful measurements would include clarification frequency, unresolved equivalence cases and the time required to reconstruct a prior decision. Those remain measurement work, rather than benefits inferred from the presence of a comparison interface.

The public procurement case documents the evidence and authority pattern behind this note.

Which attributes would your system need to establish before it is allowed to call two offers comparable?

A Useful Context Packet Should Explain What It Left Out

Ricardo Feldhaus Krisanoski — Sat, 17 Oct 2026 21:00:00 +0000

A small context packet can make an AI response easier to control. It can also hide an important decision: which available information did the system exclude, and why?

In my public context-aware reception case, the central design is a resolver before generation. It reads operational context, selects an approved commercial directive and passes a smaller package to the response agent. The model does not receive every campaign clue, historical message and response rule as an undifferentiated bundle.

This technical note explores an extension to that pattern: an exclusion record for the reviewer. It is a design proposal, not a claim that a new exclusion ledger has been deployed or improved response quality.

Selection creates a second explanation problem

When a reply is wrong, inspecting the final prompt may explain what the model saw. It cannot necessarily explain what a preceding resolver discarded.

That distinction matters when sources disagree. A campaign attribution can indicate one product family while the current conversation points elsewhere. A historical CRM field can still be populated even though its meaning is no longer current. An approved directive may exist without being applicable to this interaction.

The resolver needs a selection policy before generation. A reviewer also needs to distinguish “not available,” “available but stale,” and “available but deliberately excluded.” Those states imply different corrective work.

An absent source may require an integration check. An outdated source may need a freshness rule. A valid source excluded under the wrong policy may reveal a resolver defect. Adding more material to the prompt would not necessarily fix any of them.

An invented example

Consider a fictional interaction about a replacement accessory. The system has a campaign signal for a complete product, an old CRM category and an approved accessory-support directive.

The proposed resolver output could identify the support directive as selected, record that the campaign signal was insufficient to establish the current request, and mark the old category as requiring freshness review. It would retain sanitized source references rather than copying the conversation into another log.

The response agent would receive the relevant directive and remaining uncertainty. A reviewer could separately inspect the exclusion reasons. This example describes a proposed contract; it is not a real customer exchange or an observed outcome.

Keep the packet and the review record distinct

I would avoid solving this by placing every excluded source back into the agent's context. That would recreate the ambiguity the resolver was introduced to reduce.

Instead, the response packet would contain the selected context, the applicable directive and any uncertainty needed to answer safely. A separate review record would capture source categories considered, the selection-policy version and concise exclusion reasons.

Access rules would apply to both records. A trace reference should not become a route around document permissions, and a reason code should not accidentally reveal sensitive source content.

The record would also need a bounded vocabulary. “Lower priority” is weak evidence if nobody can explain which rule established priority. Useful reasons describe a check: outside the current request, superseded version, unresolved conflict, insufficient attribution or unavailable source.

A practical review checklist

Before adopting an exclusion record, I would check:

Can the reviewer distinguish missing input from deliberate exclusion?
Does every exclusion reason correspond to an explicit rule?
Can the selected directive be traced to an approved version?
Does unresolved conflict remain visible to the response path?
Are references sanitized and subject to the original access boundaries?
Can a reviewer reconstruct the resolver decision without expanding the model's prompt?

These checks establish an inspectable design. They do not demonstrate that the resolver selects correctly. That would require reviewed cases, disagreement analysis and a measured baseline for wrong-context replies.

The public context-aware reception case provides the underlying resolver pattern and its current evidence limits.

When you review an AI response, can you explain both the context it received and the relevant context the system chose to leave out?

An Evaluation Score Is Only as Useful as Its Target

Ricardo Feldhaus Krisanoski — Fri, 16 Oct 2026 12:09:52 +0000

My small document-evidence model agreed with its reference labels on 36 of 40 observations. The headline was 90%.

Then I inspected the confusion matrix. It missed the only reference labeled as documented evidence.

That error mattered more to the intended workflow than the headline suggested. I was building an assistant to help someone locate and check evidence in a document. Reproducing the dominant “not found” state while missing the sole positive example did not establish that the assistant was useful.

This was a local prototype, evaluated on its own training observations against AI-origin references. It was not an independent test, a measure of candidate ability, or evidence of improved hiring outcomes. Those qualifications change what the score means.

The exercise made me treat the target definition as part of the system design.

What was the model being asked to predict?

The prototype assigned a documentary state to each of four dimensions in a document. Its four possible states were documented evidence, not located, explicit absence, and ambiguous.

Each state needs a precise interpretation. “Not located” describes what the available document and process revealed. It does not establish that a person lacks a skill. A document can omit something the person knows. Extraction can fail. A relevant passage can use unexpected language.

“Documented” also requires care. A statement can mention a tool without describing completed work. An intention to learn is different from evidence of practice. Conflicting passages may require a reviewer rather than a forced positive or negative label.

The references for this run came from AI, without a new independent human annotation. Agreement with them measured reproduction of those references. Calling that number “accuracy” without naming the target would invite a much broader interpretation than the experiment supported.

Put class support beside the aggregate

The forty observations came from ten documents, with four dimensions per document. Their reference distribution was highly uneven:

Reference state	Observations
Documented evidence	1
Not located	33
Explicit absence	0
Ambiguous	6

The model reproduced all 33 “not located” references and three of the six ambiguous references. It predicted “not located” for the remaining three ambiguous observations and for the one documented-evidence observation.

Those outcomes explain the 36/40 agreement. They also show why the score could conceal a failure in the capability I wanted to investigate.

There were no explicit-absence references. A reporting convention can include that class in a four-class macro-F1 calculation, but its numerical contribution does not become measured performance on that state. The class had no evaluation support.

I want a report to make unsupported capabilities visible. A reader should be able to distinguish “the method failed on these examples” from “this dataset did not test the capability.”

Forty rows are not forty independent documents

The four dimensions within one document share text, formatting, extraction quality, and source context. That creates correlation between observations.

It would be misleading to describe this as forty independent cases. It would also be risky to split the rows randomly and allow dimensions from one document to appear in both training and evaluation. A model could encounter much of the same underlying input on both sides.

For a future document-level evaluation, I would group observations by source document before assigning splits. Closely related versions and duplicated documents need the same treatment. The unit of separation should reflect how information can cross the boundary, rather than whichever unit makes the table easiest to partition.

This particular run had an even simpler limitation: all forty observations were training observations. It demonstrated technical learning and exposed errors. It did not measure performance on unseen documents.

Keep a baseline, then ask what it measures

An earlier, separate benchmark compared a TF-IDF logistic-regression baseline with a small PyTorch model on a frozen document split. Its labels were derived from a lexical rubric, not from independently observed outcomes.

Across five additional seeds on that same split, the baseline's test macro-F1 stayed at approximately 0.902. The neural model ranged from approximately 0.576 to 0.902.

That was a useful diagnostic. The baseline was stable under that setup, while the neural model showed substantial sensitivity to initialization and optimization. Reporting only the best neural run would have hidden that sensitivity.

However, the baseline also shared vocabulary with the labeling rubric. A high score could reflect successful reproduction of the proxy. It did not establish the quality of an operational decision.

Repeated seeds examine one source of variability. They do not produce new independent documents, repair a weak target, or make an already inspected test set fresh again. I keep those historical results separate from the forty-observation prototype because they answer different questions.

More examples can introduce a different task

The next data problem was semantic compatibility. A larger synthetic corpus used eight graded criteria and levels from zero to four. The documentary prototype used four dimensions and four evidence states.

These are different targets. Concatenating their rows would not make them equivalent.

Consider this entirely invented passage:

I have explored a video-editing tool through tutorials and would
like to use it in a future team project.

A generated grading rubric might award a low level for exposure to the tool. A documentary rule might classify the passage as insufficient evidence of completed editing work. Neither label automatically specifies an operational decision.

The mapping needs an explicit definition, reviewed examples, and a record of the original target. Ambiguous or incompatible cases should remain visible. A score threshold is not a substitute for deciding what the label means.

A later local experiment preserved the synthetic corpus's own eight-criterion target rather than silently converting it into the earlier four-state task. That preserves an important boundary: a result on the synthetic task remains a result on that task.

An unseen identifier can still contain familiar language

Synthetic data makes it easier to construct coverage and contrastive examples. It also makes shortcuts easy to repeat.

Two documents can have different identifiers, names, section orders, and small wording changes while retaining the same underlying phrase template. An exact-duplicate check can pass even when the train and test examples remain linguistically very similar.

In the corpus I inspected, variants reused phrase bases with added framing. That is a reason to investigate template overlap, not evidence that every result is invalid or that all leakage has been removed.

For a stronger evaluation, I would track template families, inspect similarity across splits, and reserve genuinely different evidence formulations. I would also check whether length, section structure, or stylistic cues reveal the generated label. An unseen record is only useful to the extent that it tests the intended difficulty.

Evaluate the workflow the model is meant to help

The product question is whether the suggestion helps a reviewer find and assess evidence.

The next useful comparison would be document plus search versus document plus search plus a model suggestion. It would need independently reviewed references, positive and ambiguous examples, missing-evidence cases, and explicit handling of extraction failures.

I would measure missed evidence, misleading suggestions, corrections, abstentions, and review effort. The model suggestion and retrieved passage should remain separately inspectable: the search result is not automatically an explanation of the neural output.

Reference review should also avoid showing the model's suggestion first. Otherwise, the evaluation can inherit the model's framing. Disagreements need resolution or a recorded pending state, rather than an invented consensus.

That study has not been completed. It is the next evidence needed to judge usefulness, rather than another interpretation of the existing training score.

Questions I now put before the score

Before comparing methods, I write down:

What does the reference label mean, and who or what produced it?
Which classes and operational failures have actual support?
What is the independent unit: a row, a document, or a source family?
Has the evaluation set already influenced design decisions?
Are new labels compatible with the existing task?
Which workflow outcome would justify adding the model?

The 90% headline was useful because inspecting it changed my next step. The system needed better reference coverage and an independent workflow comparison before a claim of utility.

What does the target in your current evaluation actually represent?

A Daily Production Brief Is a Coordination Interface, Not a Dashboard

Ricardo Feldhaus Krisanoski — Wed, 23 Sep 2026 10:28:13 +0000

A production team should not reconstruct today’s priorities manually.

In a personalized-products operation, a paid order is only the start of a chain of commitments. Artwork may need review. A customer may still need to approve a design. Materials may be unavailable. An item may already be partway through production while its delivery deadline is getting closer.

The information needed to act can live across sales channels, task boards, documents, and the experience of individual team members. A status dashboard may show that an order exists without answering the more useful question: what should the team pay attention to now?

I built a daily production brief as part of an AI-assisted operating layer for an anonymized, multichannel personalized-products ecommerce operation. The brief is a coordination interface: it brings together the context needed for the team to start the day and handle exceptions with human judgment.

The brief starts with operational questions

A useful brief should make it easy to see:

what needs to be made;
by when;
which production stage each item is in;
whether artwork, materials, personalization, or approval is still a dependency;
which exceptions or risks need attention;
which cases still need a human decision.

Those details connect a commercial promise to the work required to fulfill it. Without them, the team has to rebuild the same picture manually by switching between systems and asking colleagues for context.

From channel events to a shared starting point

The operating flow is straightforward to describe:

sales channels
  -> integration and normalization
  -> operational data layer
  -> deadlines, stages, and dependencies
  -> exception detection
  -> daily production brief
  -> human prioritization and execution

Each sales platform remains authoritative for its own records. Integration turns the relevant events into consistent operational objects and keeps a reference to the source record. That gives the team a shared view without pretending that one new interface replaces every source system.

The system also needs to handle interruptions safely. Reprocessing an event should not silently create duplicate work. Missing or conflicting information should become an explicit exception rather than being filled with a plausible guess. The brief is only useful when people can tell which items are ready, which are blocked, and where the underlying context is incomplete.

A queue is more useful when it explains dependencies

Consider two orders with similar due dates. One may have approved artwork and all required materials. The other may be waiting for a customer decision. A simple chronological list hides the difference; an operational brief makes the dependency visible.

That does not mean the system should decide every production sequence. A machine constraint, quality issue, late approval, or urgent exception can change what makes sense on the floor. The system surfaces the work, deadlines, dependencies, and risks. The production team remains able to reprioritize critical work and record why.

This boundary keeps the automation useful without turning an incomplete model of the operation into an unreviewable authority.

Design the brief around exceptions

The brief should help answer three questions before work begins:

What is ready?
What is blocked?
What needs judgment?

This framing directs attention to the cases where coordination matters most. A missing material, unresolved artwork approval, or conflicting deadline is not just a bad value in a report. It is an exception with an owner and a next decision.

In the implementation, normalized order information, production stage, deadlines, dependencies, and exception signals feed the brief. The team uses it to begin execution and to identify what needs human attention. The architecture keeps source data, calculated facts, AI interpretation, and human decisions distinguishable.

Measure whether it improves the start of the day

I am not publishing a time-saved or error-reduction percentage without a validated baseline. The useful measures to collect include:

time required to prepare the daily production plan;
exceptions surfaced before work starts;
how often each brief item has an explicit dependency;
orders represented from each relevant channel;
how often a human changes a suggested priority;
whether a reprioritization has a clear reason and owner.

These measures would help answer whether the brief reduces manual reconstruction, improves visibility of blockers, and gives the team a better starting point. They also help reveal missing rules or data when human overrides happen repeatedly.

The design lesson

A production brief is not valuable because it puts more fields on a screen. It is valuable when it connects demand to operational context and gives people a reliable place to see readiness, blockers, and decisions.

That lesson applies beyond ecommerce. In any operation where a request passes through several systems and roles, a useful AI layer must make responsibilities, dependencies, uncertainty, and authority visible. The brief prepares a decision surface; the people accountable for the work still decide how to handle critical exceptions.

The full anonymized case and architecture are available in the public portfolio repository.

Which exception would you want surfaced first in your team’s daily brief?

Operational Knowledge Is a Product Surface, Not a Folder

Ricardo Feldhaus Krisanoski — Wed, 16 Sep 2026 10:03:08 +0000

When an AI workflow needs operational context, the usual instinct is to collect documents in one place and call it a knowledge base.

That is a start. It is not yet a product surface.

A folder answers a storage question: where do files live?

Operational knowledge answers a different question: what should a person or workflow be allowed to rely on while making a decision?

That distinction changes the design.

Documents are not the interface

Policies, SOPs, product notes, escalation rules and commercial guidance usually begin their lives as documents. They are written by different people, revised at different times, and stored in different places.

For a human operator, that often means searching across drives, chat history and institutional memory. For an AI workflow, it means the risk of retrieving a plausible paragraph without knowing whether it is current, approved, or relevant to the situation.

The answer is not to put every document into a larger prompt.

The answer is to make knowledge usable under explicit operating rules.

Start with source authority

Every source should have an owner and a role.

Some sources are authoritative for customer-facing language. Some are reference-only. Some are historical and useful for understanding how a decision was made, but should never guide a new action.

That is why a retrieval layer needs metadata alongside text:

source and version;
business domain;
approved use;
owner and review date;
whether the source can inform a customer-facing response;
whether a human approval is required.

This is not administrative overhead. It is the boundary between a grounded answer and an answer that merely sounds grounded.

Retrieval is a contract

I think of retrieval as a contract between knowledge and the workflow that consumes it.

The workflow needs to know what it is trying to decide, which domain applies, whose permissions are in scope, and what it must do when evidence is missing.

The knowledge layer needs to return more than passages. It should return the evidence needed to judge whether a passage can be used: source, version, allowed use, and the conditions under which the workflow should hand the decision back to a person.

That produces a different kind of system behavior.

Instead of: “Here is the most similar text,” the system can say: “Here is the approved source for this situation,” or “There is no approved source with enough confidence, so this needs human review.”

The no-answer path is not a defect. In operational systems, it is part of the product.

Permissions are part of the experience

Knowledge permissions and action permissions solve different problems.

An employee may be allowed to read a support procedure but not change an order. A workflow may be allowed to retrieve a product policy but not use it to approve a refund. A manager may see a comparison report without being the person authorized to finalize the supplier decision.

Treating all of these as one generic “access” rule creates risk.

A practical design separates:

what a role can retrieve;
what it can propose;
what it can execute;
what requires approval;
what must be logged for later review.

That keeps retrieval helpful without quietly extending authority.

Make quality observable

Knowledge systems need evaluation just as much as generation systems do.

For each important workflow, I want a small set of golden cases that describe the operating expectation:

What source should be retrieved?
What evidence must be visible in the answer or recommendation?
What response is forbidden?
When should the workflow escalate instead of answering?

Then I want the retrieval trace, the final output and the human review outcome in the same review surface.

Without that feedback loop, a team can improve the wording of an answer while missing the more important failure: the workflow retrieved the wrong policy in the first place.

Knowledge needs a maintenance loop

Operational knowledge is not finished when it is indexed.

Real work changes. A frontline correction exposes a missing SOP. A product exception reveals that a policy is unclear. A recurring handoff shows that the workflow cannot safely resolve a case with the sources it has.

Those signals should feed a controlled loop:

frontline gap → structured proposal → process owner → approver → versioned publication → reuse.

The point is not to let an AI rewrite policy on its own. The point is to make the gap visible, give the right owner a structured proposal, and preserve a versioned record of what changed.

The practical lesson

The useful question is not: “Where should we put our documents?”

It is: “What knowledge does this decision need, who owns it, how fresh must it be, and what happens when it is not enough?”

That is why operational knowledge is a product surface. It sits at the point where people, workflows and AI systems decide what is safe to do next.

The public prototype behind this approach is available on GitHub:

https://github.com/rkrisa/portfolio-ai-ops/tree/main/cases/operational-knowledge-retrieval-layer

If you operate a difficult knowledge workflow, I would be interested in the failure mode that keeps showing up.

What I log from AI workflows before I trust the automation

Ricardo Feldhaus Krisanoski — Sun, 16 Aug 2026 18:58:02 +0000

Most AI workflow posts focus on prompts, tools or output quality.

Those matter, but they are not the first thing I want to trust.

The first thing I want to trust is the audit trail.

If an AI workflow suggests a commercial follow-up, changes a queue, routes a conversation or asks for human review, I want to know:

what surface executed the decision;
what system would be affected;
whether human approval was required;
what context or evidence shaped the suggestion;
what it likely cost;
what happened after a human reviewed it.

That is why I started writing an observability contract for AI workflows before expanding the automation surface.

The Real Problem

In operational systems, the hardest bug is often not "the model answered badly."

It is "nobody can explain the path from context to action."

That creates a few predictable failures:

follow-up suggestions with no clean handoff reason;
invisible cost growth;
logs spread across workflow history, scripts and chat tools;
human edits that never become quality signals;
no reliable way to compare one workflow against another.

What I Log

I now prefer a small contract that every AI-assisted workflow can emit, whether it runs inside n8n or through a wrapper around an agent script.

At minimum, I want:

a stable workflow name;
a trace or execution ID;
the execution surface, like n8n or an agent wrapper;
the business unit and target system;
a risk level;
a flag for human approval;
the decision itself;
the outcome;
the model and estimated cost when applicable.

Then I add the fields that make review actually useful:

prompt or skill version;
retrieved sources or evidence references;
handoff reason;
human review outcome;
latency and token estimates.

That structure is much more useful than a raw dump of prompts and responses.

Why This Matters For Commercial Workflows

Imagine a workflow that suggests a commercial follow-up.

If the trace only says "AI replied" or "task created", the review loop is weak.

I want the system to answer higher-value questions:

Was this a suggestion or an action?
Which system would have been touched next?
Was approval required before any outbound step?
If it went to human review, why?
If the human edited it, was the edit approved, rejected or escalated?

That is the difference between using AI as a black box and using it as an operational component.

The Bridge Pattern

One practical detail mattered a lot in my implementation work:

I did not want workflow JSON carrying observability secrets everywhere.

So instead of wiring keys directly into each workflow, I prefer a small internal bridge:

workflow -> trace payload -> internal bridge -> observability backend

That keeps the trace contract shared while reducing secret sprawl.

It also makes it easier to instrument different surfaces with the same schema.

Human Review Is Part Of Observability

The most useful signal is often not the first model decision.

It is what happened after a person saw it.

Did they approve it?
Did they edit it?
Did they reject it?
Did they mark it as needing review?

Those outcomes turn observability into the start of an eval loop instead of a passive log archive.

Public-Safe By Design

For operational AI, public proof needs constraints.

So the public version of this work avoids:

raw customer messages;
phone numbers or candidate data;
private workflow URLs;
webhook paths;
internal secrets or auth headers;
unsupported metric claims.

Where a number is not approved, I keep it as metrics to collect.

The Practical Lesson

Before expanding an AI workflow, I want to make its decisions inspectable.

Not just its prompt.
Not just its output.
Its actual operational path.

That is why I now treat observability as part of workflow design:

trace the decision;
trace the risk;
trace the human review;
trace the cost;
keep the action path explainable.

Public case study:

https://github.com/rkrisa/portfolio-ai-ops/tree/main/cases/ai-workflow-observability-contract

AI Coding Agents Need a Decision Gate Before the Implementation Loop

Ricardo Feldhaus Krisanoski — Thu, 13 Aug 2026 13:00:00 +0000

AI-assisted development has changed the economics of implementation.

A plausible architecture can become a working prototype before the team has finished asking whether it solves the right problem.

That is useful — and dangerous.

The failure mode I kept seeing was not poor execution. It was premature commitment: a request arrived with a tool, repository, architecture, or output format already selected, and the implementation loop started before the decision had been challenged.

That is why I built Decision Path Challenger.

It is an open-source Agent Skill for OpenAI Codex and Claude Code. Its job is to create a decision gate before meaningful implementation work begins.

The solution is often baked into the request

Consider a request such as:

Build a weekly sales dashboard in Notion.

It sounds concrete. It also contains several decisions that may never have been made consciously:

the output must be a dashboard;
the dashboard must be interactive;
the work must live in Notion;
storage and delivery should happen in the same system;
someone will maintain that workspace every week.

The actual outcome may be much simpler:

Deliver a consistent weekly decision brief that managers can review, archive, and forward to the sales team.

Once the outcome is clear, the solution space changes. A collaborative dashboard remains possible, but so do a controlled HTML report, a PDF snapshot, a spreadsheet template, a concise automated briefing, or a hybrid workflow with human approval.

The point is not that Notion is wrong. The point is that the tool should not silently define the outcome.

Four layers before technology

Decision Path Challenger separates a request into four layers.

1. Outcome

What must become true for the user, team, or business?

This should be stated without inheriting the proposed tool. A weekly decision brief is an outcome. A Notion dashboard is a possible mechanism and technology choice.

2. Operating experience

How will the result be consumed, reviewed, shared, maintained, and repeated?

This layer surfaces practical questions that architecture discussions often miss. Does the result need to be editable? Must it be forwarded? Do historical copies matter? Who corrects errors? Is the user opening a workspace, receiving an alert, or approving a fixed artifact?

3. Mechanism

What is the simplest reliable way to produce that operating experience?

The mechanism may be a template, a script, a native feature, a scheduled workflow, a human review step, or a custom service. It should earn its complexity.

4. Technology

Which tools, repositories, models, APIs, and frameworks implement the chosen mechanism?

Technology comes last because it is easier to evaluate once the outcome and operating model are explicit.

Why exactly three questions

An effective decision gate cannot become another long intake form.

The skill asks exactly three high-value questions. Together they resolve three categories:

Success and consumption — what the result must enable and how people will use it.
Operating constraints — frequency, maintenance, acceptable manual work, control, cost, and reliability.
Hard boundaries and hidden flexibility — what is truly mandatory and what is merely inherited from the first proposed solution.

Three is a deliberate constraint. One question rarely exposes enough. Ten questions create friction and encourage generic answers. Three forces the skill to select the decisions that materially change the path.

If the answers already exist in the current context, the skill does not ask them again. It states the inferred answers and continues.

Compare solution classes, not cosmetic alternatives

Tool comparison often means evaluating three products from the same category. That is useful only after the category itself has been justified.

Decision Path Challenger compares materially different solution classes:

Native or no-code: built-in features, templates, exports, or scheduled functions in tools already in use.
Lightweight composition: HTML, spreadsheets, scripts, low-code workflows, serverless functions, or simple API orchestration.
Custom product: a bespoke application, agent, service, dashboard, or data pipeline.
Hybrid: native storage combined with custom rendering, or automation combined with human approval.
Process redesign: changing the workflow or output so that less technology is needed.

At least one option must challenge the original framing. Otherwise the comparison only optimizes a decision that has already been made.

Build cost is not total cost

Implementation speed is only one part of the decision.

The skill distinguishes four costs:

Build cost: what it takes to create the first working version.
Run cost: infrastructure, licenses, manual operations, monitoring, and recurring attention.
Change cost: how difficult it is to modify the system when the process evolves.
Failure cost: what happens when the workflow is wrong, unavailable, stale, or trusted too much.

It also considers reversibility. An uncertain workflow should not begin with a hard-to-exit architecture unless there is evidence that the extra commitment is justified.

This is especially important in AI-assisted development. Generating code quickly can make build cost look small while run, change, and failure costs remain hidden.

Possible is not the same as appropriate.

A decisive output

The skill is not designed to end with “it depends.”

It returns one of seven decision labels:

Proceed as proposed
Proceed with simplification
Adapt the proposed path
Use a hybrid
Run a small proof of concept first
Choose another path
Do not implement yet

Every recommendation includes the reason, the most tempting rejected alternative, what was intentionally excluded, and the smallest reversible next step.

For the recurring-report example, the decision may be Proceed with simplification: use a controlled HTML template rendered to PDF, populate it from existing sources, and preserve a human approval step before distribution.

That recommendation is not universally better than a dashboard. It wins only when consistency, portability, archival snapshots, and low weekly maintenance matter more than collaborative editing and interactive exploration.

Cross-agent by design

The canonical behavior lives in a platform-neutral SKILL.md.

The repository includes installation paths for both Codex and Claude Code, Codex-specific interface metadata, examples, evaluation references, a static validator, and a GitHub Actions workflow.

For Codex:

$decision-path-challenger

I found a repository that may help me orchestrate multiple coding agents.
Evaluate the approach before installing or modifying my project.

For Claude Code:

/decision-path-challenger

I want to build a weekly operations dashboard in Notion.
Challenge the implementation path before we start.

The skill does not replace project governance, security review, or human approval. It adds a structured pre-implementation review layer.

What the skill intentionally does not do

Decision Path Challenger does not make every task slower. It should not trigger for a small copy edit, a routine command, an emergency restoration, or a narrowly scoped bug with an already-defined fix.

It also does not treat complexity as a problem by itself. A custom product, agent, or integration can be the right answer. The requirement is that the extra burden is connected to a confirmed operating need rather than to excitement about the technology.

Finally, the skill does not make the decision on hidden assumptions. When a critical preference cannot be discovered from the environment, it asks the user. The goal is a decision-complete path, not autonomous certainty theater.

Try it on a real decision

Clone the repository:

git clone https://github.com/rkrisa/decision-path-challenger.git
cd decision-path-challenger

Install for Codex:

./scripts/install.sh codex

Install for Claude Code:

./scripts/install.sh claude

Or install for both:

./scripts/install.sh all

PowerShell users can run:

./scripts/install.ps1 -Target all

The repository is MIT licensed. Issues and contributions are welcome.

AI agents do not only need more implementation capability.

They also need better judgment about what deserves to be implemented, how the result will operate, and whether a simpler path would work better.

Decision Path Challenger is my attempt to make that judgment explicit, repeatable, and inspectable.

Try it on a real decision. If it changes or confirms your original path, I would like to hear what happened.

Why my first RAG layer starts in Postgres, not in a standalone vector database

Ricardo Feldhaus Krisanoski — Sat, 13 Jun 2026 12:30:37 +0000

When people say they are "adding RAG" to a workflow, the conversation often jumps too quickly to infrastructure choices.

Should this use a vector database?
Should there be a reranker?
Should everything go into a knowledge graph?

Those are valid questions, but they are usually not the first question.

The first question is narrower:

What approved knowledge should the workflow be allowed to retrieve before an AI decision happens?

That is why my first retrieval layer for operational AI workflows starts in Postgres, not in a standalone vector database.

The Workflow Problem

In operations-heavy systems, the model usually should not answer from raw memory or from a giant prompt dump.

The useful context already exists somewhere else:

approved response rules;
handoff criteria;
product or service notes;
source or campaign guidance;
operational decisions that were already made by humans.

The hard part is not generating fluent text.

The hard part is retrieving the right approved context, showing which source influenced the decision and refusing when no safe source exists.

Why Postgres First

For this kind of workflow, most of the surrounding data is already relational:

leads or conversations;
workflow names;
stages and owners;
human review outcomes;
source metadata;
trace logs;
document versions.

So the first technical choice is not "where do vectors live in the abstract?"

It is:

Where can I keep retrieval close to the operational data model?
Where can I log the retrieval path and the final decision together?
Where can I evolve the schema without creating a second system too early?

Postgres plus pgvector is a good first answer to that set of questions.

It lets me keep:

documents and chunks;
metadata such as allowed use and approval requirements;
retrieval traces;
cost estimates;
human review outcomes

in one place.

What The First Version Needs

The first version does not need to be broad.

It needs to be inspectable.

My narrow retrieval scope looks like this:

approved response rules;
product/service notes;
handoff and escalation criteria;
campaign/source guidance;
commercial playbooks.

Each retrieved chunk should carry more than text. It should also carry metadata such as:

source name;
document version;
business domain;
allowed use;
whether human approval is required.

That metadata matters because a workflow may be allowed to use one chunk as internal reasoning support, but not as customer-facing language.

The Eval Mindset

I do not think a retrieval layer is real until it has failure criteria.

So the public prototype includes a small golden-question set:

Does the expected source appear in the top results?
Does the workflow return no-answer or handoff when the source is missing?
Does customer-facing language come only from allowed chunks?
Can I review which chunks influenced the decision later?

That matters more than announcing that the system has embeddings.

Without retrieval checks, a RAG layer can look sophisticated while still pulling the wrong context.

Observability Is Part Of The Design

The retrieval step and the AI decision step should be traceable together.

I want the same review surface to show:

retrieved chunk IDs;
similarity or retrieval score;
model name;
token and cost estimates;
final decision;
handoff reason;
human review outcome.

That is the bridge between "the system answered" and "the system answered for a defensible reason."

When I Would Add More Infrastructure

I am not against standalone vector databases.

I would add one later if the system needed:

more search traffic;
more complex filtering boundaries;
separate deployment requirements;
recall/latency needs that justify the extra moving parts.

But before that point, I prefer a smaller stack that makes retrieval, evaluation and auditability visible.

The Practical Lesson

For AI workflows in revenue or operations contexts, the first retrieval layer should optimize for control, review and schema clarity.

Not for maximum architectural novelty.

That is why my first RAG layer starts in Postgres:

closer to operational data;
easier to trace;
easier to evaluate;
easier to keep human-in-the-loop.

The public case study is here:

https://github.com/rkrisa/portfolio-ai-ops/tree/main/cases/operational-knowledge-retrieval-layer

I added a context resolver before an AI sales agent replies

Ricardo Feldhaus Krisanoski — Thu, 28 May 2026 05:38:41 +0000

Most AI sales agents fail before the model writes a single word.

The failure is not always the prompt. It is usually the context.

In a real chat-commerce workflow, a lead can arrive with several competing signals:

the latest customer message;
CRM stage and owner;
previous conversation history;
campaign or source data;
product/category assumptions;
approved response rules;
handoff and support policies.

If all of that gets dumped into a prompt, the model may produce a fluent answer based on the wrong clue.

That is not an AI problem in the abstract. It is an operating-system problem.

The Problem

I was designing an AI reception agent for a chat-driven commerce operation.

The goal was simple: help the business reply faster and more consistently without losing the commercial context behind each lead.

But the first automated reply had a hidden risk.

A customer might send a short message like "hi" or "I want more information." On its own, that message is weak. The stronger signal may be the campaign, source, CRM stage, product page, previous conversation or approved sales rule.

If the AI agent receives all possible context at once, it still has to decide what matters.

That decision should not be left entirely to generation.

The Design Choice

I added a context-resolution step before the AI response.

Instead of asking the model to inspect every clue and improvise, the workflow first resolves a smaller object:

{
  "source_priority": "campaign_or_crm_or_message_or_fallback",
  "category": "resolved_commercial_category",
  "confidence": "high_or_medium_or_low",
  "selected_directive": "one_approved_response_rule"
}

The important part is not the exact schema. It is the order of decisions.

The system first decides which commercial context is most trustworthy. Only then does the AI agent write the reply.

The Architecture

The workflow looks like this:

Incoming chat lead.
CRM and conversation lookup.
Campaign or source-context lookup.
Context resolver.
Resolved commercial context.
AI response agent.
Structured response and routing decision.
Customer reply or human handoff.

The resolver is intentionally boring.

It is a control layer, not a creativity layer.

It exists to answer questions like:

What is the lead probably asking about?
Which source should win when signals conflict?
Is confidence high enough to answer directly?
Which approved sales rule should be used?
Should the system reply, ask a clarifying question or hand off to a human?

Why This Matters

In business workflows, "more context" is not always better.

More context can mean more ambiguity:

old messages compete with new ones;
generic playbooks compete with campaign-specific offers;
product assumptions compete with what the customer actually asked;
internal rules compete with customer-facing language.

The context resolver reduces that ambiguity before the model responds.

The AI layer becomes easier to debug because every reply can be traced back to a chosen context, confidence level and directive.

Guardrails

The workflow keeps several guardrails around the AI response:

low confidence triggers a consultative fallback;
sensitive commercial cases can be routed to human review;
approved response rules are selected before generation;
the agent receives a compact context package instead of a noisy dump;
the system logs what context was used.

This is not about making the AI sound more impressive.

It is about making the operational decision safer.

What I Would Measure Next

The current public version keeps metrics as metrics to collect, because I do not want to publish numbers that are not validated.

The useful metrics would be:

lead volume handled by the context resolver;
reduction in wrong-context replies;
human handoff rate by risk category;
response-time impact;
manual review time saved.

The Lesson

For AI agents in revenue workflows, the prompt is only one part of the system.

The harder design question is:

What should the model be allowed to know, trust and act on?

That is why I prefer designing AI agents as operating workflows: context resolution, retrieval, guardrails, structured outputs, human review and observability.

The public case study is here:

https://github.com/rkrisa/portfolio-ai-ops/tree/main/cases/context-aware-ai-reception-agent