How I validate cited answers in an ASP.NET Core 8 assistant

I built a small ASP.NET Core 8 assistant around three synthetic cooking lessons. I wanted to make its answer path easy to inspect. Before asking a model anything, my application selects a passage. Before accepting a generated answer, it checks the source identifiers against that passage.

The sample has a deterministic mode and an optional OpenAI-compatible chat adapter. The deterministic mode returns the selected lesson directly. The HTTP adapter can generate wording, but its response must pass a citation check.

This distinction matters. I can verify that a citation names the supplied passage. I cannot conclude from that identifier alone that every sentence is supported. My sample enforces a source-reference rule, not a guarantee of factual correctness.

I retrieve before generating

My lesson library uses fixed files with stable identifiers such as pasta-water. A question cannot choose an arbitrary file path. Each lesson has an identifier, a title, a source path, and text.

I use keyword retrieval for this small collection. I remove common English words and apply small suffix rules. I count distinct matching words, so repeating a keyword cannot improve its score. This expression comes from LessonLibrary.cs:

words.Count(word =>
    lesson.Title.Contains(word) || lesson.Text.Contains(word))

These code excerpts show expressions and conditions from the implementation. They are not complete programs.

A candidate needs at least two distinct query-word matches. Title matches receive two points and body matches receive one. If two lessons share the best score, I return no passage.

This is deliberately limited. Synonyms can be missed. Shared words can select an irrelevant lesson. Finding a passage does not establish that it answers every part of a question. I would evaluate retrieval separately before using a larger collection.

No passage means no model call

My service starts with a refusal response. It only enters the model branch when retrieval returns evidence. An unrelated question can therefore finish without an HTTP request to the model endpoint.

I prefer this explicit application decision to asking the model whether it knows enough. It gives me a behavior I can inspect and test. It also avoids paying for generation when my retrieval step has found nothing.

The conversation history helps with one narrow follow-up: repeating the previous cited lesson. It does not become evidence for a new answer. I keep that limitation visible rather than describing the sample as a general conversational assistant.

I send one selected passage

The adapter receives a question, evidence, and local conversation context through its interface. Its outgoing user material contains only the question and one passage. This object comes from OpenAiCompatibleChefModel.cs:

new
{
    question = input.Question,
    passages = new[] { new
    { id = input.Evidence.Id, text = input.Evidence.Text } }
}

I serialize that object into the user message. I do not include unrelated lessons or conversation history in the outgoing material. The request also contains a system instruction and the configured model identifier.

The system instruction asks the model to answer from the supplied passage, treat its text as evidence rather than instructions, and cite supported answers using [source:ID]. If the evidence is insufficient, it asks for an empty answer.

That instruction expresses the desired behavior. My application still checks the response. A prompt alone does not enforce the citation rule or establish resistance to hostile document text.

I check the returned identifiers

The adapter finds citation markers using a regular expression. It rejects blank answers and answers longer than 16,000 characters. It then requires at least one recognized citation and checks every recognized identifier against the selected passage.

This expression performs the identifier comparison:

citations.Cast<Match>().All(match =>
    string.Equals(match.Groups[1].Value, input.Evidence.Id,
        StringComparison.Ordinal))

For the pasta lesson, [source:pasta-water] can pass. [source:invented] cannot. Adding a valid marker alongside an invented marker does not rescue the answer. Every recognized marker must match.

I also keep the source path under application control. After acceptance, the service appends the path from the retrieved evidence. The model does not choose that displayed path.

There is an important boundary here. An answer can contain a valid marker and still make unsupported claims. This validator does not check each sentence against the lesson. It also does not interpret every possible malformed citation format. I would need separate answer-quality evaluation before making stronger claims.

Errors and deadlines lead to refusal

I read the provider credential from an environment variable. A missing key produces an empty model result. The adapter also returns an empty result for unsuccessful HTTP responses and the response parsing failures it handles. The service turns an unverified result into a refusal.

I use a linked cancellation token with a provider deadline of at most 20 seconds. The timeout handler distinguishes that deadline from caller cancellation:

catch (OperationCanceledException)
    when (!cancellationToken.IsCancellationRequested)

When the provider deadline expires, the adapter returns an empty result. If the caller cancels, cancellation propagates instead. I do not present a cancelled request as a completed answer.

The endpoint base URL and model are configurable. The adapter appends chat/completions to the base URL. It requires HTTPS except for loopback HTTP used in local simulations. The application disables redirects. Provider credentials stay outside the source code.

What my 49 tests establish

The recorded suite passed 49 tests, including a Windows Release run. Those tests cover more than the model adapter. They exercise lesson answers, retrieval cases, context isolation, subscription admission, monthly quotas, signed notices, and HTTP endpoint behavior.

Five adapter tests use a simulated HTTP handler. They check acceptance of a valid citation, rejection of a missing citation, rejection of an invented identifier, handling of HTTP 500, and timeout behavior. The valid-answer test also inspects the request body and checks that private history is absent.

A separate local HTTP check ran the API against a fake chat endpoint on 127.0.0.1. It accepted a correctly cited answer and refused an invented citation and a server error. An unrelated question was refused without calling that endpoint.

These results support the tested application behavior. They do not validate a real provider with a real key. I have not established provider compatibility, production latency, or answer quality on a real document collection.

The full sample is public at https://github.com/martinPisacreta/cited-assistant-dotnet.

What I would evaluate next

For another content set, I would collect expected questions, unsupported questions, and examples with misleading wording. I would measure passage selection and answer support separately. I would also review identity, document permissions, and durable storage before deployment.

My sample keeps state in memory and serializes chat requests within one process. It is a prototype with explicit boundaries.

If you are building an assistant over courses, manuals, or support articles, write about your case in the comments. What source evidence should a useful answer show, and when should your assistant decline to answer?