On most production lines, a camera can see every part, most parts are fine, and the expensive thing is a trained inspector's attention. Inspectors don't need a replacement. They need an extra set of eyes.
So I built LineSight, a demo that takes a photo or a short video clip of a manufactured part and returns three things: a PASS / REVIEW / FAIL recommendation, the visible defects with severity and location, and the most relevant entries from a library of inspection guidance. A human inspector always makes the final call. The AI's job is to focus their attention.
The constraint I gave myself: the whole stack had to have zero idle cost. No always-on servers, no managed vector database burning money overnight. Everything bills per request or not at all.
From finding faces to finding scratches
This isn't my first time pointing code at images. Years ago I worked at HP, which owned the online photo service Snapfish at the time. I was brought onto a team building image processing technologies. We started with open source libraries like OpenCV: looking at images, running interesting conversions, and pulling individual frames out of video to examine them. Then came detection: finding a face in an image, or finding the area of interest and cropping to it.
The hard part was never the code. It was the training data. To teach an algorithm what a face is, I had to break "face" down into the features that make one: eyes, eyebrows, a nose, a mouth, and where they sit relative to each other. Then I had to feed it about 100,000 sample images, some with faces and some without, so it could learn which patterns meant "face" and which were just shadows or background. Every miss meant more samples and another training run.
That toolkit is all over manufacturing, for good reason. OpenCV is fast, free to run, and deterministic. It can find a part on a conveyor, confirm a label is where it belongs, measure a hole, or flag a frame that differs from a known-good reference.
Where it struggles is judgment. A classical detector answers the question you tuned it for. Ask whether a mark is a scratch or a reflection, how bad it is, or what the inspector should do about it, and you're building a new model or a new pile of thresholds. Every new part, defect type, or lighting change means more tuning.
That's what changed most for me. Face detection took 100,000 images. LineSight was never shown a single training image. What it knows about these parts comes from a prompt and about 20 text records in the reference library. When the bike frame came in, there was no weld detector to build or retrain. The model judged the welds on its own, and the low similarity score showed the library had nothing to back it up.
A multimodal model flips that trade-off:
| Classical CV (OpenCV) | Multimodal model (Claude on Bedrock) | |
|---|---|---|
| Best at | Finding and measuring known things: faces, parts, regions, edges | Judging what it sees and explaining why |
| Training data | A large labelled image set per thing you detect | None; a prompt and ~20 reference records |
| New defect type | More samples, new detector or thresholds | A prompt change and new reference records |
| Output | Boxes, masks, numbers | Findings with severity, location, and evidence |
| Speed | Milliseconds per frame | Seconds per inspection (8 to 11 here) |
| Cost | Effectively zero once deployed | About a cent per image on Opus 5.5 |
So this isn't one replacing the other. LineSight still uses OpenCV to pull frames from video. The old toolkit does the fast, mechanical work, and the model does the part that used to need a person.
The full source (CDK stack, runtime, Streamlit app, and the synthetic reference library) is on GitHub: scooteronce/linesight-quality-inspection
Before you start
-
An AWS account in
us-west-2, with model access for Anthropic Claude and Titan Text Embeddings V2 enabled in the Bedrock console. - Tooling: Node.js 20+, npm, the AWS CDK, and Python 3.10+. Packaging the runtime uses Python 3.13 to build Linux ARM64 wheels for AgentCore.
- Cost: nothing while idle, about a cent per single-image inspection on Opus 5.5. The full breakdown is in "What it costs" below. Run the clean up at the end when you're done.
-
Setup and deploy steps are in the repo README: precompute the reference embeddings,
npx cdk deploy, thenstreamlit run web/app.py.
The architecture
Four pieces:
- Claude Opus 5.5 on Amazon Bedrock does the actual seeing. It receives the image frames and a prompt describing the inspection task, and returns structured findings.
- Amazon Titan Text Embeddings V2 turns the model's defect description into a vector so it can be matched against a reference library of defect examples and disposition guidance.
- Amazon Bedrock AgentCore Runtime hosts the inspection workflow as a serverless agent. It bills per CPU/memory-second while a request runs, and nothing while idle.
- A local Streamlit app is the inspector's review screen: upload, inspect, see the recommendation with the supporting evidence and references.
An inspection flows like this:
- The app resizes the upload and invokes the AgentCore runtime with base64 frames.
- The runtime calls Claude for structured findings.
- It embeds the defect summary with Titan and ranks it against the reference library by cosine similarity.
- Everything returns to the UI in one round trip, about 8 to 11 seconds end to end.
Decision 1: Structured outputs instead of "please return JSON"
The classic failure mode of LLM pipelines is parsing. The model wraps its JSON in prose, or invents an enum value, and your pipeline falls over at 2 a.m.
The Claude API's structured outputs feature eliminates the whole category. You attach a JSON schema to the request, and the response is guaranteed to conform: valid JSON, correct types, enums enforced.
response = claude.messages.create(
model=VISION_MODEL_ID,
max_tokens=2048,
system="You are a cautious visual quality inspection assistant.",
messages=[{"role": "user", "content": content}], # image frames + prompt
output_config={"format": {"type": "json_schema", "schema": INSPECTION_SCHEMA}},
)
The schema constrains inspection_status to PASS | REVIEW | FAIL, severities to low | medium | high, and requires evidence and a location for every defect. There is no JSON parsing code in the project beyond json.loads. No regex, no "find the first brace," no retry-on-malformed loop.
A related prompt-design point: the system is deliberately asymmetric. The prompt instructs the model to use FAIL only for clearly visible material defects, and REVIEW whenever the image is blurry, obstructed, or ambiguous. In quality inspection a false PASS is the expensive failure, so uncertainty routes to a human by design.
Here's the prompt that does it:
Inspect this manufactured part for visible surface or assembly defects.
Report inspection_status, confidence, the visible defects, and a one-sentence summary.
Use FAIL only for a clearly visible, material defect. Use REVIEW when the image is
blurry, poorly lit, obstructed, unfamiliar, or the evidence is uncertain. Use PASS
when no visible defect is apparent. Do not invent measurements or claim certainty
that the image does not support. Describe each defect location in plain language;
do not invent pixel coordinates or bounding boxes.
The same rule applies to failures. If the model declines to analyze an image, the runtime returns REVIEW with low confidence instead of an error, so the part still lands in front of a person.
Decision 2: RAG without a vector database
The retrieval layer matches each finding against a library of reference records: defect examples ("deep groove with substrate exposure is high severity; hold for review") and process guidance ("reflections can resemble scratches; use diffuse lighting").
The standard recipe is OpenSearch Serverless, and that's where the zero-idle-cost constraint bites. OpenSearch Serverless bills for minimum capacity continuously, roughly $175+/month whether you query it or not. For a reference library of a few dozen records, that's absurd.
Instead, embeddings are precomputed once at build time and shipped inside the runtime bundle as a JSON file. At inspection time, the runtime embeds only the model's one-sentence defect description (a Titan call costing a few millionths of a dollar) and ranks the library in-process:
def _cosine_similarity(a, b):
dot = sum(x * y for x, y in zip(a, b))
norm = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
return dot / norm if norm else 0.0
Pure Python, no numpy, no index, no network hop. For ~20 records of 1,024 dimensions this runs in milliseconds.
It's also honest about its limits. When I fed the system a photo of a badly welded bike frame, the top reference came back at 0.338 similarity. The library has no welding records, and the low score says so instead of pretending. When a real defect library outgrows this, the upgrade path is a managed store. That's a scale decision, not a starting requirement.
Decision 3: Video via frame sampling
Claude doesn't ingest video directly, but it doesn't need to. The app extracts evenly spaced frames from an uploaded clip (OpenCV, sampling segment midpoints so the first and last moments are represented), resizes them, and sends them as one batch.
The runtime interleaves a "Frame N:" label before each image and extends the prompt: report each distinct defect once, name the frame where it's most visible, and use agreement across frames as evidence. A mark visible from several angles is stronger evidence than something that appears in exactly one frame, which is probably glare.
That last instruction matters. Multi-frame input isn't just "more pictures." It gives the model a cheap consistency check that a single photo can't provide. The schema gains a required frame field per defect, so the UI can show the inspector exactly which sampled frame to look at.
Cost scales with frames sampled, not clip length: an 8-frame inspection costs roughly 8x a single image.
In the review app you pick 2 to 12 frames per clip. The runtime accepts up to 20 frames and 5 MB per request, and the app lowers JPEG quality until the batch fits.
Decision 4: The Anthropic SDK on Bedrock (and the IAM action nobody tells you about)
The runtime calls Claude through the official Anthropic SDK's Bedrock client rather than hand-rolling boto3 request bodies. You get the Messages API surface, typed errors, and features like structured outputs with no translation layer:
from anthropic import AnthropicBedrockMantle
claude = AnthropicBedrockMantle(aws_region=REGION)
Here's the gotcha that cost me a debugging session: this endpoint does not authorize against the classic bedrock:InvokeModel action. The runtime's first invocation failed with a 403 naming an action I'd never granted:
not authorized to perform: bedrock-mantle:CreateInference
on resource: arn:aws:bedrock-mantle:us-west-2:<account>:project/default
The fix is one extra IAM statement alongside your usual Bedrock grants:
new iam.PolicyStatement({
actions: ["bedrock-mantle:CreateInference"],
resources: [`arn:aws:bedrock-mantle:${region}:${account}:project/default`],
})
I scoped the model grants with a wildcard (anthropic.claude-* foundation models plus the us.anthropic.claude-* cross-region inference profiles), so swapping models is an environment-variable change, not an IAM change.
That matters for cost tuning. Switching VISION_MODEL_ID to Claude Haiku 5.5 makes each inspection about 40x cheaper per token (on prompts under 100K tokens, which an inspection always is). For a bounded task with a fixed schema, that's worth A/B testing against your own parts.
What it costs
Deploy it, use it twice a week, and the bill is two inspections.
| Component | Cost |
|---|---|
| AgentCore Runtime | Consumption-based, no idle charge. A few seconds of compute per inspection, well under a cent. |
| Claude Opus 5.5 | About a cent per single-image inspection ($4/M input, $20/M output on the first-party API; Bedrock rates on the AWS pricing page). An 8-frame clip: a few cents. |
| Titan embeddings | Rounding error. |
| Idle | Zero. |
New AWS accounts come with free-tier credits that cover thousands of inspections.
The honest caveat: this cost profile is a property of demo-scale choices. A production system reintroduces always-on baseline: a managed vector store once the reference library grows, a hosted authenticated review app instead of local Streamlit, VPC networking, image and audit-trail storage. The inspection workflow and the Bedrock calls stay identical. It's the infrastructure around them that starts the meter.
Pointing it at a real line
The repo inspects one upload at a time. Here's how I'd point the same pieces at a line's output. This part is a design, not something the repo does yet.
- Trigger with OpenCV. A camera at the end of the line runs continuously. Cheap frame differencing or a region-of-interest check detects when a part is in position, so frames are only captured when there's something to inspect.
- Crop before you send. Crop each frame to the part. Smaller images mean fewer tokens and less background for the model to misread.
- Capture a short burst. Grab several frames as the part moves or turns, and send them as one inspection. Cross-frame agreement is what separates a real scratch from glare.
- Route by result. PASS gets logged. REVIEW goes to an inspector's queue with the flagged frame. FAIL holds the part until a person signs off.
- Keep the evidence. Store the frames, the findings, and the inspector's final call. Confirmed defects become new reference records, so retrieval gets better at your parts over time.
At about a cent per single-image inspection on Opus 5.5, 1,000 parts a shift is roughly $10. An 8-frame burst multiplies that by about 8. If that's too much, there are two levers: inspect a sample instead of every part, or switch to Haiku 5.5. Test both against your own parts before trusting either.
Latency is the other limit. At 8 to 11 seconds per inspection, this works for sampled audits and end-of-line holds. It's too slow to gate a fast line part by part, and that's where classical CV still wins.
Clean up
Idle cost is zero, but delete the stack when you're finished so no IAM roles or runtimes linger in your account:
npx cdk destroy
That removes the AgentCore runtime and its role. No images are stored, so there's nothing else to empty. Leave the CDK bootstrap resources alone unless nothing else in the account uses them.
Wrapping up
- Structured outputs should be your default for any LLM pipeline with a downstream consumer. Schema-guaranteed responses delete an entire class of production failure.
- RAG does not automatically mean vector database. Below a few thousand records, precomputed embeddings and in-process cosine similarity are simpler, faster, and free.
- Video is frame sampling plus prompt design. Cross-frame agreement is a real signal, so tell the model to use it.
- Bias uncertain cases toward human review. In inspection, the asymmetry of error costs should be designed into the prompt, not discovered in production.
If you're putting vision models to work on the factory floor, I'd like to hear how you handle the human-review handoff. Drop a comment, or follow for more posts on AWS and applied AI.
Top comments (1)
Really like the explicit REVIEW path—on a line, false PASS and false FAIL have very different costs, so a single accuracy number can hide the risk. For a pilot, I’d log each model recommendation beside the inspector’s disposition, part or lot, lighting/setup, and whether the flagged evidence was actionable. Then track false-pass rate separately from review burden, and re-test after a camera, material, or lighting change. The suggested 2–12-frame burst is useful to explore, but I’d validate frame count against cycle time and the defect types you care about before calling it line-ready. How are you thinking about that threshold?