Short answer: normalize orientation first, crop only after the receipt bounds are known, and run metadata inspection on that clean derivative while retaining the untouched mobile photo for audit and reprocessing.
That order protects the thing an expense app actually promises: a readable receipt record tied to the photo a user submitted. A sideways JPEG can still have perfectly valid pixels. A tight crop can still remove the tax line. Treat both as data-quality decisions, not cosmetic filters.
Keep it boring.
What should a receipt capture pipeline decide before it touches pixels?
Start with the user-visible result. For a scanned receipt expense app, the result is usually a reviewable image plus extracted fields, not a single compressed bitmap. Write that contract down: the reviewer must be able to see the merchant, date, total, currency, and the edges that prove the document was not clipped. Quality beats a few kilobytes when the alternative is a rejected expense.
The source upload gets its own immutable identifier. Every rotation, crop, and processed output gets a new identifier and a pointer to its parent. That small bit of bookkeeping makes a correction safe: you can regenerate a derivative without overwriting evidence, and a support engineer can compare what the phone sent with what the extractor received.
Mobile metadata is useful but untrusted input. EXIF orientation may describe how a camera expects a viewer to rotate the pixels; some ingestion paths strip EXIF while preserving the original pixel matrix. Read the metadata, record what you found, and make the displayed orientation explicit in the derivative. Do not rely on a browser or an OCR library to agree about an implicit flag.
I first assumed a portrait upload's width and height were already its visual dimensions. The crop coordinates looked reasonable in logs, yet the preview cut off the merchant name because the orientation flag had not been applied. Then a second sample arrived with the same pixels and no EXIF at all, so the original assumption failed twice in different ways. The fix was procedural: inspect, rotate, then calculate bounds from the rotated dimensions. I now keep those cases beside a landscape receipt in the fixture set and make the test fail loudly instead of trusting a pretty preview.
That was my mistake.
How do metadata inspection, rotation, and crop choices work for mobile photos?
Use a two-track model. The safety track validates MIME type, byte limits, and decodability. The composition track applies orientation and framing. A crop must never decide whether the source is acceptable; otherwise a bad crop can hide a lifecycle problem. Persist both decisions with the asset record.
For known receipt bounds, an explicit crop is easier to test. The client or a reviewer supplies a rectangle in the normalized coordinate system, and the service checks that rectangle against the rotated width and height. Smart cropping is a different tool: it estimates a focal region when no trusted rectangle exists. It can help unattended capture, but it needs a visual acceptance set because a receipt has important content at its edges.
Bandwidth changes the trade-off. Keep the original at capture quality for audit, then produce a review derivative at a bounded size and quality. Do not repeatedly resize the same derivative for every screen; generate named variants from the source or from one canonical normalized image. A small thumbnail is fine for a list. It is not a suitable OCR input.
The following Go sketch keeps the pipeline explicit. The endpoint names are deliberately limited to the three operations in this workflow; the storage layer and authentication are left to the service that hosts them.
package receipt
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
)
type CropBox struct {
Left int `json:"left"`
Top int `json:"top"`
Width int `json:"width"`
Height int `json:"height"`
}
type imageRequest struct {
AssetID string `json:"asset_id"`
Crop *CropBox `json:"crop,omitempty"`
}
// transform calls one operation at a time so each derivative can be audited.
func transform(ctx context.Context, baseURL, route string, req imageRequest) (string, error) {
body, err := json.Marshal(req)
if err != nil {
return "", err
}
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+route, bytes.NewReader(body))
if err != nil {
return "", err
}
httpReq.Header.Set("content-type", "application/json")
resp, err := http.DefaultClient.Do(httpReq)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("image operation %s returned HTTP %d", route, resp.StatusCode)
}
var out struct {
AssetID string `json:"asset_id"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return "", err
}
if out.AssetID == "" {
return "", fmt.Errorf("image operation %s returned no derivative id", route)
}
return out.AssetID, nil
}
func prepareReceipt(ctx context.Context, baseURL, sourceID string, bounds *CropBox) (string, error) {
orientedID, err := transform(ctx, baseURL, "/v1/image/rotate", imageRequest{AssetID: sourceID})
if err != nil {
return "", err
}
framedID, err := transform(ctx, baseURL, "/v1/image/crop", imageRequest{AssetID: orientedID, Crop: bounds})
if err != nil {
return "", err
}
return transform(ctx, baseURL, "/v1/image/process", imageRequest{AssetID: framedID})
}
The caller should pass bounds == nil only when the processing service has a documented default framing rule. For a user-drawn rectangle, validate that Left, Top, Width, and Height are positive and stay inside the oriented image before making the crop request. Store the source ID, oriented ID, framed ID, and processed ID in one record; the IDs are more useful during an incident than a filename guessed from a timestamp.
How can quality and bandwidth be verified without guessing?
Build a fixture set from real capture conditions: glare, folded corners, long thermal paper, landscape phones, and receipts photographed against dark tables. Include at least one image with EXIF orientation and one with the same pixels after metadata stripping. For each fixture, assert orientation, pixel dimensions, crop containment, and whether all required fields remain visible.
A simple review table keeps the decision honest:
| Choice | Works well when | Failure signal | Safer response |
|---|---|---|---|
| Keep original | Audit or later reprocessing matters | Storage growth | Apply retention rules to derivatives separately |
| Explicit crop | Bounds are known from a user or detector | Edge text disappears | Reject the box or request a wider one |
| Smart crop | Capture is unattended and composition varies | Focal estimate clips totals | Fall back to a tested explicit box |
| Small review derivative | Network and preview latency matter | OCR confidence drops | Send the larger normalized derivative to extraction |
Measure bytes and field-level extraction accuracy together. A bandwidth win that lowers total-field recall is not a win for an expense workflow. Your mileage will vary with camera hardware and receipt stock, and I am not sure a single threshold can cover every merchant; define the acceptance threshold with the finance reviewers who approve the record. For one fixture, record the original byte count, the review derivative byte count, and the fields a reviewer can still read; repeat that comparison after every encoder or crop-policy change. A dashboard that shows only average bytes hides the one receipt that matters during reimbursement.
Keep observability attached to each derivative: source ID, operation, input dimensions, output dimensions, byte size, orientation value, and elapsed time. Log hashes or IDs, not the receipt image itself. When a user reports a missing total, those fields tell you whether the issue began in capture, geometry, or extraction.
What are the rollback and lifecycle rules for a production receipt app?
Make each stage retryable with an idempotency key derived from the source ID and operation version. A retry should find the existing derivative or create the same logical result, never replace the original. Mark a derivative as ready only after it passes decode and dimension checks; keep failed attempts out of the user-facing picker.
The catch is retention. Keeping every intermediate forever increases privacy exposure and storage obligations. Set a documented retention window for temporary rotated and cropped assets, while retaining the original and the final record for the period required by your audit policy. Delete by identifier, and record the deletion event.
This design is not suitable when you need a full document-management system with annotations, signatures, or legal holds; use a system built for those controls and keep the image pipeline as a bounded pre-processing step. Stick with a simpler on-device flow when receipts never leave the phone and offline review is the primary requirement.
Before rollout, replay the fixture set, compare derivative IDs, and verify that a failed crop leaves the source retrievable. If a new crop policy reduces field recall, roll back the policy version and regenerate derivatives from the original. That is the operational advantage of preserving the source: recovery is a controlled job, not a request for the user to photograph the receipt again.
Top comments (0)