A beginner's guide to the Dots-Ocr-With-Confidence model by Mind-Ware on Replicate

This is a simplified guide to an AI model called Dots-Ocr-With-Confidence maintained by Mind-Ware. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

dots-ocr-with-confidence is a multilingual document parser that unifies layout detection and text recognition into a single vision-language model, built by mind-ware. The model processes images and extracts text with per-word confidence scores, enabling you to identify uncertain recognitions. Unlike traditional OCR pipelines that separate layout analysis from character recognition, this model handles both tasks simultaneously using a 3B parameter vision-language model based on a 1.7B LLM backbone. The key distinction of this Replicate version is its return_confidence feature, which flags words below a configurable threshold (default 0.5) and returns individual confidence scores for each recognized word—critical for applications where recognition certainty matters as much as the text itself.

Best use cases

Document digitization with quality assurance. Convert scanned PDFs, invoices, or contracts to text while immediately identifying which words the model struggled to recognize. The confidence scores let you programmatically flag uncertain regions for human review rather than blindly trusting end-to-end results. This is valuable for legal document processing, financial record digitization, or archival work where errors carry high cost.

Multilingual text extraction from mixed-script documents. The model recognizes virtually any human script and handles documents with Chinese, Arabic, Devanagari, Cyrillic, and Latin text in the same image. Use this when processing international documents, travel permits, or scholarly papers with multiple languages on a single page. The unified approach avoids cascading errors from separate language-specific pipelines.

Structured graphics to code conversion. Beyond basic OCR, the model converts charts, diagrams, and visual graphics directly into semantic representations. Feed it a flowchart or organizational diagram and receive structured output suitable for downstream processing, rather than just raster images.

Layout-aware text extraction. The model detects reading order, multi-column layouts, headers, footers, and table structures automatically. This matters for document understanding systems where preserving spatial relationships and logical flow is required—for example, extracting Q&A pairs from documents or processing academic papers with complex formatting.

Scene text spotting and web content parsing. The model performs well on scene text (text in photographs) and web screenshots, making it suitable for accessibility tools, social media content analysis, or real-world image understanding pipelines where text appears in unpredictable locations and contexts.

Limitations

The model achieves approximately 79% accuracy on the olmOCR-bench standard, placing it below specialized models like Chandra OCR (83.1%) and roughly even with PaddleOCR-VL-1.5 (80.0%) on general benchmarks. This means approximately one error per fifty words on complex documents, which is acceptable for draft extraction but not suitable for error-critical applications without human review. On particularly challenging old scans and handwritten text, performance drops further (64.2% on "old scans math" category).

Confidence scores, while valuable for uncertainty quantification, are not calibrated to absolute reliability thresholds. A word marked as 0.6 confidence does not mean 60% probability of correctness—it reflects relative model uncertainty. The threshold filtering helps identify problematic regions but should not be treated as a statistical guarantee. The model was not evaluated on mathematical formula recognition or chemical structure parsing at scale, limiting its utility for scientific document processing.

Input images must be provided as URIs (not raw base64 in the current schema), and the maximum token output is 32,768 tokens. For extremely long documents, you may hit the context limit. The model inference runs on Replicate's infrastructure with no guarantee of execution time—expect variable latency depending on queue load. Temperature sampling is disabled when return_confidence=True, which constrains output diversity but is necessary for stable confidence calibration.

The model is 3B parameters, requiring substantial compute for local deployment. The Replicate version abstracts this away, but if you self-host, plan for several gigabytes of VRAM. License details should be verified against the GitHub repository, though the model appears to be available for research use.

How it compares

dots.ocr (HuggingFace) is the base model from the same authors. The Replicate version adds confidence scoring and threshold filtering as dedicated features, while the HuggingFace version requires manual confidence extraction if needed. Choose the Replicate version if you need out-of-the-box confidence reporting; choose HuggingFace if you want to self-host without API costs. Both share the same 79% olmOCR-bench accuracy baseline.

PaddleOCR-VL-1.5 is significantly smaller (0.9B vs. 3B) and achieves 80% olmOCR-bench accuracy with lower latency. It is production-hardened and widely deployed in Chinese tech companies. Use PaddleOCR-VL-1.5 if speed and model size matter more than extracting the last percent of accuracy; use dots-ocr-with-confidence if you need confidence-aware processing and slightly stronger performance on specialized benchmarks (1086.2 Elo vs. 920.5 Elo on OmniDocBench/XDocParse averages).

dots.mocr is the latest evolution from the same team, achieving 83.9% olmOCR-bench accuracy and state-of-the-art Elo scores. However, dots-ocr-with-confidence on Replicate may reference an earlier model version. Check the Replicate version timestamp (January 2026) against the latest HuggingFace dots.mocr release—if you need the newest performance gains, the HuggingFace version may be newer.

ocr (datalab-to) is a multi-task Replicate model supporting 90 languages with bounding box output and reading order analysis. It is more general-purpose and may be optimized for raw speed. Use dots-ocr-with-confidence if you need per-word confidence; use the datalab-to model if you need bounding box coordinates without confidence metadata.

Technical specifications

The model is a 3-billion parameter vision-language model with a 1.7B LLM backbone, trained for unified document parsing. It processes images of arbitrary resolution (no explicit maximum stated) and generates text with optional per-word confidence scores. The model supports multiple prompt modes for different extraction objectives: layout_all (default, full layout and text), grounding_ocr (text within a specific bounding box), and custom (user-defined instructions).

Key technical parameters include:

• Architecture: Vision-language model with unified layout + text backbone
• Parameter count: 3B total (1.7B LLM)
• Input format: Image URI (JPEG, PNG, or other common formats)
• Output format: JSON with text string, optional per-word confidence objects, aggregated statistics
• Maximum tokens: 32,768 (configurable via max_tokens, default 8,192, minimum 256)
• Temperature: 0.1 default (range 0–2; disabled when confidence scoring is enabled)
• Top-p sampling: 0.9 default (range 0–1)
• Confidence threshold: 0.5 default (range 0–1), flags words below this as low-confidence
• Bounding box input: Optional 'x1,y1,x2,y2' format for region-specific extraction
• Prompt modes: layout_all, grounding_ocr, custom
• Training dataset: Trained on multilingual document parsing benchmarks including olmOCR-bench, OmniDocBench, and XDocParse
• Benchmark performance: 1086.2 Elo average (OmniDocBench + olmOCR + XDocParse), 79.1% olmOCR-bench accuracy

Model inputs and outputs

Inputs

• image (string, URI, required): Input image for OCR; must be a valid image URL
• prompt_mode (enum, default layout_all): Controls extraction behavior; options include layout_all (full layout + text), grounding_ocr (text within bounding box), custom (custom prompt)
• custom_prompt (string, default empty): Custom instruction text when prompt_mode='custom'
• bbox (string, default empty): Bounding box in format 'x1,y1,x2,y2' for region-specific extraction (used with prompt_mode='grounding_ocr')
• max_tokens (integer, default 8,192): Maximum output tokens; range 256–32,768
• return_confidence (boolean, default false): If true, returns per-word confidence scores and disables temperature sampling
• confidence_threshold (number, default 0.5): Threshold for flagging low-confidence words; range 0–1
• temperature (number, default 0.1): Sampling temperature for text generation (ignored when return_confidence=True); range 0–2
• top_p (number, default 0.9): Nucleus sampling parameter; range 0–1

Outputs

• text (string): Extracted text from the image
• prompt_mode (string, nullable): Echo of the prompt mode used
• word_count (integer, nullable): Total number of words extracted
• bbox_used (array of integers, nullable): Bounding box coordinates if region-specific extraction was performed
• avg_confidence (number, nullable): Average confidence score across all words (only when return_confidence=True)
• word_confidences (array of objects, nullable): Per-word confidence data; each object maps word text to confidence score
• low_confidence_count (integer, nullable): Count of words below the confidence_threshold (only when return_confidence=True)

Getting started

import replicate

client = replicate.Replicate()

result = client.run(
    "mind-ware/dots-ocr-with-confidence:91ce60f4885d7ca6e095755e25d0f9ff2bcfe963c816937ece4be50d811f26c4",
    input={
        "image": "https://example.com/document.jpg",
        "prompt_mode": "layout_all",
        "return_confidence": True,
        "confidence_threshold": 0.5,
        "max_tokens": 8192
    }
)

print(result["text"])
print(f"Average confidence: {result.get('avg_confidence', 'N/A')}")
print(f"Low confidence words: {result.get('low_confidence_count', 0)}")

if result.get("word_confidences"):
    for word_data in result["word_confidences"]:
        print(word_data)

Frequently asked questions

Q: Should I enable return_confidence for production use?

A: Yes, if uncertainty quantification matters. Enabling it disables temperature sampling (fixing it at 0.1), which produces more deterministic output and provides actionable confidence metadata for downstream validation. Without it, you have no signal for which words are unreliable.

Q: Can I use this model to extract text from a specific region of an image?

A: Yes, use prompt_mode='grounding_ocr' and set the bbox parameter to 'x1,y1,x2,y2' coordinates. This constrains extraction to that bounding box region, useful for focusing on specific fields in forms or documents.

Q: How does the confidence threshold actually work?

A: Words with model confidence scores below the confidence_threshold (default 0.5) are counted in low_confidence_count and marked in the word_confidences array. This is a filtering signal, not a statistical guarantee—use it to identify candidates for human review, not as a probability bound.

Q: What is the difference between this model and the base dots.ocr on HuggingFace?

A: This Replicate version is purpose-built with the return_confidence feature and confidence threshold filtering as first-class outputs. The HuggingFace version requires custom code to extract confidence scores. Both use the same core model, but the Replicate wrapper adds API-level support for uncertainty reporting.

Q: Is this model still maintained?

A: The latest version was deployed January 9, 2026. The parent team has released newer variants (dots.mocr) on HuggingFace, but the Replicate version remains available. Check the GitHub repository for the latest research updates and model variants.

Q: What image formats are supported?

A: The schema specifies URI format, meaning you must provide a URL-accessible image. Common formats (JPEG, PNG, WebP) are supported via standard image libraries. Local files must be uploaded to an accessible server first.

Q: How does this compare to general-purpose vision models for OCR?

A: This is specialized for document parsing, not general vision understanding. It achieves ~1086 Elo on document benchmarks but is optimized for structured text extraction, not image captioning or visual question-answering. Use it when OCR and layout analysis are the primary task.

Q: What happens if I hit the 32,768 token limit on a long document?

A: The model will truncate output at that limit. For very long documents, consider splitting them into pages and running multiple API calls, then stitching results together using the detected reading order.

Click here to read the full guide to Dots-Ocr-With-Confidence