This is a simplified guide to an AI model called Grounding-Dino maintained by Hautechai. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
grounding-dino is a zero-shot, text-prompted object detector based on Grounding DINO with a SwinT-OGC backbone. The Replicate version is maintained by hautechai and runs as an H100 build. You provide an image and a text query containing object names; the model returns detected regions and can render bounding boxes on the source image. The key point before adoption is that this is an open-set detector, not a general image-understanding or segmentation model: it can localize objects described by language without task-specific retraining, but its results depend on prompt wording, thresholds, image content, and tokenization. The underlying project reports 52.5 AP on zero-shot COCO without COCO training data and 63.0 AP after COCO fine-tuning. The supplied materials do not state parameter count, image-resolution limits, inference latency, VRAM consumption, or a structured output schema beyond a ModelOutput reference.
Best use cases
Zero-shot dataset annotation. Use it to create initial bounding-box labels for categories that do not exist in a fixed detector’s class list, such as “red safety helmet,” “forklift,” “damaged package,” or “person wearing a backpack.” The language interface lets you change categories without retraining a detector. Treat the results as proposals that require review, especially for production datasets.
Open-vocabulary image search and indexing. Run queries such as “laptop, coffee mug, notebook” over an image collection and store the detected regions or rendered visualizations. This fits applications where users define categories at query time rather than selecting from a fixed taxonomy.
Visual inspection prototypes. Use prompts for visible conditions such as “cracked screen,” “missing label,” or “open cabinet” to test whether a concept can be localized before investing in a labeled training set. The model can support rapid feasibility studies, but threshold tuning and human validation remain necessary for high-consequence inspection.
Grounded image editing workflows. The README describes integrations with Stable Diffusion and GLIGEN, where detected boxes provide spatial grounding for image editing. This makes the model useful for selecting regions such as “the person,” “the car,” or “the background sign” before passing them to another editing or segmentation system.
Object localization before segmentation or tracking. Use its boxes as prompts for a downstream segmentation or tracking model. The project highlights Grounded SAM and Grounded SAM 2 workflows. This detector supplies language-driven localization; it does not return pixel masks or tracking trajectories itself.
Limitations
The Replicate schema accepts one image URI and one query string. It does not expose batch inputs, image resizing controls, non-maximum-suppression settings, maximum detections, or a model-size selector. The query description says “Comma seperated names of the objects,” so use comma-separated category names. The upstream README recommends separating category names with periods, which creates an interface mismatch worth testing: the hosted schema asks for commas, while the project guidance recommends periods.
Prompt wording affects results. Grounding DINO scores image regions against text tokens, and one word can split into multiple tokens. The number of words in a sentence does not necessarily equal the number of text tokens. The README says the model produces 900 boxes by default, assigns similarity scores across input words, keeps boxes whose highest similarity exceeds box_threshold, and extracts words whose similarity exceeds text_threshold as labels. This can produce missed detections, duplicate or weak boxes, and labels that do not match a developer’s intended phrase.
The model detects boxes, not masks. It cannot provide precise object boundaries, depth, pose, identity, attributes with guaranteed reliability, or temporal consistency across video frames. Small, occluded, unusual, abstract, or visually ambiguous objects can fail. Natural-language descriptions that combine multiple concepts may also behave differently from separate category prompts.
The schema allows both thresholds from 0 to 1, with defaults of 0.25. Lower values can increase recall and false positives; higher values can reduce false positives while missing valid objects. The source materials do not provide a recommended threshold for a particular domain, latency benchmark, maximum image size, output file format, or cost. Do not assume the H100 build guarantees a fixed response time or a particular price.
The upstream repository states that CPU-only execution is supported, but this hosted deployment is described as an H100 build. The README’s local installation instructions require CUDA setup for GPU compilation and warn that incorrect installation can cause NameError: name '_C' is not defined. The project also notes that training code was not released in the referenced README, so adapting the original implementation may require working with inference code and external training approaches.
The repository license is available through the project’s license, but the supplied material does not reproduce its terms. Review that license and any Replicate terms before commercial deployment. The model card and README do not provide a complete safety or bias assessment. Avoid using detections as the sole basis for decisions about people, access, employment, law enforcement, medical care, or other high-impact outcomes.
How it compares
GroundingDINO is the closest equivalent: it represents the same Grounding DINO family and is useful when you want a Hugging Face deployment path, model-library integration, or more control over local preprocessing and post-processing. Choose this Replicate version when you want a hosted API with a small input surface and an H100 build; choose the Hugging Face version when you need to run inside your own infrastructure or integrate with Transformers. The supplied information does not establish a reliable speed, cost, or quality difference between them.
grounding-dino is another Replicate-hosted listing with the same broad purpose, described as “Detect everything with language!” Choose this grounding-dino listing when its exposed schema, version, deployment hardware, or operational behavior fits your application; choose the alternative after benchmarking both on your images and prompts. The available data does not provide a verified cost, latency, checkpoint, or accuracy comparison.
grounding-dino-tiny is the tiny variant and is the better candidate when lower compute use and faster local inference matter more than maximum detector capacity. Choose this H100 SwinT-OGC deployment when detection quality and a managed endpoint matter more than a smaller model footprint. No parameter count or benchmark table is supplied here, so validate the speed-quality tradeoff on representative images.
grounding-dino-base is the base variant and may suit users who need a local Hugging Face model with a different capacity or deployment profile. Choose this Replicate endpoint for a managed API and the documented hosted controls; choose the base variant when infrastructure control, offline execution, or custom preprocessing is more important. The supplied sources do not state enough to claim a numerical quality, speed, or cost advantage.
grounding-dino-swinb uses a Swin-B variant, while this listing is described as SwinT-OGC. Choose the Swin-B listing if your evaluation shows better localization on difficult images and its additional compute fits your budget; choose this model when the H100 deployment and SwinT-OGC configuration meet your latency and quality targets. No direct benchmark, price, or latency data is provided.
Technical specifications
The model comes from the Grounding DINO project, titled “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.” The relevant research paper describes the open-set detection approach: a conventional object detector is extended with language grounding so text can define detection targets.
Confirmed details include:
- Architecture: Grounding DINO with a SwinT-OGC backbone, according to the Replicate description.
- Task: zero-shot, text-prompted object detection.
- Deployment: Replicate public model; latest supplied version ID is
e1ab4da0c9a2841ed63fa4343c4a1bd42ba413a80146b763a77c644e12289b02. - Latest version creation timestamp:
2026-09-16T14:17:56.683858Z. - Cog version:
0.16.2. - Hardware description: H100 build.
- Upstream reported performance: 52.5 AP on zero-shot COCO without COCO training data; 63.0 AP after COCO fine-tuning.
- Upstream default proposal count: 900 boxes.
- Upstream input concept: an
(image, text)pair. - Upstream scoring: each box has similarity scores across input words;
box_thresholdfilters boxes andtext_thresholdextracts predicted labels. - Upstream prompt guidance: separate category names with periods; the hosted schema describes comma-separated names.
- Local upstream support: CPU-only mode exists; CUDA compilation is supported when CUDA is available.
- Training code: marked as unreleased in the supplied README.
- License: the repository license is linked at
https://github.com/IDEA-Research/GroundingDINO/blob/main/LICENSE. - No parameter count, training-set size, maximum resolution, quantization option, context-window limit, output file format, latency, or VRAM requirement is stated in the supplied sources.
Model inputs and outputs
Inputs
-
image: string with URI format. Required by the schema description as the input image to query. -
query: string. The schema describes it as comma-separated names of objects to detect. The README recommends separating category names with periods. -
box_threshold: number from0to1; default0.25. Controls the confidence threshold for retaining detected boxes. -
text_threshold: number from0to1; default0.25. Controls the confidence threshold used for object labels. -
show_visualisation: boolean; defaulttrue. When enabled, the service draws and visualizes bounding boxes on the image. - No enum values, image-dimension constraints, batch field, or output-format selector is exposed.
Outputs
- The OpenAPI output schema references
ModelOutputbut does not expand its fields or type in the supplied schema. - The README establishes that the underlying detector produces bounding boxes and text-associated similarity scores, with predicted labels derived from text thresholds.
- When
show_visualisationis enabled, expect a visualization of bounding boxes on the image, but the exact returned URI, object structure, serialization, and file format are not specified by the supplied output schema. - Build downstream code against the actual returned value from the deployed version rather than assuming a particular JSON field layout.
Getting started
import replicate
output = replicate.run(
"hautechai/grounding-dino:e1ab4da0c9a2841ed63fa4343c4a1bd42ba413a80146b763a77c644e12289b02",
input={
"image": "https://example.com/image.jpg",
"query": "person, backpack, bicycle",
"box_threshold": 0.25,
"text_threshold": 0.25,
"show_visualisation": True,
},
)
print(output)
Use a publicly reachable image URI in the placeholder. Inspect the returned object before production integration because the supplied output schema does not enumerate the ModelOutput fields.
Frequently asked questions
Q: What inputs does grounding-dino require?
A: It accepts an image URI and a text query. The query describes the object names to detect; the hosted schema specifies comma-separated names.
Q: What do the two threshold parameters control?
A: box_threshold controls which candidate boxes remain, while text_threshold controls which text-associated labels are extracted. Both accept values from 0 to 1 and default to 0.25.
Q: Does it return segmentation masks?
A: No. The documented output is object detection with bounding boxes and text-associated scores. Use a downstream segmentation model when you need pixel-level masks.
Q: What output format does the Replicate endpoint return?
A: The OpenAPI schema references ModelOutput without listing its fields or serialization. The README confirms boxes and predicted labels conceptually, but you should inspect the live response before defining a strict client schema.
Q: Can I use this model commercially?
A: The project license is linked in the repository license file, but its terms are not reproduced in the supplied material. Review that license and Replicate’s applicable terms before commercial use.
Q: Why might a valid object be missed?
A: Failures can result from prompt wording, tokenization, object size, occlusion, visual ambiguity, or thresholds that are too high. Try separate category prompts, the README’s period-separated category style, and threshold evaluation on representative images.
Q: Is this model suitable for production use?
A: It can support production prototypes and managed detection services, but validate recall, false-positive rates, latency, output serialization, and cost on your own data. Do not use it as the sole decision mechanism in high-impact applications.
Q: Is it still actively maintained?
A: The supplied Replicate metadata gives a latest version creation timestamp of 2026-09-16T14:17:56.683858Z, and the upstream README references newer projects such as Grounding DINO 1.5 and Grounded SAM 2. That does not establish a maintenance schedule or guarantee that this specific checkpoint is the newest or most capable option.
Top comments (0)