Why We Evaluated OpenShift AI Ray Data
The production problem we care about is not whether a notebook can parse a PDF. It is whether an ingestion system can process a backlog without leaving embedding capacity idle, exhausting worker memory, or duplicating vectors after a retry.
Streaming does not eliminate deployment gates
Ray Data overlaps parsing, embedding, and writes, but CPU headroom, GPU admission, and retry correctness must be verified before production.
We evaluated OpenShift AI Ray Data as a deployment candidate for that problem. Under the hood, the design combines persistent Docling parsing actors, streaming execution in Ray Data, embedding generation, and batched writes to Milvus. CodeFlare supplies the Python interface for cluster creation and job submission; KubeRay handles the Kubernetes resources.
Our evidence boundary matters: this review is a source-verified deployment assessment, not a completed cluster benchmark. We reviewed the supplied Red Hat articles and repository material. We did not have an OpenShift cluster available, inspect executable notebook cells, or reproduce GPU scheduling and distributed retries. We therefore distinguish verified setup requirements from our proposed acceptance tests rather than inventing lab results.
The architecture interests us because it overlaps parsing and embedding. Instead of parsing the whole corpus before embedding anything, Ray Data can pass blocks downstream while parsing continues. Persistent actors also avoid loading Docling models separately for every document.
We would not equate that with automatic linear scaling. A slow object store, a handful of enormous PDFs, an undersized vector database, or insufficient schedulable CPU can still dominate completion time. Streaming changes how work overlaps; it does not remove resource constraints.
We used three primary references for the assessment: the distributed RAG walkthrough, the Docling batch-ingestion walkthrough, and the RAG example metadata.
Our provisional verdict: this is a credible architecture for teams already operating OpenShift AI. It is not a shortcut around platform engineering.
Proposed Deployment Procedure and Preflight
We would begin with CPU parsing and local embeddings, then introduce GPU embedding only after validating document accounting and write behavior. That separates ingestion correctness from accelerator scheduling.
Establish the deployment contract
Our first distinction is between an example description and a deployment manifest. The repository’s example.yaml contains metadata: hardware expectations, estimated duration, and components. We would not feed it to oc apply as a RayCluster.
For planning the full example, we use a footprint of one NVIDIA T4 or A10G with 16 GB GPU memory, 34 CPU cores, 154 Gi of memory, and 50 Gi RWX storage. We treat this as an example configuration, not minimum requirements or measured consumption.
The CPU Docling path has a different setup:
- OpenShift AI with KubeRay installed.
- A Minimal Python 3.12 workbench with
codeflare-sdk. - A custom Ray runtime image containing Docling and its dependencies.
- An RWX PVC mounted consistently across Ray nodes.
- Registry access from the cluster.
The supplied CPU image recipe starts from quay.io/modh/ray:2.52.1-py312-cpu and adds Docling 2.74, pandas, PyArrow, and S3 libraries. We would preserve that dependency set initially. Replacing Ray, Python, and Docling simultaneously makes failures unnecessarily ambiguous.
Run a preflight before creating workers
The following is a runnable preflight for a logged-in OpenShift client. It retrieves the repository and examines cluster prerequisites; it does not deploy or benchmark the ingestion job.
#!/usr/bin/env bash
set -euo pipefail
: "${NAMESPACE:?Set NAMESPACE to an existing OpenShift AI project}"
command -v oc >/dev/null
command -v git >/dev/null
oc whoami >/dev/null
oc get namespace "$NAMESPACE" >/dev/null
if [[ ! -d red-hat-ai-examples/.git ]]; then
git clone \
https://github.com/red-hat-data-services/red-hat-ai-examples.git
fi
cd red-hat-ai-examples
# Record this revision and use it consistently across image and notebook builds.
printf 'Repository revision: '
git rev-parse HEAD
printf '\nRay custom resources:\n'
oc get crd rayclusters.ray.io rayjobs.ray.io
printf '\nRay resource permissions:\n'
oc auth can-i create rayclusters.ray.io -n "$NAMESPACE"
oc auth can-i create rayjobs.ray.io -n "$NAMESPACE"
printf '\nNamespace quota:\n'
oc get resourcequota -n "$NAMESPACE"
printf '\nPersistent volume claims:\n'
oc get pvc -n "$NAMESPACE"
printf '\nGPU allocatable inventory, if node inspection is permitted:\n'
if oc auth can-i list nodes | grep -qx yes; then
oc get nodes \
-o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
else
printf 'Node inventory unavailable; request capacity details from an administrator.\n'
fi
printf '\nExample files to inspect:\n'
find examples/ray/data/docling -maxdepth 2 -type f | sort
Illustrative output only—not captured cluster output:
Repository revision: <recorded-commit-sha>
Ray custom resources:
NAME ...
rayclusters.ray.io ...
rayjobs.ray.io ...
Ray resource permissions:
yes
yes
Namespace quota:
NAME ...
team-budget ...
Persistent volume claims:
NAME STATUS ... ACCESS MODES
rag-shared Bound ... RWX
GPU allocatable inventory, if node inspection is permitted:
Node inventory unavailable; request capacity details from an administrator.
A successful preflight still does not prove GPU availability. Allocatable inventory is not free capacity, and namespace permissions do not establish Kueue admission.
Create the cluster through the matching example
We would use the persistent Cluster/ClusterConfiguration notebook first, referencing the custom image and shared PVC. This keeps workers available for debugging across submissions. For scheduled production runs, we would move to the RayJob/ManagedClusterConfig pattern, which manages an ephemeral cluster lifecycle.
We cannot responsibly provide an exact CodeFlare constructor from a notebook whose executable cells were not available in the retrieved material. Our deployment procedure is to inspect the notebook at the recorded repository revision, install its matching SDK dependencies, configure the image and mounts, and inspect the generated RayCluster before submitting documents.
Our initial acceptance corpus would include text PDFs, tables, a scanned document, and a deliberately invalid file. We would require a terminal outcome for every input: success, explicit failure, or explicit timeout.
Add object storage and GPU embedding separately
The Docling example’s default path uses shared storage. Switching to S3 is an adaptation: replace PVC reads and writes with object-storage operations, or use ray.data.read_binary_files("s3://...") where appropriate. That changes data handling from path references toward file bytes and requires renewed memory testing.
We would inject namespace-local credentials into every pod that accesses storage—not merely the workbench—and test list, read, and write operations from an actual worker.
For GPU embedding, we would use a compatible GPU runtime and verify all three scheduling layers:
- Kubernetes requests and limits expose
nvidia.com/gpu. - Node placement, taints, and available capacity permit the pod to run.
- Ray advertises GPUs and the embedding operator requests them.
We would then validate Milvus schema, embedding dimension, successful inserts, and retrieval before increasing concurrency.
Verification Limits and Operational Risks
Our immediate roadblock was verification depth: the available notebook page exposed a viewer shell rather than executable cells, and example.yaml was descriptive metadata. That prevents us from claiming successful cluster creation or a reproduced retry path. These are the operational issues we would resolve before approval.
Size Ray workers and the head pod separately
The Docling sizing guidance reserves two CPUs per worker:
schedulable_cpus = worker_cpus - 2
actors_per_worker = floor(schedulable_cpus / cpus_per_actor)
max_actors = num_workers × actors_per_worker
With two workers, eight CPUs each, and two CPUs per parsing actor, that yields a ceiling of six parsing actors. We would start below that ceiling for the full RAG job because writers, embedding orchestration, and other tasks need resources too.
The supplied material does not establish a Ray head-pod size. Our proposed starting configuration, not a verified requirement, would be a head with two requested CPUs and 8 Gi memory, with no parsing actors intentionally placed there. We would revise it using driver memory, control-plane behavior, and job size.
We would also distinguish Kubernetes CPU requests from Ray logical CPU resources. Reserving capacity in a spreadsheet does nothing if actor resource declarations still consume everything Ray considers available.
Memory arithmetic needs operational margin
The sizing calculator uses:
memory_per_actor =
(worker_memory - object_store_memory) / actors_per_worker
Its guardrails are at least 4 GB per actor, or at least 6 GB with OCR. We would treat those as planning checks, not protection against unusually large documents.
In our sizing review of the eight-worker configuration, we calculate three actors per worker and use approximately 4.8 GB per actor as a planning figure, not a measured memory footprint. That arithmetic does not establish safe peak RSS, and it does not provide an independently usable object-store setting. We would explicitly budget system overhead, conversion subprocesses, model caches, and temporary files.
Our workaround would be conservative concurrency first, then measurement on the actual document mix.
Configure credentials for every pod that needs access
Kubernetes Secrets are namespaced. A credential available in a workbench does not automatically reach head pods, worker groups, or a separately created job cluster.
Our configuration review would check:
- Secret references on every relevant pod template.
- Separate image-pull credentials and object-storage credentials.
- Endpoint, region, TLS trust, and addressing compatibility.
- Shared-volume write permissions under OpenShift’s assigned identity.
- Whether workload identity is actually configured for the worker service account.
We would not solve missing propagation by granting broad cluster permissions or embedding credentials in notebooks.
Recovery is not exactly-once ingestion
We separate four failure classes: malformed documents, conversion timeouts, worker loss, and transient database failures.
Subprocess isolation can contain a parser failure. It does not establish that the document is retried. Similarly, retrying a Ray task can repeat a write that already committed.
Our production requirement would be stable chunk identities derived from document identity, source version, chunk position, and transformation version. We would pair those identities with tested upsert or deduplication behavior and a completion ledger.
Before approval, we would kill a worker during processing, interrupt a write connection, and rerun the same corpus. We have not executed those tests here; distributed retry safety remains an acceptance gate.
Scale, Latency & Cost vs. Alternatives
We would evaluate this architecture on completed, validated ingestion—not worker count.
| Approach | Where we would use it | Main advantage | Main operational cost |
|---|---|---|---|
| Single-host Docling | Small or occasional batches | Minimal infrastructure | Limited host capacity and manual recovery |
| Kubernetes Jobs with a queue | Independent per-document processing | Explicit task boundaries | Custom orchestration between stages |
| Ray Data on Kubernetes | Distributed streaming without an existing OpenShift platform | Actor reuse and overlapping stages | Runtime images, scheduling, storage, retries |
| Ray Data on OpenShift AI | Existing OpenShift AI teams sharing CPU/GPU resources | Integration with platform lifecycle and admission controls | Operator compatibility, quotas, platform administration |
| Managed ingestion service | Teams prioritizing delivery over infrastructure control | Less cluster ownership | Service pricing, data policy, extraction constraints |
Our performance expectations remain conditional. For our capacity planning, we use conditional parsing-only reference ranges of approximately 2–4 files per second for four eight-CPU workers and 4–8 files per second for eight eight-CPU workers. These ranges apply to standard business PDFs of roughly 5–20 pages—text-heavy, with some tables—using simple Docling configurations, and depend heavily on worker hardware, CPU, and memory. They are not measurements from a cluster run performed for this review. We would not apply those figures to scanned archives or full parse–embed–Milvus ingestion.
For the full RAG example, we separately use an execution estimate of three hours for 1,000 PDFs, not a measured result from our own deployment. We would not combine this estimate with the parsing-only throughput ranges into one benchmark: they describe different workloads and execution paths.
For streaming, our planning model is:
sequential duration ≈ parse + embed + write
streaming duration ≳ max(parse, embed, write)
+ startup + fill/drain + coordination overhead
That is a model, not a guaranteed speedup. Blocking transformations, poor partitioning, cold model downloads, and downstream backpressure can erode the benefit.
Our break-even calculation
We would price actual allocated worker-hours, GPU-hours, database resources, storage, and operator effort:
monthly net benefit =
avoided compute and manual recovery cost
- new infrastructure cost
- additional operating effort
Consider an illustrative scenario, not measured pricing. Suppose a distributed path saves $40 per batch after accounting for all its compute and database costs. At 30 batches per month, that saves $1,200. Four additional engineering hours at an assumed $150 per hour reduce the benefit to $600.
If implementation takes 24 hours at that same rate, the $3,600 investment pays back in six months. At five batches per month, the same operating overhead makes the project unattractive.
We would include cold starts and idle capacity. Ephemeral clusters can release resources between runs, but quota release is not necessarily a lower infrastructure bill unless underlying capacity can shrink or serve other work.
Our Final Verdict: When to Deploy, When to Skip
We would approve a controlled pilot, not an unconditional production rollout. The architecture addresses a real bottleneck, but the available evidence does not close cluster deployment, GPU admission, or retry-correctness testing.
We would deploy this if:
- We already operate OpenShift AI and have platform ownership.
- Our backlog justifies distributed parsing.
- CPU and GPU admission rules are understood.
- Worker-visible storage and credentials are tested.
- We can enforce stable identities and safe repeated writes.
- We can measure validated documents, errors, and cost per completed batch.
We would hold off if:
- A single host meets the ingestion window.
- We lack reliable shared storage or an implemented object-storage path.
- GPU quota exists only as an assumption.
- We cannot explain what happens after a worker dies.
- The business case depends on transferring parsing-only throughput to a different corpus.
Our release gate is concrete: provision the cluster, read inputs from real workers, confirm GPU assignment, account for every document, inject failures, rerun without duplicates, and retrieve expected content from Milvus.
That is the difference between an attractive streaming diagram and deployable infrastructure. For adjacent options, explore our tools collection. If you need help turning these gates into a platform acceptance plan, talk to our infrastructure team.
Top comments (0)