Portofino

Posted on Sep 4 AI-assisted

Ray Core vs. Data, Train, Tune, and Serve: A Practical Mental Model

#python #machinelearning #distributedcomputing #ray

Ray is easy to describe badly.

Call it a "distributed Python framework," and it sounds like a faster multiprocessing. Call it an "AI platform," and it sounds like it should manage users, approvals, datasets, and model releases. Call it a "cluster manager," and people understandably ask why they still need Kubernetes.

None of those labels is sufficient by itself.

A more useful description is this:

Ray is a distributed compute layer for Python and AI workloads. It gives you low-level primitives for running Python across processes and machines, plus higher-level libraries for data processing, training, tuning, reinforcement learning, and serving.

That distinction matters. Once you understand which problems Ray solves—and which problems remain yours—the project becomes much easier to reason about.

This article builds that mental model from the smallest possible Ray program, then follows it through an end-to-end machine learning workflow. It closes by examining the boundary between Ray's compute runtime and the lifecycle contracts an application or platform may add above it.

Version scope: This article targets Ray 2.55.1, the compatibility baseline for the SDK case study at the end. API details and behavior can change, so use the versioned Ray documentation for the environment you deploy.

The task and actor examples use small in-memory inputs so you can run them locally. They adapt the task and actor patterns from the versioned Ray Core documentation to one small scoring example. The rest of the article focuses on how Ray's libraries and boundaries fit together rather than presenting a complete application.

The distributed systems problem hiding inside Python

Suppose you have several data partitions to score. A normal Python program might process them one at a time, with one process owning execution order, memory, errors, and return values.

Distributing the same work introduces a surprising number of questions:

Which machine should run each function?
Does that machine have the required CPU, GPU, memory, code, and packages?
How do inputs and outputs move between machines?
What happens when a worker dies?
How do we avoid sending the same large object over the network repeatedly?
Who decides whether to retry, wait, or fail?

Ray's value starts here. It lets your code describe work and resource requirements while the runtime handles scheduling, process management, and distributed object movement.

The three ideas at the center of Ray Core

The smallest useful Ray vocabulary contains three nouns:

A task is a stateless remote function.
An actor is a stateful remote class instance.
An object reference is a handle to a value that may live elsewhere in the cluster.

Here is the earlier loop as Ray tasks:

import ray

ray.init()


@ray.remote(num_cpus=1)
def score_partition(partition: list[int]) -> dict:
    return {"count": len(partition), "sum": sum(partition)}


partitions = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]
result_refs = [score_partition.remote(partition) for partition in partitions]

# Work has already been submitted. This is the synchronization point.
results = ray.get(result_refs)

The @ray.remote decorator turns the function into a task definition. Calling .remote() submits work asynchronously and immediately returns an ObjectRef. Calling ray.get() resolves the references and waits only when the concrete values are needed.

This is more important than the decorator makes it look. The driver no longer chooses a process or machine. It declares work, and Ray's scheduler places that work according to the resources available in the cluster.

There is also a common performance trap hidden in this example. If you call ray.get() immediately inside the submission loop, you serialize independent work again. Submit independent calls first; synchronize later.

Use actors when state should stay warm

Tasks are a good fit for independent transformations. They are less convenient when initialization is expensive or state must survive across calls. A model server, database connection pool, or simulation environment often fits the actor model better:

import ray

ray.init()


@ray.remote(num_cpus=1)
class ModelWorker:
    def __init__(self, bias: float = 0.0):
        self.bias = bias

    def predict(self, batch: list[float]) -> list[float]:
        return [value + self.bias for value in batch]


worker = ModelWorker.remote(0.5)
prediction_ref = worker.predict.remote([1.0, 2.0, 3.0])
predictions = ray.get(prediction_ref)

The actor's state is initialized once inside the actor process and reused by later method calls. A real model worker can use the same pattern to load a model once and keep it warm. Ray AI libraries build many of their higher-level abstractions from the same task, actor, and object primitives.

An ObjectRef is a claim ticket, not the object itself

An ObjectRef is best understood as a location-independent claim ticket. The value can live in a node's shared-memory object store rather than inside the driver's Python heap. Tasks can pass object references to other tasks without forcing the driver to download and re-upload every value.

That does not make data movement free. Large datasets still consume object-store memory and network bandwidth. Ray removes much of the coordination burden; it does not repeal physics.

Ray Core tells you how; AI libraries let you declare what

There is a useful progression in Ray's API design.

With Core primitives, you control more of how work runs: which calls become tasks, which state lives in actors, how many calls may remain pending, when to wait, and where object references flow. That flexibility is valuable when an application does not fit a fixed data-processing pattern.

With an AI library, you usually declare more of what you want: transform this dataset in batches, start this training worker group, explore this search space, or serve this deployment. The library translates that declaration into tasks, actors, object transfers, scheduling requests, and recovery behavior.

Neither level is universally better. Start at the highest level that expresses the workload correctly, then drop to Core only when you genuinely need tighter execution control. This avoids rebuilding batching, backpressure, actor-pool management, or retry logic that a Ray library already owns.

A practical mental model: libraries, Core, and clusters

Ray's official overview presents the framework in three layers. Thinking in those layers prevents many category errors.

At the top, Ray AI Libraries solve recognizable workload problems. In the middle, Ray Core supplies distributed tasks, actors, objects, scheduling, resource declarations, and runtime environments. At the bottom, Ray Clusters provide the processes and machines on which the work runs.

The same Python application can start a local Ray runtime on a laptop or connect to an existing cluster. The code may stay similar, but production operation does not happen automatically: networking, storage, observability, security, dependency images, and failure policy still need deliberate design.

What each Ray AI library is for

Ray's native libraries share a runtime, but they do not form one mandatory pipeline. You can use Data without Train, Tune with a custom function, or Serve without training the model in Ray.

Library	The question it answers	Core abstraction
Ray Data	How do I read, transform, and write distributed tabular or tensor data while streaming blocks through execution?	Lazy `Dataset` plans executed through tasks or actor pools
Ray Train	How do I coordinate distributed training workers?	A training function, workers, `ScalingConfig`, and a `Trainer`
Ray Tune	How do I run and manage many experiments efficiently?	Trainables, search spaces, search algorithms, schedulers, trials, and `Tuner`
Ray Serve	How do I expose Python and models as scalable online services?	Controller, proxies, deployments, replicas, and deployment handles
RLlib	How do I scale reinforcement-learning sampling and learning?	Algorithms, EnvRunners, Learners, environments, and RLModules

Ray Data: distributed data movement and transformation

A Ray Dataset represents a distributed collection split into blocks. Operations such as read_parquet(), filter(), and select_columns() are lazy: Ray first builds a logical plan, then optimizes and converts it into physical operators when the dataset is consumed.

The point is not merely parallel file reading. Ray Data can stream blocks through operators so a pipeline does not have to materialize its full intermediate state at every step. For vectorized transforms and model inference, map_batches() is usually the key API. Callable classes can run in an actor pool, which allows each actor to load a model once and reuse it across batches.

Ray Data is not a data catalog, data-governance product, or universal transaction layer. The selected data source still defines consistency, snapshot, schema-evolution, and commit behavior.

Ray Train: coordinate workers without rewriting the model framework

Ray Train organizes distributed training around four ideas: a user training function, worker processes, a scaling configuration, and a Trainer. In a typical PyTorch integration, TorchTrainer launches the worker group from a train_loop_per_worker function and uses ScalingConfig to request the required workers and GPUs.

Ray creates the worker group, configures the framework's distributed environment, and runs the function on every worker. PyTorch still owns tensor operations, gradients, and the model. Ray owns distributed execution around that training code.

This separation is one of Ray's strongest design choices: it generally integrates with the Python ML ecosystem instead of asking you to replace it.

Ray Tune: distribute the search, not just the training

Training one model faster is only part of the problem. Model development often means running dozens or thousands of trials with different hyperparameters.

Tune separates several concerns that are frequently mixed together:

the trainable computes and reports metrics;
the search space defines possible configurations;
the search algorithm chooses configurations;
the scheduler can pause or stop unpromising trials;
the Tuner launches trials and returns a ResultGrid.

The function API reports intermediate metrics with tune.report(). The important distinction is that Tune selects and manages experiments; it does not by itself turn the best trial into a production release.

Tune can launch distributed Ray Train runs as trials, but a tuning result is not automatically a production model release. Selecting the best trial, reproducing it, validating artifacts, and publishing a deployable model remain lifecycle decisions.

Ray Serve: stateful actors become an online application

Serve runs on Ray actors. A controller manages deployments; HTTP or gRPC proxies receive requests; replicas execute application code; deployment handles connect components inside a composed application.

That architecture gives Serve several useful properties:

each deployment can request different CPU and GPU resources;
replicas can batch requests and autoscale with load;
models and ordinary Python business logic can live in one application graph;
within a running Ray cluster, actor restart and controller state provide some recovery below the application boundary; cluster-wide recovery remains an infrastructure or KubeRay concern.

Serve is a serving runtime, not a model-governance system. It does not decide which model version passed your approval process or whether a schema change is safe.

RLlib: a specialized distributed system for reinforcement learning

Reinforcement learning has two expensive loops: collecting experience from environments and learning from that experience. RLlib can scale those axes independently with EnvRunner and Learner actors. It also provides algorithms, multi-agent support, offline data paths, and model abstractions through RLModules.

RLlib belongs in the overall Ray map even if your organization never uses reinforcement learning. It demonstrates why general-purpose tasks and actors are valuable: a complex domain-specific runtime can be assembled on top of a small distributed foundation.

A Ray Job is larger than a Ray task

The word job is often confused with task.

A Ray task is one remote function call. A Ray Job is the application entrypoint and all tasks, actors, and objects created recursively by that entrypoint.

For a remote cluster, the Ray Jobs API is a common boundary for submitting an application. A submission contains an entrypoint command and a runtime environment describing code and package dependencies. The CLI form is ray job submit --no-wait --working-dir . -- python train.py; with --no-wait, the client can disconnect while the cluster-owned job continues. The Jobs API also exposes status, logs, and stop operations.

KubeRay solves a different problem. It is a Kubernetes operator that manages RayCluster, RayJob, and RayService resources. Ray schedules Python work inside the Ray cluster; Kubernetes and KubeRay manage the cluster's pods and infrastructure lifecycle.

Where Ray sits in the open AI compute stack

The broader AI stack becomes clearer when we separate its scheduling grains. PyTorch and vLLM optimize model execution. Ray coordinates the distributed processes and stages inside an application. Kubernetes or Slurm allocates infrastructure resources across applications and users. A company platform or SDK adds the domain policy that none of those lower layers can guess.

Layer	Examples	Primary responsibility
Application and lifecycle contracts	Internal ML platforms, workflow SDKs	Valid requests, provenance, artifacts, release semantics, organization policy
Model runtimes and frameworks	PyTorch, XGBoost, vLLM, Hugging Face	Tensor computation, model logic, model parallelism, engine-level optimization
Distributed compute engine	Ray Core and Ray AI Libraries	Tasks, actors, objects, data movement, worker coordination, workload-aware recovery
Infrastructure orchestration	Kubernetes, KubeRay, Slurm, cloud VMs	Nodes, containers, multi-tenant resource allocation, infrastructure lifecycle

The boundaries are compositional rather than competitive. Ray Train can coordinate PyTorch workers. Ray Serve can orchestrate vLLM replicas. KubeRay can manage the Kubernetes resources that host both. In October 2025, Ray became a PyTorch Foundation-hosted project, alongside projects including PyTorch, vLLM, and DeepSpeed—an organizational reflection of this layered technical relationship.

How the pieces form an ML lifecycle

Imagine a customer-churn system trained from daily Parquet data.

A sensible flow might look like this:

Submit the application through Ray Jobs using an entrypoint command and a reproducible runtime environment.
Use Ray Data to read bounded input and prepare distributed shards.
Use Ray Tune to run parallel trials, report metrics, and optionally stop weak candidates.
Run a separate Ray Train training run with the selected parameters when the release policy requires retraining.
Validate and publish the model artifact.
Use Ray Data for throughput-oriented batch inference.
Use Ray Serve for latency-oriented online requests.
Record metrics, lineage, and model metadata in external systems.

Ray can execute every compute-heavy box in that picture. However, the arrows between the boxes contain product-specific rules:

Which input snapshot produced this model?
Which preprocessing schema belongs to it?
Which files make up the model, and have they been tampered with?
Which exported format is actually executable by which runtime?
Is a tuning trial allowed to publish a production artifact?
Can the batch and online paths interpret the same tensor signature?
Which credentials may cross from source storage to model storage?

Those are not merely scheduling questions. They are lifecycle-contract questions.

What production teams teach us

Production stories are useful because they reveal both Ray's strengths and the engineering that still surrounds it. The following systems include company-specific extensions; they should not be read as a list of features available automatically in an unmodified Ray installation.

ByteDance: move from Core control to Data abstractions

ByteDance described using Ray for large-scale audio and video data pipelines serving multimodal-model development. Its teams initially used Ray Core to distribute pipeline nodes, then adopted Ray Data for automatic block management, data loading, common transforms, and actor-pool scaling. This is the how-to-what progression in a real system: Core made the migration possible, while Data removed infrastructure code that did not differentiate the application.

The case also exposes an important boundary. Moving large video objects through the object store introduced serialization and spilling costs, so one packaging path fused download, processing, Parquet writing, and upload inside multithreaded actors. On preemptible infrastructure, the team added higher-level task reassignment and lineage handling around its particular workload. The lesson is not "always fuse operators" or "Ray handles every failure." It is to profile object movement and assign recovery ownership at the correct layer. See the Ray Summit 2024 case summary and recorded talk.

Tencent and WeChat: AI data pipelines are often batch inference

WeChat's AI workloads include search, recommendation, content understanding, and generative-media processing. Its published Ray platform work emphasizes fine-grained CPU/GPU resource declarations and the ability to express a multi-model application as Python rather than a mesh of separately deployed services. At its scale, however, the team built substantial platform machinery for heterogeneous resources, rapid failure routing, runtime distribution, and federated clusters. That is strong evidence for Ray as a compute foundation—and equally strong evidence that Ray is not the whole enterprise platform.

A later Tencent Hunyuan pipeline architecture makes another useful point: many "data pipelines" in modern AI are really batch-inference systems. Table metadata may identify the input, but most compute is spent decoding multimodal objects and running CPU or GPU models. A streaming-batch execution model can keep heterogeneous stages busy without turning the workload into an online service. See the streaming-batch research paper, WeChat's large-scale Ray practice, and Tencent Hunyuan's heterogeneous pipeline architecture.

Spotify: developer experience lives above the runtime

Spotify built its Hendrix ML platform and a cloud development environment around Kubernetes, Ray, PyTorch, and internal SDKs. The surrounding platform standardized environments, integrated company data services, exposed remote compute, and added access control, telemetry, availability, and cost-management behavior.

This is the clearest answer to a common question: if Ray already scales Python, why build another SDK? Because a compute runtime does not know how an organization names datasets, packages models, authorizes users, or defines a successful release. Spotify's platform work illustrates that upper-layer problem. See the Spotify Engineering account of its Ray platform.

Where Ray intentionally stops

Ray is often most useful when you treat it as a compute substrate rather than expecting it to become your entire ML platform.

Ray provides	Your application or platform still decides
Tasks, actors, objects, scheduling, and retries	Business-level state transitions and idempotency
Dataset execution	Source governance, credentials, snapshot guarantees, and schema policy
Training workers and checkpoints	A portable, validated model-delivery contract
Tune trials and result selection tools	Whether and how a selected trial becomes a formal release
Serve deployments and autoscaling	Approval, promotion, canary, rollback, and tenant policy
Cluster resource requests	Organizational quota, RBAC, cost ownership, and infrastructure security

This boundary is healthy. A general compute framework should not guess every organization's governance model.

It also creates room for higher-level frameworks that add opinionated contracts while continuing to delegate distributed execution to Ray.

When Ray is a good fit

Ray is worth evaluating when several of these statements are true:

Your workload is naturally expressed in Python.
You need both stateless parallel work and long-lived stateful workers.
CPUs and GPUs must be scheduled across multiple nodes.
You want Data, Train, Tune, Serve, or RLlib to share one runtime.
Existing ML libraries should remain in control of model logic.
You need to move from laptop development to cluster execution without rewriting the application around a different programming model.

Ray may be unnecessary or counterproductive when:

the workload fits comfortably in one process and distribution overhead dominates;
the computation is primarily a SQL query already handled well by a database or warehouse;
tasks have strong sequential dependencies and little useful parallelism;
your only requirement is a conventional stateless web API;
your organization needs a fully managed platform and does not want to operate distributed runtime infrastructure;
you expect Ray alone to supply multi-tenancy, governance, and organization-specific release policy.

"Can Ray run it?" is usually the wrong first question. Ask whether the workload has enough parallelism, state, or heterogeneous resource demand to justify a distributed runtime.

Practical advice before scaling out

Start locally, but test the real boundary

ray.init() is excellent for learning and local development. Before production, test the actual submission, dependency, storage, and failure boundaries you will use on the cluster. A local success does not prove that workers have the right packages or network access.

Declare resources honestly

Ray schedules declared resources. If a task consumes four CPUs but requests one, the scheduler cannot make a good placement decision. The same applies to GPUs, custom resources, and memory-sensitive actors.

Keep the driver out of the data path

The driver should coordinate work, not collect every large intermediate result into its heap. Prefer distributed transformations and pass object references between workers where possible.

Understand failure semantics at every layer

A retried task, restarted actor, failed training worker, partially published artifact, and timed-out HTTP request are different failures. Define which layer owns retry, cleanup, idempotency, and the final source of truth.

Make environments reproducible

Runtime environments can ship code and Python packages, but dynamic installation is not a substitute for a tested runtime image in sensitive or large deployments. Pin versions and verify that the driver and workers see the same dependency closure.

Measure before adding nodes

Distribution adds serialization, coordination, object-store pressure, and network transfer. Profile the single-node path first. More machines help only when useful parallel work exceeds those costs.

Profile the whole pipeline as well as the model kernel. GPU utilization can remain low because CPU decoding, tokenization, network reads, or serialization cannot feed the accelerator quickly enough. Conversely, sending very large intermediate objects through many actor boundaries can make the object store or spill path the bottleneck. Tune stage concurrency, batch size, CPU-to-GPU ratios, and object lifetime together rather than optimizing each operator in isolation.

The shortest useful summary

Ray becomes much less mysterious when you keep four boundaries in mind:

Ray Core turns Python functions, classes, and objects into distributed tasks, actors, and references.
Ray AI Libraries assemble those primitives into data, training, tuning, serving, and reinforcement-learning runtimes.
Ray Clusters and Jobs provide the execution and application-submission boundary.
Your framework or platform still owns domain contracts, artifacts, governance, and business-level lifecycle policy.

A concrete implementation example: Tributo

Disclosure: I maintain Tributo. It is an Apache-2.0, Ray-native SDK shown here as one implementation of the upper layer described above, not as a required part of Ray. Its current compatibility baseline is Ray 2.55.1. Its design keeps distributed execution inside Ray while adding typed requests, provider routing, provenance, validated model Bundles, and explicit batch or online inference contracts.

The project deliberately does not provision Kubernetes, implement a custom scheduler, manage tenants or quotas, or provide approval and rollout workflows. Its support matrix separates verified, beta, alpha, adapter-only, and unsupported paths. The broader lesson is independent of this project: keep policy and delivery semantics above Ray, and keep physical distributed execution inside the runtime that owns it.

References

Disclosure: This article was drafted with AI assistance for research organization, structure, language, and diagram production. I reviewed the technical claims against the linked Ray documentation and the public Tributo source tree. The human author remains responsible for rechecking every claim and code sample before publication.

DEV Community