CDP-6001 in PySpark: Six Code Blocks Behind 58% of the Cloudera ML Engineer Exam

CDP-6001, Cloudera's Machine Learning Engineer exam, gives you 45 questions in 90 minutes and needs 60% to pass. Three of its five domains (Spark 18%, Spark MLlib 22% and Deploying a Machine Learning Model 18%) add up to 58% of the exam, and all three are code you can run yourself in a Cloudera Machine Learning session.

This post walks through that code as one small churn project: load events, build window features, fit a pipeline, tune it, track the run with MLflow and deploy a scoring function. Each block maps to a line of the official topic list.

What does the CDP-6001 exam measure?

Cloudera lists five domains and a weight for each. The exam costs $330, is taken online with a proctor, and allows no reference material at all, so you cannot look up an API while you answer.

Domain	Weight	Topics listed
Cloudera Machine Learning	31%	Workspaces, Projects, Experiments, Accelerators for ML Projects, Data Visualizations, Runtimes, GPUs
Spark MLlib	22%	Model Selection and Tuning, Fitting and Evaluating Models, Pipelines
Spark	18%	DataFrames, File Types, Window Functions
Deploying a Machine Learning Model	18%	Applications/API, Autoscaling and Performance, Model Metrics and Monitoring, ML Flow
Deep Learning and General Machine Learning	11%	General Machine Learning, Supervised and unsupervised learning, Algorithms

The weights above are copied from the CDP Machine Learning Engineer exam guide, which also names the suggested course, Predicting with Cloudera Machine Learning.

How fast do you need to answer?

Ninety minutes for 45 questions is two minutes a question. If every question counts the same, 60% of 45 is 27 correct answers, so you can miss 18. Cloudera publishes the pass score only as a percentage, so treat 27 as arithmetic, not a rule.

Two minutes is enough to read a short code snippet and spot the wrong argument, but not enough to work out an API from scratch. That is why writing the code below once, by hand, pays off. This short from the AnalyticsExam channel covers the same pacing:

Which file and DataFrame skills does the Spark domain test?

The Spark domain names DataFrames and File Types. The pattern to know: read raw CSV with an explicit schema, write it once as partitioned Parquet, and read the Parquet copy from then on.

from pyspark.sql import SparkSession, functions as F, types as T

spark = SparkSession.builder.appName("cdp6001-churn").getOrCreate()
BASE = "/tmp/churn"  # point this at storage your Spark executors can read

schema = T.StructType([
    T.StructField("customer_id", T.StringType()),
    T.StructField("event_ts", T.TimestampType()),
    T.StructField("plan", T.StringType()),
    T.StructField("monthly_spend", T.DoubleType()),
    T.StructField("support_tickets", T.IntegerType()),
    T.StructField("churned", T.IntegerType()),
])

raw = spark.read.option("header", True).schema(schema).csv(f"{BASE}/events.csv")

(raw.withColumn("event_date", F.to_date("event_ts"))
    .write.mode("overwrite")
    .partitionBy("event_date")
    .parquet(f"{BASE}/events_parquet"))

events = spark.read.parquet(f"{BASE}/events_parquet")
events.printSchema()

Three details tend to come up:

• An explicit schema saves the extra pass over the file that inferSchema needs, and it stops a column of IDs from turning into integers.
• Parquet stores the schema with the data and is columnar, so a query that reads three columns does not scan the other ten.
• partitionBy("event_date") writes one folder per date, and a filter on event_date can skip whole folders.

How do window functions turn events into features?

Window functions are the third Spark topic, and they are how raw events become one row of features per customer. A window has a partition, an order and, for running totals, a frame.

from pyspark.sql import Window

w_time = Window.partitionBy("customer_id").orderBy("event_ts")
w_last7 = w_time.rowsBetween(-6, Window.currentRow)
w_latest = Window.partitionBy("customer_id").orderBy(F.desc("event_ts"))

features = (events
    .withColumn("prev_spend", F.lag("monthly_spend", 1).over(w_time))
    .withColumn("spend_change", F.col("monthly_spend") - F.col("prev_spend"))
    .withColumn("tickets_last7_events", F.sum("support_tickets").over(w_last7))
    .withColumn("rn", F.row_number().over(w_latest))
    .filter("rn = 1")
    .drop("rn")
    .fillna({"prev_spend": 0.0, "spend_change": 0.0}))

lag returns null on the first row of every partition, which is why fillna follows it. rowsBetween(-6, 0) counts rows, so the sum covers the last seven events, not the last seven days; a time-based frame needs rangeBetween over a numeric ordering column. The Spark SQL reference on window functions and frame clauses lists the ranking functions (ROW_NUMBER, RANK, DENSE_RANK, NTILE, PERCENT_RANK) and the analytic ones (LAG, LEAD, FIRST_VALUE and others).

What does a Spark MLlib pipeline look like end to end?

Pipelines are one of the three MLlib topics, and the exam's tuning questions build on them. A pipeline chains stages; Estimators get fit(), Transformers get transform().

from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler
from pyspark.ml.classification import LogisticRegression

train, test = features.randomSplit([0.8, 0.2], seed=42)

indexer = StringIndexer(inputCol="plan", outputCol="plan_idx", handleInvalid="keep")
encoder = OneHotEncoder(inputCols=["plan_idx"], outputCols=["plan_vec"])
assembler = VectorAssembler(
    inputCols=["plan_vec", "monthly_spend", "spend_change", "tickets_last7_events"],
    outputCol="features")
lr = LogisticRegression(featuresCol="features", labelCol="churned")

pipeline = Pipeline(stages=[indexer, encoder, assembler, lr])
model = pipeline.fit(train)       # a PipelineModel
scored = model.transform(test)    # adds rawPrediction, probability, prediction

StringIndexer, OneHotEncoder and LogisticRegression are Estimators here; VectorAssembler is a Transformer. After fit(), every Estimator in the chain has become a fitted Transformer inside the PipelineModel, so training and test data go through exactly the same steps. The Spark guide to ML Pipelines, Estimators and Transformers has the diagram worth remembering for this part.

How do CrossValidator and TrainValidationSplit differ?

Model Selection and Tuning is where a counting question is easy to set: how many models does this code fit?

from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
from pyspark.ml.evaluation import BinaryClassificationEvaluator

grid = (ParamGridBuilder()
        .addGrid(lr.regParam, [0.01, 0.1, 1.0])
        .addGrid(lr.elasticNetParam, [0.0, 0.5])
        .build())                                    # 3 x 2 = 6 settings

evaluator = BinaryClassificationEvaluator(labelCol="churned")  # default metric: areaUnderROC

cv = CrossValidator(estimator=pipeline, estimatorParamMaps=grid,
                    evaluator=evaluator, numFolds=3, parallelism=2, seed=42)
cv_model = cv.fit(train)

test_auc = evaluator.evaluate(cv_model.transform(test))
best_lr = cv_model.bestModel.stages[-1]
print(test_auc, best_lr.getRegParam(), best_lr.getElasticNetParam())

Six settings times three folds is 18 fits, and CrossValidator then re-fits the best setting on the whole training set, so 19 in total. TrainValidationSplit evaluates each setting once on a single split set by trainRatio, so the same grid costs 6 fits plus the final one. parallelism=2 evaluates two settings at a time; it changes the run time, not the result.

Because the estimator is the whole pipeline, the encoder and indexer are re-fitted inside every fold too, which keeps test-fold data out of training. The Spark page on model selection and hyperparameter tuning also lists the three evaluators: regression, binary classification and multiclass.

How do you track runs with MLflow in Cloudera Machine Learning?

ML Flow sits under the deployment domain. In Cloudera Machine Learning an experiment run is ordinary MLflow code inside a project session.

import mlflow

mlflow.set_experiment("cdp6001-churn")
with mlflow.start_run():
    mlflow.log_param("numFolds", 3)
    mlflow.log_param("best_regParam", best_lr.getRegParam())
    mlflow.log_param("best_elasticNetParam", best_lr.getElasticNetParam())
    mlflow.log_metric("test_auc", test_auc)

Points from Cloudera's page on running an experiment with MLflow that make good exam questions:

• Experiment runs need a session on ML Runtimes; a Legacy Engine session cannot create them.
• The MLflow client is installed by default, but you import it in every session.
• Run artifacts are stored in the project under .experiments/<experiment_id>/<run_id>.
• mlflow.autolog() before the training code logs parameters, metrics and models without explicit calls.

How does a model become a REST endpoint?

Applications/API and Autoscaling and Performance are about Cloudera's model serving. A model is a Python function in a project file; Cloudera builds it into a container and puts a REST endpoint in front of it.

# predict.py
import cml.models_v1 as models
import joblib

model = joblib.load("churn_model.joblib")  # module level: loaded once when the replica starts
FEATURES = ["monthly_spend", "spend_change", "tickets_last7_events"]

@models.cml_model
def predict(args):
    missing = [f for f in FEATURES if f not in args]
    if missing:
        return {"error": "missing fields", "fields": missing}
    row = [[float(args[f]) for f in FEATURES]]
    return {"churn_probability": float(model.predict_proba(row)[0][1])}

In this sketch the endpoint serves a small scikit-learn model trained on the same three features, since a Spark PipelineModel needs a Spark session to score. Test it in the session first with predict({"monthly_spend": 42.5, "spend_change": -8.0, "tickets_last7_events": 3}), then go to Models, New Model, and enter predict.py as the file, predict as the function and the same JSON as the example input. Cloudera's walkthrough on creating and deploying a model adds the replica settings: the default maximum is 9 replicas per deployment, and an administrator can raise it to 199.

The serving rules behind the autoscaling questions:

• Every model function takes one JSON object and returns one JSON object.
• A replica handles one request at a time; extra requests wait in a queue.
• If no replica frees up within 30 seconds, the call fails with model.busy and HTTP 429, and the fix is more replicas.
• Callers need the model's access key and, with authentication on (the default), their own model API key.

Size limits matter too. The page on model request and response formats says requests should stay under 5 MB, dates should travel as ISO-8601 strings, and images or video should be passed by URL rather than inline.

What about the 42% that is not code?

The Cloudera Machine Learning domain (31%) is the platform itself: workspaces, projects, experiments, Accelerators for ML Projects (AMPs), Data Visualizations, Runtimes and GPUs. The fastest preparation is to do each once: create a project, start a session on an ML Runtime, pick a GPU resource profile if your workspace has one, launch an AMP, and build one dashboard in Data Visualization.

The last 11% is general machine learning: supervised versus unsupervised learning and the common algorithms. If you can say why the churn model above is supervised and when you would reach for clustering instead, you have the level of that domain.

To check yourself against the full topic list, the CDP-6001 syllabus and practice exam page repeats the five weights with sample questions.

Frequently asked questions

1. How many questions are on the CDP-6001 exam?

CDP-6001 has 45 questions in 90 minutes, with a pass score of 60%. It costs $330 and is taken online with a proctor.

2. Can I use documentation during the CDP-6001 exam?

No. Cloudera allows no resources: no reference material, white papers, user guides or anything else during the exam.

3. Which Spark skills does CDP-6001 test?

The Spark domain (18%) covers DataFrames, file types and window functions. The Spark MLlib domain (22%) covers model selection and tuning, fitting and evaluating models, and pipelines.

4. Does CDP-6001 cover MLflow?

Yes. ML Flow is listed under Deploying a Machine Learning Model (18%), next to Applications/API, Autoscaling and Performance, and Model Metrics and Monitoring.

5. How many questions can I miss on CDP-6001?

If every question counts the same, 27 correct answers out of 45 reach 60%, which leaves room for 18 misses. Cloudera publishes the pass mark only as a percentage.

Written by the team behind AnalyticsExam.