DEV Community

Cover image for Enhancing Google Cloud Data Quality Pipelines with Context-Aware Rule Discovery for Great Expectations with DataBuck SDK
Utkarsh Nihal
Utkarsh Nihal

Posted on

Enhancing Google Cloud Data Quality Pipelines with Context-Aware Rule Discovery for Great Expectations with DataBuck SDK

Use the DataBuck SDK to discover candidate rules from your data and policy documents, optionally review them, and run them as Great Expectations validations in a serverless Dataproc job on Google Cloud, orchestrated with Cloud Composer.

Google Cloud gives data teams strong building blocks for data quality: auto data quality scans in Knowledge Catalog (formerly Dataplex Universal Catalog) check BigQuery tables against built-in and SQL rules, and many teams also run Great Expectations (GX) on Spark. Both share one bottleneck: someone has to decide what "valid" means for every dataset.

A data profile shows what the data looks like, not what the business requires, such as a field a policy makes mandatory. That knowledge lives in policy documents. This article shows a pattern for bringing it into the pipeline: the DataBuck SDK reads the dataset and your policy PDFs and proposes candidate rules, you can review them and remove the ones you do not want, and DataBuck converts them into GX Expectations and runs them.

In one run on a 10,000-row transactions dataset, a single DataBuck.discover call produced 89 rules. The Dataproc job converted them to Great Expectations and ran them: 72 passed, 17 failed and one breached the quality gate.

The example reads from Cloud Storage, but the SDK discovers rules from a Spark DataFrame, so any source Spark on Dataproc can load will work. You will build a Dataproc Serverless batch that discovers rules, writes them to Cloud Storage for review and writes a per-rule validation report back, with an optional Cloud Composer DAG to schedule it.

1. How it works

DataBuck saves discovered rules as a JSON file. Each rule is a named SQL condition that is true when a row is valid, and a row violates a rule when its condition is false or NULL. Rules are grouped by what happens when they fail: fail, warn or drop.
Checks that need the whole dataset sit under dataset_checks and carry a description and the context they came from. The excerpt below is abridged from a discovery run on a banking transactions dataset; the expressions are verbatim.

{
  "fail": {
    "BuckGPT_Rule_005": "NOT COALESCE((customer_country_code <> 'US'), FALSE)"
  },
  "drop": {
    "not_null_transaction_date": "`transaction_date` IS NOT NULL",
    "BuckGPT_Rule_030": "NOT COALESCE((sar_filed_ind = 'Y' AND sar_reference_number IS NULL), FALSE)"
  },
"dataset_checks": [{
    "name": "BuckGPT_Rule_006",
    "description": "Ensures that each transaction has a unique identifier.",
    "reference_context": "BuckGPT_Intel: The transaction_id is the primary key for the transaction record and must be unique ...",
    "columns": ["transaction_id"],
    "sql": "SELECT `transaction_id`, COUNT(*) AS duplicate_count FROM {{DATAFRAME}} WHERE `transaction_id` IS NOT NULL GROUP BY `transaction_id` HAVING COUNT(*) > 1"
  }]
}

Enter fullscreen mode Exit fullscreen mode

What the DataBuck SDK does for you

• Discovers rules. It profiles the dataset and proposes rules in one call, from column checks to cross-field business rules.
• Uses your context. It takes a domain name and your policy PDFs, so rules can reflect what the business requires.
• Prepares rules for use. It groups rules by severity (fail, warn or drop) and saves them as a portable JSON file.
• Converts and runs them. It turns the rules into GX Expectations, runs them, applies a quality gate and reports what each rule became.

The flow has five steps. One batch runs them in one go, or in two runs with your review in between:

  1. Read the dataset. from Cloud Storage into a Spark DataFrame.
  2. Discover. Call DataBuck.discover with a context: the domain, the policy PDFs and a Gemini API key. The rules are written to Cloud Storage as a JSON file.
  3. Review and approve. A person opens the file, keeps the rules that fit, edits the ones that do not and removes the rest.
  4. Convert and run. DataBuck converts the approved rules into GX Expectations and runs them on the DataFrame.
  5. Report. The per-rule results are written to Cloud Storage.

Diagram

You need the following before you start.

• A Google Cloud project with billing enabled and permission to create Cloud Storage buckets, service accounts, Secret Manager secrets, Artifact Registry repositories and Dataproc batches. Enable the Dataproc, Artifact Registry, Cloud Build, Secret Manager and Compute Engine APIs (add Cloud Composer for the optional last step).
• The DataBuck SDK. This article uses the Python package databuck-spark-sdk (version 0.5.44) and its jar file, databuck-spark-sdk-slim.jar.
• Great Expectations 1.x. The converter uses the 1.x API, including data sources, validation definitions and Checkpoints.
• A Gemini API key, for the LLM-assisted discovery used in this article, which takes a domain name and policy documents as context.
• Data in Cloud Storage. A CSV, Parquet or JSON dataset to profile and validate, and optionally the policy PDFs that describe your domain.

3. Set up Google Cloud

The setup needs one Cloud Storage bucket, one service account, one secret and two network settings. Create the bucket with these prefixes.

GCS Bucket Directory Structure

Prefix Contents
data/ The dataset to validate (CSV, Parquet, or JSON)
policies/ Policy PDFs passed to rule discovery
jobs/ Job scripts and the helper library databuck_gx_lib.py
jars/ The DataBuck SDK JAR
rules/ Rules JSON, written by discovery or supplied by you
results/ Validation reports, with one folder per run
export PROJECT=<project-id> REGION=<region> BUCKET=<bucket>
gcloud storage buckets create gs://$BUCKET --project=$PROJECT --location=$REGION

Enter fullscreen mode Exit fullscreen mode

The batch runs as a service account that you create. Give it the following roles.

gcloud iam service-accounts create dataproc-batch-sa --project=$PROJECT
SA=dataproc-batch-sa@$PROJECT.iam.gserviceaccount.com
for r in roles/dataproc.worker roles/storage.objectAdmin \
         roles/secretmanager.secretAccessor roles/artifactregistry.reader; do
  gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA --role=$r
done
Enter fullscreen mode Exit fullscreen mode

The person who submits the batch also needs the Service Account User role on dataproc-batch-sa. Without it, batch creation fails.
Store the Gemini API key in Secret Manager. The job reads it at run time, so the key never appears in the script, the batch arguments or the logs. Check afterwards that the secret has an enabled version; a secret with no versions returns a "not found or has no versions" error.

printf '%s' '<your-gemini-key>' | gcloud secrets create gemini-api-key --data-file=- --project=$PROJECT

Enter fullscreen mode Exit fullscreen mode

Two network settings matter for a Dataproc Serverless batch.
• Private Google Access must be on for the subnet the batch uses. Workers have no external IP, and this setting lets them reach Cloud Storage and Secret Manager. The subnet must also allow traffic between its own instances, which the default network does out of the box.
• Cloud NAT is needed for discovery. The SDK calls the Gemini API on the public internet, and workers without a NAT gateway cannot reach it. A run that only converts and runs an approved rules file makes no external calls and needs no NAT.

4. Build the runtime image and attach dependencies

Dataproc Serverless has no job parameter that installs pip packages when a batch starts, so you build a custom container image once and point every batch at it. The packages install at build time, so the running batch needs no PyPI access.
List the packages in a requirements.txt next to the Dockerfile.

great-expectations==<GX version>
databuck-spark-sdk==0.5.44
pyyaml
google-cloud-storage
google-cloud-secret-manager
Enter fullscreen mode Exit fullscreen mode

Pin one exact GX version, and do not install pyspark, because the Dataproc runtime provides it. Install the SDK as databuck-spark-sdk and import it as databuck.

FROM debian:12-slim
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
      procps tini libjemalloc2 wget ca-certificates \
    && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

ENV CONDA_HOME=/opt/miniforge3
RUN wget -q https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O /tmp/mf.sh \
    && bash /tmp/mf.sh -b -p ${CONDA_HOME} && rm /tmp/mf.sh
ENV PATH=${CONDA_HOME}/bin:${PATH}
ENV PYSPARK_PYTHON=${CONDA_HOME}/bin/python

COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt

RUN groupadd -g 1099 spark && useradd -u 1099 -g 1099 -d /home/spark -m spark
USER spark
Enter fullscreen mode Exit fullscreen mode
gcloud artifacts repositories create databuck-gx --repository-format=docker --location=$REGION
gcloud builds submit --tag $REGION-docker.pkg.dev/$PROJECT/databuck-gx/gx-spark:latest .
Enter fullscreen mode Exit fullscreen mode

The remaining files are attached to the batch, not baked into the image.

Dependencies and How They Are Attached

Component How It Is Attached
GX, YAML parsing, and the SDK Installed in the container image
Converter and validator library (databuck_gx_lib.py) --py-files gs://<bucket>/jobs/databuck_gx_lib.py
SDK JAR on the Spark classpath --jars with the JAR in jars/ (see Section 5)
Local copy of the JAR that the SDK finds through its environment variable Job argument --sdk_jar_gcs_path, pointing to the same GCS path

DataBuck's GX integration library, databuck_gx_lib.py, converts approved DataBuck rules into GX Expectations and runs them. Upload it to jobs/ as a single file.

5. Discover rules with the DataBuck SDK

DataBuck.discover takes a Spark DataFrame, an output path for the rules file and a context. The context carries the domain name, the policy PDFs and the Gemini API key.

from databuck import DataBuck

context = {
    "gemini_api_key": get_gemini_key("gemini-api-key"),  # read from Secret Manager
    "domain_name": "banking",
    "pdf_paths": local_pdf_paths,                        # downloaded from Cloud Storage
}
DataBuck.discover(df, "/tmp/databuck/aml_rules.json", context=context)

Enter fullscreen mode Exit fullscreen mode

On Dataproc, the job prepares three things around that call:
• Policy PDFs. The SDK takes local file paths, so the job first downloads every PDF under the policies/ prefix to the driver's disk.
• The jar. The SDK locates its jar through the DATABUCK_SPARK_SDK_JAR environment variable. The job downloads the jar from Cloud Storage and sets the variable before it imports databuck.
• The output file. Discovery writes to a local path on the driver. The job then uploads the file to --rules_path in Cloud Storage and reads it back, so the file in Cloud Storage is exactly what GX runs.

What discovery returns

On the transactions dataset, discovery produced 88 row rules (1 in fail, 87 in drop) and one dataset check. Most are column checks: 42 not-null, 13 pattern and 4 length rules. The 29 rules that relate columns to each other are the more useful ones. These are some of them, with the number of rows each flagged out of 10,000:

Data Validation Rules and Results

Rule What It Enforces Rows Flagged
BuckGPT_Rule_026 (drop) For a credit, closing balance equals opening balance plus amount minus fee 86
BuckGPT_Rule_021 (drop) For non-USD transactions, the USD amount equals the amount multiplied by the exchange rate 42
BuckGPT_Rule_028 (drop) If an alert was generated, an alert ID must exist and the alert status must not be NOT_APPLICABLE 25
BuckGPT_Rule_030 (drop) A filed SAR must have a SAR reference number 25
BuckGPT_Rule_024 (drop) A transaction between different countries must be marked as cross-border 22

Use the slim SDK jar
Use databuck-spark-sdk-slim.jar, the build of the SDK jar that is provided for Dataproc. Upload it to the jars/ prefix and pass it twice: with --jars, so that it is on Spark's classpath, and with --sdk_jar_gcs_path, so that the SDK finds its local copy.

6. Review the rules (optional)

Discovery produces candidate rules. A review is optional, but it is the quickest way to remove rules you do not need. Without it, the job runs the discovered rules straight away. To review, run discovery on its own, then run the job again from the approved file.

  1. Run the job with --rules_source=sdk and --rules_path set to a file under rules/. The job writes the discovered rules there, then converts and runs them so that you get a first report.
  2. Open the rules file in Cloud Storage. Keep the rules that match your intent, correct expressions or thresholds that are off, and delete the rules you do not want. A rule's group (fail, warn or drop) decides whether a violation blocks the pipeline, so move a rule to another group if its severity is wrong.
  3. Save the approved file, for example under rules/approved/. It is a plain JSON file, so you can keep it in version control and review changes to it like code.

Then run the job again with--rules_source=gcs and --rules_path pointing at the approved file. That run needs no SDK jar and no Gemini key, and every rule it runs is one that a person has approved.

A rule can be well formed and still wrong for your data or policy. In our run, BuckGPT_Rule_003 expected the transaction fee to be between 2% and 5% of the amount and flagged 9,929 of 10,000 rows. A rule that flags almost everything is a good candidate to correct or remove. Only a person who knows the policy can make that call, which is why the rules file is plain JSON that you can keep in version control and review like code.

Open item: add a --discover_only option that stops after discovery, so that the first run does not validate unreviewed rules.

7. Convert the approved rules and run them

One class, DataBuckGXValidator, converts the approved rules into GX Expectations and runs them on the DataFrame. With the builtin engine, a rule becomes a native GX Expectation only when the translation is provably identical to running its SQL.

validator = DataBuckGXValidator(rules=rules_doc, config=GateConfig())
report = validator.run(df, engine="builtin")

A NULL result counts as a violation, as it does in DataBuck. Rules that cannot be translated exactly into a native Expectation run as custom SQL, and the report records why.

Severity maps from DataBuck to GX as fail to critical, warn to warning and drop to info. The quality gate blocks by default when a failrule has any failing row, when a dataset check fails, or when a fail rule cannot be evaluated. warn and droprules never block.
Choose how the rules execute with --engine. Use builtin for readable native Expectations, or boolean_columns when you have many complex SQL rules and efficient execution matters most.

8. Create and run the Dataproc Serverless batch

A Dataproc Serverless batch is one run. You cannot edit a submitted batch, so every change is a new batch with a new ID. Submit it from the console (Dataproc, Serverless, Batches, Create) or with gcloud, using the settings below.

The script reads its configuration from batch arguments, written as --name=value.

Job Arguments and Example Values

Argument Example Value
--source_path gs://<bucket>/data/aml.csv
--output_path gs://<bucket>/results
--rules_source sdk to discover rules, gcs to use an approved file
--rules_path gs://<bucket>/rules/aml_rules.json
--dataset_name aml
--domain_name banking
--pdf_gcs_prefix gs://<bucket>/policies/
--gemini_secret_id gemini-api-key (a secret name or a full resource path)
--sdk_jar_gcs_path gs://<bucket>/jars/databuck-spark-sdk-slim.jar
--engine builtin
--fail_on_breach false (set to true to fail the run on a breach)

The same submission as one command:

gcloud dataproc batches submit pyspark gs://$BUCKET/jobs/databuck_gx_poc.py \
  --project=$PROJECT --region=$REGION --batch=gx-poc-001 --version=2.3 \
  --container-image=$REGION-docker.pkg.dev/$PROJECT/databuck-gx/gx-spark:latest \
--py-files=gs://$BUCKET/jobs/databuck_gx_lib.py \
  --jars=gs://$BUCKET/jars/databuck-spark-sdk-slim.jar \
  --subnet=default --service-account=$SA \
  --user-workload-authentication-type=SERVICE_ACCOUNT \
  -- --rules_source=sdk \
     --source_path=gs://$BUCKET/data/aml.csv \
     --output_path=gs://$BUCKET/results \
     --dataset_name=aml --domain_name=banking \
     --pdf_gcs_prefix=gs://$BUCKET/policies/ \
     --gemini_secret_id=gemini-api-key \
     --sdk_jar_gcs_path=gs://$BUCKET/jars/databuck-spark-sdk-slim.jar

Enter fullscreen mode Exit fullscreen mode

With --rules_source=gcs, the job reads an approved file instead of discovering, so it needs no SDK jar and no Gemini key. The smaller databuck_gx_convert_run.py does only that second job.

9. Orchestrate with Cloud Composer (optional)

Cloud Composer is managed Apache Airflow. It decides when and in what order jobs run, retries failures and keeps a run history, while Dataproc does the processing. You can skip it and submit batches by hand, but it is the natural next step when this job becomes part of a larger pipeline, for example "validate after the data lands, and load only if the quality gate passes".

The DAG databuck_gx_validation has a mode parameter. A branch task reads it and runs exactly one of two jobs: convert_runstarts databuck_gx_convert_run.py on an existing rules file, and discover_run starts databuck_gx_poc.py, which discovers the rules first. Each job is one DataprocCreateBatchOperatortask, and the DAG parameters mirror the job arguments from section 8.

Set it up in four steps.

  1. Create a service account for Composer, composer-sa, with the Composer Worker and Dataproc Editor roles. Give it Service Account User on dataproc-batch-sa, so that it can start batches as that account.
  2. Create a Composer 3 environment (listed as Managed Airflow Gen 3 in some consoles) with Standard resilience, the Small size and composer-sa as its service account. Creation takes 20 to 30 minutes.
  3. Upload the DAG file to the environment's DAGs folder, and unpause the DAG in the Airflow UI.
  4. Trigger the DAG with a config: choose the mode and adjust the paths.

10. Read the results

Each run writes three files to results/<dataset>/<timestamp>/ in Cloud Storage: dq_report_<run id>.json, the same report as dq_report_<run id>.csv, and run_summary.json with the counts and any fallback reasons. The discovered rules are in the file at --rules_path.

The report has one row per rule.

Column Meaning
rule The rule name from the DataBuck rules file
kind row_rule or dataset_check
sdk_severity fail, warn, drop, or dataset_check
status PASS, FAIL, or ERROR (the rule could not be evaluated)
total_rows Total number of rows checked
invalid_rows Number of rows that violate the rule
invalid_pct Percentage of rows that violate the rule
gate_breach Indicates whether this rule breaches the quality gate
gx_expectation The GX Expectation that the rule was converted into
error, sample Error text for an ERROR row and a few violating rows for a failed dataset check

In a run on 10,000 transaction rows with 89 rules (88 row rules and one dataset check), 72 rules passed, 17 failed and none errored. Of the 88 row rules, 42 became native ExpectColumnValuesToNotBeNull Expectations and 46 ran as custom SQL, with the reason recorded for each. One rule breached the quality gate: the duplicate-transaction dataset check, which found 25 transaction IDs that appear twice. drop failures are reported without blocking.

Selected rows from the run report:

Rule Kind Severity Status Invalid Rows Gate Breach
not_null_transaction_date row_rule drop PASS 0 No
valid_pattern_ip_address row_rule drop FAIL 360 (3.6%) No
BuckGPT_Rule_003 row_rule drop FAIL 9,929 (99.29%) No
BuckGPT_Rule_005 row_rule fail PASS 0 No
BuckGPT_Rule_006 dataset_check dataset_check FAIL 25 Yes

Where this fits alongside Google Cloud data quality
This pattern complements Google Cloud's data quality tooling. Auto data quality in Knowledge Catalog scans BigQuery tables with managed built-in and custom SQL rules, and can recommend rules from a data profile scan. If those rules cover your needs, keep them.
Use this pattern when your team has standardized on GX for Spark, or when you need rules that reflect your policy documents and not only the data, with the option to review every rule first. See Google's documentation for auto data quality and data quality scans in BigQuery.

Conclusion

Writing rules by hand is the slow part of data quality, and statistics alone leave out what the business requires. The DataBuck SDK discovers candidate rules from your data and policy documents, you can review them, and DataBuck runs them as GX Expectations, all in one Dataproc Serverless batch. A discovered rule is still a candidate, so the review step is where your domain knowledge decides what goes into production. To try it, run discovery on one dataset and read the rules it proposes.

Source code and documentation

  • databuck_gx_poc.py: the batch script that discovers rules, converts them and runs them in GX, or starts from an approved rules file.
  • databuck_gx_convert_run.py: a smaller script that only converts and runs an approved rules file.
  • databuck_gx_lib.py: the converter and validator library that both scripts import.
  • databuck_gx_dag.py: the optional Cloud Composer DAG.
  • Dockerfile and requirements.txt: the runtime image.

Source code:

https://github.com/angsumandutta/databuck-gx-bridge/tree/main/GCP_dataproc_workflow.

Frequently asked questions

Do I have to review the discovered rules before they run?

No. Review is optional. Without it, the job converts and runs the discovered rules straight away. If you want a review, run discovery on its own, edit the rules file in Cloud Storage, then run the job again from the approved file.

Does this work only with data in Cloud Storage?

The example reads from Cloud Storage, but the DataBuck SDK discovers rules from a Spark DataFrame. Any source that Spark on Dataproc can load into a DataFrame can use the same approach.

Do I need Cloud Composer?

No. You can submit the Dataproc Serverless batch from the console or with gcloud. Composer is useful when you want to schedule the job, trigger it from other pipelines or choose between the two modes at trigger time.

What leaves my environment during discovery?

Discovery sends content from your policy documents to the LLM API that your Gemini key belongs to, so apply your organization's data-handling rules before you use it with sensitive documents. A run from an approved rules file needs no Gemini key and makes no LLM calls.

How is this different from Google Cloud auto data quality?

It complements it. Auto data quality scans BigQuery tables with managed rules. This pattern runs Great Expectations on Spark, and it discovers rules from your policy documents as well as from the data.

About DataBuck

DataBuck is the data quality platform from FirstEigen. The DataBuck SDK, used in this article, brings its automated rule discovery to Spark workloads such as Dataproc batches. Learn more at firsteigen.com.

Disclosure: FirstEigen develops DataBuck. This article describes a proof of concept built by the author.

Top comments (0)