Comparing DataVow Vs CsvPath

DataVow and CsvPath Framework are fellow travelers. As with Soda.io, they are validation and contracts oriented. Also like Soda.io, at least to some degree, DataVow is focused on SQL. But its goal is to prevent bad data that may originate in files. In this it is like CsvPath. CsvPath is, of course, focused only on preboarding tabular data files. I.e. CSV, JSONL, Excel, and DataFrames.

DataVow's primary claims to fame, by my read of the project site, are:

• ODCS native
• Tightly integrated with dbt
• The ability to create CI-blocking data checks
• Useful for the project team, not just engineers

CsvPath Framework's goals are a bit broader, but overlap:

• Strong validation and upgrading to eliminate manual SME data checks
• Integrated with OpenLineage, OTEL, and a large number of backends and systems
• Edge Data Governance control at entry point, not just validation
• Acts as a permanent source of truth for inbound data

CsvPath Framework and DataVow both have Airflow operators and are intended for broad team use, not just data engineers. Both have a CLI. CsvPath Framework also powers FlightPath Data and FlightPath Server, open source GUI and no-code REST APIs-driven server, respectively.

Let's look at a simple example from DataVow's documentation. The data is a sample of orders. In DataVow, the docs example seems to be validating data housed in a database. CsvPath Framework handles inbound orders data as one of its typical use cases, but as files, not database tables. To that end, we'll create a toy orders file.

In DataVow:

apiVersion: datavow/v1
kind: DataContract
metadata:
  name: orders
  version: 1.0.0
  owner: [e-mail]
  domain: sales

schema:
  type: table
  fields:
    - name: order_id
      type: integer
      required: true
      unique: true
    - name: customer_email
      type: string
      required: true
      pii: true

quality:
  rules:
    - name: no_negative_totals
      type: sql
      query: "SELECT COUNT(*) FROM {table} WHERE total_amount < 0"
      threshold: 0
      severity: CRITICAL

Vs. in CsvPath:

Let's assume our orders CSV file looks like:

order_id, total_amount, customer_email
1, .99, [e-mail]
2, 5.89, [e-mail] 
3, 10.05, [e-mail] 

Our CsvPath validation would become

$[*][ 
  line.orders(
    integer.notnone.distinct(#order_id), 
    decimal.notnone(#total_amount, none(), 0),
    email.notnone(#customer_email)
  )
]

I.e. a little schema defining an orders entity. The none() in decimal.notnone(#total_amount, none(), 0) is saying we don't have an upper bound, only a lower one.

You could also create this same validation using only concise rules. But this is probably the most readable, while being more SQL-like. Or you could avoid that decision by downloading FlightPath and have its AI generate your validation.

So, very similar tools, in some ways. DataVow is narrowly focused on data contracts enforcement. CsvPath Framework is all about Edge Data Governance, including data contracts enforcement. ODCS is a core value add for DataVow. CsvPath doesn't yet support ODCS, but likely will later this year. At a high level, both tools validate data; CsvPath also manages data.

Data contracts validation is important. Particularly at the edge of the organization and in protecting the data lake or data warehouse. And importantly, data contracts management can reduce firefighting, protect customer relationships, and lower ops costs.

Check out DataVow and CsvPath Framework and, for the love of Pete, choose one of them! You'll be glad you did.