DEV Community: Wei Lee

DVC - Pipeline Versioning

Wei Lee — Fri, 02 Jul 2021 10:51:07 +0000

This post also lives on GitHub

We've versioned our data in the previous post. This article will demonstrate how we could define a data pipeline and version it through DVC.

Pipeline versioning

We'll continue using dvc_example. You can checkout to tag v3-remove-2-rows to follow along.

Split training logic into different stages

In the original design, we use pipenv run python digit_recognizer/digit_recognizer.py to run the whole training process. We'll split them into process-data, train, and report stages.

def main():
    ......

    if args.command == "process-data":
        X, y = load_data("data/digit_data.csv", "data/digit_target.csv")
        X_train, X_test, y_train, y_test = process_data(X, y)
        export_processed_data((X_train, y_train), "output/training_data.pkl")
        export_processed_data((X_test, y_test), "output/testing_data.pkl")
    elif args.command == "train":
        X_train, y_train = load_processed_data("output/training_data.pkl")
        model = train_model(X_train, y_train)
        export_model(model, "output/model.pkl")
    elif args.command == "report":
        X_test, y_test = load_processed_data("output/testing_data.pkl")
        model = load_model("output/model.pkl")
        predicted_y = model.predict(X_test)
        output_test_data_results(y_test, predicted_y)
        output_metrics(y_test, predicted_y)

You can view the complete code change on v3-remove-2-rows...v4-split-pipeline-logic.

After these changes, we'll use the following 3 commands to run the pipeline.

pipenv run python digit_recognizer/digit_recognizer.py process-data
pipenv run python digit_recognizer/digit_recognizer.py train
pipenv run python digit_recognizer/digit_recognizer.py report

Add the first stage in our pipeline to DVC

We add stages through dvc run command. Let's add our first stage process-data.

# add process-data stage
$ pipenv run dvc run --name process-data \
      -d digit_recognizer/digit_recognizer.py \
      -d data/digit_data.csv \
      -d data/digit_target.csv \
      -o output/training_data.pkl \
      -o output/testing_data.pkl \
      "pipenv run python digit_recognizer/digit_recognizer.py process-data"

Running stage 'process-data':
> pipenv run python digit_recognizer/digit_recognizer.py process-data
Creating 'dvc.yaml'
Adding stage 'process-data' in 'dvc.yaml'
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'

To track the changes with git, run:

git add dvc.yaml output/.gitignore dvc.lock


Next, we add these DVC files into git to track.

# add DVC configuration to git and commit
$ git add dvc.yaml dvc.lock output/.gitignore
$ pipenv run cz commit

See what's composed of this command

--name: the name of this stage
- It must be unique throughout the project.
-d: the dependencies of this stage
- All the files related to running this stage should be counted as dependencies.
- DVC won't these dependency files into it storage but only store the hashes of them.
- In this example, we need digit_recognizer/digit_recognizer.py to load data/digit_data.csv and data/digit_target.csv to process the data. Thus, these 3 files are added as dependencies.
-o: the output files of this stage
- DVC stores these files in its storage. If you want to track it through git or simply don't want to track it, you can use -O instead.

dvc run runs the stage right after adding it. If you don't want DVC to run it, you can add --no-exec flag or dvc stage add with the same arguments

After adding a stage, DVC updates dvc.yaml, output/.gitignore and dvc.lock

`dvc.yaml`: the definition of stages

stages:
  process-data:
    cmd: pipenv run python digit_recognizer/digit_recognizer.py process-data
    deps:
    - data/digit_data.csv
    - data/digit_target.csv
    - digit_recognizer/digit_recognizer.py
    outs:
    - output/testing_data.pkl
    - output/training_data.pkl

DVC transforms what we defined in dvc run to a human-readable format and store it. But if you already know how to define the stage, you can edit dvc.yaml directly. In addition, there're advanced techniques like Templating and foreach stages that can help us define complicated stages.

`dvc.lock`: the hashes of dependencies and outputs

schema: '2.0'
stages:
  process-data:
    cmd: pipenv run python digit_recognizer/digit_recognizer.py process-data
    deps:
    - path: data/digit_data.csv
      md5: 942481fce846fb9750b7b8023c80a5ef
      size: 490582
    - path: data/digit_target.csv
      md5: 2a6cfa13365ac9b3af5146133aca6789
      size: 3590
    - path: digit_recognizer/digit_recognizer.py
      md5: 65ecf27479538a74ade42462b1566db1
      size: 3629
    outs:
    - path: output/testing_data.pkl
      md5: 78be1761d227f71b1a8f858fed766982
      size: 529016
    - path: output/training_data.pkl
      md5: f95e8f978a05395ba23479ff60eda076
      size: 528427

DVC uses these hashes to know whether there's any modification on our stages. Therefore, we only add deterministic files. Randomness might make this lock file meaningless. Take a look at the "Avoiding unexpected behavior" in dvc run - Description could save your time debugging unexpected failure.

`output/.gitignore`: Add files that DVC should track to gitignore

Define the whole pipeline

With similar command, we can add train and report stages to our pipeline.

# add train stage
pipenv run dvc run --name train \
      -d digit_recognizer/digit_recognizer.py \
      -d output/training_data.pkl \
      -o output/model.pkl \
      "pipenv run python digit_recognizer/digit_recognizer.py train"

# add report stage
pipenv run dvc run --name report \
      -d digit_recognizer/digit_recognizer.py \
      -d output/testing_data.pkl \
      -d output/model.pkl \
      -o output/metrics.json \
      -o output/test_data_results.csv \
      "pipenv run python digit_recognizer/digit_recognizer.py report"

# add DVC configuration to git and commit
git add dvc.yaml dvc.lock model/.gitignore
pipenv run cz commit

Similar to our previous step, DVC updates dvc.yaml, dvc.lock and output/.gitignore.

$ cat dvc.yaml

...
  train:
    cmd: pipenv run python digit_recognizer/digit_recognizer.py train
    deps:
    - digit_recognizer/digit_recognizer.py
    - output/training_data.pkl
    outs:
    - output/model.pkl
  report:
    cmd: pipenv run python digit_recognizer/digit_recognizer.py report
    deps:
    - digit_recognizer/digit_recognizer.py
    - output/model.pkl
    - output/testing_data.pkl
    outs:
    - output/metrics.json
    - output/test_data_results.csv

We can visualize the pipeline through dvc dag

$ pipenv run dvc dag

    +----------+
    | data.dvc |
    +----------+
          *
          *
          *
  +--------------+
  | process-data |
  +--------------+
     **        **
   **            *
  *               **
+-------+           *
| train |         **
+-------+        *
     **        **
       **    **
         *  *
     +--------+
     | report |
     +--------+

If you pay attention to each parameter passes to dvc run, you might have noticed that train stage depends on the output output/training_data.pkl from process-data stage. This is how DVC decides the order of each stage in our pipeline.

Run the pipeline

Contradict to its naming, dvc run is only used for defining the stage and run it for the first time. dvc repro (reproduce) is what we use to run the pipeline,

$ pipenv run dvc repro

'data.dvc' didn't change, skipping
Stage 'train' didn't change, skipping
Data and pipelines are up to date.

Because we've not yet made any changes since we define our pipeline, DVC won't waste time and resources to generate results it has already known. However, you can add a -f flag to force DVC to rerun the pipeline.

Next, we'll change gamma to 0.01 to see how dvc repro works.

def train_model(X_train, y_train, params):
    ...
    clf = svm.SVC(gamma=0.01)
    ...

Because our dependency digit_recognizer/digit_recognizer.py has been modified, DVC expects the result might be different. Therefore, we can now run dvc repro.

$ pipenv run dvc repro

'data.dvc' didn't change, skipping
Running stage 'process-data':
> pipenv run python digit_recognizer/digit_recognizer.py process-data
Updating lock file 'dvc.lock'

Running stage 'train':
> pipenv run python digit_recognizer/digit_recognizer.py train
Updating lock file 'dvc.lock'

Running stage 'report':
> pipenv run python digit_recognizer/digit_recognizer.py report
Updating lock file 'dvc.lock'

To track the changes with git, run:

git add dvc.lock
Use `dvc push` to send your updates to remote storage.

By running git diff, you'll find out that the hashes of digit_recognizer/digit_recognizer.py, output/model.pkl, output/metrics.json, output/test_data_results.csv inside dvc.lock has been changed.

Track parameters

In the previous section, even though we change only the parameter related to the train stage, DVC still reruns the whole pipeline. To make DVC runs only the stages affect by the changed parameters, we can refactor our code to load parameters from a separate file params.yaml.

def main():
    params = load_params("params.yaml")
    X, y = load_data("data/digit_data.csv", "data/digit_target.csv")
    X_train, X_test, y_train, y_test = process_data(
        X, y, params["process_data"]
    )

    model = train_model(X_train, y_train, params["train"])
    export_model(model)
    ......

This is how params.yaml looks like.

process_data:
  test_size: 0.5
  shuffle: false
train:
  gamma: 0.01

Full code changes can be found on v5-parameters-in-separate-file.

To add parameters to a stage, we'll need to run the previous dvc run command again with -f and -p flag.

-f: overwrite the stage with the same name
-p: parameters
- Use "," to separate parameters

# Add parameters process_data.test_size and process_data.shuffle to process-data stage
pipenv run dvc run -f --name process-data \
      -d digit_recognizer/digit_recognizer.py \
      -d data/digit_data.csv \
      -d data/digit_target.csv \
      -o output/training_data.pkl \
      -o output/testing_data.pkl \
      -p process_data.test_size,process_data.shuffle \
      "pipenv run python digit_recognizer/digit_recognizer.py process-data"

# Add parameters train.gamma to train stage
pipenv run dvc run -f --name train \
      -d digit_recognizer/digit_recognizer.py \
      -d output/training_data.pkl \
      -o output/model.pkl \
      -p train.gamma \
      "pipenv run python digit_recognizer/digit_recognizer.py train"  

# add DVC configuration to git and commit
git add dvc.yaml dvc.lock model/.gitignore
pipenv run cz commit

DVC adds params key to both process-data and train stages in dvc.yaml.

stages:
  process-data:
    ......
    params:
    - process_data.shuffle
    - process_data.test_size
  train:
      ......
    params:
    - train.gamma

params.yaml is the default parameter file name, but DVC also supports YAML, JSON, TOML, and Python files. We only need to add the file name as an additional layer to params to use it. e.g.,

  # this is an example of using different parameter file name
  # we don't need to make changes to our code
  train:
      ......
    params:
    - params.json
      - train.gamma

Let's change gamma to 0.1. We can check this change through dvc params diff. By providing git reference, we can even see parameters difference between different git commits. (e.g., dvc params diff HEAD~1)

$ pipenv run dvc params diff

Path     Param        Old    New
params.yaml  train.gamma  0.01   0.1

If we run dvc repro now, DVC reruns only train and report stages. train stage is affected by train.gamma change. Due to this change, the output file from the train stage has been updated. Thus, DVC reruns report stages as well.

$ pipenv run dvc repro

'data.dvc' didn't change, skipping
Stage 'process-data' didn't change, skipping
Running stage 'train':
> pipenv run python digit_recognizer/digit_recognizer.py train
Updating lock file 'dvc.lock'

Running stage 'report':
> pipenv run python digit_recognizer/digit_recognizer.py report
Updating lock file 'dvc.lock'

To track the changes with git, run:

    git add dvc.lock
Use `dvc push` to send your updates to remote storage.

# reset gamma back to 0.01
$ git checkout dvc.lock params.yaml

We're not going to store this parameter change. Run git checkout out params.yaml dvc.lock to restore the previous state.

Track metrics

We now know how to track parameters. Next, we'll see how changing these parameters affect the performance of our models. You may have already noticed that we've outputted a output/metrics.json file. Although we could track it as the output file, DVC has better support for metrics files.

Like adding parameters, we add -m flag for DVC to recognize the output as metrics. Instead of using -M as the official tutorial did, I use -m because I prefer tracking metrics through DVC remote storage instead of saving it to git as part of our source code.

# Add output/metrics.json as metrics to report stage
$ pipenv run dvc run -f --name report \
      -d digit_recognizer/digit_recognizer.py \
      -d output/testing_data.pkl \
      -d output/model.pkl \
      -o output/test_data_results.csv \
      -m output/metrics.json \
      "pipenv run python digit_recognizer/digit_recognizer.py report"

# add DVC configuration to git and commit
$ git add dvc.yaml dvc.lock model/.gitignore
$ pipenv run cz commit

# metrics have been added to the report stage as expected
$ cat dvc.yaml

...
  report:
    ......
    metrics:
    - metrics.json:

Use dvc metrics show to see how well our model performs
Note that values are not calculated through DVC. DVC only provides a way to display values in file organized as tree hierarchies and compare them throughout different git commits.

$ pipenv run dvc metrics show

Path             accuracy_score    weighted_f1_score    weighted_precision    weighted_recall
output/metrics.json  0.69265       0.74567              0.91941               0.69265

Change gamma to 0.1 again and use dvc metrics diff to see if model performance is improved after this change.

# reruns the pipeline with new parameters
$ pipenv run dvc repro

# check metrics differences between unstaged and HEAD
$ pipenv run dvc metrics diff

Path             Metric              Old      New      Change
output/metrics.json  accuracy_score  0.69265  0.10134  -0.59131
output/metrics.json  weighted_f1_score   0.74567  0.01865  -0.72702
output/metrics.json  weighted_precision  0.91941  0.01027  -0.90914
output/metrics.json  weighted_recall 0.69265  0.10134  -0.59131

# reset gamma back to 0.01
$ git checkout dvc.lock params.yaml

We don't need this change either. Reset gamma back to 0.01 through git checkout

plotting

There's only one left output output/test_data_results.csv that has not yet been used. This file stores the ground truth and the predicted result from our model. We're going to use it to see how DVC plots our data. Before plotting, let's change gamma to 0.001 first and run dvc repro. Otherwise, the output plot will look a bit odd due to the low model performance.

$ cat output/test_data_results.csv

actual,predicted
4.0,4.0
8.0,8.0
......

Add --plots flag and specify output/test_data_results.csv as the file to plot

# add output/test_data_results.csv as the file to plot to report stage
$ pipenv run dvc run -f --name report \
      -d digit_recognizer/digit_recognizer.py \
      -d output/testing_data.pkl \
      -d output/model.pkl \
      -o output/test_data_results.csv \
      -m output/metrics.json \
      --plots output/test_data_results.csv \
      "pipenv run python digit_recognizer/digit_recognizer.py report"

# plots have been added to dvc.yaml
$ cat dvc.yaml
  ......
  plots:
  - output/test_data_results.csv

DVC generates plots through Vega, a declarative grammar that can define interactive graph in JSON format. It supports linear plot, scatter plot, and confusion matrix by default. These templates are stored in .dvc/plots. We can also define our plots. (Read
dvc plots - Custom templates to find out more)

In the following example, we'll plot a confusion matrix through dvc plots show.

$ pipenv run dvc plots show output/test_data_results.csv --template confusion -x actual -y predicted --out confusion_matrix.html

file:///....../confusion_matrix.html

--template: name of the plot template
-x: field name of the data for the X-axis
-y: field name of the data for the y axis
--out: output file name

The following is a screenshot of the generated plot.

As of now, DVC does not track our plot (i.e. confusion-matrix.jpg) but only our data to plot (i.e., output/test_data_results.csv). Let's add plot as the final stage of our pipeline.

# Add stage plot
pipenv run dvc run -f --name plot \
          -d output/test_data_results.csv \
          -o confusion_matrix.html \
          "pipenv run dvc plots show output/test_data_results.csv --template confusion -x actual -y predicted --out confusion_matrix.html"

Conclusion

In this post, we create a data pipeline that process data, train the model, generate the report and visualize it.

$ pipenv run dvc dag

       +----------+
       | data.dvc |
       +----------+
             *
             *
             *
     +--------------+
     | process-data |
     +--------------+
         *        *
       **          *
      *             **
+-------+             *
| train |           **
+-------+          *
         *        *
          **    **
            *  *
        +--------+
        | report |
        +--------+
             *
             *
             *
         +------+
         | plot |
         +------+

We also see how to use DVC to track each component and provide an easy way to run the pipeline. In the following article, we will discuss how to run experiments with different parameters and compare the results in an even more convenient way.

One more thing: When should we save files to DVC instead of git?

Short answer: It depends.

When defining pipeline we can decide whether to save our outputs (-o / -O), metrics (-m / -M) and plots (--plots / --plots-no-cache) to DVC storage. DVC document suggests not storing metrics and plots to DVC as they are typically small enough for git to track. But I'd prefer storing only thing relates to our logic to git. That's why I use -m and --plots in the examples. If you don't want to track these, you could just pass -O, -M, or --plots-no-cache and add them to both .gitignore and .dvcignore.

Reference

DVC - Data Versioning

Wei Lee — Fri, 02 Jul 2021 10:49:33 +0000

This article also lives on GitHub

About DVC (Data Version Control)

What's DVC?
- version control system for data science and machine learning
- compatible with git (it's based on git)
What can DVC do?
- track
  - data
  - model
  - pipeline
  - metrics
- use storage directly
- no external services needed
Who are the targeted users of DVC?
- ML research / engineer
- DevOps & Engineers
Why DVC?
- It links your data, model, and pipelines with your metrics.
  - reproducibility
  - trackable

Read DVC - Versioning Data and Models for more use cases

How to use DVC?

Install DVC globally

I suggest using pipx if you're to install DVC globally. However, an even better way is to install it inside the virtual environment within your project.

$ pip install pipx
$ pipx install dvc
$ dvc --version

2.3.0

DVC also provides Shell Completion and Syntax Highlighting Plugins for popular editors.

Take a look at the example project

I'll use dvc_example to demonstrate how I applied DVC to an existing machine learning project. The example is based on Recognizing hand-written digits from scikit-learn documentation. All the DVC parts start from v1-base. You can git checkout to the tag to follow along.

$ git clone https://github.com/Lee-W/dvc_example/ --branch v1-base
$ cd dvc_example
$ tree
.
├── LICENSE
├── Pipfile
├── Pipfile.lock
├── digit_recognizer
│   ├── __init__.py
│   └── digit_recognizer.py
├── docs
│   └── README.md
├── mkdocs.yml
├── output
└── tasks.py

To set up the development environment, you'll need pipenv and invoke. If you run into an error when running pipenv install, you can run export SYSTEM_VERSION_COMPAT=1 before it. It's an open issue (Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip? #4564) of pipenv as of now. Or, you can just run the following commands.

# install needed tools
pipx install pipenv invoke

# set up environments
invoke init-dev

We'll use digit_recognizer/digit_recognizer.py for training a model that can recognize handwritten digits.

def main():
    X, y = load_data()
    X_train, X_test, y_train, y_test = process_data(X, y)
    model = train_model(X_train, y_train)
    predicted_y = model.predict(X_test)
    output_results(y_test, predicted_y)
    output_metrics(y_test, predicted_y)

Install DVC into the virtual environment

pipenv install dvc

If you're to save data to remote storage, you might need to install extra dependencies.
(e.g., pipenv install dvc[s3])

Supported types
- [s3]
- [azure]
- [gdrive]
- [gs]
- [oss]
- [ssh]

Or, use pipenv install dvc[all] to install them all

Read dvc remote for more information

Initialize DVC

# initialize DVC configurations
$ pipenv run dvc init

# see what's created by DVC
$ tree .dvc

.dvc
├── config
└── plots
    ├── confusion.json
    ├── confusion_normalized.json
    ├── default.json
    ├── linear.json
    ├── scatter.json
    └── smooth.json

# track DVC configuration through git
$ git add .dvc

# git commit
$ pipenv run cz commit

Add DVC remote

I'll use another local directory ../dvc_remote as our remote storage. You can change it to s3 or other remote storage.

mkdir ../dvc_remote
dvc remote add --default local ../dvc_remote

Through --default flag, we can push/pull from local remote without specifying remote name.

Let see what's changed in .dvc/config.

$ cat .dvc/config

[core]
remote = local
['remote "local"']
url = ../../dvc_remote

The url is ../../dvc_remote instead of ../dvc_remote because it's the relative path to .dvc. As we've not yet push anything to our pseudo remote, ../dvc_remote is still empty.

Track data through DVC

By this time, the data is loaded through sklearn.datasets.load_digits. We're going to change it to read from static file in data/.

def load_data():
    # Load data
    digits = datasets.load_digits()
    ...

We can use the following script to output the digit data into data/. Note that it's a one-time use script. We won't add it into git.

import os

import pandas as pd
from sklearn import datasets

os.mkdir("data")

digits = datasets.load_digits()

df = pd.DataFrame(digits.data)
df.to_csv("data/digit_data.csv", header=False, index=False)

df = pd.DataFrame(digits.target)
df.to_csv("data/digit_target.csv", header=False, index=False)

We'll need to make changes to load_data and main functions to read data from these files.

def load_data(X_path, y_path):
    with open(X_path) as input_file:
        csv_reader = csv.reader(input_file, quoting=csv.QUOTE_NONNUMERIC)
        X = list(csv_reader)

    with open(y_path) as input_file:
        csv_reader = csv.reader(input_file, quoting=csv.QUOTE_NONNUMERIC)
        y = [row[0] for row in csv_reader]

    return X, y

......

def main():
    X, y = load_data("data/digit_data.csv", "data/digit_target.csv")
    ......

Run pipenv run python digit_recognizer/digit_recognizer.py to check whether everything works as we expected. If so, add these code changes into git.

Next, add data/ to DVC.

$ pipenv run dvc add data

100% Add|████████████████|1/1 [00:00,  2.14file/s]

To track the changes with git, run:

git add data.dvc .gitignore

dvc add creates a data.dvc file to track data/ and add it into .gitignore so that data/ will only be tracked through DVC but not git.

# Add DVC files into git track
git add .gitignore data.dvc

# git commit
pipenv run cz commit

In data.dvc, we can see 2 files (digit_data.csv and digit_target.csv) are tracked.

$ cat data.dvc

outs:
- md5: b8d81f4964ecb86739c79c833fb491f3.dir
  size: 494728
  nfiles: 2
  path: data

Push these tracked data into DVC remote

dvc push

See what's changed in our repo storage ../dvc_remote

$ tree ../dvc_remote

../dvc_remote
├── 02
│   └── b861b6dc8e08da6d66547860f69277
├── 8c
│   └── ba569595920d230ade453b150f372b
└── b8
    └── d81f4964ecb86739c79c833fb491f3.dir

3 directories, 3 files

The md5 value of our tracked data is b8d81f4964ecb86739c79c833fb491f3.dir. There's also a corresponding file in ../dvc_remote/b8/d81f4964ecb86739c79c833fb491f3.dir.

$ cat ../dvc_remote/b8/d81f4964ecb86739c79c833fb491f3.dir

[{"md5": "02b861b6dc8e08da6d66547860f69277", "relpath": "digit_data.csv"}, {"md5": "8cba569595920d230ade453b150f372b", "relpath": "digit_target.csv"}]%

This file indicates where the actual data sources are stored in ../dvc_remote.

In conclusion, if we want to know how data is stored through DVC,

find the md5 value in *.dvc in our project
find the path that matches this md5 value in our remote storage
use the md5 value specified in the previous step to find the data sources in our remote storage

But most of the time, we don't need to do so. We can leave the tracking work to DVC.

Fetch data from DVC remote storage

# temporary delete our data locally
$ rm -rf data

# check whether DVC actually tracks our data
$ dvc status

data.dvc:
changed outs:
    deleted:            data

# bring our data back from remote storage
$ dvc checkout data

data
├── digit_data.csv
└── digit_target.csv

Add data changes into DVC

To demonstrate how DVC track data changes, let's remove the last 2 rows from data/digit_data.csv and data/digit_target.csv.

# check what's changed
$ dvc status

data.dvc:
changed outs:
    modified:           data

# Add these changes to DVC and git
$ dvc add
$ git add data.dvc
# git commit
$ pipenv run cz commit

# Push these changes to our remote storage
$ dvc push

The md5 value has been changed, and the size of our data is smaller than our previous record, 494728.

$ cat data.dvc

outs:
- md5: a333e114a49194e823ab9a4fa9e33ee9.dir
  size: 494172
  nfiles: 2
  path: data

More files are added to ../dvc_remote due to the data changes. You can follow the steps in the previous section to see what're actually store.

$ tree ../dvc_remote

../dvc_remote
├── 02
│   └── b861b6dc8e08da6d66547860f69277
├── 2a
│   └── 6cfa13365ac9b3af5146133aca6789
├── 8c
│   └── ba569595920d230ade453b150f372b
├── 94
│   └── 2481fce846fb9750b7b8023c80a5ef
├── a3
│   └── 33e114a49194e823ab9a4fa9e33ee9.dir
└── b8
    └── d81f4964ecb86739c79c833fb491f3.dir

6 directories, 6 files

Let's git checkout to the previous git commit to see what happens if we only revert the changes in data.dvc.

# or "git checkout v2-track-data"
git checkout HEAD~1

After running wc -l data/digit_data.csv, we'll still find 1795 rows instead of 1797 rows in the previous stage. That's because we need to run dvc checkout as well.

We might easily forget this step. Thus, DVC implements a git-hook that can trigger dvc checkout right after git checkout. You can install these git-hooks through dvc install. These hooks are added into .git/hooks. If you want to know the detail of what's added, read dvc install.

Test these steps again. There should be an additional line after running git checkout. This is the output message of dvc checkout.

M       data/

Push our code to a remote git repository

git remote add origin <REMOTE GIT REPO>
git push origin main

Fetch code and data changes from remote

We've already pushed all the code and data changes to remote. Let's see how we could reproduce in another environment.

# check what's in our repo
$ dvc list <REMOTE GIT REPO>

.dvcignore
.github
.gitignore
LICENSE
Pipfile
Pipfile.lock
data
data.dvc
digit_recognizer
docs
mkdocs.yml
output
tasks.py

Although git does not track data/, we can still list it through DVC.

Because we use relative path ../dvc_remote as DVC remote storage, we need to create the new project in the same layer as dvc_example. We'll clone the project into ../dvc_example_on_another_machine.

# Clone repo git repo
$ git clone <YOUR REMOTE GIT REPO> ../dvc_example_on_another_machine
$ cd ../dvc_example_on_another_machine
$ tree .

.
├── LICENSE
├── Pipfile
├── Pipfile.lock
├── data.dvc
├── digit_recognizer
│   ├── __init__.py
│   └── digit_recognizer.py
├── docs
│   └── README.md
├── mkdocs.yml
├── output
└── tasks.py

3 directories, 9 files

As you can see, data/ has not yet been added to the project. We can now pull data from our DVC remote storage.

# pull data from default DVC remote storage
$ dvc pull

A   data/
1 file added and 2 files fetched

# `data` has now been added to the project
$ tree .

.
├── LICENSE
├── Pipfile
├── Pipfile.lock
├── data
│   ├── digit_data.csv
│   └── digit_target.csv
├── data.dvc
├── digit_recognizer
│   ├── __init__.py
│   └── digit_recognizer.py
├── docs
│   └── README.md
├── mkdocs.yml
├── output
└── tasks.py

4 directories, 11 files

That's all for data versioning in DVC. In the next post, We'll continue on versioning a data pipeline, tracking parameters and metrics. We won't need dvc_example_on_another_machine for the following steps. Feel free to remove it and change directory back to dvc_example.

DEV Community: Wei Lee

DVC - Pipeline Versioning

Pipeline versioning

Split training logic into different stages

Add the first stage in our pipeline to DVC

dvc.yaml: the definition of stages

dvc.lock: the hashes of dependencies and outputs

output/.gitignore: Add files that DVC should track to gitignore

Define the whole pipeline

Run the pipeline

Track parameters

Track metrics

plotting

Conclusion

One more thing: When should we save files to DVC instead of git?

Reference

DVC - Data Versioning

How to use DVC?

Install DVC globally

Take a look at the example project

Install DVC into the virtual environment

Initialize DVC

Add DVC remote

Track data through DVC

Fetch data from DVC remote storage

Add data changes into DVC

Fetch code and data changes from remote

Reference

`dvc.yaml`: the definition of stages

`dvc.lock`: the hashes of dependencies and outputs

`output/.gitignore`: Add files that DVC should track to gitignore