Hello there! Here's a detailed introduction to Data version control (DVC) from a junior data scientist's experience. This is a tool for managing and versioning data that is akin to version control systems like Git. I have outlined simple steps to get started with DVC for data science:
DVC Installation: To begin with install DVC on your pc. DVC can be installed via pip or conda for those using anaconda.
DVC Initialisation: Following installation, you can initialize it in your project working directory by running the dvc init command. This creates a .dvc directory in your project that will hold information about your data and the versions of your data.
Set up a memory location: You will need to decide on a memory location either locally or on a cloud-based service like Amazon S3. DVC can be configured to store data by running the dvc remote add command.
Adding data to DVC: Adding a file to DVC involves running the command dvc add <file_you_intend_to_add>. This creates a .dvc file that will be used to track the changes made to the data. One may add multiple files to DVC by running this command on each file.
To version data: Creating a version of your data involves running the dvc commit command. A new Git commit is created and the .dvc file is updated with the new version information. You can then push this commit to your Git repository.
To retrieve previous versions of your data the dvc checkout command is called. This will replace the current version of your data with the specified version.
If you are working on a project with other developers/data scientists, DVC comes in handy as well in collaborating on the same data. You can share the .dvc directory with your collaborators and they can use the same commands to manage the data.
The Data Version Control has an edge over github in that it allows storage of large data sets unlike github which has a cap of 100 mb for a single file that can be pushed.
Top comments (0)