How to migrate Firestore schema

Intro

How you store your data changes over time. Some cases, you need to modify current data to be fit in a new data model structure. If your using SQL DB, you can write a migration script and run it. But if you’re using Firebase Firestore, this task can be tricky since itself has no concept of migration.

In order to do that, you can use Firebase Cloud Function, write some code using Firebase Admin SDK, and run it. It’s possible, but what if you’re using free plan(you can’t use Cloud Function unless you give them a payment information) or using Dart?(according to pub.dev, you can’t use official firebase package in pure dart, you must use Flutter)

I looked it up on this topic and found an awesome package called fireway. It inspired me to work on a same problem, but in this case, with Pure Dart.

In this post, I would like to share what I’ve built and how to use it.

What I’ve used

• Dart console application
• google api related packages
• service-account.json for authentication
• dart_console for display text in a console
• version for dealing with migration version

Github repository for source code

You can clone this public repository and modify freely.

• https://github.com/swimmingkiim/firestore_migration

How to use this Dart application

(Assum you already cloned the repository)

1. Create service account json file

First, you need service account json file. It is needed in order to automate authentication(without login all the time). You can create service account file in Google Developer Console. The link below explains how to do it.

• https://cloud.google.com/iam/docs/creating-managing-service-account-keys

The content is something like this.

{
  "type": "service_account",
  "project_id": "PROJECT_ID",
  "private_key_id": "KEY_ID",
  "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n",
  "client_email": "SERVICE_ACCOUNT_EMAIL",
  "client_id": "CLIENT_ID",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL"
}

WARNING → DO NOT upload this file to github publically! Anyone who has this file can lookup your project and do dirty thing. Add it to .gitignore!

You may want to move the service account file to your project root.

2. flutter pub get

Of course.

flutter pub get

3. Write your migration code

Under the lib/migrations/versions , you’ll find 0_0_1, 0_0_2, 0_0_3 dart files. You must follow semver versioning rules. But in the file name, use “_” instead of “.”

In 0_0_1.dart , there’s a same code.

import 'package:dart_console/dart_console.dart';
import 'package:firebaseapis/firestore/v1.dart';
import 'package:firestore_migration/config/config.dart';
import 'package:firestore_migration/src/crud_document.dart';
import 'package:firestore_migration/src/crud_field.dart';
import 'package:version/version.dart';

import '../migration.dart';

class Migration_0_0_1 implements MigrationFunc {
  @override
  Version get version => Version(0, 0, 1);

  @override
  Future<void> execute(FirestoreApi firestoreApi, Console console) async {
    // [[ YOUR_MIGRATION_CODE_HERE ]]
  }
}

Above code is a basic structure of a migration file.

• You implements MigrationFunc (in lib/migrations/migration.dart)
• You override version and execute function.
• execute function takes two parameters,
  • firestoreApi
    • from firebaseapis package
      • https://pub.dev/packages/firebaseapis
    • use directly with the help of official document, or use prebuilt utils which I’ll explain below
  • console
    • from dart_console package
    • use when you write some text(success, error etc)

In order to modify your Firestore scheme, I made util classes (CRUDDocument and CRUDField). I only created some methods that I need for now. You can freely add other method. Maybe if you think it has general use cases, then please make a PR, so that others may get your help.

Sample code I wrote is something like below.

...

final crudField = CRUDField(
      firestoreApi: firestoreApi,
      console: console,
      projectId: projectId,
    );
    final crudDocument = CRUDDocument(
      firestoreApi: firestoreApi,
      console: console,
      projectId: projectId,
    );
    final documents = await crudDocument.readDocuments('test');
    for (var document in documents) {
      await crudField.createField(
          document, 'hello', Value(stringValue: 'world'));
    }
    console.writeLine('done!');
...

Explain :

• Read collection called test
• Create field called hello with string value “world” to each document under test collection.
• Print “done!” in console

4. Run it

Now you can run it. But before, It’s better for you to read basic commands and options in this application.

firestore_migration                                                                                                                                                                                                               


A Dart console application for firestore migration                                                                                                                                                                                




[Commands]                                                                                                                                                                                                                        


    migration       : Start migration work                                                                                                                                                                                            


[Options]                                                                                                                                                                                                                         


    --version, -v                                                                                                                                                                                                                     
                    : Specify migration file version to run. Format should be match to semver(x.y.z) Default is latest.                                                                                                                              
    --service-account, -s                                                                                                                                                                                                             
                    : Path of service account json file. (required option)                                                                                                                                                                           


[Flags]                                                                                                                                                                                                                           


    --help          : Display helper text on terminal

Now, You can run it.

dart run ./bin/firestore_migration.dart migrate -s service_account.json -v 0.0.1

And if you got this, then the job you asked for is done! Check your Firestore in Firebase console to see if data changed correctly.

Start : Firestore Migration...                                                                                                                                                                                                    


this is 0.0.1
done!                                                                                                                                                                                                                             


Finish : Firestore Migration!

Conclusion

Note that you need to test it on the test project is safe. And I strongly recommend you to backup first before make any changes with this application.

I hope that this repository can help with those who has this specific need.

Any new feature idea or PR is very welcomed.

Cheers!