Complete Guide: Transferring Firestore Data Between Databases (Including Subcollections)

Introduction

If you've ever needed to move data between Firestore databases - whether for migrating to a new project, creating a backup, or setting up a development environment - you know it's not as simple as copy and paste. In this comprehensive guide, I'll walk you through the entire process of transferring Firestore data between databases, including those tricky nested subcollections that many developers forget about.

Recently, I needed to transfer a complex Firestore database with deeply nested collections. After spending hours figuring out the best approach, I've compiled everything I learned into this step-by-step guide.

---

Table of Contents

1. Understanding the Challenge
2. Prerequisites
3. Setting Up Service Accounts
4. Basic Transfer Script
5. Handling Subcollections (The Hard Part)
6. Complete Production-Ready Script
7. Security Best Practices
8. Troubleshooting Common Issues

---

Understanding the Challenge

Firestore data isn't just a flat list of documents. It's a hierarchical structure where documents can have subcollections, which can have their own subcollections, and so on.

When you fetch a document, Firestore does not automatically include its subcollections. This is a common pitfall that leads to incomplete data transfers.

Categories (Collection)
  └── category_id_1 (Document)
      ├── name: "Technology"
      └── Subcategories (Subcollection)
          └── sub_id_1 (Document)
              ├── name: "Programming"
              └── Items (Subcollection)
                  └── item_id_1 (Document)
                      └── name: "JavaScript Course"

If you only transfer the top-level categories documents, you'll lose all the subcategories and items nested underneath them.

---

Prerequisites

Before we dive in, make sure you have:

• Node.js installed (v12 or higher)
• A Firebase/Google Cloud project for both source and destination databases
• The firebase-admin npm package installed
• Basic familiarity with JavaScript and the command line

---

Setting Up Service Accounts

Step 1: Generate Service Account Keys

For each project (source and destination), you need a service account key:

1. Go to your Firebase Console
2. Click on Project Settings (gear icon)
3. Navigate to the Service Accounts tab
4. Click Generate New Private Key
5. Choose JSON as the key type
6. Download and save the file

File naming convention:

• Source project: source-key.json
• Destination project: dest-key.json

Step 2: Assign the Right Permissions

For security, follow the principle of least privilege:

Database	Recommended Role	Why
Source	Cloud Datastore Viewer	Only needs to read data
Destination	Cloud Datastore User	Needs to write data

To assign these roles:

1. Go to Google Cloud Console → IAM & Admin
2. Find your service account
3. Click the edit icon (pencil)
4. Remove default roles and add the recommended ones
5. Save changes

---

Basic Transfer Script

Here's a basic script that transfers a collection (but not subcollections):

const admin = require("firebase-admin");

// Initialize source and destination
const sourceApp = admin.initializeApp({
  credential: admin.credential.cert(require("./source-key.json")),
  projectId: "source-project-id"
}, "source");

const destApp = admin.initializeApp({
  credential: admin.credential.cert(require("./dest-key.json")),
  projectId: "dest-project-id"
}, "dest");

const sourceDb = sourceApp.firestore();
const destDb = destApp.firestore();

async function transferCollection(collectionName) {
  const snapshot = await sourceDb.collection(collectionName).get();

  let batch = destDb.batch();
  let count = 0;

  for (const doc of snapshot.docs) {
    batch.set(
      destDb.collection(collectionName).doc(doc.id), 
      doc.data()
    );
    count++;

    // Firestore allows max 500 operations per batch
    if (count % 500 === 0) {
      await batch.commit();
      batch = destDb.batch();
    }
  }

  if (count % 500 !== 0) {
    await batch.commit();
  }

  console.log(`Transferred ${count} documents from '${collectionName}'`);
}

// Run the transfer
transferCollection("categories");

⚠️ Limitation: This only transfers top-level documents. Subcollections are left behind!

---

Handling Subcollections (The Hard Part)

To transfer subcollections, we need to:

1. List all subcollections under each document
2. Recursively transfer each subcollection
3. Maintain the same path structure in the destination

Here's the key function that handles recursion:

async function transferDocumentWithSubcollections(sourceDocRef, destDocRef) {
  // 1. Transfer the document itself
  const doc = await sourceDocRef.get();
  if (doc.exists) {
    await destDocRef.set(doc.data());
  }

  // 2. Get all subcollections
  const collections = await sourceDocRef.listCollections();

  // 3. Transfer each subcollection recursively
  for (const collection of collections) {
    await transferCollection(collection.path);
  }
}

---

Complete Production-Ready Script

Here's the full, battle-tested script that handles:

• ✅ Top-level documents
• ✅ Nested subcollections (any depth)
• ✅ Batch writes (500 documents per batch)
• ✅ Error handling
• ✅ Progress logging
• ✅ Configurable collection paths

const admin = require("firebase-admin");

// Initialize source and destination
const sourceApp = admin.initializeApp({
  credential: admin.credential.cert(require("./source-key.json")),
  projectId: "online-jobs-e01c0"
}, "source");

const destApp = admin.initializeApp({
  credential: admin.credential.cert(require("./dest-key.json")),
  projectId: "online-jobs-23402"
}, "dest");

const sourceDb = sourceApp.firestore();
const destDb = destApp.firestore();

/**
 * Transfers a collection and all its subcollections recursively
 * @param {string} collectionPath - Path of collection to transfer
 * @param {Firestore} db - Destination Firestore instance
 */
async function transferCollection(collectionPath, db = destDb) {
  try {
    const snapshot = await sourceDb.collection(collectionPath).get();

    if (snapshot.empty) {
      console.log(`📭 No documents found in '${collectionPath}'`);
      return 0;
    }

    let batch = db.batch();
    let count = 0;
    let batchCount = 0;
    const docsToProcess = [];

    // First pass: collect all documents and add to batch
    for (const doc of snapshot.docs) {
      const destDocRef = db.collection(collectionPath).doc(doc.id);

      batch.set(destDocRef, doc.data());
      count++;

      docsToProcess.push({
        sourceRef: sourceDb.doc(`${collectionPath}/${doc.id}`),
        destRef: destDocRef
      });

      // Commit batch if at limit
      if (count % 500 === 0) {
        await batch.commit();
        console.log(`  ✅ Committed batch ${++batchCount} for '${collectionPath}'`);
        batch = db.batch();
      }
    }

    // Commit remaining documents
    if (count % 500 !== 0) {
      await batch.commit();
      console.log(`  ✅ Committed final batch for '${collectionPath}'`);
    }

    console.log(`📊 Transferred ${count} documents from '${collectionPath}'`);

    // Second pass: process subcollections
    for (const { sourceRef, destRef } of docsToProcess) {
      const subcollections = await sourceRef.listCollections();

      for (const subcollection of subcollections) {
        console.log(`  📁 Processing subcollection: ${subcollection.id} under ${sourceRef.path}`);
        await transferCollection(subcollection.path, db);
      }
    }

    return count;

  } catch (error) {
    console.error(`❌ Error transferring '${collectionPath}':`, error.message);
    throw error;
  }
}

/**
 * Transfer a specific document and all its subcollections
 * @param {string} documentPath - Full path of document
 */
async function transferDocumentTree(documentPath) {
  const sourceDocRef = sourceDb.doc(documentPath);
  const destDocRef = destDb.doc(documentPath);

  console.log(`\n🚀 Transferring document tree: ${documentPath}`);
  await transferDocumentWithSubcollections(sourceDocRef, destDocRef);
  console.log(`✅ Completed transfer of ${documentPath}\n`);
}

/**
 * Transfer a single document with its subcollections
 */
async function transferDocumentWithSubcollections(sourceDocRef, destDocRef) {
  // Transfer document
  const doc = await sourceDocRef.get();
  if (doc.exists) {
    await destDocRef.set(doc.data());
    console.log(`  ✓ Document: ${sourceDocRef.path}`);
  }

  // Transfer subcollections
  const collections = await sourceDocRef.listCollections();
  for (const collection of collections) {
    console.log(`  → Processing subcollection: ${collection.id}`);
    await transferCollection(collection.path);
  }
}

// ============================================
// RUN THE TRANSFER
// ============================================

async function main() {
  console.log("🔥 Starting Firestore Data Transfer\n");
  console.log(`📤 Source: ${sourceApp.options.projectId}`);
  console.log(`📥 Destination: ${destApp.options.projectId}\n`);

  try {
    // Transfer the main collection and all subcollections
    const totalDocs = await transferCollection("categories");

    console.log(`\n🎉 Transfer complete!`);
    console.log(`📊 Total documents transferred: ${totalDocs}`);

  } catch (error) {
    console.error("💥 Transfer failed:", error);
    process.exit(1);
  }
}

// Uncomment the line below to run
main();

// ============================================
// USAGE EXAMPLES
// ============================================

// Option 1: Transfer a specific document tree
// await transferDocumentTree("categories/technology");

// Option 2: Transfer only subcollections of existing documents
// async function transferOnlySubcollections(documentPath) {
//   const sourceDocRef = sourceDb.doc(documentPath);
//   const collections = await sourceDocRef.listCollections();
//   for (const collection of collections) {
//     await transferCollection(collection.path);
//   }
// }

---

Security Best Practices

🚨 Never Commit Keys to Version Control

Add this to your .gitignore:

*.key.json
*-key.json
!example-key.json

🔐 Use Environment Variables

Instead of hardcoding file paths:

const sourceCredentials = JSON.parse(process.env.SOURCE_CREDENTIALS);
const destCredentials = JSON.parse(process.env.DEST_CREDENTIALS);

const sourceApp = admin.initializeApp({
  credential: admin.credential.cert(sourceCredentials),
  projectId: sourceCredentials.project_id
}, "source");

🛡️ Rotate Keys Regularly

• Generate new keys every 90 days
• Revoke old keys immediately after rotation
• Monitor service account activity in Google Cloud Console

👁️ Enable Audit Logging

gcloud services enable cloudaudit.googleapis.com

---

Troubleshooting Common Issues

Issue 1: "Permission Denied"

Solution: Verify your service account has the correct roles:

• Source: Cloud Datastore Viewer
• Destination: Cloud Datastore User

Issue 2: Batch Limit Exceeded

Firestore batches have a 500 operation limit. The script handles this, but if you're modifying it, remember:

// ✅ Correct
if (count % 500 === 0) {
  await batch.commit();
}

// ❌ Incorrect (will cause errors)
if (count === 500) {
  await batch.commit();
}

Issue 3: Timeout for Large Transfers

For very large databases, consider:

1. Using export/import via Cloud Storage instead
2. Paginating with query cursors
3. Running in smaller chunks

Issue 4: Circular References

Firestore doesn't allow circular references in document data. If you encounter this, sanitize your data before writing.

Issue 5: Missing Subcollections

Make sure you're using listCollections() properly:

// Correct: on document reference
const subcollections = await sourceDb.doc(path).listCollections();

// Incorrect: on collection reference
const subcollections = await sourceDb.collection(path).listCollections(); // ❌

---

Alternative Approaches

Option A: Firebase Export/Import (Best for Large Datasets)

# Export
gcloud firestore export gs://your-bucket/export-folder

# Import
gcloud firestore import gs://your-bucket/export-folder --project=dest-project

Pros: Fast, handles everything atomically

Cons: Requires Cloud Storage, costs money

Option B: Firebase Extensions

Install the "Firestore Import/Export" extension from Firebase Extensions marketplace.

Pros: GUI interface, scheduled backups

Cons: Less control, additional configuration

---

Conclusion

Transferring Firestore data between databases isn't just about copying documents - it's about preserving the entire data hierarchy. The recursive approach outlined in this guide ensures that every document, subcollection, and nested structure is preserved.

Key takeaways:

1. Always account for subcollections - they're easy to miss!
2. Use batch writes to respect Firestore limits
3. Apply the principle of least privilege for security
4. Test with a single document before full transfer

What's Next?

• 📖 Read the Official Firebase Documentation
• 🛠️ Consider setting up automated backups
• 🔍 Monitor your transfer with Firebase Analytics

---

Resources

• Firebase Admin SDK Documentation
• Cloud Firestore Data Model
• Firestore Pricing
• Github Repository with Script

---

Did this guide help you? Have questions or suggestions? Let me know in the comments below!

---