DEV Community

Cover image for From 1 Hour to 90 Seconds: High-Performance Data Migration with Appwrite & Go
Etop - Essien Emmanuella Ubokabasi
Etop - Essien Emmanuella Ubokabasi

Posted on • Edited on

From 1 Hour to 90 Seconds: High-Performance Data Migration with Appwrite & Go

#appwrite #go #tutorial #datamigration

Introduction

Migrating 10,000 records from CSV to a database took over an hour with sequential processing. Using Appwrite's Go SDK and worker pools, I was able to reduce it to 90 seconds—about 40x faster.

This post shows how Appwrite's Go SDK and features make it straightforward to build a concurrent data migration tool.

Why Appwrite for Data Migration?

1. Official Go SDK

Appwrite provides an official, well-maintained Go SDK (github.com/appwrite/sdk-for-go) that simplifies database operations.

2. Developer-Friendly API

  • Clear, consistent API design
  • Strong typing with Go structs
  • Straightforward error handling
  • Built-in authentication and security

3. Scalable Infrastructure

  • Cloud-hosted or self-hosted
  • Handles high concurrency
  • No database management overhead
  • Built-in rate limiting and security

4. Perfect for Backend Services

  • Server-side SDKs for production workloads
  • API keys with granular permissions
  • RESTful API that works well with Go's concurrency model

The Challenge

  • 10,000 CSV records
  • Upload to Appwrite Database
  • Sequential processing was too slow

The Solution: Worker Pool Pattern + Appwrite Go SDK

A worker pool uses a fixed number of goroutines to process jobs from a queue, enabling controlled concurrency. Appwrite's Go SDK integrates cleanly with this pattern.

Why Worker Pools Work Well with Appwrite

  • Appwrite's API handles concurrent requests efficiently
  • The Go SDK is thread-safe and designed for concurrent use
  • Clear error responses make error handling straightforward
  • No connection pooling needed—Appwrite handles it

Implementation with Appwrite

Setting Up Appwrite

First, we set up our Appwrite project:

  1. Create a Database and Collection in Appwrite Console
  2. Define attributes matching our CSV structure
  3. Generate an API key with documents.write scope
  4. Use the official Go SDK

Architecture Overview 

CSV File → Job Channel → Worker Pool (50 workers) → Appwrite Go SDK → Appwrite API
Enter fullscreen mode Exit fullscreen mode

Key Components

  1. Appwrite client initialization: simple configuration
  2. Job channel: distributes CSV rows to workers
  3. Worker pool: 50 goroutines processing concurrently
  4. WaitGroup: ensures completion before exit
  5. Error handling: Appwrite provides clear error messages

The Code Structure 

import (
    "github.com/appwrite/sdk-for-go/appwrite"
    "github.com/appwrite/sdk-for-go/databases"
)

// Initialize Appwrite client - clean and simple!
client := appwrite.NewClient(
    appwrite.WithEndpoint(endpoint),
    appwrite.WithProject(projectID),
    appwrite.WithKey(apiKey),
)

database := appwrite.NewDatabases(client)

// Worker pool processes jobs using Appwrite SDK
func worker(id int, jobChan <-chan Job, wg *sync.WaitGroup, 
            database *databases.Databases, databaseID, collectionID string) {
    // Process jobs with Appwrite's clean API
    database.CreateDocument(databaseID, collectionID, documentID, documentData)
}
Enter fullscreen mode Exit fullscreen mode

What Makes Appwrite's Go SDK Great

  1. Type-safe operations: compile-time safety
  2. Functional options pattern: clean configuration
  3. Clear error messages: easier debugging
  4. Well-documented: easy to get started
  5. Active maintenance: regular updates and improvements

Benchmark Results

Test Setup

  • Dataset: 10,000 records
  • Platform: Appwrite Cloud
  • SDK: Official Appwrite Go SDK v0.16.0
  • Network: Standard internet connection

Results Comparison

Configuration Total Time Avg Time/Record Speedup
1 Worker (Sequential) 1h 1m 12s 367.23ms 1x
50 Workers (Concurrent) 1m 37s 9.78ms 37.5x

Visual Comparison 

Sequential (1 Worker):  ████████████████████████████████████████████████████ (61 minutes)
Concurrent (50 Workers): █ (1.6 minutes)
Enter fullscreen mode Exit fullscreen mode

Key Findings

1. Appwrite Handles Concurrency Well

  • 50 concurrent workers processed 10,000 records without issues
  • No rate limiting problems
  • Stable performance throughout

2. Go SDK Performance

  • Low overhead per request
  • Efficient connection handling
  • Fast response times

3. Developer Experience

  • Simple setup and configuration
  • Clear error messages
  • Easy to debug and monitor

4. Production Ready

  • Reliable under load
  • Built-in security features
  • Scalable infrastructure

Technical Deep Dive

Why Appwrite + Go is a Great Combination

Appwrite's Strengths:

  • RESTful API that works well with HTTP clients
  • Stateless design fits Go's concurrency model
  • Clear authentication with API keys
  • Comprehensive documentation

Go's Strengths:

  • Excellent concurrency primitives
  • Efficient HTTP client
  • Strong typing and error handling
  • Fast compilation and execution

Together, they enable high-performance, maintainable backend services.

The Worker Pool Pattern with Appwrite 

// Appwrite client setup - one time initialization
client := appwrite.NewClient(
    appwrite.WithEndpoint(endpoint),
    appwrite.WithProject(projectID),
    appwrite.WithKey(apiKey),
)
database := appwrite.NewDatabases(client)

// Worker pool - Appwrite SDK is thread-safe!
for i := 0; i < WorkerCount; i++ {
    wg.Add(1)
    go worker(i, jobChan, &wg, database, databaseID, collectionID)
}
Enter fullscreen mode Exit fullscreen mode

Appwrite's Error Handling

Appwrite provides clear error messages that make debugging straightforward:

// Appwrite returns descriptive errors
err := database.CreateDocument(...)
if err != nil {
    // Clear error messages help identify issues quickly
    log.Printf("Failed: %v", err)
}
Enter fullscreen mode Exit fullscreen mode

Best Practices for Appwrite + Go

  1. Use API keys with appropriate scopes
  2. Leverage Appwrite's built-in security
  3. Handle errors gracefully
  4. Monitor API usage
  5. Use worker pools for bulk operations

Real-World Applications

This pattern works well for:

  • Database migrations to Appwrite
  • Bulk data imports
  • ETL pipelines
  • Batch processing jobs
  • Data synchronization

Why Choose Appwrite?

For Backend Developers

  • Official Go SDK with active maintenance
  • Simple, consistent API
  • Production-ready infrastructure
  • Strong security defaults
  • Comprehensive documentation

For This Use Case

  • Handles high concurrency
  • Fast response times
  • Clear error messages
  • Easy to scale
  • No database management overhead

Conclusion

Appwrite's Go SDK and infrastructure make it straightforward to build high-performance data migration tools. In this case, 50 workers reduced migration time from 61 minutes to 98 seconds.

Takeaways

  • Appwrite's Go SDK is production-ready and developer-friendly
  • Worker pools enable efficient concurrent processing
  • Appwrite handles high concurrency reliably
  • The combination of Appwrite + Go is powerful for backend services

Conclusion

This is the kind of backend workload Appwrite is well-suited for: concurrent, production-grade services without database overhead

Resources

Top comments (3)

Collapse
 
marcello_h profile image
Marcelloh • • Edited

I looked suprised at the one hour it took for that dataset. I tested the same set to put it sequentially into my own database and it took much less:
LargeDataset20260103: 78.9375ms with 10000 records
So where does the speed loss happen then?
Is it the network?
Is it Appwrite?

I couldn't find the Go SDK documentation btw and I also think that any documentation could benefit from examples.

Collapse
 
ubcodes profile image
Etop - Essien Emmanuella Ubokabasi •

You're right, the 78ms you’re seeing is likely the raw speed of the database engine (like MariaDB or Postgres) performing a bulk insert on a local machine.

The speed loss in my example comes primarily from network latency and API overhead. Also, Appwrite here isn't just a database; it's an API layer. That 'one hour' I started with was the result of making those API calls one by one sequentially.

The jump down to ~90 seconds was achieved by applying Go’s concurrency (worker pools), effectively amortizing that API and network overhead across many parallel requests.

Regarding the Go SDK documentation, I have corrected the link, and great suggestion, I agree with you that the documentation would benefit from examples/

Collapse
 
marcello_h profile image
Marcelloh • • Edited

my 78ms was done sequential, record by record, no bulk insert.
10.000 records in 3600 seconds, that's 3 records per second roughly. I find that way too slow.
Perhaps I should test this myself. But okay, the thing is not the data migration, but the actual usage of getting records and modifying them while there "in there".

Decided to try it out for myself, but the documentation is not that great. I wanted to create a database by code, but I already got:
The maximum number of databases allowed for the selected plan has reached. Upgrade to increase the limit.
So much for a free trail. (this was when creating the table.)
and this is where I leave the bus....