Day 2: Enterprise Testing Infrastructure with TestContainers
Welcome back to the 30-day challenge where I'm building an enterprise-grade AI-native observability platform using Claude Code and documentation-driven development. We're now 6% through the journey!
Today's Focus: Real Databases, Real Confidence
Today was all about implementing comprehensive testing infrastructure. This represents a critical milestone in building a production-ready observability platform that can compete with solutions that typically take 12+ months to develop.
🎯 Goals & Achievements
Completion Rate: 2/4 goals achieved
✅ Completed
- Set up development environment (GitHub, Docker Compose)
- Generate and test core storage package code
🔄 In Progress
- Validate OTel Demo integration works end-to-end (75% complete - remaining: Fix remaining Effect-TS type issues in config package)
🎉 Bonus Achievements
Sometimes the best progress comes from unexpected discoveries:
- Fixed all Effect-TS compilation issues blocking storage package
- Created comprehensive TestContainers integration with ClickHouse and MinIO
- Established two-tier testing strategy (unit + integration tests)
🔧 Technical Deep Dive
Today's implementation focused on several critical areas:
TestContainers Integration
We now have real database testing with ClickHouse and MinIO containers that spin up for integration tests.
Integration Test Suite
Created 12 comprehensive integration tests that validate our storage layer against actual databases.
Effect-TS Compilation Fixes
Resolved type compatibility issues by adjusting TypeScript configuration, particularly around readonly arrays.
Configurable S3 Encryption
Made server-side encryption optional to support both MinIO testing and production S3.
Why TestContainers Matters
Traditional testing approaches often rely on mocks or in-memory databases that don't reflect production behavior. TestContainers changes this by providing:
- Real Database Engines: Testing against actual ClickHouse and MinIO instances
- Deterministic Environments: Fresh containers for each test run
- CI/CD Compatible: Works seamlessly in Docker-enabled pipelines
- Zero Configuration: No need for developers to install databases locally
Here's a glimpse of our integration test setup:
const clickhouseContainer = await new ClickHouseContainer()
.withDatabase('otel')
.withUsername('otel')
.withPassword('otel123')
.start()
// Now we have a real ClickHouse instance for testing!
This approach caught several issues that unit tests missed, including timestamp formatting incompatibilities and connection pooling edge cases.
💡 Key Learnings
Every day brings new insights that shape the project's direction:
TestContainers provides deterministic testing environments
No more "works on my machine" - every test run gets fresh, isolated containers.
Effect-TS readonly array issues can be resolved
Disabling exactOptionalPropertyTypes in TypeScript config resolves most compatibility issues without sacrificing type safety.
Two-tier testing strategy pays off
Fast unit tests for rapid feedback, thorough integration tests for confidence.
🚧 Challenges & Solutions
Software development is about solving problems. Here are today's battles:
Challenge: S3 server-side encryption incompatible with MinIO in tests
Solution: Made S3 encryption configurable, allowing MinIO compatibility in test environments while maintaining security in production.
Challenge: ClickHouse timestamp formatting required nanosecond to second conversion
Solution: Implemented proper nanosecond to second conversion for ClickHouse DateTime64(9) columns.
Challenge: Git merge conflicts from simultaneous main branch updates
Solution: Created new feature branch and rebased changes onto updated main branch.
📊 Project Metrics
- Overall Progress: Day 2 of 30 (6% complete)
- Current Phase: Foundation & Infrastructure (Week 1)
- Test Coverage: 30 unit tests + 12 integration tests passing
- Momentum: 🚀 Strong
🔮 Tomorrow's Priority
Continuing with Week 1 foundation work:
- Complete storage package implementation with Effect-TS
- Set up Docker Compose for local development
- Integrate with OpenTelemetry Demo for real telemetry data
- Begin AI analyzer package specification
🤖 The Claude Code Advantage
This project continues to demonstrate the power of AI-assisted development. Today's session with Claude Code:
- Generated comprehensive TestContainers integration tests
- Resolved complex TypeScript compilation issues
- Maintained consistency across the codebase
- Kept documentation in sync with implementation
The documentation-driven approach means every line of code has a purpose, and every architectural decision is captured for future reference.
💭 Reflection
Building an enterprise-grade observability platform in 30 days seemed impossible just yesterday. But with each passing day, the vision becomes clearer and the implementation more solid. The combination of Claude Code's capabilities with documentation-driven development is proving to be a game-changer.
The TestContainers integration added today isn't just about testing—it's about confidence. When you know your code works against real databases, you can move faster with less fear. This is how we compress 12 months into 30 days: by building on solid foundations and leveraging AI to handle the implementation details while we focus on architecture and design.
💭 Additional Insights
One key realization today: upgrading to Claude Pro Max for $100/month was essential. The standard Pro version ran out of tokens after just 3-4 hours of intensive development. For a project of this scope, having unlimited capacity is crucial for maintaining momentum.
The alignment between our Docker Compose setup and the runtime OTLP ingestion still needs work, but having TestContainers in place means we can iterate quickly with confidence.
🔗 Resources & Links
Project Resources:
- GitHub Repository: otel-ai
- Documentation: Living specs in Dendron
- CI/CD: GitHub Actions with TestContainers
Technologies Featured Today:
- TestContainers - Integration testing with real databases
- Effect-TS - Type-safe functional programming
- ClickHouse - Real-time analytics database
- MinIO - S3-compatible object storage
- Claude Code - AI pair programming
- OpenTelemetry - Observability framework
What challenges are you facing in your observability stack? How would AI-native observability change your workflow? Drop a comment below—I'd love to hear your thoughts!
Follow the series for daily updates on this 30-day challenge. Tomorrow we dive deeper into completing the foundation layer.
Top comments (0)