DEV Community: Alejandro Sosa

Your Test Coverage Is Lying to You

Alejandro Sosa — Wed, 01 Apr 2026 15:07:05 +0000

$ testreg trace auth.login

  Feature: User Login (auth.login)
  Priority: critical

route:/login                                              apps/web/src/router.tsx:142
└─ LoginPage                                              apps/web/src/pages/auth/LoginPage.tsx:13
   └─ useAuth                                             apps/web/src/hooks/useAuth.ts:19
      └─ authApi.login                                    apps/web/src/services/api/auth.ts:46
         └─ POST /api/v1/auth/login                       src/infrastructure/http/handlers/auth_handler.go:576
            └─ AuthHandler.Login                          src/infrastructure/http/handlers/auth_handler.go:249
               └─ authService.Login                       src/application/services/auth_service.go:172
                  ├─ JWTGenerator.GenerateTokenPair        src/infrastructure/auth/jwt_generator.go:70
                  │  ├─ JWTGenerator.GenerateAccessToken   src/infrastructure/auth/jwt_generator.go:97
                  │  └─ JWTGenerator.GenerateRefreshToken  src/infrastructure/auth/jwt_generator.go:123
                  ├─ authRepository.StoreRefreshToken      src/domain/repositories/auth_repository.go:329
                  ├─ repositories.HashToken                src/domain/repositories/auth_repository.go:90
                  └─ sql:GetUserByEmail                    src/domain/repositories/queries/user.sql:21

That's one command. It traces every function your login feature touches — from the React route to the SQL query. It tells you which of those 11 nodes have tests, which don't, and which are critical enough to block a release.

Your coverage tool tells you a percentage. This tells you where you'll get paged at 2 AM.

It's not a magic wand. The tool auto-maps roughly 95% of call edges through AST parsing, but the remaining edges need lightweight annotations — a comment above a handler, a tag on a test file. The kind of overhead that takes seconds per file. The initial setup is real. What it replaces is realer.

Here's a question most teams can't answer: which business features does your test suite actually validate?

Not which files have tests. Not which lines were executed. Which features — the things your users pay for, the things your PM tracks, the things that wake you up at night — are meaningfully covered across every layer of the stack?

I worked on a full-stack nutrition platform — 749 test files. CI was green. Coverage sat at a comfortable number. And I couldn't tell you whether the login flow was tested end-to-end or whether the meal logging feature had a single integration test that hit a real database.

The problem wasn't a lack of tests. It was a lack of visibility. The metrics we use — line coverage, branch coverage, "all tests pass" — answer the wrong question. They tell you how much code was executed during tests. They say nothing about which business outcomes those tests protect.

When I dug in, here's what I found:

78% of tests used mocks. Only 22% hit real services.
22 source files had zero test files — invisible behind the aggregate coverage number.
A load test had been "passing" for weeks. It was testing a rate limiter that was never wired into the middleware chain. The test could not fail because the feature it tested didn't exist in production.
A Maestro E2E test for the training session screen checked for the presence of labels (assertVisible: text: 'Total Volume') but not their values. Total Volume: 0 kg was a passing test.

Every one of these problems was invisible to standard coverage tools. Green dashboard. Broken assumptions.

Full-Stack Dependency Tracing: From React Route to SQL Query

I built testreg to answer the question coverage tools can't: for each business feature, what is the test coverage across every layer of the stack?

It works by combining three things:

1. Static analysis of the actual source code. testreg parses Go source files using go/ast and go/parser — the same packages the Go compiler uses. It doesn't instrument your code or run your tests. It reads the AST, discovers functions and methods, resolves struct field types, follows call chains through handlers → services → repositories → SQL queries. For projects that need exact cross-package type resolution, an experimental type_checking: true option enables go/types — the same type checker the Go compiler uses. It's not battle-tested yet, but the direction is clear: opt-in precision when the default heuristic isn't enough. For TypeScript, it runs a scanner using the TypeScript compiler API to trace React Router routes → components → hooks → API service calls.

2. Dependency injection and query resolution. testreg parses DI framework registrations to resolve interface-to-concrete mappings automatically — Google Wire (wire.Bind()), Uber Fx (fx.Provide(), fx.Options()), and Dig (dig.Provide()). If you use SQLC for database queries, it maps generated Go methods back to their source SQL files. These integrations mean the graph doesn't stop at an interface boundary — it follows the real implementation.

3. A feature registry. You define your business features in YAML — auth.login, meals.log-create, billing.checkout — with their API surfaces and priority levels. Test files get a // @testreg auth.login annotation. The tool maps features to tests, traces the dependency graph from each feature's entry point, and produces a per-feature health score weighted by architectural layer:

Health = (handler_coverage × 0.30)
       + (service_coverage × 0.30)
       + (repository_coverage × 0.25)
       + (query_coverage × 0.15)

The output looks like this:

Feature: auth.login (critical)  Health: 74%
═══════════════════════════════════════════════════════

  Dependency Chain:
    route:/login                                            ✓ tested
    └─ LoginPage                                            ✓ tested
       └─ useAuth                                           ✓ tested
          └─ authApi.login                                  ✓ tested
             └─ POST /api/v1/auth/login                     ✓ tested
                └─ AuthHandler.Login                        ✓ tested
                   └─ authService.Login                     ✓ tested
                      ├─ JWTGenerator.GenerateTokenPair     ✓ tested
                      ├─ authRepository.StoreRefreshToken   ✘ NO TEST
                      ├─ repositories.HashToken             ✘ NO TEST
                      └─ sql:GetUserByEmail                 ✘ NO TEST

  Gaps (3):
    ✘ [CRITICAL] authRepository.StoreRefreshToken — no unit test
    ✘ [HIGH]     repositories.HashToken — no unit test
    ✘ [MEDIUM]   sql:GetUserByEmail — no query-level test

That's not a percentage. That's a map.

The --format prompt flag produces the same gap data as a structured work order, designed for AI agents:

## Feature: auth.login
Priority: critical | Health: 74% | Target: 100%

### Gaps (3):
1. CRITICAL: authService.Login has no unit test
   - Source: src/application/services/auth_service.go:172
   - Write: unit test for authService.Login
   - Annotation: // @testreg auth.login #real

Every field is actionable: source file path, line number, what test to write, where to put it, the exact annotation to add. The AI receives a work order, not a reading list.

The graph powers more than gap analysis. testreg contract auth.login shows the full API contract and implementation chain — not just which functions are called, but what data flows through each layer with exact function signatures:

$ testreg contract auth.login

  Feature: Login (auth.login)
  Entry:   POST /api/v1/auth/login
  ═══════════════════════════════════════════════════════════════

  Layer 1: Endpoint
    File: apps/web/src/router.tsx:142
    Delegates to: LoginPage

  Layer 2: Component
    File: apps/web/src/pages/auth/LoginPage.tsx:13
    Delegates to: useAuth

  Layer 3: Hook
    File: apps/web/src/hooks/useAuth.ts:19
    Delegates to: authApi.login

  Layer 4: Service
    File: apps/web/src/services/api/auth.ts:46
    Delegates to: POST /api/v1/auth/login

  Layer 5: Handler
    File: src/infrastructure/http/handlers/auth_handler.go:249
    func (*AuthHandler) Login(w http.ResponseWriter, r *http.Request)
    Delegates to: authService.Login

  Layer 6: Service
    File: src/application/services/auth_service.go:172
    func (*authService) Login(ctx context.Context, email string,
                              password string) (*AuthResponse, error)
    Also calls: JWTGenerator.GenerateTokenPair,
                authRepository.StoreRefreshToken,
                repositories.HashToken, sql:GetUserByEmail

  Layer 7: Service
    File: src/infrastructure/auth/password.go:68
    func (*Argon2Hasher) Verify(password string,
                                encodedHash string) (bool, error)

  Test Coverage: 31 test files across Go, TypeScript, Playwright,
                 and Maestro — unit, integration, and E2E

Eight layers, from the React route to the password hasher, with exact function signatures and file locations. A frontend developer runs one command and sees the complete implementation chain without reading Go source. An AI agent implementing a new consumer of this endpoint gets the exact function signatures and delegation chain. The contract is always current because it's derived from source code, not documentation.

How Static Analysis Speeds Up AI-Assisted Development

Without structured tooling, every AI-assisted session starts the same way: scan the codebase, load files into context, figure out what's tested and what isn't. The same orientation work, repeated every session, burning context window before any productive work begins.

The capability gap matters more than the token savings. An AI reading source files as text cannot replicate what AST-based dependency tracing produces. It can't systematically resolve Wire bindings across 839 Go files. It can't trace struct field types through multi-level call chains. It can't map SQLC-generated methods to their source SQL files. These require parsed abstract syntax trees, not pattern matching on text. No amount of context window changes this.

But the token savings are real too. I measured them on a 2,122-file production codebase (184 features, 16 domains):

testreg command	Output size (chars)	~Tokens
`sprint -n 10` (prioritized gap list)	939	~235
`audit --summary` (health by priority tier)	386	~97
`gaps --feature auth.login --format prompt` (work order)	3,645	~911

That sprint + summary is roughly 330 tokens of structured, prioritized output — exact file paths, line numbers, gap severities, weighted health scores — for a codebase that totals approximately 5 million tokens of source code.

An AI exploring that same codebase from scratch to answer "what should I work on next?" needs to read directory structures, sample test files, cross-reference source directories, and reason about coverage. Conservative estimate: 10,000-25,000 tokens of exploration, producing a qualitative approximation. The ratio is roughly 30x-75x for orientation tasks, depending on how selectively the AI reads. For single-feature work on a codebase the AI already knows, the ratio narrows to roughly 5x-12x — still meaningful, but the real value is that the output is structured and actionable, not approximate. (The "with testreg" numbers are measured from actual CLI output at ~4 chars/token, ±30%. The "without" numbers are conservative estimates, not benchmarks.)

Where the savings compound most: parallel agent dispatch. In a sprint where I dispatched 25 agents across 7 batches, each agent received a structured work order of ~900 tokens. Without testreg, each agent independently explores the codebase — multiplying the orientation cost by the number of agents. Five agents exploring independently: five times the orientation cost. Five work orders from one testreg run: the analysis is paid once, offline, in milliseconds.

The pattern: cheap deterministic analysis (static tool, milliseconds) feeds expensive creative execution (AI agent, minutes). The tool finds gaps in milliseconds. The AI writes tests in minutes. Neither alone achieves what both together produce.

This isn't just about testing. The same dependency graph powers testreg diagnose auth.login --symptom "401 unauthorized" — match an error pattern against the dependency chain and rank which files to check first. It powers testreg trace for understanding a feature before refactoring it. It powers testreg sprint for data-driven sprint planning instead of gut-driven guessing.

The graph is the product. The commands are lenses.

Bugs That Test Coverage Metrics Missed

Theory is cheap. Here's what happened when I applied this methodology across production codebases.

The 5-Bug Cascade Nobody Could See

On the nutrition platform, testreg audit flagged training.record-exercise at 0% health despite having 20+ passing unit tests at the service layer. The audit showed why: the outer GraphQL resolver — the entry point that sits between the API and the business logic — had zero tests.

I traced the dependency chain:

mutationResolver.TrainingLogSet        ← NO TESTS (resolver layer)
  → TrainingResolver.LogSet            ← delegation
    → sessionService.LogSet            ← tested, but...
      → setRepo.Create                 ← repository
        → sql:CreateExerciseSet        ← SQL

The unit tests covered the service layer and below. Nobody had tested the resolver — the integration seam where data transformations happen. Five bugs were hiding there:

Layer	Bug
Go resolver	`set.exercise` always null — a comment said "resolved via DataLoader" but no DataLoader existed
Go service	`ListHistory` returned sessions without segments — stats showed 0 kg, 0 exercises
React Native state	Stale closure captured initial values — `handleCompleteSet` sent weight=0, reps=0
React Native navigation	Race condition on summary screen — re-fetch competed with cache invalidation
React Native persistence	Unhandled SQLite foreign key errors polluting error state

All existing tests passed. The Maestro E2E test passed — it checked for label presence, not values. Traditional code coverage would show the resolver as "covered" because other test paths touched it. testreg showed it was uncovered for this specific feature's entry point.

No single test, no code review, no coverage tool would catch all five. They required tracing the full stack and finding the untested seam.

The Rate Limiter That Was Never There

On a different project, testreg audit flagged load test files as "unknown language." Adding JavaScript assertion patterns made them auditable — which led to actually running the load tests. Which revealed:

All GraphQL load tests returned 401 (no auth headers in the test)
The rate limiter load test always passed — because the rate limiter was never wired into the middleware chain

A "passing" test that can never fail is worse than no test at all. The chain reaction: testreg audit → unknown language flag → add JS support → run tests → find 401s → fix auth → find rate limiter passing with 0 rejections → discover it was never wired → wire it → verify.

Without the audit flagging an "unknown language," none of this surfaces.

The Numbers

In one sprint against the nutrition platform:

Metric	Before	After
Features at 80%+ health	29	46
Critical features at 100%	10 of 23	20 of 23
Tests written	—	500+
Production bugs found	—	5 (multi-layer cascade)
Audit time (184 features)	1m 52s	14s

The 7.9x performance improvement came from a single architectural change: build the dependency graph once and query it per-feature, instead of rebuilding it 184 times. A 2-minute command doesn't get integrated into CI. A 14-second command runs on every push.

Running Static Analysis on a Codebase You've Never Seen

Everything above was tested on projects I built. Here's what happens on a project I didn't.

Metro-Grama is a university transit app built with Echo (Go) and React/TypeScript — a stack I didn't build and hadn't seen before. I cloned it and ran testreg with zero annotations.

The entire setup:

# 1. Clone the project
git clone https://github.com/MetroTech-UNIMET/Metro-Grama.git
cd Metro-Grama

# 2. Create .testreg.yaml (4 lines)
cat > .testreg.yaml << 'EOF'
graph:
  backend_root: server
  frontend_roots:
    - client/src
  max_depth: 10
EOF

# 3. Auto-discover routes and generate features
testreg init --discover

# 4. Scan for existing tests
testreg scan

Results — no annotations written, no features manually defined:

Discovered 43 routes → 43 features across 15 domains
  auth: 5 features
  careers: 5 features
  enroll: 5 features
  subjects: 4 features
  ...

Scan complete.
  Total test files: 6
  Mapped:           3 (auto-mapped by directory proximity)
  Unmapped:         3

testreg init --discover parsed the Echo router, found all 43 routes, grouped them by module, and generated features with correct API surfaces. testreg scan found 6 existing test files and auto-mapped 3 of them to features by matching directory names to domains — server/modules/auth/auth_test.go mapped to auth features, client/src/features/student/Profile/Profile.test.tsx mapped to student features.

  ┌──────────────────────┬───────┬──────────┬──────────┬──────────┐
  │ Domain               │ Total │ Unit     │ Integ.   │ E2E      │
  ├──────────────────────┼───────┼──────────┼──────────┼──────────┤
  │ auth                │ 5    │ 5/5 OK  │ 0/5 !!  │ 0/5 !!  │
  │ careers             │ 5    │ 0/5 !!  │ 0/5 !!  │ 0/5 !!  │
  │ enroll              │ 5    │ 3/5 ✓   │ 0/5 !!  │ 0/5 !!  │
  │ student             │ 3    │ 3/3 OK  │ 0/3 !!  │ 0/3 !!  │
  │ ...                 │      │          │          │          │
  ├──────────────────────┼───────┼──────────┼──────────┼──────────┤
  │ TOTAL               │ 43   │ 26% !!  │ 0% !!   │ 0% !!   │
  └──────────────────────┴───────┴──────────┴──────────┴──────────┘

In under a minute, with 4 lines of configuration, testreg produced a complete coverage dashboard for a foreign codebase. The dependency graph was auto-discovered from the Echo router. The test mapping used directory proximity, not annotations. The coverage gaps are immediately visible: auth has unit tests, careers has nothing, nobody has integration or E2E tests.

The annotations would make it more precise — mapping specific test functions to specific features. But even without them, the tool produces actionable output on day zero.

What This Tool Can't Do (And Why That's the Point)

testreg makes specific assumptions about your codebase. Understanding them upfront determines whether it's useful to you.

Router support:

Router	Support Level
Chi	Auto-detected
Echo	Auto-detected
stdlib `net/http`	Auto-detected
Gin, Fiber, gorilla/mux	Via `@api` annotations

Chi is battle-tested across 184 features on a production monorepo. Echo was validated against Metro-Grama (43 routes). Other routers are supported through @api annotations but haven't seen the same volume of real-world usage.

Dependency injection support:

DI Framework	Support Level
Google Wire	Full — parses `wire.Bind()` and provider functions
Uber Fx / Dig	Full — parses `fx.Provide()`, `fx.Options()`, `dig.Provide()`
Manual wiring (struct fields)	Full
Manual wiring (constructor params)	Partial — constructor visible, internal calls not traced
Closures / globals	None

Type resolution:

By default, testreg uses go/ast heuristics — fast, works on any source code, no build required. An experimental type_checking: true option enables go/types for cross-package resolution — not yet battle-tested, but the foundation for exact interface resolution in future versions.

Data access:

Tool	Support
SQLC	Full — maps generated methods to SQL source files
GORM, ent, raw SQL	None — too dynamic for AST analysis

Frontend:

Framework	Support
React Router + TanStack Query	Full tracing
Next.js, Vue, Svelte, Angular	None

Architecture assumptions:

The health score weights assume a layered architecture: handler → service → repository → query. Flat architectures get skewed scores. The graph still builds, but the weights won't reflect your actual risk distribution.

What's heuristic, not deterministic:

Test status is file existence, not pass/fail. A broken test still shows as "tested."
partial coverage counts the same as tested in the health score — which can inflate it.
Gap severity is fixed by architectural layer, not code complexity or change frequency.

The hidden dependency: coding conventions.

None of this works without disciplined engineering practices underneath it. testreg can classify a function as a "handler" because the directory is named handlers/. It can trace h.service.Login() because dependencies are struct fields, not closures. It can weight health scores by layer because the architecture has layers. The tool doesn't create structure — it reads structure that's already there.

Naming conventions, folder organization, layered architecture, struct field injection, code-generation DI — these aren't just "clean code" preferences. They're machine-readable metadata. Every convention your team follows is a signal that static analysis can consume. Every shortcut — a global variable here, a closure-captured dependency there — is an edge the graph can't trace.

This is why testreg rewards teams that already follow good practices and offers less to codebases that don't. The tool is a consequence of disciplined engineering, not a substitute for it.

I'm explicit about these limitations because they are the design. I built testreg on tools that already exist in the Go ecosystem — go/ast, go/parser, Wire, SQLC, the TypeScript compiler API (with experimental go/types support in progress). I didn't invent a new static analysis framework. I composed existing ones into a workflow that answers a question they couldn't answer individually.

Why Your Stack Choice Matters for AI-Assisted Development

There's a thesis hiding inside this tool that has nothing to do with testing.

The reason testreg works is not that the approach is clever. It's that Go's ecosystem provides the raw materials for deterministic static analysis:

go/ast and go/parser are standard library — zero dependencies to parse any Go file
go/types provides full type resolution — opt-in, first-party, currently experimental in testreg but the capability exists in the stdlib
Wire and SQLC use code generation, not runtime reflection — their bindings are visible to static analysis
Even Uber Fx/Dig, which use runtime reflection at execution time, register providers through static Go code that AST parsing can trace

Compare this to ecosystems where the same approach is structurally harder:

Static Analysis Feasibility	Examples
Traceable — static typing + stdlib AST tooling	Go, Rust, Java, C#
Partially traceable — provider registration parseable, runtime behavior not	Uber Fx/Dig (Go), Spring (Java)
Not statically resolvable — dynamic typing, metaprogramming	Python, Ruby, plain JavaScript

This doesn't mean Go is the only viable stack for AI-assisted development. Java and C# have excellent static analysis ecosystems. Rust's type system is even stricter. TypeScript's compiler API powers testreg's frontend scanning. Python has the ast module and tools like mypy.

But the principle is clear: stacks with accessible AST tooling and deterministic DI resolution give you a structural advantage for building the kind of tooling that makes AI-assisted development efficient. The graph testreg builds is possible because Go was designed with tooling in mind. Every piece — the parser, the type checker, the convention of struct field injection — contributes to making the codebase statically analyzable.

The next time you evaluate a tech stack, consider asking: does this ecosystem give me the tools to build deterministic static analysis? Because in AI-assisted development, the stack that's easiest to analyze statically is the stack where AI delivers the most value.

The Future: OpenAPI, Richer Metadata, and Self-Describing Codebases

What follows are directions, not shipped features.

testreg uses a minimal set of custom annotations — @testreg for feature mapping, @api for route binding, #mocked/#real for test classification. Even this minimal metadata layer produces significant value. But there's more on the table.

Projects that already have structured metadata — OpenAPI specs, GraphQL schemas, Swagger comments — are sitting on information that tools like testreg could consume. An OpenAPI spec already defines routes, request/response schemas, and authentication requirements. A GraphQL schema already defines the type contracts between layers. Today, testreg doesn't parse these standards. It could. Integrating with OpenAPI would eliminate the need for @api annotations entirely on projects that already document their APIs — one less custom tag, one less thing to maintain.

Beyond existing standards, richer annotations could unlock more precise AI work orders. If a test annotation included the expected invariant (@invariant "never stores plaintext password"), the AI could write assertion-rich tests targeting specific guarantees instead of generic happy-path coverage. If features declared dependencies (@depends-on auth.session), the tool could answer "if auth breaks, which downstream features are at risk?"

There's also the reverse direction. The same conventions testreg reads for analysis — directory structure, DI patterns, test naming — could drive generation in reverse. Define meals.log-create in the registry YAML, run a scaffold command, and get backend stubs — handler, service, repository, SQL query, test files — all wired, all annotated, all following the conventions the tool already understands. Frontend scaffolding from the same definition is a longer-term goal that requires bridging the Go-to-TypeScript generation boundary. The feature registry becomes a blueprint, not just documentation. Scaffolding generates the structure; an AI agent fills in the implementation; testreg verifies the coverage. The tool bookends the creative work.

Each additional annotation makes AI work orders more precise. Each one is also maintenance burden. An annotation that drifts from reality is worse than no annotation: it gives false confidence. The right strategy is: derive metadata from code wherever possible (testreg already does this with AST parsing and init --discover), build on existing standards where they cover the use case (OpenAPI, GraphQL schemas), and add custom annotations only for what no standard covers (business feature mapping, test classification).

The more self-describing your codebase is, the less an AI agent needs to explore. testreg is one implementation of this principle for one stack. The principle is stack-agnostic. And we're just getting started.

This is still early. testreg solves visibility into test coverage — not test quality, not code correctness, not security. It's a memory bank and a gap finder for specific workflows: sprint planning, codebase onboarding, bug triage, agent dispatch. One tool in a larger toolkit that's still being built.

But the piece it solves sits at a foundation layer. Knowing which business features are at risk, across every layer of the stack, without re-exploring the codebase every session — that's what other tools can build on. And the pattern underneath it — structured metadata feeding AI execution, static analysis compressing exploration into structured output, ecosystem tooling making codebases self-describing — that pattern has a long way to run.

Your coverage percentage is lying to you. Now you know what to ask instead.

testreg on GitHub

Building Applications with the Right Tools

Alejandro Sosa — Tue, 27 May 2025 23:17:06 +0000

Throughout my career, I've worked in a variety of environments, from fast-paced startups to established software factories. Whether dealing with in-house code or outsourced teams (let's move away from the term "talent"), one constant remains: development processes are often optimized for small or greenfield projects, which serve as the foundation for a company's codebase. While it's relatively easy to swap out developers on a project, this does not mean that technical debt—especially from choosing the wrong language or tool for the job—will be less costly than investing in the right technology from the outset.

As a case study, I'll discuss a project I've been developing over the past year, which I intend to build into a company. The backend was initially written entirely in Golang, chosen for its efficiency and speed in cloud environments. Go compiles quickly and is exceptionally performant, making it ideal for scalable cloud applications. However, as I began integrating Gremlin (Apache TinkerPop) for graph database operations, I encountered significant challenges: the Gremlin API is written in Groovy, a JVM language, and is best supported by other JVM languages like Java and Scala.

Adhering strictly to the Gremlin API using Go proved difficult. I found myself relying on Java documentation to understand which parameters were accepted, which quickly became unsustainable. Eventually, I decided to refactor and migrate this repository layer to a JVM language. Not being particularly fond of Java, I explored Scala, which offers strong integration with other Apache projects like Kafka—opening up future possibilities for seamless tool integration. Although new to Scala, a few days of studying documentation, consuming "Rock the JVM" content, and leveraging Copilot helped me grasp the basics. My prior experience with the Gremlin API, combined with Scala's clear method overloads, allowed me to rapidly improve both my integration and my understanding of the framework.

I'll skip the technical details of integrating the Go-based domain codebase, generating services in Go, and consuming the repository layer in Scala—suffice it to say that gRPC, wrappers, and facades abstract the flows and unify both languages within a single codebase. The key takeaway is this: while Go is excellent for cloud-native development, it cannot do everything. Choosing the right tool for each part of your system not only streamlines development but also future-proofs your codebase. Too often, teams defer these decisions, thinking, "We'll migrate and improve the codebase later," but in reality, such migrations rarely happen. Selecting the right tools from the beginning ensures that any future refactoring is an enhancement, not a costly migration between languages or frameworks.

System Design - Building a Scalable Real-Time Blog Platform with Kafka and Cassandra

Alejandro Sosa — Wed, 13 Nov 2024 18:12:42 +0000

In the world of modern web applications, achieving scalability and real-time responsiveness is essential. Two technologies that excel in these areas are Apache Kafka and Apache Cassandra. This post explores how to integrate Kafka and Cassandra to build a robust blog platform, driven by data access patterns and designed for future scalability and analytics.

Understanding the Components

Apache Kafka

Kafka is a distributed streaming platform that enables the building of real-time data pipelines and streaming applications.
It excels at handling high-throughput, low-latency data feeds.
Acts as a real-time messaging system, decoupling data producers from consumers.

Apache Cassandra

Cassandra is a distributed NoSQL database designed for handling large amounts of data across many servers.
Provides high availability with no single point of failure.
Optimized for write-heavy workloads and linear scalability.

Modeling Data Based on Access Patterns

---
title: Blog example
---
erDiagram
    User {
        string firstName
        string lastName
        string userName
        string email
    }
    Post {
        string title
        string content
        int likes
    }
    Comment {
        string content
        int likes
    }
    Category {
        string name
    }
    User only one to zero or more Post : has
    Post one or more to one or more Category : in
    User only one to zero or more Comment : makes
    Post only one to zero or more Comment : has

In Cassandra, data modeling starts with understanding data access patterns, which are essentially the questions your application needs to answer efficiently. Unlike traditional relational databases, Cassandra encourages designing tables around queries to optimize performance.

Identifying Key Questions

We will cover the flow and process for designing the insertion of a post.

Here are some critical questions for our blog platform that dictate the table designs:

How to retrieve all posts by a specific user?
How to retrieve all posts within a specific category?
How to count the total number of posts in each category?
How to track a user's activity and engagement over time?
How to maintain user-specific post counts?
How to manage active posts and their categories?
How to count active posts in each category?
How to log user activities by category?
How to count user activities overall and by category?
How to log and count category-specific activities?

Mapping Questions to Tables

For each question, we design tables to provide efficient answers:

Posts by User
- Table: posts_by_user
- Stores posts keyed by user ID for quick retrieval of a user's posts.
Posts by Category
- Table: posts_by_category
- Indexes posts by category for efficient category-based queries.
Post Counts by Category
- Table: posts_count
- Table: post_count_by_category
- Maintains counts of posts overall and per category for analytics.
User Activity Tracking
- Table: user_activity
- Table: user_activity_by_category
- Logs user actions and behaviors over time.
User Post Counters
- Table: user_posts_count
- Table: user_post_count_by_category
- Maintains counts of user posts overall and per category.
Active Posts Management
- Table: active_posts
- Table: user_active_posts
- Table: user_active_posts_by_category
- Table: active_posts_by_category
- Keeps track of currently active posts and categorizes them accordingly.
Active Post Counters
- Table: active_posts_count
- Table: active_posts_count_by_category
- Maintains counts of active posts overall and per category.
User Active Post Counters
- Table: user_active_posts_count
- Table: user_active_posts_count_by_category
- Maintains counts of active posts overall and per category.
User Activity Counters
- Table: user_activity_count
- Table: user_activity_count_by_category
- Maintains counts of user activities overall and per category.
Category Activity Logs
- Table: category_activity
- Records activities within each category for analysis.
Category Activity Counters
- Table: category_activity_count
- Maintains counts of activities per category.

This comprehensive mapping results in approximately 20 tables, each optimized to answer specific queries essential for the blog platform's functionality and scalability.

High-Level Architecture Overview

To visualize how all the components interact, let's examine the system's high-level architecture using a C4 Containers diagram.

Components of the System

User: The end-user interacting with the blog platform via web or mobile applications.
API Gateway: Manages incoming HTTP requests, handling routing, authentication, and authorization.
Write Service: Handles write operations such as creating posts and comments, persisting data to Cassandra, and producing events to Kafka.
Read Service: Manages read operations, retrieving data from Cassandra based on various queries.
Kafka Cluster: Facilitates asynchronous communication by streaming events between services.
Cassandra Cluster: Serves as the main data store for user data, posts, comments, and activity logs.
Analytics Processor: Consumes events from Kafka to perform analytics and updates Cassandra with aggregated data.

Architecture Containers Diagram

@startuml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml

LAYOUT_TOP_DOWN()

Person(user, "User", "Interacts with the blog platform via web or mobile.")

System_Boundary(blogPlatform, "Blog Platform") {
    Container(apiGateway, "API Gateway", "Nginx", "Routes incoming HTTP requests to appropriate services and handles authentication and authorization.")

    Container(writeService, "Write Service", "Golang", "Handles all write operations such as creating posts, comments, and logging user activities. Persists data to Cassandra and produces events to Kafka.")

    Container(readService, "Read Service", "Golang", "Handles all read operations like retrieving posts, comments, and personalized recommendations by querying Cassandra.")

    Container(kafka, "Apache Kafka", "Kafka Cluster", "Facilitates asynchronous communication between services by storing and managing event streams.")

    Container(cassandra, "Cassandra Cluster", "Apache Cassandra", "Primary data store for user data, posts, comments, and analytics data. Utilizes separate keyspaces for operational and analytics data.")

    Container(analyticsProcessor, "Analytics Processor", "Apache Spark", "Consumes events from Kafka, processes user activity data, and updates analytics tables in Cassandra.")

    Container(newPostConsumer, "NewPostConsumer", "Golang", "Processes new_post events and inserts data into relevant Cassandra tables.")
    Container(newPostCountersConsumer, "NewPostCountersConsumer", "Golang", "Updates post counts in Cassandra based on new_post events.")
    Container(userPostCountersConsumer, "UserPostCountersConsumer", "Golang", "Updates user post counts in Cassandra based on new_post events.")
    Container(newActivePostConsumer, "NewActivePostConsumer", "Golang", "Handles active post data insertion into Cassandra.")
    Container(logUserActivityConsumer, "LogUserActivityConsumer", "Golang", "Logs user activities into Cassandra.")
    Container(logCategoryActivityConsumer, "LogCategoryActivityConsumer", "Golang", "Logs category-specific activities into Cassandra.")
}

Rel(user, apiGateway, "Uses")
Rel(apiGateway, writeService, "Routes write requests to")
Rel(apiGateway, readService, "Routes read requests to")
Rel(writeService, cassandra, "Writes data to")
Rel(writeService, kafka, "Produces events to")
Rel(kafka, newPostConsumer, "Delivers new_post events to")
Rel(kafka, newPostCountersConsumer, "Delivers new_post_counters events to")
Rel(kafka, userPostCountersConsumer, "Delivers user_post_counters events to")
Rel(kafka, newActivePostConsumer, "Delivers new_active_post events to")
Rel(kafka, logUserActivityConsumer, "Delivers log_user_activity events to")
Rel(kafka, logCategoryActivityConsumer, "Delivers log_category_activity events to")
Rel(newPostConsumer, cassandra, "Inserts data into posts_by_user and posts_by_categories")
Rel(newPostCountersConsumer, cassandra, "Updates posts_count and post_count_by_category")
Rel(userPostCountersConsumer, cassandra, "Updates user_posts_count and user_post_count_by_category")
Rel(newActivePostConsumer, cassandra, "Inserts data into active_posts and related tables")
Rel(logUserActivityConsumer, cassandra, "Inserts data into user_activity and related tables")
Rel(logCategoryActivityConsumer, cassandra, "Inserts data into category_activity and related tables")
Rel(writeService, analyticsProcessor, "Sends events for analytics")
Rel(analyticsProcessor, cassandra, "Updates analytics tables in")
Rel(readService, cassandra, "Queries data from")
@enduml

Data Flow Example: Creating a New Post

Let's walk through the process of a user creating a new post and see how Kafka and Cassandra interact in this scenario.

Step-by-Step Process

User Submits a Post
- The user creates a new post via the platform's interface.
- The request is sent to the API Gateway.
API Gateway Routes the Request
- Validates the request and forwards it to the Write Service.
Write Service Processes the Post
- Handles business logic, such as validating the content.
- Inserts data into Cassandra:
  - Adds the post to posts_by_user and posts_by_categories tables.
  - Each insertion accounts for all categories associated with the post.
Event Production to Kafka
- The Write Service produces events to Kafka topics such as new_post.
- Events contain information about the new post and its associated categories.
Kafka Distributes Events
- Kafka efficiently distributes events to all subscribed consumers.
Consumers Process Events
- NewPostConsumer:
  - Inserts data into posts_by_user and posts_by_categories.
  - Produces additional events to topics like new_post_counters, user_post_counters, etc.
- NewPostCountersConsumer:
  - Updates posts_count and post_count_by_category tables.
- UserPostCountersConsumer:
  - Updates user_posts_count and user_post_count_by_category tables.
- NewActivePostConsumer:
  - Inserts data into active_posts and related tables.
- LogUserActivityConsumer:
  - Logs user activities into user_activity and related tables.
- LogCategoryActivityConsumer:
  - Logs category-specific activities into category_activity and related tables.
Analytics Processor Updates Aggregates
- Consumes events from Kafka to perform real-time analytics.
- Updates aggregate data in Cassandra tables like user_activity_count.
Read Service Serves Data to Users
- When other users request data, the Read Service queries Cassandra.
- Retrieves posts, counts, and activity logs efficiently via designed tables.

Sequence Diagrams Explained

To further illustrate the interactions, let's explore how the sequence diagrams fit into the data flow example.

1. New Post Creation

sequenceDiagram
    participant Client as HTTP Client
    participant Server as HTTP Server
    participant Kafka as Kafka Cluster
    participant Consumer as NewPostConsumer
    participant DB as Cassandra Database

    Client->>Server: POST /api/v1/posts (new post data)
    Server->>Kafka: Produce message to "new_post" topic
    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch insert into posts_by_user and posts_by_categories tables
    Consumer->>Kafka: Produce message to "new_post_counters" topic
    Consumer->>Kafka: Produce message to "user_post_counters" topic
    Consumer->>Kafka: Produce message to "new_active_post" topic
    Consumer->>Kafka: Produce message to "log_user_activity" topic
    Consumer->>Kafka: Produce message to "log_category_activity" topic

2. Post Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as NewPostCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update posts_count and post_count_by_category tables

3. User Post Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as UserPostCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update user_posts_count and user_post_count_by_category tables

4. New Active Post Creation

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as NewActivePostConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch insert into active_posts, user_active_posts, user_active_posts_by_category, and active_posts_by_category tables
    Consumer->>Kafka: Produce message to "new_active_post_counters" topic
    Consumer->>Kafka: Produce message to "user_active_post_counters" topic

5. New Active Post Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as NewActivePostCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update active_posts_count and active_posts_count_by_category tables

6. User Active Post Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as UserActivePostCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update user_active_posts_count and user_active_posts_count_by_category tables

7. Log User Activity Creation

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as LogUserActivityConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch insert into user_activity and user_activity_by_category tables
    Consumer->>Kafka: Produce message to "log_user_activity_counters" topic

8. Log User Activity Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as LogUserActivityCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update user_activity_count and user_activity_count_by_category tables

9. Log Category Activity Creation

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as LogCategoryActivityConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch insert into category_activity tables
    Consumer->>Kafka: Produce message to "log_category_activity_counters" topic

10. Log Category Activity Counters Update

sequenceDiagram
    participant Kafka as Kafka Cluster
    participant Consumer as LogCategoryActivityCountersConsumer
    participant DB as Cassandra Database

    Kafka->>Consumer: Deliver message to consumer group
    Consumer->>DB: Batch update category_activity_count tables

These sequence diagrams demonstrate how each Kafka topic interacts with its respective consumer and how data flows into Cassandra. This modular approach ensures that each component has a single responsibility, enhancing maintainability and scalability.

Benefits of This Architecture

Scalability

Kafka and Cassandra are both designed to scale horizontally.
The architecture handles increased load by adding more nodes to the Kafka cluster and Cassandra ring.

Real-Time Processing

Kafka's event streaming enables real-time data processing.
Consumers can react to events as they occur, providing up-to-date information.

High Availability

Cassandra's replication across multiple nodes ensures no single point of failure.
Kafka's distributed nature provides fault tolerance in message processing.

Optimized Queries

Designing tables around specific queries allows Cassandra to retrieve data efficiently.
Denormalized data models reduce the need for complex joins and enable fast reads.

Key Takeaways

Data Access Patterns Drive Design: Start by identifying the questions your application needs to answer. Design your Cassandra tables to provide efficient responses to these queries.
Asynchronous Processing with Kafka: Use Kafka to decouple services and handle event-driven processing. This ensures that write operations don't block read operations and vice versa.
Denormalization for Performance: Embrace denormalization in Cassandra to optimize read performance. Store data in the format that best suits your query patterns.
Scalability and Resilience: Build with technologies that support horizontal scalability and fault tolerance to future-proof your application.
Monitoring and Maintenance: Implement robust monitoring for both Kafka and Cassandra to maintain performance and quickly address issues.

Conclusion

By integrating Apache Kafka and Apache Cassandra, you can build a blog platform that is both scalable and capable of real-time data processing. The key lies in modeling your data based on the application's access patterns and leveraging the strengths of both technologies to handle high-throughput workloads. Leveraging advanced data modeling techniques, as demonstrated in Advanced Data Modeling in Apache Cassandra by DataStax Developers, ensures that your Cassandra database is structured efficiently to support diverse query requirements. Additionally, Cassandra builds upon the foundational concepts introduced in Amazon's Dynamo, offering a more flexible approach without the constraints of single table design inherent in DynamoDB. This architectural synergy not only meets current demands but also provides a solid foundation for future analytics and feature expansions.

Remember, the success of such a system hinges on thoughtful design and a clear understanding of how each component interacts within the architecture. By focusing on the core questions your application needs to answer, you can tailor your data models and services to work harmoniously, delivering a responsive and reliable user experience.

Comparative Benchmarking: ILP, A*, and Branch and Bound Algorithms in High-Throughput Scenarios

Alejandro Sosa — Tue, 05 Nov 2024 13:19:06 +0000

In this blog post, we will compare the performance of three different algorithms used in a recent personal project: the ILP (Integer Linear Programming) algorithm, the Local algorithm utilizing the A* algorithm, and an optimized solution using the Branch and Bound algorithm. All algorithms were tested using the same dataset, with the ILP and Branch and Bound implementations sharing the same workload, while the A* implementation was limited due to performance constraints.

Disclaimer: While I will not delve into the project's specific code details, I will share some insights from it. The codebase is not intended for public disclosure, and this serves as a disclaimer to respect its confidentiality.

Benchmark Results

Here are the benchmark results for all three algorithms:

goos: linux
goarch: amd64
pkg: github.com/sosalejandro/<my-project>/<my-package>/pkg
cpu: 13th Gen Intel(R) Core(TM) i7-13700HX

BenchmarkGenerateReportILP-24                            724       1694029 ns/op       30332 B/op        181 allocs/op
BenchmarkGenerateReportILPParallel-24                   6512        187871 ns/op       34545 B/op        184 allocs/op
BenchmarkGenerateReportLocal-24                            2     851314106 ns/op    559466456 B/op   7379756 allocs/op
BenchmarkBranchGenerateReportLocal-24                 101449         12106 ns/op       29932 B/op        165 allocs/op
BenchmarkGenerateReportLocalParallel-24                    3     349605952 ns/op    559422440 B/op   7379837 allocs/op
BenchmarkBranchGenerateReportLocalParallel-24         120543         10755 ns/op       29933 B/op        165 allocs/op
PASS
coverage: 81.4% of statements
ok      github.com/sosalejandro/<my-project>/<my-package>/pkg   11.121s

Workload Configuration

All algorithms were tested using the same set of data, but the workload (i.e., the number of times each item is processed) differed between the implementations.

ILP and Branch and Bound Implementation Workload:

plan := []Plan{
    {ID: "1", Times: 100},
    {ID: "2", Times: 150},
    {ID: "3", Times: 200},
    {ID: "8", Times: 50},
    {ID: "9", Times: 75},
    {ID: "10", Times: 80},
    {ID: "11", Times: 90},
    {ID: "12", Times: 85},
    {ID: "13", Times: 60},
    {ID: "14", Times: 110},
}

A* Implementation Workload:

plan := []Plan{
    {ID: "1", Times: 1},
    {ID: "2", Times: 1},
    {ID: "3", Times: 5},
    {ID: "8", Times: 5},
    {ID: "9", Times: 5},
    {ID: "10", Times: 5},
    {ID: "11", Times: 9},
    {ID: "12", Times: 5},
    {ID: "13", Times: 5},
    {ID: "14", Times: 5},
}

Workload Analysis

To understand the impact of these workloads on the benchmark results, let's calculate the total number of iterations (i.e., the sum of the Times values) for each implementation.

Total Iterations:

ILP and Branch and Bound Implementations:

  100 + 150 + 200 + 50 + 75 + 80 + 90 + 85 + 60 + 110 = 1000

A* Implementation:

  1 + 1 + 5 + 5 + 5 + 5 + 9 + 5 + 5 + 5 = 46

Workload Ratio:

ILP Iterations / A* Iterations = 1000 / 46 ≈ 21.74

This means the ILP and Branch and Bound implementations are handling approximately 21.74 times more iterations compared to the A* implementation.

Performance Comparison

Let's break down the benchmark results in relation to the workload differences.

Benchmark	Runs	ns/op	B/op	allocs/op	Total Time (ns)
BenchmarkGenerateReportILP-24	724	1,694,029	30,332	181	≈ 1,225,836,996
BenchmarkGenerateReportILPParallel-24	6,512	187,871	34,545	184	≈ 1,223,607,552
BenchmarkBranchGenerateReportLocal-24	101,449	12,106	29,932	165	≈ 1,224,505,394
BenchmarkGenerateReportLocal-24	2	851,314,106	559,466,456	7,379,756	≈ 1,702,628,212
BenchmarkGenerateReportLocalParallel-24	3	349,605,952	559,422,440	7,379,837	≈ 1,048,817,856
BenchmarkBranchGenerateReportLocalParallel-24	120,543	10,755	29,933	165	≈ 1,295,219,065

Observations

Execution Time per Operation:
- BenchmarkGenerateReportILP-24 vs BenchmarkBranchGenerateReportLocal-24:
  - Branch and Bound is 99.29% faster than ILP, reducing execution time from 1,694,029 ns/op to 12,106 ns/op.

BenchmarkGenerateReportILP-24 vs BenchmarkGenerateReportLocal-24:
- ILP is 99.80% faster than Local, reducing execution time from 851,314,106 ns/op to 1,694,029 ns/op.
BenchmarkGenerateReportILPParallel-24 vs BenchmarkBranchGenerateReportLocalParallel-24:
- Branch and Bound Parallel is 94.28% faster than ILP Parallel, reducing execution time from 187,871 ns/op to 10,755 ns/op.
BenchmarkGenerateReportILPParallel-24 vs BenchmarkGenerateReportLocalParallel-24:
- ILP Parallel is 99.95% faster than Local Parallel, reducing execution time from 349,605,952 ns/op to 187,871 ns/op.

Memory Allocations:
- ILP Implementations: Slight increase in memory usage and allocations when running in parallel.
- Branch and Bound Implementations: Lower memory usage and allocations compared to the A* implementations.
- A* Implementations: Extremely high memory allocations, leading to inefficient resource utilization.
Throughput:
- ILP Parallel and Branch and Bound Parallel can handle approximately 21.74 times more iterations due to the higher workload.
- A* Implementations struggle with throughput not due to the significantly lower number of iterations but due to inefficient memory usage and implementation.

Impact of Varying Workload on Performance

Given that the ILP and Branch algorithms handle 21.74 times more throughput per test iteration, this difference in workload impacts each algorithm's performance and efficiency:

ILP and Branch Algorithms: As these handle a greater throughput, they are optimized for higher workloads. Despite handling more operations, they maintain faster execution times. This suggests they are not only computationally efficient but also well-suited for high-throughput scenarios.
Local Algorithm: With a smaller throughput and higher execution time, this algorithm is less efficient in handling increased workloads. If scaled to the same throughput as ILP or Branch, its execution time would increase significantly, indicating it’s not ideal for high-throughput cases.

In scenarios where workload is increased, ILP and Branch would outperform Local due to their ability to manage higher throughput efficiently. Conversely, if the workload were reduced, the Local algorithm might perform closer to ILP and Branch but would still likely lag due to fundamental differences in algorithmic efficiency.

Algorithm Overview

To provide a clearer understanding of how each algorithm approaches problem-solving, here's a general overview of their mechanisms and methodologies.

Integer Linear Programming (ILP)

Purpose:

ILP is an optimization technique used to find the best outcome (such as maximum profit or lowest cost) in a mathematical model whose requirements are represented by linear relationships. It is particularly effective for problems that can be expressed in terms of linear constraints and a linear objective function.

General Workflow:

Define Variables:

Identify the decision variables that represent the choices to be made.
Objective Function:

Formulate a linear equation that needs to be maximized or minimized.
Constraints:

Establish linear inequalities or equalities that the solution must satisfy.
Solve:

Utilize an ILP solver to find the optimal values of the decision variables that maximize or minimize the objective function while satisfying all constraints.

Pseudocode:

function ILP_Solve(parameters):
    variables = define_decision_variables(parameters)
    objective = define_objective_function(variables)
    constraints = define_constraints(parameters, variables)

    solver = initialize_ILP_solver()
    solver.set_objective(objective)
    for constraint in constraints:
        solver.add_constraint(constraint)

    result = solver.solve()
    return result

A* Algorithm (Local Implementation)

Purpose:

A* is a pathfinding and graph traversal algorithm known for its performance and accuracy. It efficiently finds the shortest path between nodes by combining features of uniform-cost search and pure heuristic search.

General Workflow:

Initialization:

Start with an initial node and add it to the priority queue.
Loop:
- Remove the node with the lowest cost estimate from the priority queue.
- If it's the goal node, terminate.
- Otherwise, expand the node by exploring its neighbors.
- For each neighbor, calculate the new cost and update the priority queue accordingly.
Termination:

The algorithm concludes when the goal node is reached or the priority queue is empty (indicating no path exists).

Pseudocode:

function AStar(initial_state, goal_state, heuristic):
    open_set = PriorityQueue()
    open_set.push(initial_state, priority=heuristic(initial_state))

    came_from = {}
    g_score = map with default value infinity
    g_score[initial_state] = 0

    while not open_set.is_empty():
        current = open_set.pop()

        if current == goal_state:
            return reconstruct_path(came_from, current)

        for neighbor in current.neighbors:
            tentative_g_score = g_score[current] + cost(current, neighbor)
            if tentative_g_score < g_score[neighbor]:
                came_from[neighbor] = current
                g_score[neighbor] = tentative_g_score
                f_score = tentative_g_score + heuristic(neighbor)
                open_set.push(neighbor, priority=f_score)

    return failure

Branch and Bound Algorithm

Purpose:

Branch and Bound is an optimization algorithm that systematically explores the solution space. It divides the problem into smaller subproblems (branching) and uses bounds to eliminate subproblems that cannot produce better solutions than the current best (bounding).

General Workflow:

Initialization:

Start with an initial solution and set the best known solution.
Branching:

At each node, divide the problem into smaller subproblems.
Bounding:

Calculate an optimistic estimate (upper bound) of the best possible solution in each branch.
Pruning:

Discard branches where the upper bound is worse than the best known solution.
Search:

Recursively explore remaining branches using depth-first or best-first search.
Termination:

When all branches have been pruned or explored, the best known solution is optimal.

Pseudocode:

function BranchAndBound():
    best_solution = None
    best_score = negative_infinity
    initial_state = create_initial_state()
    stack = [initial_state]

    while stack is not empty:
        current_state = stack.pop()
        current_score = evaluate(current_state)

        if current_score > best_score:
            best_score = current_score
            best_solution = current_state

        if can_branch(current_state):
            branches = generate_branches(current_state)
            for branch in branches:
                upper_bound = calculate_upper_bound(branch)
                if upper_bound > best_score:
                    stack.push(branch)
    return best_solution

Comparative Analysis

Feature	ILP Implementation	Local (A*) Implementation	Branch and Bound Implementation
Optimization Approach	Formulates the problem as a set of linear equations and inequalities to find the optimal solution.	Searches through possible states using heuristics to find the most promising path to the goal.	Systematically explores and prunes the solution space to find optimal solutions efficiently.
Scalability	Handles large-scale problems efficiently by leveraging optimized solvers.	Performance can degrade with increasing problem size due to the exhaustive nature of state exploration.	Efficient for combinatorial problems, with pruning reducing the search space significantly.
Development Time	Faster implementation as it relies on existing ILP solvers and libraries.	Requires more time to implement, especially when dealing with complex state management and heuristics.	Moderate development time, balancing complexity and optimization benefits.
Flexibility	Highly adaptable to various linear optimization problems with clear constraints and objectives.	Best suited for problems where pathfinding to a goal is essential, with heuristic guidance.	Effective for a wide range of optimization problems, especially combinatorial ones.
Performance	Demonstrates superior performance in handling a higher number of iterations with optimized memory usage.	While effective for certain scenarios, struggles with high memory allocations and longer execution times under heavy workloads.	Shows significant performance improvements over ILP and A* with optimized memory usage and faster execution times.
Developer Experience	Improves developer experience by reducing the need for extensive coding and optimization efforts.	May require significant debugging and optimization to achieve comparable performance levels.	Balances performance with manageable development effort, leveraging existing strategies for optimization.
Integration	Currently integrates a C++ ILP module with Golang, facilitating efficient computation despite cross-language usage.	Fully implemented within Golang, but may face limitations in performance and scalability without optimizations.	Implemented in Golang, avoiding cross-language integration complexities and enhancing performance.

Implications for Server Performance

Scalability:
- The Branch and Bound implementation demonstrates excellent scalability, efficiently handling a large number of concurrent requests with reduced latency.
- The ILP Parallel implementation also shows excellent scalability, efficiently handling a large number of concurrent requests with reduced latency.
- The A* implementation is unsuitable for high-load environments due to performance limitations.
Resource Utilization:
- Branch and Bound Implementations utilize resources efficiently, with low memory consumption and fast execution times.
- ILP Parallel effectively utilizes multi-core CPUs, providing high throughput with manageable memory consumption.
- A* Implementations consume excessive memory, potentially leading to resource exhaustion.

Workload Impact on Performance

The workload differences influence the performance of the algorithms:

Branch and Bound Implementation handles the same workload as the ILP implementation efficiently, providing fast execution times and low memory usage, making it suitable for scaling.
ILP Implementation handles a larger workload efficiently due to optimized solvers.
A* Implementation struggles with performance due to high execution times and memory usage.

Conclusion

An extra comparison was added using an optimized solution with the Branch and Bound algorithm, which shows how it significantly improved over the ILP and the A* algorithms in terms of performance and resource utilization. The workload used on the Branch and Bound Algorithm is the same as the ILP algorithm.

The Branch and Bound-based BenchmarkBranchGenerateReportLocalParallel function showcases exceptional performance improvements, making it highly suitable for server environments demanding high concurrency and efficient resource management.

By focusing on leveraging the strengths of the Branch and Bound approach and optimizing it for the specific problem, we can ensure that the project remains both performant and scalable, capable of handling increasing demands with ease.

Final Thoughts

Balancing performance, scalability, and developer experience is crucial for building robust applications. The Branch and Bound approach has proven to be the most efficient in the current setup, offering substantial performance gains with reasonable development effort.

By continuously profiling, optimizing, and leveraging the strengths of each algorithmic approach, we can maintain a high-performance, scalable, and developer-friendly system.

Open Telemetry: Observing and Monitoring Applications

Alejandro Sosa — Fri, 08 Mar 2024 03:25:22 +0000

Introduction to OTEL (Open Telemetry)

Open Telemetry, often abbreviated as OTEL, stands as a versatile, open-source, and vendor-agnostic framework empowering developers and DevOps specialists with comprehensive tools for observing and monitoring applications. Covering a spectrum from logging to metrics and traces, Open Telemetry provides a robust foundation for gaining insights into application performance.

While logging is a familiar concept, metrics and traces may require clarification, especially for developers unfamiliar with the term "observability."

Metrics: Unveiling Application Insights

Metrics in Open Telemetry involve aggregated information over the runtime of an application. This encompasses the ability to observe various aspects, such as the number of requests made to a server, the success or failure of requests, resource utilization, and even specifics like garbage collection (GC) metrics, heap usage, and CPU usage. The flexibility of metrics creation extends to anything observable within the application.

While some frameworks offer default metrics, they still need to be gathered and exported through a collector for monitoring on services like Prometheus.

Traces: Navigating Request Lifecycles

Traces, a captivating component of observability, enable the visualization of a request's journey through services like Jaeger or Zipkin. Each request is assigned a unique traceId as the trace begins. Traces facilitate the tracking and observation of a call-chain of requests, akin to tracking a package's journey in a courier and delivery system.

Consider the following questions when analyzing a trace:

How long did the request take?
What entities were involved in handling the request?
Which transport methods or vehicles were employed?

Traces allow for a granular understanding of breaks within the route, their duration, reasons for stops, and the frequency of stops. Open Telemetry's SDK and Trace Provider, along with the utilization of Spans, Span Events, and Span Records, play a crucial role in answering these questions.

Baggage: Enhancing Contextual Information

In the context of Open Telemetry, baggage is akin to the context API in React. It serves as an object that can be propagated through the context to extract data relevant only to a specific request. While suitable for storing small values for tracing and logging, it is essential to avoid sending sensitive data through baggage.

The baggage serves as a valuable tool to enrich spans with detailed information, particularly accessible in specific services or layers of an application.

Putting Knowledge into Practice: Hands-On Experience

To leverage Open Telemetry effectively, it's imperative to engage in hands-on activities. Setting up a Trace Provider, working with Spans, Span Events, and Span Records, and understanding the nuances of baggage usage contribute to a comprehensive and practical understanding of Open Telemetry in action. Dive into the world of Open Telemetry to enhance your application observability and monitoring capabilities.

While many programming languages provide robust support for Open Telemetry, this instance focuses on Golang. It's important to note that, in the current context, the logs SDK for Golang is not implemented. For future reference consult the list of supported languages and explore the Open Telemetry repositories. Always prioritize the main repository and its contrib repository, housing extensions and instrumentation libraries crucial to the Open Telemetry framework. Stay updated with the latest developments to ensure seamless integration and enhanced functionality.

Before delving into hands-on activities, it's crucial to grasp the concept of a "Resource." In the context of Open Telemetry, a resource refers to metadata from our application that is added to the Tracer. Tracers, which are singleton objects, play a pivotal role in manipulating the Trace API and creating Spans.

The core idea behind observability is the effective propagation of context. Later, we will explore how to set up propagation to seamlessly gather data at different points throughout the lifespan of a request.

The following snippet, extracted from the Otel's Golang documentation, exemplifies the creation of resources in Golang:



resources := resource.New(context.Background(),
    resource.WithFromEnv(), // Pull attributes from OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables
    resource.WithProcess(), // Configure a set of Detectors that discover process information
    resource.WithOS(), // Configure a set of Detectors that discover OS information
    resource.WithContainer(), // Configure a set of Detectors that discover container information
    resource.WithHost(), // Configure a set of Detectors that discover host information
    resource.WithDetectors(thirdparty.Detector{}), // Bring your own external Detector implementation
    resource.WithAttributes(attribute.String("foo", "bar")), // Specify resource attributes directly
)

This data is then attached to the parent span:

This code snippet showcases the flexibility of defining resources in Golang. It allows for pulling attributes from environment variables, configuring detectors for various types of information (process, OS, container, and host), and even incorporating custom external detectors. Additionally, specific resource attributes can be specified directly for a more tailored approach. Understanding and effectively utilizing resources is a fundamental step in maximizing the capabilities of Open Telemetry in your application.

The setup

First, we proceed to set up telemetry helpers for our main server. Let's break down each part.

Before configuring the resource, we obtain the application context, which is the very first context as the application starts. We create our resource settings, which we'll later use when setting up the Tracer and Metric Providers.



ctx := context.Background()

res, err := resource.New(ctx,
    resource.WithFromEnv(),
    resource.WithProcess(),
    resource.WithTelemetrySDK(),
    resource.WithHost(),
    resource.WithAttributes(
        // the service name used to display traces in backends
        semconv.ServiceNameKey.String(serverName),
        attribute.String("environment", os.Getenv("GO_ENV")),
    ),
)

handleErr(err, "failed to create resource")

This is an optional step, but the Agent can be configured through an Environment Variable, dynamically setting the exporter address.



otelAgentAddr, ok := os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT")
if !ok {
    otelAgentAddr = "0.0.0.0:4317"
}

Now, we set up the metrics exporter and the metric provider. We specify that we are not using TLS, so the WithInsecure option from otlptracegrpc is passed as a configuration parameter.



metricExp, err := otlpmetricgrpc.New(
    ctx,
    otlpmetricgrpc.WithInsecure(),
    otlpmetricgrpc.WithEndpoint(otelAgentAddr),
)
handleErr(err, "Failed to create the collector metric exporter")

meterProvider := sdkmetric.NewMeterProvider(
    sdkmetric.WithResource(res),
    sdkmetric.WithReader(
        sdkmetric.NewPeriodicReader(
            metricExp,
            sdkmetric.WithInterval(2*time.Second),
        ),
    ),
)
otel.SetMeterProvider(meterProvider)

Finally, we set up the gRPC trace client to manage the exportation of our traces, create the exporter, set up a batch span processor, and instantiate our Trace Provider. We then set a propagation strategy and the trace provider.



traceClient := otlptracegrpc.NewClient(
    otlptracegrpc.WithInsecure(),
    otlptracegrpc.WithEndpoint(otelAgentAddr),
    otlptracegrpc.WithDialOption(grpc.WithBlock()),
)
traceExp, err := otlptrace.New(ctx, traceClient)
handleErr(err, "Failed to create the collector trace exporter")

bsp := sdktrace.NewBatchSpanProcessor(traceExp)
tracerProvider := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(getSampler()),
    sdktrace.WithResource(res),
    sdktrace.WithSpanProcessor(bsp),
)

// set global propagator to tracecontext (the default is no-op).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
otel.SetTracerProvider(tracerProvider)

All together create the following file

otel.go



package telemetry

import (
    "context"
    "log"
    "os"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
    "google.golang.org/grpc"
)

// Initializes an OTLP exporter, and configures the corresponding trace and
// metric providers.
func InitProvider(serverName string) func() {
    ctx := context.Background()

    res, err := resource.New(ctx,
        resource.WithFromEnv(),
        resource.WithProcess(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
        resource.WithAttributes(
            // the service name used to display traces in backends
            semconv.ServiceNameKey.String(serverName),
            attribute.String("environment", os.Getenv("GO_ENV")),
        ),
    )
        handleErr(err, "failed to create resource")

    otelAgentAddr, ok := os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if !ok {
        otelAgentAddr = "0.0.0.0:4317"
    }

    metricExp, err := otlpmetricgrpc.New(
        ctx,
        otlpmetricgrpc.WithInsecure(),
        otlpmetricgrpc.WithEndpoint(otelAgentAddr))
    handleErr(err, "Failed to create the collector metric exporter")

    meterProvider := sdkmetric.NewMeterProvider(
        sdkmetric.WithResource(res),
        sdkmetric.WithReader(
            sdkmetric.NewPeriodicReader(
                metricExp,
                sdkmetric.WithInterval(2*time.Second),
            ),
        ),
    )
    otel.SetMeterProvider(meterProvider)

    traceClient := otlptracegrpc.NewClient(
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint(otelAgentAddr),
        otlptracegrpc.WithDialOption(grpc.WithBlock()))
    traceExp, err := otlptrace.New(ctx, traceClient)
    handleErr(err, "Failed to create the collector trace exporter")

    bsp := sdktrace.NewBatchSpanProcessor(traceExp)
    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(getSampler()),
        sdktrace.WithResource(res),
        sdktrace.WithSpanProcessor(bsp),
    )

    // set global propagator to tracecontext (the default is no-op).
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
    otel.SetTracerProvider(tracerProvider)

    return func() {
        cxt, cancel := context.WithTimeout(ctx, time.Second)
        defer cancel()
        if err := traceExp.Shutdown(cxt); err != nil {
            otel.Handle(err)
        }
        // pushes any last exports to the receiver
        if err := meterProvider.Shutdown(cxt); err != nil {
            otel.Handle(err)
        }
    }
}

func handleErr(err error, message string) {
    if err != nil {
        log.Fatalf("%s: %v", message, err)
    }
}

// Helper function to define sampling.
// When in development mode, AlwaysSample is defined,
// otherwise, sample based on Parent and IDRatio will be used.
func getSampler() sdktrace.Sampler {
    ENV := os.Getenv("GO_ENV")
    switch ENV {
    case "development":
        return sdktrace.AlwaysSample()
    case "production":
        return sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.5))
    default:
        return sdktrace.AlwaysSample()
    }
}

Now for our example server, we'll use the following code:

First, we instantiate our providers from the telemetry package, deferring the provider's cleanup function to be executed prior to stopping our program.



otelShutdown := telemetry.InitProvider(serverName)
defer otelShutdown()

On this program we'll be using the Gorilla Mux and the instrumentation package from the opentelemetry-go-contrib otelmux. Explained in detail, after instantiating the router we are adding the otelmux middleware for adding the option which formats the Span Name with the method and the route used.

In this program, we utilize the Gorilla Mux and the instrumentation package from opentelemetry-go-contrib (otelmux). After instantiating the router, we add the otelmux middleware to format the Span Name with the method and the route used.



router := mux.NewRouter()
router.Use(
    otelmux.Middleware(
        serverName,
        otelmux.WithSpanNameFormatter(func(routeName string, r *http.Request) string {
            return fmt.Sprintf("%s %s", r.Method, routeName)
        }),
    ),
)

Moving forward to our handler, streamlined with a few objects and methods from the OpenTelemetry framework. We leverage the baggage package to obtain the baggage propagated through the context. After obtaining the span from the context, we use the getter method Member from the baggage package to retrieve the values of destination and transportation (defined initially). We then create trace attributes for them and add them to a Span Event.



router.HandleFunc("/packages/{id:[0-9]+}", func(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    id := vars["id"]
    // package response
    pr := getPackage(r.Context(), id)

    baggage := baggage.FromContext(r.Context())

    // late acquisition of the span to add attributes
    span := trace.SpanFromContext(r.Context())
    destination := baggage.Member("destination").Value()
    transportation := baggage.Member("transportation").Value()
    destinationAttr := trace.WithAttributes(attribute.String("destination", destination))
    transportationAttr := trace.WithAttributes(attribute.String("transportation", transportation))
    span.AddEvent("Obtaining package", destinationAttr, transportationAttr)

    reply := fmt.Sprintf("package is %s (id %s)\n", pr, id)
    _, _ = w.Write(([]byte)(reply))
})

At last, we define the getPackage method, taking the context as the first argument. The context must be propagated at all times, not just for handling cancellation, but also playing a key role in telemetry. We instantiate a new child span from our Provider and the current context. We defer its cleanup function when we call defer span.End() to ensure no resources are leaked. We create a Span Event named getPackage and add the trace attribute for the package and the id. If the package is found, we add another event; if it fails, we record the exception within the span.



func getPackage(ctx context.Context, id string) string {
    _, span := trace.SpanFromContext(ctx).TracerProvider().Tracer(serverName).Start(ctx, "getPackage")
    defer span.End()

    span.AddEvent("getPackage", trace.WithAttributes(attribute.String("package", id)))
    if id == "123" {
        span.AddEvent("found package")
        return "found package"
    }
    span.RecordError(fmt.Errorf("package not found"))
    return "unknown"
}

All put together, it creates the following file:

main.go



package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "time"

    "github.com/gorilla/mux"
    "github.com/sosalejandro/otel-example/commons/telemetry"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/baggage"
    "go.opentelemetry.io/otel/trace"
)

const serverName = "otel-example-server"

func main() {
    // ...

    otelShutdown := telemetry.InitProvider(serverName)
    defer otelShutdown()

    router := mux.NewRouter()
    router.Use(
        otelmux.Middleware(
            serverName,
            otelmux.WithSpanNameFormatter(func(routeName string, r *http.Request) string {
                return fmt.Sprintf("%s %s", r.Method, routeName)
            })),
    )

    router.HandleFunc("/packages/{id:[0-9]+}", func(w http.ResponseWriter, r *http.Request) {
        vars := mux.Vars(r)
        id := vars["id"]
        // package response
        pr := getPackage(r.Context(), id)

        baggage := baggage.FromContext(r.Context())

        // late adquisition of the span to add attributes
        span := trace.SpanFromContext(r.Context())
        destination := baggage.Member("destination").Value()
        transportation := baggage.Member("transportation").Value()
        destinationAttr := trace.WithAttributes(attribute.String("destination", destination))
        transportationAttr := trace.WithAttributes(attribute.String("transportation", transportation))
        span.AddEvent("Obtaining package", destinationAttr, transportationAttr)

        reply := fmt.Sprintf("package is %s (id %s)\n", pr, id)
        _, _ = w.Write(([]byte)(reply))
    })

    server := &http.Server{
        Addr:         ":8080",
        Handler:      router,
        ReadTimeout:  500 * time.Millisecond,
        WriteTimeout: 1 * time.Second,
        IdleTimeout:  15 * time.Second,
    }

    if err := runServer(server); err != nil {
        log.Fatalf("Failed to start server: %v", err)
    }
}

func runServer(server *http.Server) error {
    // Start the server in a separate goroutine
    go func() {
        if err := server.ListenAndServe(); err != nil {
            log.Printf("Server error: %v", err)
        }
    }()

    // Wait for an interrupt signal to gracefully shut down the server
    stop := make(chan os.Signal, 1)
    signal.Notify(stop, os.Interrupt)
    <-stop

    log.Println("Shutting down server...")

    // Create a context with a timeout of 5 seconds to allow outstanding requests to finish
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    // Shut down the server gracefully
    if err := server.Shutdown(ctx); err != nil {
        log.Printf("Server shutdown error: %v", err)
        return err
    }

    log.Println("Server shut down.")

    return nil
}

func getPackage(ctx context.Context, id string) string {
    _, span := trace.SpanFromContext(ctx).TracerProvider().Tracer(serverName).Start(ctx, "getPackage")
    defer span.End()

    span.AddEvent("getPackage", trace.WithAttributes(attribute.String("package", id)))
    if id == "123" {
        span.AddEvent("found package")
        return "found package"
    }
    span.RecordError(fmt.Errorf("package not found"))
    return "unknown"
}

Now, for the client application, we define a simple initTracer within the main.go file for our client application:



func initTracer() (*sdktrace.TracerProvider, error) {
    ctx := context.Background()

    res, _ := resource.New(ctx,
        resource.WithFromEnv(),
        resource.WithProcess(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
        resource.WithAttributes(
            semconv.ServiceNameKey.String(serverName),
            attribute.String("environment", os.Getenv("GO_ENV")),
        ),
    )

    otelAgentAddr, ok := os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT")

    if !ok {
        otelAgentAddr = "0.0.0.0:4317"
    }

    traceClient := otlptracegrpc.NewClient(
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint(otelAgentAddr),
        otlptracegrpc.WithDialOption(grpc.WithBlock()))
    traceExp, err := otlptrace.New(ctx, traceClient)
    telemetry.HandleErr(err, "Failed to create the collector trace exporter")

    bsp := sdktrace.NewBatchSpanProcessor(traceExp)
    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(telemetry.GetSampler()),
        sdktrace.WithResource(res),
        sdktrace.WithSpanProcessor(bsp),
    )

    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
    otel.SetTracerProvider(tracerProvider)
    return tracerProvider, nil
}

Moving forward to the main function, we declare our tracer and a cleanup function:



tp, err := initTracer()
if err != nil {
    log.Fatal(err)
}
defer func() {
    if err := tp.Shutdown(context.Background()); err != nil {
        telemetry.HandleErr(err, "Error shutting down tracer provider")
    }
}()

Next, we create our URL and an HTTP client using the otelhttp instrumentation package and its Transport:



url := flag.String("server", "http://localhost:8080/packages/123", "server url")
flag.Parse()

client := http.Client{
    Transport: otelhttp.NewTransport(
        http.DefaultTransport,
        otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {
            return otelhttptrace.NewClientTrace(ctx)
        }),
    ),
}

Now, we set the baggage values and add it to the context:



bag, _ := baggage.Parse("destination=newyork,transportation=truck")
ctx := baggage.ContextWithBaggage(context.Background(), bag)

Finally, we define our body variable, obtain the tracer, and create an anonymous function, which will be our handler in this case. We start our span, define its name, and add the trace attribute using the semantic convention for Peer Service, declaring otel-example-server as our peer service. We defer the span and create the request:



var body []byte

tr := otel.Tracer(serverName)
err = func(ctx context.Context) error {
    ctx, span := tr.Start(
        ctx,
        "Otel propagation example: sending package from Boston",
        trace.WithAttributes(semconv.PeerService("otel-example-client")))
    defer span.End()
    req, _ := http.NewRequestWithContext(ctx, "GET", *url, nil)

    fmt.Printf("Sending request...\n")
    span.AddEvent("Sending request...")
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    body, err = io.ReadAll(res.Body)
    span.AddEvent("Request received")
    _ = res.Body.Close()

    return err
}(ctx)

At last, we handle the error and wait for 10 seconds while the spans are exported:



if err != nil {
    telemetry.HandleErr(err, "Error executing handler request")
}

fmt.Printf("Response Received: %s\n\n\n", body)
fmt.Printf("Waiting for a few seconds to export spans ...\n\n")
time.Sleep(10 * time.Second)
fmt.Printf("Inspect traces on Jaeger\n")

Everything put together within the main.go looks like this:



package main

import (
    "context"
    "flag"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/http/httptrace"
    "os"
    "time"

    "github.com/sosalejandro/otel-example/commons/telemetry"
    "go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/baggage"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
    "go.opentelemetry.io/otel/trace"
    "google.golang.org/grpc"
)

const serverName = "otel-example-client"

func initTracer() (*sdktrace.TracerProvider, error) {
    ctx := context.Background()

    res, _ := resource.New(ctx,
        resource.WithFromEnv(),
        resource.WithProcess(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
        resource.WithAttributes(
            // the service name used to display traces in backends
            semconv.ServiceNameKey.String(serverName),
            attribute.String("environment", os.Getenv("GO_ENV")),
        ),
    )

    otelAgentAddr, ok := os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT")

    if !ok {
        otelAgentAddr = "0.0.0.0:4317"
    }

    traceClient := otlptracegrpc.NewClient(
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint(otelAgentAddr),
        otlptracegrpc.WithDialOption(grpc.WithBlock()))
    traceExp, err := otlptrace.New(ctx, traceClient)
    telemetry.HandleErr(err, "Failed to create the collector trace exporter")

    bsp := sdktrace.NewBatchSpanProcessor(traceExp)
    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(telemetry.GetSampler()),
        sdktrace.WithResource(res),
        sdktrace.WithSpanProcessor(bsp),
    )

    // set global propagator to tracecontext (the default is no-op).
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
    otel.SetTracerProvider(tracerProvider)
    return tracerProvider, nil
}

func main() {
    tp, err := initTracer()
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            telemetry.HandleErr(err, "Error shutting down tracer provider")
        }
    }()

    url := flag.String("server", "http://localhost:8080/packages/123", "server url")
    flag.Parse()

    client := http.Client{
        Transport: otelhttp.NewTransport(
            http.DefaultTransport,
            otelhttp.WithClientTrace(func(ctx context.Context) *httptrace.ClientTrace {
                return otelhttptrace.NewClientTrace(ctx)
            }),
        ),
    }

    bag, _ := baggage.Parse("destination=newyork,transportation=truck")
    ctx := baggage.ContextWithBaggage(context.Background(), bag)

    var body []byte

    tr := otel.Tracer(serverName)
    err = func(ctx context.Context) error {
        ctx, span := tr.Start(
            ctx,
            "Otel propagation example: sending package from boston",
            trace.WithAttributes(semconv.PeerService("otel-example-server")))
        defer span.End()
        req, _ := http.NewRequestWithContext(ctx, "GET", *url, nil)

        span.AddEvent("Sending request...")
        res, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, err = io.ReadAll(res.Body)
        span.AddEvent("Request received")
        _ = res.Body.Close()

        return err
    }(ctx)

    if err != nil {
        telemetry.HandleErr(err, "Error executing handler request")
    }

    fmt.Printf("Response Received: %s\n\n\n", body)
    fmt.Printf("Waiting for few seconds to export spans ...\n\n")
    time.Sleep(10 * time.Second)
    fmt.Printf("Inspect traces on jaeger\n")
}

The Containers Setup

jaeger-ui.json



{
    "monitor": {
      "menuEnabled": true
    },
    "dependencies": {
      "menuEnabled": true
    }
  }

otel-collector-config.yml



receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

  logging:

  zipkin:
    endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
    format: proto

  otlp:
    endpoint: jaeger-all-in-one:4317
    tls:
      insecure: true

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, zipkin, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

prometheus.yml



scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8889']
      - targets: ['otel-collector:8888']

docker-compose.yml



version: '3'

services:
  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    restart: always
    ports:
      - "16686:16686"
      - "14268"
      - "14250"

  # Zipkin
  zipkin-all-in-one:
    image: openzipkin/zipkin:latest
    restart: always
    ports:
      - "9411:9411"

  # Collector
  otel-collector:
    image: otel/opentelemetry-collector:latest
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "55679:55679" # zpages extension
    depends_on:
      - jaeger-all-in-one
      - zipkin-all-in-one

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

At last, a Makefile to help us setup and clean things easily:



# Build stage
build:
    @echo "Creating docker compose..."
    docker compose create
    @echo "Building server app..."
    go build -o server_app ./app1/main.go 
    @echo "Building client app..."
    go build -o client_app ./app2/main.go 
    @echo "Build stage completed."

setup:
    @echo "Setting up docker compose..."
    docker compose up -d
    @echo "Setting up server app..."
    ./server_app & echo $$! > server_app.pid
    @echo "Setup stage completed."

run:
    @echo "Running client app..."
    ./client_app 
    @echo "Run stage completed."

clean:
    @echo "Cleaning up..."
    docker compose down
    @echo "Cleaning up server app..."
    kill `cat server_app.pid`
    rm -f server_app server_app.pid
    @echo "Cleaning up client app..."
    rm -f client_app
    @echo "Clean stage completed."

We'll wrap-up with a demonstration of the following code when ran and how it looks on either Jaeger and Zipkin.



make build
Creating docker compose...
docker compose create
[+] Creating 5/5
 ✔ Network otel-example-go_default                Created                                                                                                                                                                                            0.1s 
 ✔ Container otel-example-go-zipkin-all-in-one-1  Created                                                                                                                                                                                            0.2s 
 ✔ Container otel-example-go-jaeger-all-in-one-1  Created                                                                                                                                                                                            0.2s 
 ✔ Container prometheus                           Created                                                                                                                                                                                            0.2s 
 ✔ Container otel-example-go-otel-collector-1     Created                                                                                                                                                                                            0.1s 
Building server app...
go build -o server_app ./app1/main.go 
Building client app...
go build -o client_app ./app2/main.go 
Build stage completed.



make setup
Setting up docker compose...
docker compose up -d
[+] Running 4/4
 ✔ Container otel-example-go-zipkin-all-in-one-1  Started                                                                                                                                                                                            1.5s 
 ✔ Container prometheus                           Started                                                                                                                                                                                            1.5s 
 ✔ Container otel-example-go-jaeger-all-in-one-1  Started                                                                                                                                                                                            1.5s 
 ✔ Container otel-example-go-otel-collector-1     Started                                                                                                                                                                                            1.0s 
Setting up server app...
./server_app & echo $! > server_app.pid
Setup stage completed.



make run
Running client app...
./client_app 
Response Received: package is found package (id 123)



Waiting for few seconds to export spans ...

Inspect traces on jaeger
Run stage completed.

Jaeger:
The Jaeger dashboard is simple to use, we'll just select the service we want to observe, and Jaeger will retrieve all the spans related to this service.

Zipkin:
We'll need to query the otel-example-client within the Zipkin dashboard to retrieve the traces related to our sender app and visualize how the propagation is handled.

At last, we clean up:



make clean
Cleaning up...
docker compose down
[+] Running 5/5
 ✔ Container prometheus                           Removed                                                                                                                                                                                            0.5s 
 ✔ Container otel-example-go-otel-collector-1     Removed                                                                                                                                                                                            0.6s 
 ✔ Container otel-example-go-zipkin-all-in-one-1  Removed                                                                                                                                                                                            2.7s 
 ✔ Container otel-example-go-jaeger-all-in-one-1  Removed                                                                                                                                                                                            0.6s 
 ✔ Network otel-example-go_default                Removed                                                                                                                                                                                            0.2s 
Cleaning up server app...
kill `cat server_app.pid`
rm -f server_app server_app.pid
Cleaning up client app...
rm -f client_app
Clean stage completed.

Now, we are able to trace how the petitions work, are handled, if any error occurred. We are able to enrich the traces with metadata for observability which adds more context to the transaction so we can make the right guesses about how the system works, if the values are sent, what is sent and how long it takes to fulfill the request.

We'll wrap-up for now and we'll continue with metrics on a new post.

The URL for accessing the Jaeger and Zipkin dashboards are
localhost:16686 and localhost:9411 respectively.

sosalejandro / otel-example-go

Open Telemetry demostration.

Introduction to OTEL (Open Telemetry)

While logging is a familiar concept, metrics and traces may require clarification, especially for developers unfamiliar with the term "observability."

Metrics: Unveiling Application Insights

While some frameworks offer default metrics, they still need…

View on GitHub