DEV Community: aymane

OBDA and Virtual Knowledge Graphs: A Practical Introduction

aymane — Sat, 12 Sep 2026 22:45:05 +0000

When working with enterprise data, one problem appears again and again: the data is spread across different systems, but applications need to work with it as if it belonged to one coherent model.

A company might have:

PostgreSQL databases
Oracle databases
data warehouses
legacy systems
APIs
CSV files
different databases owned by different teams

The schemas are usually different, even when the systems describe related things.

For example, one system might have:

customer_id

another:

client_number

and another:

account_id

All three might represent the same business concept: a customer.

One approach is to copy everything into a new database or knowledge graph.

Another approach is to leave the data where it is and put a semantic layer on top of it.

This is where Ontology-Based Data Access (OBDA) and Virtual Knowledge Graphs (VKGs) come in.

What is OBDA?

Ontology-Based Data Access is an approach where users access data through an ontology instead of directly interacting with the schemas of the underlying databases.

The ontology provides the conceptual model.

For example:

Customer
Order
Product

Customer ── places ──> Order
Order ── contains ──> Product

The database doesn't have to use these names.

It could look like:

crm_customer
------------
cust_no
cust_nm
country_code

and:

sales_order
-----------
order_no
cust_no
total

The ontology can still expose these tables as:

Customer
Order

The connection between the physical schema and the ontology is defined through mappings.

The important idea is:

Ontology
   ↓
Mappings
   ↓
Data sources

The ontology describes what the data means.

The mappings describe where that data comes from.

The actual data remains in the original systems.

The Three Main Components

An OBDA system can be understood through three main components:

Ontology
Mappings
Data sources

1. Ontology

The ontology describes the concepts and relationships that users work with.

For example:

Customer
Order
Product

with relationships:

Customer ── places ──> Order
Order ── contains ──> Product

It can also contain additional semantic information.

For example:

PremiumCustomer subclassOf Customer

or:

Order hasCustomer Customer

The ontology represents the domain from the point of view of the application or user.

It does not need to mirror the database schema.

2. Mappings

Mappings connect the ontology to the physical data.

Suppose the database contains:

SELECT id, name, country
FROM customers;

A mapping can define that the rows returned by this query represent instances of:

Customer

and that:

customers.name

represents:

Customer.name

while:

customers.country

represents:

Customer.country

The database might therefore look like:

customers
---------
id
name
country

while the semantic model looks like:

Customer
 ├── id
 ├── name
 └── country

The mapping connects the two.

This separation is important because the physical schema can change without necessarily changing the conceptual model.

3. Data Sources

The data sources are the systems containing the actual data.

For example:

PostgreSQL
Oracle
MySQL
SQL Server
Data warehouse
Legacy database
API

An OBDA system can sit above these sources and provide a semantic interface.

The important point is that the data does not have to be copied into the semantic layer.

What is a Virtual Knowledge Graph?

A knowledge graph represents information as entities and relationships.

For example:

Customer123 ── places ──> Order456

Order456 ── contains ──> Product789

With RDF, this can be represented as triples:

Customer123 → places → Order456
Order456    → contains → Product789

A Virtual Knowledge Graph provides this graph representation without requiring the underlying data to be physically stored as RDF.

The database remains the source of truth.

Conceptually:

Relational database
        ↓
     Mappings
        ↓
Virtual RDF graph

The graph is virtual because it is derived from the source data when needed.

Materialized Knowledge Graph vs Virtual Knowledge Graph

There are two different approaches.

Materialized Knowledge Graph

The data is extracted and loaded into a graph database.

Database
   │
   │ ETL
   ▼
Knowledge Graph

The graph contains its own copy of the data.

This can be useful when graph queries need to be served independently of the original database.

But it introduces another copy of the data.

You now have to think about:

Source database
       ↓
ETL
       ↓
Knowledge graph

and keeping the two synchronized.

Virtual Knowledge Graph

With a VKG:

Database
   │
   ▼
Mappings
   │
   ▼
Virtual graph

The graph is a semantic view over the existing data.

There is no requirement to first copy every row into a graph database.

If the database changes, queries against the virtual graph can see the new data because the data is still being obtained from the source.

This is one of the main reasons VKGs are useful when organizations already have large relational systems.

Querying a Virtual Knowledge Graph

A VKG is normally queried using SPARQL, the standard query language for RDF.

For example:

SELECT ?customer
WHERE {
    ?customer :hasCountry "Morocco" .
}

The user is querying the semantic model.

The database does not necessarily contain a column called:

hasCountry

The mapping might tell the system that:

:hasCountry

corresponds to:

customers.country

The VKG system can then generate a database query similar to:

SELECT id
FROM customers
WHERE country = 'Morocco';

The user works with the ontology.

The database executes SQL.

The system connects the two.

Query Rewriting

Query rewriting is one of the central mechanisms behind OBDA.

Consider this query:

SELECT ?customer
WHERE {
    ?customer :hasCountry "Morocco" .
}

The query is expressed using the ontology.

The OBDA system has to determine how that semantic query can be answered using the underlying database.

A simplified process is:

SPARQL query
     ↓
Ontology-level rewriting
     ↓
Mapping expansion
     ↓
SQL query
     ↓
Database

For example:

:hasCountry

could be mapped to:

customers.country

and:

:Customer

could be mapped to:

customers

The resulting SQL can then be executed directly by the database.

The VKG Query Pipeline

A simplified VKG query pipeline looks like this:

SPARQL query
      ↓
Query rewriting
      ↓
Rewritten query
      ↓
Mapping unfolding
      ↓
SQL query
      ↓
Database
      ↓
SQL result
      ↓
SPARQL result

There are two important transformations here.

Rewriting

The ontology is used to expand the query according to the semantic relationships defined in the ontology.

Unfolding

The mappings are used to translate the ontology-level query into a query over the actual data sources.

The result is a database query that can be executed by the underlying system.

Why Use an Ontology?

A database schema tells you how data is stored.

An ontology is concerned with what the data means.

Consider two databases:

Database A

client_id
client_name

and:

Database B

customer_number
customer_name

From the database perspective, these are different schemas.

From the business perspective, both might represent:

Customer

An ontology can provide that common conceptual layer.

             Customer
             /      \
            /        \
       Database A   Database B

The applications don't need to know that one system calls it client_id and another calls it customer_number.

OBDA vs Data Warehouse

OBDA and a data warehouse solve different problems.

A data warehouse physically stores integrated data.

Source A ─┐
Source B ─┼──> ETL ──> Data Warehouse
Source C ─┘

OBDA instead creates a semantic access layer:

                 Ontology
                    │
                 Mappings
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
       Source A  Source B  Source C

This doesn't mean one approach is better than the other.

They can also be used together.

For example:

                    Semantic Layer
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
         PostgreSQL    Warehouse     Data Lake

The ontology provides a common vocabulary while the underlying systems remain responsible for storing and processing the data.

OBDA vs Knowledge Graph

The terms OBDA and knowledge graph are related, but they are not the same thing.

A knowledge graph is a way of representing information as entities and relationships.

OBDA is an architecture for accessing data through an ontology.

A knowledge graph can be materialized:

Database
   ↓
ETL
   ↓
Graph database

or virtual:

Database
   ↓
Mappings
   ↓
Virtual Knowledge Graph

So a VKG can be viewed as a knowledge graph that is exposed virtually rather than requiring the entire graph to be materialized first.

OBDA Architecture

Putting the pieces together:

                 Application
                      │
                      │ SPARQL
                      ▼
                 ┌───────────┐
                 │ Ontology  │
                 └─────┬─────┘
                       │
                    Mappings
                       │
              ┌────────┼────────┐
              ▼        ▼        ▼
           Database  Database  Database

A real OBDA implementation adds more components around this basic model.

For example:

SPARQL
   ↓
Query Rewriter
   ↓
Mapping / Query Unfolder
   ↓
SQL Generator
   ↓
Database

The exact architecture depends on the implementation.

Where the Complexity Actually Goes

At first glance, the architecture looks simple:

Ontology
   +
Mappings
   +
Database

The difficult part starts when the semantic query becomes complicated.

For example:

Customer
    ↓
Order
    ↓
Product
    ↓
Supplier

might correspond to several joins in the underlying relational schema.

The OBDA engine needs to transform the semantic query into an efficient database query.

That means the system has to deal with things such as:

query rewriting
SQL generation
join optimization
database constraints
query planning
source capabilities
different SQL dialects
large intermediate results

This is why OBDA is closely related to database query processing.

The ontology gives the system the semantic model, but the database still has to execute the final query efficiently.

A Simple Mental Model

A useful way to think about the whole architecture is:

Ontology
"What does this data mean?"

        ↓

Mappings
"Where does this information come from?"

        ↓

VKG
"What does the integrated data look like semantically?"

        ↓

Query rewriting
"How can this semantic query be answered?"

        ↓

SQL
"How do I ask the underlying database?"

        ↓

Database
"Execute it."

That separation is the core idea behind OBDA and Virtual Knowledge Graphs.

Conclusion

OBDA provides a way to put a semantic layer over existing data sources.

The basic model is:

Ontology
    +
Mappings
    +
Data Sources

A Virtual Knowledge Graph builds on this idea by exposing the underlying data as a virtual RDF graph.

The data can remain in relational databases while users query it through concepts such as:

Customer
Order
Product
Supplier

The system then handles the translation:

SPARQL
   ↓
Ontology reasoning / rewriting
   ↓
Mapping unfolding
   ↓
SQL
   ↓
Database

The main advantage is the separation between how data is stored and how data is understood.

The database can continue using tables and columns designed for storage and operations, while the semantic layer provides a model designed around the domain.

That separation is what makes OBDA and Virtual Knowledge Graphs useful when working with heterogeneous data sources without immediately moving everything into a new physical knowledge graph.

My Journey Into Ontology Engineering

aymane — Thu, 10 Sep 2026 20:29:34 +0000

Over the past few months, I have been exploring a topic that has changed the way I think about software, data, and complex systems: ontology engineering.

My journey started with theory.

I began reading C. Maria Keet's An Introduction to Ontology Engineering, and the book helped me build a much stronger foundation for understanding what an ontology actually is, how concepts and relationships can be formally represented, and how ontology engineering connects to practical software systems.

But the more I read, the more I wanted to move beyond theory.

I started asking a simple question:

What happens when ontology engineering meets real-world software?

That question led me into several fascinating areas.

I began exploring Ontology-Based Data Access (OBDA) and the idea of using an ontology as a semantic layer between applications and underlying data sources.

Instead of applications only thinking in terms of tables, columns, and database schemas, an ontology can provide a higher-level representation of concepts and relationships.

I also started exploring knowledge graphs and how connected entities can provide context that isolated data often cannot.

From there, my curiosity moved toward another challenge: decision-making.

How can a system use ontology-based knowledge to understand a situation?

How can relationships between entities provide context for a decision?

And how can language contribute to that process?

This led me to explore the idea of interpreting decision phrases and lexical expressions and connecting them to concepts represented in an ontology.

A phrase can contain intent, context, entities, relationships, and domain-specific terminology.

The challenge becomes connecting human language with structured knowledge.

I also became increasingly interested in how ontology-driven approaches are used in modern systems involving AI, knowledge representation, complex operational environments, and large-scale data integration.

I have been particularly interested in publicly available examples of ontology-driven platforms such as Palantir and the broader challenge of creating systems that can connect people, events, organizations, assets, and information into a meaningful operational model.

This exploration eventually became practical.

At my company, I have been working on building an ontology engine with a focus on resilience and performance.

That engineering work has also given me an opportunity to explore the strengths of both Go (Rust for incoming usage since im still beginner in Rust ;) ).

Go has helped with building pragmatic, concurrent, and resilient backend components.

Rust has helped me think more deeply about performance, correctness, memory safety, and strongly structured domain models.

Together, they have made me increasingly interested in a larger engineering question:

How can we build fast, resilient systems that don't just store information—but understand the relationships and meaning behind it?

This is the beginning of a series where I will share what I am learning and building.

I'll explore ontology engineering from both perspectives:

theory and practice.

Topics will include:

Ontology engineering fundamentals
C. Maria Keet's work and the foundations that helped me get started
Ontology-Based Data Access
Knowledge graphs
AI and knowledge representation
Ontology-driven operational systems
Lexical interpretation and decision phrases
Decision-making and contextual reasoning
Building resilient ontology engines
Go and Rust for high-performance ontology systems

I'm not writing this series because I believe I have all the answers.

I'm writing it because I believe the best way to understand a complex subject is to study it, build with it, question it, and share what you learn.

This is the beginning of my journey into ontology engineering.

And I'm excited to explore where theory meets production software.

This post was handwritten and enhanced with AI, but the entire content was written by me. :)

Reference

C. Maria Keet — An Introduction to Ontology Engineering

worker pool in golang - part 2

aymane — Sat, 08 Aug 2026 11:59:21 +0000

In Part 1 we
built a worker pool: a jobs channel, three workers, a results channel, and a
closer goroutine that calls wg.Wait() then close(results). We also walked
into a deadlock on purpose, because that is how you learn to read a concurrent
program.

Part 1 was easy in one specific way: the workers never shared anything. Each
worker read from a channel and wrote to a channel, and channels do the
synchronization for you. Channels are Go's "share memory by communicating"
answer.

In this part we do the opposite. We add a counter that every worker touches at
the same time, and we protect it with sync.Mutex. Then we prove it works with
go test, go test -race and go test -bench, and - the interesting part - we
delete the mutex and watch the race detector tear the program apart.

The full source is in
worker-pool-v2, and
the workflow diagram for this part is in diagram.md.

What we are adding

Four steps:

A Stats struct that counts processed and failed jobs.
A sync.Mutex inside it, so concurrent increments are safe.
Tests: unit, concurrent, race, benchmark.
Remove the mutex, run -race, and read the report.

Step 1: the Stats struct

// adding stats struct to follow the number of jobs successed and failed

type Stats struct {
    JobProcessed byte
    JobFailed    byte
    mu           sync.Mutex
}

func (s *Stats) processed() {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.JobProcessed++
}

func (s *Stats) failed() {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.JobFailed++
}

Three things worth saying out loud.

The mutex lives inside the struct it protects. Not next to it, not in a
global. When the lock and the data travel together, nobody can pass you the
counter without also passing you the thing that guards it.

The methods have pointer receivers. func (s *Stats), not func (s Stats).
A value receiver would copy the struct - including the mutex - and you would lock
a private copy that nobody else can see. go vet catches this one for you
(passes lock by value), and it is a mistake worth catching, because the code
still compiles and still runs and is simply wrong.

defer s.mu.Unlock() right after Lock(). The unlock runs even if the body
panics or returns early. In a two-line function this looks like ceremony. In a
twenty-line function with three return statements it is the difference between
working code and a permanently locked mutex.

Step 2: wiring it into the worker

func worker(id byte, jobs <-chan Job, results chan<- Result, stats *Stats, wg *sync.WaitGroup) {
    defer wg.Done()
    //iterate over incoming jobs from the channel and process them
    for job := range jobs {
        stats.processed()
        fmt.Printf("job %d is being processed by worker %d\n", job.ID, id)
        //simulate a failure
        //if the random value is eq to 3 then fail
        if rand.Intn(7) == 3 {
            stats.failed()
            continue
        }
        //sleep to simulate work
        time.Sleep((time.Duration(rand.Intn(3)) + 1) * time.Second)
        fmt.Printf("job %d is done by worker %d\n", job.ID, id)
        results <- Result{WorkerID: id, JobID: job.ID}
    }
    fmt.Println("worker ", id, " is done")
}

stats is a *Stats - one struct, shared by all three workers. That is the
whole point. If we passed Stats by value, each worker would count its own jobs
into its own copy and main would print zeros.

And in main:

stats := &Stats{}

for i := range NumOfWorkers {
    wg.Add(1)
    go worker(byte(i+1), jobs, results, stats, &wg)
}

go initJobs(jobs)

//using the clsoing go routines
go func() {
    wg.Wait()
    close(results)
}()

for result := range results {
    fmt.Printf("result of job %d is done by worker %d\n", result.JobID, result.WorkerID)
}
fmt.Printf("Stats - Processed: %d, Failed: %d\n", stats.JobProcessed, stats.JobFailed)

Notice the last line reads stats.JobProcessed without locking. Is that a
bug? No - and the reason is the closer goroutine we built in Part 1:

workers write stats ──► wg.Done() ──► wg.Wait() ──► close(results)
                                                         │
                                                         ▼
                                      range results ends ──► safe read

The for range results loop can only exit after close(results), which can only
happen after wg.Wait() returns, which can only happen after every worker has
returned. By the time we print, there is no other goroutine left to race with.
The Go memory model gives us that ordering for free. This is the same coordination
we built in Part 1, now paying a second dividend.

Step 3: testing it

Concurrency bugs do not show up reliably when you run the program by hand. They
show up when you write tests that try to break the code.

The unit tests

func TestStats(t *testing.T) {
    //testing stats struct methods
    t.Run("Test processed method", func(t *testing.T) {
        stats := &Stats{}
        stats.processed()
        if stats.JobProcessed != 1 {
            t.Errorf("expected JobProcessed to be 1, got %d", stats.JobProcessed)
        }
    })
    t.Run("Test failed method", func(t *testing.T) {
        stats := &Stats{}
        stats.failed()
        if stats.JobFailed != 1 {
            t.Errorf("expected JobFailed to be 1, got %d", stats.JobFailed)
        }
    })

Boring, and that is fine. They pin down the single-goroutine behaviour so that
when the concurrent test fails you know the arithmetic itself was never the
problem.

The concurrent test - the one that matters

    t.Run("Test concurrent access", func(t *testing.T) {
        stats := &Stats{}
        var wg sync.WaitGroup
        for range 100 {
            //waiting for two goroutines in each iteration
            wg.Add(2)
            go func() {
                defer wg.Done()
                stats.processed()
            }()
            go func() {
                defer wg.Done()
                stats.failed()
            }()
        }
        wg.Wait()
        if stats.JobProcessed != 100 {
            t.Errorf("expected JobProcessed to be 100, got %d", stats.JobProcessed)
        }
        if stats.JobFailed != 100 {
            t.Errorf("expected JobFailed to be 100, got %d", stats.JobFailed)
        }
    })
}

200 goroutines launched as fast as the runtime can spawn them, all hammering the
same two bytes. If the locking is wrong, the counters land below 100.

wg.Add(2) sits before the two go statements, not inside them. Calling
Add inside the goroutine is a classic race: wg.Wait() can return before the
goroutine has even started counting itself.

Also note the loop is for range 100 - a Go 1.22+ range-over-int, same as
for i := 0; i < 100; i++ without the unused variable.

The end-to-end test

func TestWorker(t *testing.T) {
    jobs := make(chan Job, NumOfJobs)
    results := make(chan Result, NumOfJobs)
    stats := &Stats{}
    var wg sync.WaitGroup
    //testing with one worker
    wg.Add(1)
    go worker(byte(1), jobs, results, stats, &wg)
    for i := range NumOfJobs {
        jobs <- Job{ID: byte(i)}
    }
    close(jobs)
    wg.Wait()
    close(results)
    //check if all jobs are processed
    if stats.JobProcessed != NumOfJobs {
        t.Errorf("expected %d jobs processed, got %d", NumOfJobs, stats.JobProcessed)
    }
}

One detail that is easy to miss: results is buffered here
(make(chan Result, NumOfJobs)), while in main it is unbuffered. In the test
there is no consumer ranging over results, so an unbuffered channel would block
the worker on its first send and wg.Wait() would hang forever - the exact
deadlock from Part 1, rebuilt by accident inside a test. The buffer gives the
worker somewhere to put its results so it can finish and call wg.Done().

We assert on JobProcessed only, not on JobFailed, because failures are
random (rand.Intn(7) == 3). Every job is counted as processed regardless, so
that number is deterministic. Asserting on random output is how you get a test
that fails once a week and teaches the team to ignore it.

Run it:

go test .

Step 4: `go test -race`

A passing test proves the result was right this time. It does not prove the
program is correct. Two goroutines can race and still, by luck, produce 100.

-race answers the stronger question. It instruments every memory access and
records which goroutine touched which address while holding which locks. If two
goroutines touch the same address, at least one writes, and nothing orders them -
it reports, even if the output happened to be correct.

go test -race .

Green. The mutex is doing its job.

The race detector is not free - roughly 5-10x slower and much more memory - so it
is a CI and development tool, not a production build flag. But every concurrent
package you write should have at least one test that runs under -race, and
that test needs enough concurrency to give the detector something to observe.
That is why TestStats/Test_concurrent_access exists.

Step 5: `go test -bench`

Locks cost something. Let's measure it rather than guess.

func BenchmarkStatsProcessed(b *testing.B) {
    //sequential cost of one Lock/Unlock pair
    stats := &Stats{}
    for b.Loop() {
        stats.processed()
    }
}

func BenchmarkStatsProcessedParallel(b *testing.B) {
    //same counter hammered from every available core -> lock contention
    stats := &Stats{}
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            stats.processed()
        }
    })
}

b.Loop() is the Go 1.24+ form of the benchmark loop. It replaces
for i := 0; i < b.N; i++ and keeps the compiler from optimizing away work whose
result you never use.

b.RunParallel is the one that tells you something you did not already know: it
runs the body on GOMAXPROCS goroutines at once, so you are measuring lock
contention, not just lock overhead.

go test -bench . -run XXX .

(-run XXX matches no test name, so only benchmarks run.)

On my M1 the two numbers are roughly:

BenchmarkStatsProcessed-8           54193095    22.13 ns/op
BenchmarkStatsProcessedParallel-8    8557812   157.4  ns/op

Uncontended, a lock/unlock pair is ~22 ns - cheap. Contended across 8 goroutines
it is ~157 ns, about 6.5x worse, because the goroutines are now queuing behind
each other and paying for cache-line ping-pong between cores.

That is the real lesson of mutex performance: the lock itself is cheap, waiting
for it is not. Our workers hold the lock for one increment and spend seconds
sleeping outside it, so contention is irrelevant here. If you ever find yourself
doing I/O inside a critical section, this is the number that will bite you.

(For a plain counter like this, sync/atomic would be faster still - but that is for later chapter)

Now the fun part: what if we drop the mutex?

Let's break it deliberately, the same way Part 1 broke on a deadlock. Comment out
the locks:

func (s *Stats) processed() {
    // s.mu.Lock()
    // defer s.mu.Unlock()
    s.JobProcessed++
}

func (s *Stats) failed() {
    // s.mu.Lock()
    // defer s.mu.Unlock()
    s.JobFailed++
}

Why this is broken

s.JobProcessed++ looks atomic. It is not. On any real CPU it is three steps:

        read  JobProcessed   ->  register
        add   1              ->  register
        write register       ->  JobProcessed

Two goroutines can interleave those steps:

 time ──────────────────────────────────────────────────►

 Worker 1   read(5) ──── add ──── write(6)
 Worker 2        read(5) ──── add ──── write(6)
                                          ▲
                                two increments, counter moved by 1
                                          = lost update

Both read 5, both compute 6, both write 6. One increment vanished. This is a
lost update, and with 200 goroutines it does not happen once - it happens
constantly.

It gets worse than lost counts. Without synchronization there is no
happens-before relationship between the goroutines at all, so the Go memory
model gives the compiler and the CPU permission to reorder and cache these
accesses. A goroutine can keep JobProcessed in a register and never publish it.
The behaviour is undefined, not merely "slightly off" - which is exactly why you
cannot debug this by adding fmt.Println and staring at it.

Running it

go test .

This may well pass. That is the trap. With only 100 iterations and a fast
machine, the goroutines often serialize by accident and you get 100 and 100.
Data races are not deterministic; a green test here proves nothing.

Now the real check:

go test -race .

==================
WARNING: DATA RACE
Read at 0x00c0000123d5 by goroutine 13:
  worker-pool.(*Stats).failed()
      .../worker-pool-series/golang/worker-pool-v2/main.go:46 +0x78
  worker-pool.TestStats.func3.2()
      .../worker-pool-series/golang/worker-pool-v2/main_test.go:52 +0x74

Previous write at 0x00c0000123d5 by goroutine 15:
  worker-pool.(*Stats).failed()
      .../worker-pool-series/golang/worker-pool-v2/main.go:46 +0x8c
  worker-pool.TestStats.func3.2()
      .../worker-pool-series/golang/worker-pool-v2/main_test.go:52 +0x74

Goroutine 13 (running) created at:
  worker-pool.TestStats.func3()
      .../worker-pool-series/golang/worker-pool-v2/main_test.go:50 +0x78
  testing.tRunner()
...
==================
--- FAIL: TestStats/Test_concurrent_access
    expected JobProcessed to be 100, got 97
FAIL

How to read that report, because it is genuinely useful once you know the shape:

Read at 0x... / Previous write at 0x... - the same memory address, hit by two different goroutines. Same address is the whole finding.
The two stacks - where each access happened. Here both point at main.go:42, the bare s.JobFailed++. The detector puts your bug on a line number.
Goroutine N created at - where each goroutine was spawned, so you can trace it back to the go statement in the test.
Then the assertion failure underneath: got 97 instead of 100. Three lost updates in a single run.

Put the two s.mu lines back, rerun go test -race ., and it is green again.
That round trip - break it, see the report, fix it, see it clear - is the fastest
way I know to build a real intuition for what a mutex is actually buying you.

In Part 3 we keep going through sync: sync/atomic for counters like this one
(no lock at all), sync.RWMutex for read-heavy state.

GitHub: https://github.com/aymane-belassiria/worker-pool-series/tree/main/golang/worker-pool-v2

Worker pool in Golang - part1

aymane — Sun, 02 Aug 2026 14:14:34 +0000

Building a Worker Pool in Go

In this series of articles, we'll explore one of the most widely used concurrency patterns in Go: the Worker Pool.

We'll learn what a worker pool is, why it exists, and when you should use it. More importantly, we'll implement several versions using Go's concurrency primitives, compare different approaches, and discuss the trade-offs and performance characteristics of each implementation.

What Is a Worker Pool?

Between the 1960s and the 1980s, computer scientists developed techniques to efficiently manage the creation and scheduling of threads and processes. Since creating these resources is relatively expensive, they introduced the concept of worker pools (also known as thread pools), where a fixed number of workers are created in advance and reused to execute multiple tasks.

Instead of continuously creating and destroying threads, applications reuse existing workers, significantly reducing overhead and improving performance for CPU-intensive and I/O-intensive workloads.

Later, with the rise of web servers, developers faced the challenge of handling thousands of incoming client requests while making efficient use of the server's hardware resources. The worker pool pattern became a natural solution because it limits the amount of concurrent work while maximizing resource utilization.

Today, worker pools remain one of the fundamental concurrency patterns. Almost every modern web server, runtime, and application framework uses some variation of this technique to efficiently serve requests.

Let's Implement Our First Worker Pool in Go

Now that we have a basic understanding of the worker pool pattern, it's time to build one in Go.

We won't build anything fancy just yet—our goal is simply to understand the core idea and see how everything works behind the scenes.

Our implementation will follow four simple steps:

Create a fixed number of workers.
Send jobs through a channel.
Let each worker process incoming jobs.
Collect the results.

By the end of this implementation, you'll have a solid understanding of how worker pools work internally. Later in this series, we'll extend it with graceful shutdown, context cancellation, error handling, dynamic worker scaling, and more.

Defining the `Job` and `Result` Types

Let's begin by defining the data structures our worker pool will use.

type Job struct {
    ID byte
}

type Result struct {
    WorkerID byte
    JobID    byte
}

Job represents a unit of work. In a real application, this could be a database query, an HTTP request, image processing, file compression, or any other task.

For now, we'll keep it simple and only store an ID.

Result represents the output of a processed job. In real-world applications, it may also include processed data, errors, execution time, or any additional metadata.

Implementing the Worker

Now let's implement the heart of our worker pool: the worker function.

func worker(id byte, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()

    // Process jobs until the channel is closed.
    for job := range jobs {
        fmt.Printf("Job %d is being processed by worker %d\n", job.ID, id)

        // Simulate some work.
        time.Sleep((time.Duration(rand.Intn(3)) + 1) * time.Second)

        fmt.Printf("Job %d completed by worker %d\n", job.ID, id)

        results <- Result{
            WorkerID: id,
            JobID:    job.ID,
        }
    }

    fmt.Printf("Worker %d has stopped\n", id)
}

As you can see, there's nothing particularly complicated here.

The worker receives:

its ID,
a receive-only jobs channel,
a send-only results channel,
and a sync.WaitGroup used to notify the main goroutine when the worker exits.

Inside the loop, the worker continuously receives jobs from the jobs channel, processes them, and sends the corresponding Result to the results channel. Once the jobs channel is closed, the loop exits and the worker terminates.

Initializing the Jobs

Let's create a helper function that generates our jobs.

func initJobs(jobs chan<- Job) {
    for i := range NumOfJobs {
        jobs <- Job{ID: byte(i)}
    }

    // No more jobs.
    close(jobs)
}

This function simply sends NumOfJobs jobs into the channel and closes it afterward so the workers know there is no more work to process.

Assembling Everything

Now let's wire everything together inside main().

func main() {
    jobs := make(chan Job, NumOfJobs) // buffered channel
    results := make(chan Result)      // unbuffered channel

    for i := range NumOfWorkers {
        wg.Add(1)
        go worker(byte(i+1), jobs, results, &wg)
    }

    go initJobs(jobs)

    wg.Wait()
    close(results)

    for result := range results {
        fmt.Printf("Job %d was processed by worker %d\n", result.JobID, result.WorkerID)
    }

    fmt.Println("All jobs have been processed.")
}

Everything looks correct at first glance.

Let's run it.

Oops! We immediately hit a deadlock.

I intentionally kept this mistake because it's a great reminder that concurrent programming requires careful coordination. If you're not careful, you'll eventually run into problems such as deadlocks or race conditions.

Why Does the Deadlock Happen?

Inside the worker, we send the result through an unbuffered channel:

results <- Result{
    WorkerID: id,
    JobID:    job.ID,
}

Sending on an unbuffered channel blocks until another goroutine receives the value.

Meanwhile, the main goroutine is waiting here:

wg.Wait()

before it starts reading from the results channel.

This creates a circular dependency:

Every worker is blocked trying to send a result.
The main goroutine is blocked waiting for every worker to finish.
The workers cannot finish because they're blocked on the channel send.

Since no goroutine can make progress, the Go runtime detects a deadlock.

Fixing the Deadlock

Fortunately, the fix is simple.

We'll introduce what I like to call a closer goroutine. Its only responsibility is to wait until every worker finishes and then close the results channel.

go func() {
    wg.Wait()
    close(results)
}()

Our final main() function now looks like this:

func main() {
    jobs := make(chan Job, NumOfJobs) // buffered channel
    results := make(chan Result)      // unbuffered channel

    for i := range NumOfWorkers {
        wg.Add(1)
        go worker(byte(i+1), jobs, results, &wg)
    }

    go initJobs(jobs)

    // Closer goroutine.
    go func() {
        wg.Wait()
        close(results)
    }()

    for result := range results {
        fmt.Printf("Job %d was processed by worker %d\n", result.JobID, result.WorkerID)
    }

    fmt.Println("All jobs have been processed.")
}

Now the main goroutine immediately starts consuming results, allowing workers to continue sending values without blocking indefinitely. Once all workers exit, the closer goroutine closes the results channel, which naturally terminates the range loop.

Another way to solve this problem is to start consuming the results channel before calling wg.Wait(). The important takeaway is that an unbuffered channel always requires a receiver. Otherwise, every sender will eventually block.

Conclusion

For this first implementation, I intentionally kept everything simple so we could focus on understanding the worker pool pattern itself.

Even with this small example, we've already covered several important Go concurrency concepts:

Goroutines
Channels
Directional channels
sync.WaitGroup
Closing channels
Understanding and fixing deadlocks

In the next part of this series, we'll continue improving our worker pool by exploring more of the sync package, writing proper tests with Go's testing package, and benchmarking different implementations to better understand their performance characteristics.
The complete source code for this article is available on GitHub:

GitHub: https://github.com/aymane-belassiria/worker-pool-series/tree/main/golang/worker-pool-v1

DEV Community: aymane

OBDA and Virtual Knowledge Graphs: A Practical Introduction

What is OBDA?

The Three Main Components

1. Ontology

2. Mappings

3. Data Sources

What is a Virtual Knowledge Graph?

Materialized Knowledge Graph vs Virtual Knowledge Graph

Materialized Knowledge Graph

Virtual Knowledge Graph

Querying a Virtual Knowledge Graph

Query Rewriting

The VKG Query Pipeline

Rewriting

Unfolding

Why Use an Ontology?

OBDA vs Data Warehouse

OBDA vs Knowledge Graph

OBDA Architecture

Where the Complexity Actually Goes

A Simple Mental Model

Conclusion

My Journey Into Ontology Engineering

Reference

worker pool in golang - part 2

What we are adding

Step 1: the Stats struct

Step 2: wiring it into the worker

Step 3: testing it

The unit tests

The concurrent test - the one that matters

The end-to-end test

Step 4: go test -race

Step 5: go test -bench

Now the fun part: what if we drop the mutex?

Why this is broken

Running it

Next

Worker pool in Golang - part1

Building a Worker Pool in Go

What Is a Worker Pool?

Let's Implement Our First Worker Pool in Go

Defining the Job and Result Types

Implementing the Worker

Initializing the Jobs

Assembling Everything

Why Does the Deadlock Happen?

Fixing the Deadlock

Conclusion

Step 4: `go test -race`

Step 5: `go test -bench`

Defining the `Job` and `Result` Types