When we started thinking about keeping repository data updated in WebiU, the first solution seemed pretty obvious:
GitHub has webhooks.
Repository changes.
GitHub sends an event.
We update our database.
Done :))
Simple.
Clean.
Real-time.
What could possibly go wrong?
Well...
Apparently, quite a lot XD
Because somewhere between:
"Let's add GitHub webhooks."
and:
"Let's make sure our database actually represents what exists on GitHub."
I learned something important:
Real-time synchronization and reliable synchronization are two completely different problems.
And that realization changed how we designed the entire synchronization system behind WebiU.
The Original Problem: GitHub Was Becoming Our Data Source
WebiU started with a much simpler architecture.
A lot of project information could live inside static files.
That works perfectly well when your data changes occasionally.
But GitHub repositories definitely don't behave like that :')
Repositories are created.
Repositories are renamed.
Descriptions change.
Topics change.
Contributors join.
Repositories get archived.
Metadata evolves constantly.
Once WebiU started moving toward becoming an Open Source Intelligence Platform, keeping that information manually synchronized stopped making sense.
So we needed a system where GitHub remained the source of truth...
while WebiU maintained its own persistent representation of that data.
And that creates a surprisingly interesting problem:
How do you keep two systems synchronized when only one of them is actually authoritative?
The Obvious Solution: Just Use Webhooks
GitHub webhooks are almost perfect for this.
Something changes on GitHub.
GitHub sends an HTTP request to your backend.
Your backend processes the event.
Your database gets updated.
Instead of constantly asking GitHub:
"Anything changed?"
"Anything changed now?"
"How about now?"
"...now?" ._.
GitHub simply tells you when something happens.
This is event-driven architecture in its simplest form.
And compared to repeatedly polling hundreds of repositories, it has some obvious advantages.
Fewer unnecessary API requests.
Lower latency.
Near real-time updates.
Better use of GitHub API rate limits.
Everything looks great.
Until you ask one slightly uncomfortable question:
What happens if we miss an event?
:0
Webhooks Tell You What Happened.
They Don't Prove Your Current State Is Correct.
Imagine GitHub sends a webhook.
Your server is temporarily unavailable.
The request fails.
Or your application receives the webhook but crashes while processing it.
Or the database transaction fails.
Or someone accidentally removes a webhook configuration.
Or a repository existed before your webhook system was even deployed.
Now GitHub knows one thing.
Your database knows another.
And neither system is going to magically fix that difference for you :')
This is called data drift.
Your local representation slowly starts becoming different from the actual source of truth.
And the dangerous part is that your application might continue working perfectly.
No errors.
No crashes.
No giant red warning saying:
HELLO. YOUR DATABASE IS WRONG NOW. XD
It simply serves outdated information.
That's much harder to notice.
Real-Time Does Not Mean Correct
This was probably the biggest mental shift for me while working on synchronization.
Webhooks optimize for freshness.
They tell us about changes quickly.
But freshness and correctness aren't the same thing.
You can have a system receiving updates within milliseconds...
and still have incorrect data because of one event you missed three weeks ago.
That means relying entirely on webhooks creates an architectural assumption:
Every important event will always be delivered and successfully processed.
That's a very strong assumption.
And distributed systems generally have a habit of destroying strong assumptions XD
So instead of asking:
How do we receive updates quickly?
We started asking:
How do we guarantee that our system eventually returns to the correct state?
That question led us to reconciliation.
Enter Scheduled Reconciliation
The idea behind reconciliation is surprisingly simple.
Every once in a while, don't trust your own database.
Go back to the source of truth.
Compare reality with what you think reality looks like.
Then repair the differences.
In our case:
GitHub represents the authoritative repository state.
WebiU maintains a persistent local representation.
So periodically, the system can fetch the actual repository state from GitHub and compare it against what exists locally.
If everything matches?
Great :))
Move on.
If something doesn't?
Now we know drift has occurred.
And the system can recover.
Webhooks + Reconciliation
This is where the architecture started making much more sense to me.
Webhooks and scheduled synchronization aren't competing approaches.
They solve different problems.
Webhooks give us:
- Fast updates
- Event-driven processing
- Fewer unnecessary API requests
- Near real-time synchronization
Reconciliation gives us:
- Recovery from missed events
- Drift detection
- Eventual correctness
- Protection against historical inconsistencies
Together, they create something much stronger.
The architecture becomes roughly:
GitHub changes
→ Webhook event
→ Backend processes event
→ Database updates immediately
Then, separately:
Scheduled reconciliation
→ Fetch authoritative GitHub state
→ Compare against persisted state
→ Detect differences
→ Repair inconsistencies
The webhook path handles the normal case.
The reconciliation path handles the:
"Something somewhere went wrong three days ago and nobody noticed." XD
case.
And those cases matter more than I originally expected.
Idempotency Suddenly Matters
Once you have multiple synchronization paths, another problem appears.
The same repository update might arrive through a webhook...
and later be discovered again during reconciliation.
Should the system update everything twice?
Create duplicate records?
Trigger duplicate downstream work?
Hopefully not :')
This is where idempotency becomes important.
An operation should be safe to perform multiple times without creating incorrect results.
For example:
Synchronize repository X with GitHub.
Whether that operation runs once or five times, the final state should still represent GitHub correctly.
This sounds obvious when written in one sentence.
Designing your code so that it actually behaves this way is slightly less obvious XD
But once synchronization systems become asynchronous, retryable, and event-driven, idempotency stops being a nice architectural word.
It becomes a survival mechanism.
Failures Should Become State
Another interesting lesson came from thinking about failed synchronization.
The naive approach is:
Synchronization fails.
Log the error.
Done.
But logs are mostly useful to developers who already know something is wrong.
What about the system itself?
If a repository failed synchronization yesterday, shouldn't the application know that?
Instead of treating synchronization failures as temporary console messages, we started representing synchronization state explicitly.
A repository can have information about whether it is synchronized correctly.
Whether synchronization failed.
Why it failed.
Where the latest reconciliation came from.
This turns operational problems into queryable application state.
And that opens the door to much better observability.
Instead of searching through logs wondering:
"Which repository is broken?" ._.
The system itself can answer that question.
The Database Became More Than Storage
This was another shift in how I thought about persistence.
Initially, it's easy to think:
Database = place where data lives.
But in a synchronization-heavy system, the database also becomes a record of what the application believes about the outside world.
That means metadata becomes important.
When was this repository last synchronized?
Was the synchronization successful?
Where did this update come from?
Is this repository still active?
Did reconciliation detect drift?
Suddenly, your persistence model isn't just representing repositories.
It's representing the health of your relationship with the external system.
I definitely wasn't thinking about databases like that when I first started building CRUD applications xD
GitHub API Rate Limits Still Matter
Of course, reconciliation introduces its own problem.
Remember why we didn't want to poll GitHub constantly in the first place?
Rate limits.
If you have hundreds of repositories and periodically fetch everything from GitHub without thinking about efficiency...
Congratulations.
You've reinvented aggressive polling with extra steps XD
So reconciliation needs to be intentional.
Not every piece of data needs the same synchronization frequency.
Not every repository needs to be refreshed constantly.
Bulk operations can reduce unnecessary requests.
Cached data can prevent repeated work.
The system needs to understand which information is actually worth refreshing.
This is where architecture becomes less about:
"Can we build this?"
and more about:
"Can we build this without creating another problem somewhere else?"
That question keeps appearing suspiciously often in software engineering :')
Synchronization Is a Consistency Problem
Before working on this system, I mostly thought about GitHub integration as an API problem.
Call endpoint.
Get JSON.
Save JSON.
Done.
But once you're maintaining persistent state, it becomes something else entirely.
It's a consistency problem.
You have:
- An external source of truth
- Your local representation
- Events arriving asynchronously
- Operations that can fail
- Network requests that can time out
- Rate limits
- Retries
- Historical data
- Multiple synchronization mechanisms
Now you're not simply consuming an API.
You're designing how two systems agree with each other over time.
That is a much more interesting problem.
And also a much better way to accidentally spend an entire evening staring at architecture diagrams :')
The Architecture We Ended Up With
At a high level, the synchronization strategy became:
1. PostgreSQL for persistent repository state
WebiU keeps a local representation of repository information rather than fetching everything directly from GitHub for every request.
2. GitHub webhooks for fast updates
When important events occur, GitHub can notify the backend and allow the system to react quickly.
3. Scheduled reconciliation for correctness
The system periodically compares persisted state against GitHub to detect and repair drift.
4. Synchronization metadata for observability
Repositories can expose their synchronization status instead of hiding failures inside logs.
5. Idempotent synchronization operations
Updates should remain safe even when the same state is processed multiple times.
Together, these mechanisms create something much more resilient than:
GitHub webhook → database update.
Because production systems need to account for what happens when the happy path isn't so happy XD
What I'd Do Differently Now
One thing I'm slowly learning is that the first architecture you imagine is usually optimized for the happy path.
The more you work on production systems, the more you start designing for everything surrounding that path.
What happens when this fails?
Can it retry?
What happens if the retry runs twice?
How do we detect stale data?
Can the system repair itself?
How do we know it's broken?
What happens when the external API disappears for ten minutes?
Those aren't questions I naturally asked when I was building my first React applications.
Back then, my biggest architectural concern was probably:
"Should this component go inside the components folder or the pages folder?" XD
Now I find myself thinking about synchronization, consistency, persistence, failure recovery, and observability.
And I'm still very much learning all of it.
But that's probably been one of the most interesting parts of working on WebiU through Google Summer of Code.
The problems keep getting deeper.
And every time I think:
"Okay, I understand this architecture now."
another edge case quietly walks into the room :')
The Biggest Lesson
If there's one thing I'd take away from building this system, it's this:
Don't design only for how data gets updated. Design for how your system recovers when it doesn't.
Webhooks are excellent.
We use them because they solve an important problem.
But they aren't a complete synchronization strategy.
Reliable systems need a way to question their own state.
To compare it against reality.
To detect when something went wrong.
And ideally...
to fix themselves without waiting for a developer to discover three weeks later that something has been broken the entire time XD
I'm still learning.
Still designing.
Still finding edge cases I didn't know existed.
Still occasionally looking at a simple feature and realizing it has somehow become a distributed systems problem :')
But I think that's the part of software engineering I'm starting to enjoy the most.
The code is one part.
Understanding how the entire system behaves when things go wrong?
That's where things start getting really interesting :))
Top comments (1)
Warning: This one got a bit long! But You are a patient soul!
XD