DEV Community: Josephat Macharia

When Your Codebase Blocks Agile

Josephat Macharia — Fri, 22 Aug 2025 13:23:03 +0000

One of the biggest mistakes companies make when they adopt Agile is assuming that changing the process is enough. Daily stand-ups, sprints and Kanban boards are introduced — but the code remains untouched, structured for a world that predates Agile.

The result? An architecture mismatch that quietly undermines agility. Teams may be “Agile” in ceremony, but in practice, they are slowed down so much that they can hardly respond to change in any meaningful sense.

The Hidden Bottleneck

Agility is about responding quickly to new requirements, shifting priorities and customer needs. But if your codebase is still organized in a way that resists change, every new feature feels like a battle. Instead of building, teams spend disproportionate time:

Untangling tightly coupled modules

Reworking brittle dependencies

Fighting against legacy design decisions

This isn’t a process problem. It’s an architectural problem.

Why Architecture Shapes Agility

A system’s architecture sets the boundaries for speed, adaptability and collaboration. If the architecture reflects an older way of working — where long release cycles and rigid silos were the norm — Agile practices will only expose those weaknesses faster.

Agile thrives when the codebase itself is:

Modular – enabling teams to work independently without constant conflicts.

Flexible – designed for extensions and iterations, not rigid perfection.

Evolvable – allowing refactoring and improvement as part of everyday work.

Without these qualities, your Agile transformation risks becoming Agile in name only.

The Road Ahead

If you are considering a move to Agile, be prepared: restructuring the fundamental architecture of your system is almost certainly in your future. The sooner you address it, the faster your teams can deliver value without being slowed down by the very code they’re meant to improve.

Agility isn’t just about process. It’s about aligning process and architecture so that your teams — and your software — can truly move at the speed of change.

Rethinking Estimation In Agile Development

Josephat Macharia — Thu, 15 May 2025 18:35:50 +0000

Estimation

Perhaps we should rethink and reassess our ritual of story estimation.

Direct from the Scrum creator himself:

Most story estimation is a waste of everybody's time and a recipe for organisational disappointment.

One of the ongoing debates in agile circles is that of story sizing. For years, it has seemed to be a settled topic, but there are always new people entering the field that don't know that it's settled, go out looking for how long something is going to take and find a bunch of information that says you should use story points (or t-shirt sizing, or something similar) to figure out how big something is and get an idea of how long it will take.

So, let's see if we can bring people up to speed. When XP was developed, its creators thought it was important to be able to have an indicative value of the relative complexity of work.
They started with ideal hours, then padded them and then settled on points to abstract away from time, because they found that sizing by hours didn't work very well. Points were never meant to be a predictor of duration.

After a number of years, just about everybody involved in formalising story points realised it didn't work particularly well, for lots of reasons and suggested we move to something else.

Here's Ron Jeffries on how his views of story points have changed:
"I like to say that I may have invented story points and if I did, I’m sorry now."
From Story Points Revisited

For a lot of people, that isn't enough. Knowing that the creators thought it was a mistake isn't enough to enable people to change.
To change, people also need to know what to change to. For that, a detailed breakdown of what to do instead is really valuable. So, here it is:

Why estimating doesn't work:
✔ Parkinson's Law (the idea that work expands to fill the time allotted for its completion. This may mean you take longer than necessary to complete a task or you procrastinate and complete the task right before the due date.)
✔ Optimism
✔ Inherent uncertainty of creative work (This uncertainty stems from the open-ended nature of creative challenges, where there is rarely a single "right" answer or a clear, pre-defined path to a successful outcome. )
✔ Humans are awful at estimating anything bigger than about 2 days of work

What to do instead:
💡 Make stories smaller. No, even smaller. (Hint: See "2 days" above)
💡 Actively seek to understand the uncertainty
💡 Use exploratory stories
💡 Write genuine user stories
💡 Remember that user stories are a placeholder for a conversation
💡 Go talk to the user
💡 Utilizes short iterations of 1-2 weeks while embracing continuous releases.
💡 Incorporates sprint planning and review while allowing flexibility and no commitment.
💡 No notion of story points or “points per sprint.”

The Fallacy of Estimates

Estimates are inherently based on guesswork.
Estimation leads to dysfunction and false commitments.
It's not that things are late 80% of the time; it's that we don't know how to do estimates 80% of the time, so our estimates are always wrong.
Making decisions based on erroneous estimations leads to significant process breakdown.
Story points have no meaning, any more than time-based estimates would have.

Velocity

What if you need to understand velocity? Simple: count the stories. That's your velocity. Once you make stories really small, their variability decreases and you get through more. The Law of Large Numbers will help eliminate some of the variability you used to see in your work, so counting stories becomes a perfectly reasonable way to measure velocity, if you need to (you probably don't).

Velocity is a ratio that compares the hypothetical time needed to complete a project without interruptions to the actual time it takes in practice.
Essentially, it's a decimal figure ranging from 0 to 1, similar to a percentage.
Our wrong estimates are tied into this.
We wrongly attribute decreased velocity to team inefficiency rather than acknowledging systemic barriers (e.g., bureaucratic processes and unnecessary tasks).
Velocity is another destructive force, often misinterpreted as a measure of efficiency or productivity.

Sprint "Spillover"

In Agile methodology, there's no concept of Sprint "spillover."
Agile embraces change, even during coding, allowing for feedback-driven adjustments.
Insisting on estimation often means misunderstanding Agile, resorting to what is termed as "fake Scrum."
True Agile doesn't require committing to a fixed set of stories within a Sprint; instead, it focuses on committing to overall goals, emphasizing adaptability over rigid planning, similar to traditional waterfall approaches.

Conclusion

Some people will say, "Aha! You said estimating was a waste, but you must estimate in order to figure out if a story is less than 2 days." You're right. But that's the extent of estimation's value. If you understand what's needed well enough, you'll be able to get the story to be smaller than 2 days. If you don't, you won't.

It's time to move away from the culture of estimates and focus on delivering valuable software.
If we are obsessed with estimates, let's employ a difficulty scale that is difficult to translate into time.
- For example, categorize tickets as trivial, easy, normal, hard, don't know, etc.
- This could provide valuable information for prioritization.

The Rewrite Trap: A Programmer's Survival Guide

Josephat Macharia — Sun, 22 Dec 2024 16:47:25 +0000

In the world of software development, there's a constant temptation that lurks in the shadows of every legacy codebase: the urge to throw everything away and start fresh.

It's a siren song that has led many projects to their doom, yet it continues to seduce developers across the industry.

Let's dive deep into the art of code refactoring and explore why the "rewrite everything" approach is often a costly mistake.

The Temptation of the Blank Slate

There’s a unique allure in starting fresh.
Like an architect yearning to demolish and rebuild, programmers often see existing codebases as messy, convoluted, or just plain ugly.

It’s natural to feel this way.
Reading someone else’s code (or even your own, months later) is harder than writing new code from scratch. But this impulse to bulldoze can be catastrophic.

Here’s the truth: existing code is more than just text on a screen. It’s the culmination of countless hours of debugging, testing and adapting to real-world problems.

Each line however inelegant—represents accumulated knowledge.

Tossing it aside without due diligence is like throwing away a treasure map because you dislike the handwriting.

Why Complete Rewrites Are Dangerous

Rewriting an entire codebase might seem like a clean solution, but history shows it’s often a grave mistake.

Consider these cautionary tales:

Netscape's decision to rewrite their browser from scratch led to a three-year gap between versions 4.0 and 6.0, giving Internet Explorer the opportunity to dominate the browser market.
Borland's attempt to rewrite Quattro Pro ended in failure.
Even Microsoft, with all its resources, had to abandon a complete rewrite of Word.

Rewriting not only wastes time and money but also introduces unknown bugs while losing the hard-earned stability of the original code.

Refactor, Don’t Rewrite

So, what’s the alternative? Incremental refactoring. This approach allows you to:

Preserve Battle-Tested Code: Even ugly code has survived production, been reviewed and patched over time. Respect the lessons it holds.
Make Incremental Changes: Small, focused updates make it easier to track the impact of each change. It’s far less intimidating to fix two failing tests than to face a hundred.
Leverage Existing Tests: Never discard old tests without understanding their purpose. These tests may uncover subtle issues you might overlook.

The Right Way to Approach Refactoring

Before refactoring, ask yourself: Why am I doing this?

Performance Issues: Profile the code first. Identify bottlenecks and optimize strategically.
Difficulty Adding Features: If tight coupling or spaghetti code is the issue, focus on encapsulation and better interfaces.
Code Maintainability: Start small, rename variables, clean up formatting and clarify comments. Gradual improvements add up.

Practical Tips for Refactoring Success

Understand the Existing System: Take stock of the codebase and tests. Identify strengths to preserve and weaknesses to address.
Refactor in Small Steps: Focus on one area at a time. Verify changes by running tests after each iteration.
Avoid Personal Bias: Restructure for functionality, not style. If it isn’t broken, don’t fix it.
Beware of New Technology: Don’t refactor just to use the latest framework or language. Unless there’s a clear, measurable benefit, it’s not worth the risk.
Respect the Past: Every “ugly” workaround or patch likely exists for a reason. Investigate before replacing it.

Conclusion: Progress Without Pain

Refactoring isn’t just about writing better code, it’s about making smarter decisions.

By respecting the lessons embedded in your existing system and taking a methodical incremental approach, you’ll avoid the pitfalls of hubris and set yourself up for long-term success.

Remember, code doesn’t need to be perfect, it just needs to work well for the people who use and maintain it.

So next time you’re tempted to rewrite everything, stop and think: What’s the real problem I’m solving? The answer might save you countless hours—and a lot of headaches.

Introduction To RabbitMQ

Josephat Macharia — Sun, 31 Oct 2021 14:25:26 +0000

What is RabbitMQ

RabbitMQ is a message broker that accepts and forwards messages.
It accepts messages from producers, and delivers them to consumers. It acts like a middleman which can be used to reduce loads and delivery times taken by web application servers.

Message flow in RabbitMQ

The producer publishes a message to an exchange. When creating an exchange, the type must be specified.
The exchange receives the message and is now responsible for routing the message. The exchange takes different message attributes into account, such as the routing key, depending on the exchange type.
Bindings must be created from the exchange to queues. In this case, there are two bindings to two different queues from the exchange. The exchange routes the message into the queues depending on message attributes.

4.The messages stay in the queue until they are handled by a consumer

5.The consumer handles the message

Types of exchanges in RabbitMQ:

Fanout
This exchange basically sends a copy of messages to all the queues it knows. (in the dashboard every queue should be bind ed to one exchange)
Direct
A direct exchange delivers messages to queues based on a message routing key. The routing key is a message attribute added to the message header by the producer. Think of the routing key as an “address” that the exchange is using to decide how to route the message.
Topic
Topic exchanges route messages to one or many queues based on matching between a message routing key and the pattern that was used to bind a queue to an exchange.

4.Headers
A header exchange is an exchange that route messages to queues based on message header values instead of routing key. If a match is found, it routes the message to the queue whose binding value is matched and if a match is not found, it ignored the message.

RabbitMQ Terminologies and Concepts

Producer: A producer is a user application that sends messages to the consumers.

Consumer: A consumer is a user application that receives messages.

Message: It’s a simple serialized object from the producer!

Exchange: An exchange is a very simple thing. On one side it receives messages from producers and the other side pushes them to queues. This is a place that messages come first and its main purpose is to decide which consumer application, the message should go!

Queue:
A queue is a buffer that stores messages basically, the exchange routes the message to queues, then the consumer clients receive messages from the queue. We have queues as many as the consumer’s applications
The producers send the message to the exchange first, the exchange decides which queues, the message should go.

When to Use RabbitMQ

For long-running tasks, when you need to run reliable background jobs. e.g such as file scaning, image scaling or PDF
communication and integration within, and between applications, i.e as middleman between microservices; where a system simply needs to notify another part of the system to start to work on a task, like ordering handling in a webshop (order placed, update order status, send order, payment, etc.)

17 days of code complete Day 2 - Chapter 3 and 4

Josephat Macharia — Fri, 29 Oct 2021 06:18:08 +0000

On day 2 we will look into chapter 3 and 4.
For more in depth notes of the book, please purchase the book!

Chapter 3 Measure Twice, Cut Once Upstream Prerequisites

This chapter describes the work that must be done to prepare for software construc-
tion. As with building construction, much of the success or failure of the project has
already been determined before construction begins. If the foundation hasn’t been
laid well or the planning is inadequate, the best you can do during construction is to
keep damage to a minimum.
This chapter discusses software equivalent of blueprints
and construction permits. You might not have much control over preparations, so the focus of this chapter is on assessing what you have to work with when
construction begins.

Key take away

It is good to plan to specify requirements up front.
Essentially, you’ll need to pick the right balance for your project. If it’s something more stable, where the requirements are well understood and unlikely to change, a more sequential approach is appropriate. But if the requirements may change, or the structure isn’t as well understood, then a more iterative approach will be more beneficial.
The overarching goal of preparation is risk reduction
Most common project risks in software devel-
opment are poor requirements and poor project planning, thus preparation tends to
focus on improving requirements and project plans.
From a technical point of
view, planning means understanding what you want to build so that you don’t waste
money building the wrong thing.
Explicit requirements keep you from guessing
what the user wants.
Explicit requirements also help to avoid arguments. You decide on the scope of the
system before you begin programming. If you have a disagreement with another pro-
grammer about what the program is supposed to do, you can resolve it by looking at
the written requirements.
Software being what it is, iterative approaches are useful much more often than sequential approaches are.

Handling Requirements Changes During Construction

Use the requirements checklist at the end of the section to assess the quality of your requirements If your requirements aren’t good enough, stop work, back up, and make them right before you proceed.
Make sure everyone knows the cost of requirements changes, work up a revised schedule and cost estimate so that you can decide whether you want to do it now or later.
Set up a change-control procedure If your client’s excitement persists, consider establishing a formal change-control board to review such proposed changes. It’s all right for customers to change their minds and to realize that they need more capabilities. The problem is their suggesting changes so frequently that you can’t keep up

Chapter 4 Key Construction Decisions

This chapter focuses on preparations that individual programmers and technical leads are responsible for, directly or indirectly. It discusses the software equivalent of how to select specific tools for your tool belt and how to load your
truck before you head out to the job site.

Programmers may be similarly influenced by their languages. The words available in a
programming language for expressing your programming thoughts certainly deter-
mine how you express your thoughts and might even determine what thoughts you
can express.

Your programming tools don’t have to
determine how you think about programming.
Programmers who program “in” a language limit their thoughts to constructs that the language
directly supports. If the language tools are primitive, the programmer’s thoughts will also be primitive.
Programmers who program “into” a language first decide what thoughts they want to express, and then they determine how to express those thoughts using the tools provided by their specific language.
Your position on the technology wave determines what approaches will be effective or even possible. Identify where you are on the technology wave, and adjust your plans and expectations accordingly.
Establish programming conventions before you begin programming. It’s nearly
impossible to change code to match them later.
Every programming language has strengths and weaknesses. Be aware of the specific strengths and weaknesses of the language you’re using.

My thoughts

Its hard for programmers to adopt to the test first approach.
Ensuring the programmer has stepped through their code in the debugger before they check it in is not easy.

How To Build APIs That Are Resilent From 3rd Party Failures

Josephat Macharia — Thu, 28 Oct 2021 20:27:52 +0000

There are many possible reasons for failures, from an outage of a third-party service (API, DB, etc.) to a hardware failure, to the “classic” software bug (after all, software developers are humans too, aren’t we? 🤔).

Fault tolerance is a requirement, not a feature.

This post is an a attempt to address handling system failures during 3rd Party API communications.

What are some of the failure points?

The network connectivity between the application and the third party system might be disrupted causing communication timeouts or lost information.
The third party system has an internal error or machine failure causing the application not to receive a response.
Third party system has not been paid for, e.g late in account payment renewal.
We have an internal error or machine failure. Depending on the timing of this failure, two things could happen:
```
 1. Inability to send the request.

 2. Inability to receive the response from the third party.
```

How do we respond to a user request when failure occurs?

The diagram below attempts to show a process that would be followed to achieve resilience in API communication failure

In each of the options described above a timeout, internal error or connection error will result in a request not retrieving the optimal response for customers.

What are the possible actions to take when retry will not succeed and is unlikely to help?

Notifications: when such 3rd party failure occurs, notifications should be sent for quick action such as calling the customer back to manage expectation and give way out that lets them know that you have their back.
Showing descriptive messages that will aid to let the customer know what to do. e.g refresh, retry

To improve the resilience of the application we should consider the following patterns:

1.Retry
2.Caching
3.Persistence
4.Circuit Breaker

Retry

For a reliable and robust retrying solution it should be:

Smart - To answer this, we use exponential backoff retrying.
Customizable - Standard way to handle errors, we should control which errors should be retried and which shouldn’t by throwing exceptions for the errors that should be retried and by catching the others.

Caching

We record each request in the database before sending the information to an external party.
We should have a status attribute that tracks which part of the process the request is in.
We update the request with a new status based on the response, normally either created, declined or successful.

Persistence

To get more inspiration Uber manages the reprocessing of their data using kafka topics and dead-letter queues. See https://eng.uber.com/reliable-reprocessing/.

Uber’s idea would fit many of our needs, so we can design a solution based on it.

First, our error handling solution should identify that there is an error. Then, it needs to grab the relevant context of the error (in this example, the handled HTTP request).
After that, it should send that context to some persistence layer, for later re-execution of the service flow.
When a failure occurs in one of the HTTP requests, the request context (request body, query params, etc.) will be sent as a Kafka message to the first retry Kafka topic.

As for the re-execution part, a polling of the first retry topic is added and if that processing fails too, it will be sent to the second topic and so forth, until it’s sent to the dlq (“dead-letter queue”) topic, for manual analysis. (The number of retry topics is arbitrary in this example, it can be any number).
Let’s see how the solution I described applies to service A below:

Summary

If we implement the solutions I’ve described, we can be much more confident now when our system has failures, the system requires fewer manual interventions when it fails, and most importantly, we sleep well at night!

I hope this post helped you by providing you with some new ideas for handling failures in your applications.

Feel free to comment below with any questions/thoughts :-)

When To Use NoSQL Document Databases: MongoDB, CouchDB

Josephat Macharia — Sat, 08 May 2021 10:33:59 +0000

Why NoSQL Databases Were Invented

When traditional databases were introduced they were able to handle the size by running on bigger machines
NoSQL means that the database does not employ tables, rows and columns for data organization
Relational databases could horizontally scale which means being able to copy their data to other machines so reads could be distributed.
There was no easy way to distribute writes
To fix this problem, leading technological companies introduced new databases referred to as NoSQL.
Therefore NoSQL databases were designed to scale large datasets horizontally which means scaling large datasets was the primary motivator to create NoSQL databases.

Types Of NoSQL Databases

Key/value
Graph databases
Column-oriented
Document-oriented

Industry examples of Document Databases

MongoDB
CouchDB
Cosmos DB
CrateDB
Couchbase Server

What Is A Document

A way to organize and store data as a set of field value pairs.

Features Of Document databases

Relationships are obvious using embedded arrays and documents.
The biggest benefit to developers of using a document-oriented database is that developers use structure like maps, hashes, dictionaries, and JSON objects.
The document model use a similar structure resulting in the representation that is easier and more natural, as it is the same in the code and the application.
This means there is no complex mapping between your applications data object and the database.
Today, we know that modern data sets can grow to be huge, that data is polymorphic by nature, and that our modern applications have to be resilient and always up.
NoSQL databases, and in particular a document-oriented database like MongoDB, are a more natural way to work with modern data.

Why Use Document Databases

Applications that take records that are fairly complex and need to access those records in a fairly complex way
Allows one to have a lot of records with a lot of engineering
Allows you to have different categories of data and access them the same way
For rapid application development
When you need a flexible schema

When To Use Document Databases

When the schema is easily changeable.
When you want to scale (especially horizontal): they are built for scale since their horizontal partitioning is inbuilt since they expect a lot of scale.
When you require simplicity e.g fewer tables, fewer relationships
When you want to aggregate data since they are built for aggregation

When Not To Use Document Databases

When you require to make a lot of updates since consistency is not guaranteed, the data may not be consistent (two nodes might have different data for the same ID)
They are not read optimized since they read each block and filter out the rows
They do not have implicit information of relations in data, therefore joins are hard if you have more than one table.

How Monitoring Works: Prometheus, Grafana and StatsD

Josephat Macharia — Tue, 27 Apr 2021 07:15:04 +0000

What is Monitoring?

The Periodic tracking (for example, daily, weekly, monthly, quarterly, annually) of any activity’s progress by systematically gathering and analyzing data and information

Why monitor?

Helps to answer the following:(Imagine or picture this):

What is happenning right now?
When did things change or what has changed (has it gone up or down did it change when we did this deploy)
When did it happen
How often does it happen e.g how many times a second, a day when do people use this feature, how many queries am i sending to the database
How long did it happen.
Enhance reliability: software bugs can be prevented via:
- testing
- measuring
- monitoring
- analyzing.

What is a metric?

The unit that is being monitored for a specific target.

Things to monitor?

request latency
request count
number of exceptions
cpu utilization

How to Monitor?

we can use traditional methods but why reinvent the wheel use monitoring tools.

What do monitoring tools offer?

collection
- Describes how the measurements are taken.
storage
- how to get metrics you measured from where you measured into something that will store them long term (usually called time series databases).
- This storage also needs to be capable of fetching that stuff quickly for analysis/graphing/alerting etc.
graphing
- visualizing and analyzing metrics collected.
aggregation windows
- how long you can keep metrics before you run out of space or the storage becomes slow.
alerting
- channels used to notify users when things happen.

What are some of the common measurement/metric types?

Counter
- Use this for situations where you want to know “how many times has x happened”.
- for example total number of HTTP requests, or the total number of bytes sent in HTTP
Gauge
- A representation of a metric that can go both up and down.
- “what is the current value of x now”.
- what is the current value of logged in users now.
Histogram

This represents observed metrics sharded into distinct buckets.
Think of this as a mechanism to track “how long something took” or “how big something was”.

Monitoring patterns for reporting metrics

Pull model
- the monitoring system "scrapes" the application at a predefined HTTP endpoint.
Push model, the application sends the data to the monitoring system.

An example of a monitoring system working in the pull model is Prometheus. StatsD is an example of a monitoring system where the application pushes the metrics to the system.

Monitoring tools

We will focus on the following:

StatsD
Prometheus

Whats is StatsD

System for instrumenting metrics
statsd is semi real time and little historical
a funnel that collects data

Why it was creatd?

Devops guys did not understand the internals of an application apart from its CPU utilization and network IO.

How it works?

StatsD will accept measurements from all over your network with UDP.
Aggregates them into 10-seconds chunks.
Sends them off to somewhere that will store the data e.g Graphite

What are the components of StatsD?

API Client --> this integrates with your actual application.
UDP Protocol --> API client just wraps over the UDP protocol.
Daemon --> this deamon runs on your actual application service.
Metrics collection: you fit your application with metrics which will send UDP packets to a local daemon the local daemon will aggregate these packets together and send them in batch to a backend

What metric types are available?

counters - how many x times something happened
timers - it took this long e,g on average
gauges - % completion
sets - keep track of unique values

How does it integrate with my app

Push metrics from your code
clients are available for many languages e.g ntest statsd

Which backends it can integrate with

graphite
mysql, influxdb, mongodb

Prometheus

A monitoring tool that actively scrapes data, stores it, and supports queries, graphs, and alerts, as well as provides endpoints to other API consumers like Grafana or even Graphite itself.

Why it exists

Created to monitor highly dynamic container environments like kubernates and docker swarm etc
However it can used in a traditional non container infrastructure.
It has become the mainstream monotoring tool of choice in container and microservice world.

Where can i use it?

When running multiple servers that run containerized applications and there are x(processes) running on that infrastructure and things are interconnected.
In complex infrastructure with lots of servers distributed

What can go wrong in complex infrastructure?

One service can crush and cause failure to others.This can be difficult to debug manually.
Application downtime
Errors
Overloaded and running out of resources
Response latency

What is the remedy for problems in complex infrastructure

Have a tool that constantly monitors all the services.
Alerts the maintainers as soon as one service crashes e.g you know what happened.
Identify problems before they occur and alert the system admin responsible for that infrastructure to prevent that issue e.g
- when the application is about to run out of storage space
- when an application becomes too slow, have tool that detects network spikes

What it offers

Automated monitoring and alerting for devops workflow

Architecture/How it works

Prometheus Server

Does the actual monitoring work and made up of three components
1. Storage: time series database --> stores all the metrics data e.g current cpu usage, no_of_requests
2. Data retrieval worker responsible for getting or pulling those metrics and pushing them to the database
3. Server API That accepts queries for that stored data and used to display the data/visualize inside prometheus or other graph tool

What is the work of prometheus alert Manager

Responsible for firing alerts via different channels e.g email, slack.
Determines who receives? how it triggers.
Reads the rule, if the condition is met then an alert is fired.

Where is data stored

Stores the data on disk also integrates with remote storage.

What are the characteristics

Designed to be reliable even when other services have an outage.
Standalone and self containing --> does not depend on network storage

What it Monitors

A particulat thing e.g a linux server, windows, database server
Things it monitors are called targets
- For linux server it can be, CPU status, memory disk, space usage
- For application server it can be exceptions count, number of request and request duration

There are 3 primitive metrics types

Counter --> How many times x happened
Gauge --> what is the current value of x now or what is the current capacity of disk space now
Histogram/Timer ---> How long or how big

How it collects metrics

It pulls metrics from a http endpoint whose host adress exposes a /metrics endpoint. for that to work:
- The target must expose the /metrics endpoint
- Data available at /metrics must be in a format that prometheus understands

Target Endpoints and exporters

Some servers are already exposing the /metrics endpoint so no extra work is needed to get metrics from them
Many services dont have native prometheus endpoint and need an extra component known as an exporter

Exporter is a script or service that fetches metrics from ur target and converts them into a format prometheus understands
and exposes these metrics to its own /metrics endpoint

How to monitor a linux/windows server

Download a node exporter
untar and execute
converts metrics of the server
exposes /metrics endpoint
configure promethues to scrape this endpoint

How to monitor an application

At application level you can monitor the following:

How long request are taking
- How many requests Use client libraries to expose /metrics endpoint

Why care

In push mechanism applications/servers push to a centralized platform
high load of network traffic: when working with many microservices and you have each service pushing their metrics to the monitoring system it creates a high load of traffic in your infrastructre and your monitoring can be your bottleneck
must install a daemon on each target to push data to the monitoring server.
Better detect/insight if service is up and running

note: for shortlived gateway it offers a pushgateway

How it know what to scape and when?

All that is confugured in promethues.yml:
- define what targets to scrape and at what interval.
- It uses a service discovery to discover those endpoints.

What is it does not do well

Dificult to scale when you have 100 of servers you might want to have multiple prometheus servers that somewhat aggregate all these metrics data and configuring that can be dificlut

Workarounds

Increase prometheus server capacity.
Limit number of metrics.

Example

Graphana

Allows you to query, visualize alert and understand your metrics no matter where they are stored.

Enables you to create, explore and share dashboards with your team.

Example:
Explore metrics

Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query

1.tns_request_duration_seconds_count

is a counter, a type of metric whose value only ever increases. Rather than visualizing the actual value, you can use counters to calculate the rate of change, i.e. how fast the value increases.
1. Add the rate function to your query to visualize the rate of requests per second. Enter the following in the Query editor and then press Shift + Enter.

rate(tns_request_duration_seconds_count[5m])

Immediately below the graph there’s an area where each time series is listed with a colored icon next to it. This area is called the legend.

PromQL lets you group the time series by their labels, using the sum function.

Add the sum function to your query to group time series by route:

sum(rate(tns_request_duration_seconds_count[5m])) by(route)

Add a logging data source

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

Loki

Grafana Loki is a set of components that can be composed into a fully featured logging stack.

Grafana supports log data sources, like Loki. Just like for metrics, you first need to add your data source to Grafana.

Explore logs

{filename="/var/log/tns-app.log"}

Grafana displays all logs within the log file of the sample application. The height of each bar encodes the number of logs that were generated at that time.

{filename="/var/log/tns-app.log"} |= "error"

Logs are helpful for understanding what went wrong

you can correlate logs with metrics from Prometheus to better understand the context of the error.

Build a Dashboard

A dashboard gives you an at-a-glance view of your data and lets you track metrics through different visualizations.

Dashboards consist of panels, each representing a part of the story you want your dashboard to tell.

Every panel consists of a query and a visualization. The query defines what data you want to display, whereas the visualization defines how the data is displayed.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels).

Annotate events

When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs.

query annotations
{filename="/var/log/tns-app.log"} |= "error"

Alerts

Alerts allow you to identify problems in your system moments after they occur. By quickly identifying unintended changes in your system, you can minimize disruptions to your services.

Alerts consists of two parts:

Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.
Alert rules - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.

Supported Alert Notification Channels
Email
Slack
Kafka
Google Hangouts Chat
Microsoft Teams
etc.

Others to look at

TICK stack
Google cloud Monitoring

Document Databases e.g MongoDB

Josephat Macharia — Tue, 27 Oct 2020 14:25:36 +0000

This course focuses on mapping the knowledge of:

architects designing a database schema
developers accessing the database from an application
from SQL or RDBMS (Relational DataBase Management Systems) to MongoDB

17 days of code complete Day 1 - Chapter 1 and 2

Josephat Macharia — Mon, 14 Sep 2020 21:19:04 +0000

In the coming days i will be giving a summary and my own review and thoughts of Code Complete book by Steve McConnell chapter by chapter. I hope you find this book review helpful.

Chapter 1

What Is Software Construction?

In common usage, “construction” refers to
the process of building. The construction process might include some aspects of
planning, designing, and checking your work, but mostly “construction” refers to
the hands-on part of creating something

Construction activities:

coding and debugging
detailed design
unit testing
integration testing

Tasks involved in construction activities:

Verifying that the groundwork has been laid so that construction can proceed successfully.
Determining how your code will be tested.
Designing and writing classes and routines.
Creating and naming variables and named constants.
Selecting control structures and organizing blocks of statements.
Unit testing, integration testing, and debugging your own code.
Reviewing other team members’ low-level designs and code and having them review yours.
Polishing code by carefully formatting and commenting it.
Integrating software components that were created separately.
Tuning code to make it smaller and faster.

Nonconstruction activities:

Management.
Requirements development.
Software architecture.
User-interface design.
System testing and maintenance

Why Is Software Construction Important?

Putting construction in its context with other activities helps
keep the focus on the right tasks during construction and appropriately
emphasizes important nonconstruction activities.
Construction is a large part of software development.
Construction is the central activity in software development.
individual programmer’s productivity can improve enormously.

My thoughts:

When tuning code to make it smaller and faster one should be pragmatic about choosing algorithms, the fastest is not always the best for the job
Design to test, build the tests before you implement the code.build testability into the software from the very beginning.
Continuous integration and delivery should be part of the construction process to prevent the integration hell, although integrating code frequently does not by itself offer any guarantees about the quality of the new code or functionality.
Although not mandatory including acceptance testing to be part of the construction process minimizes the amount of bugs that could be realized.
Well written code should be self documenting and does not need comments.

Chapter 2: Metaphors for a Richer Understanding of Software Development

A software metaphor is more like a searchlight than a roadmap. It doesn’t tell
you where to find the answer; it tells you how to look for it. A metaphor serves
more as a heuristic than it does as an algorithm.

An algorithm is predictable, deterministic, and not subject to chance

A heuristic is a technique that helps you look for an answer. Its results are
subject to chance because a heuristic tells you only how to look, not what to find.
It doesn’t tell you how to get directly from point A to point B; it might not even
know where point A and point B are. In effect, a heuristic is an algorithm in a
clown suit. It’s less predictable, it’s more fun, and it comes without a 30-day
money-back guarantee

The main difference between the two is the level of indirection from the solution.
An algorithm gives you the instructions directly. A heuristic tells you how to discover the instructions for
yourself, or at least where to look for them.

How to Use Software Metaphors

Use them to give you insight into your programming problems and processes.
Use them to help you think about your Programming activities and to help you imagine better ways of doing things.

The Importance of Metaphors

Metaphors have the virtue of an expected behavior that is understood by all.
Unnecessary communication and misunderstandings are reduced.
Learning and education are quicker.
In effect, metaphors are a way of internalizing and abstracting concepts allowing one’s thinking to be on a higher plane and low-levelmistakes to be avoided.

Common Software Metaphors

Software Penmanship: Writing Code
- “writing code.” The writing metaphor suggests that developing a program is like writing a casual letter—you sit down with pen, ink, and paper and write it from start to finish.
Software Farming: Growing a System
- Creating software as something like planting seeds and growing crops. You design a piece, code a piece, test a piece, and add it to the system a little bit at a time. By taking small steps, you minimize the trouble you can get into at any one time.
- If you buy into the farming metaphor, you might find yourself talking about fertilizing the system plan, thinning the detailed design, increasing code yields through effective land management, and harvesting the code itself.
Software Oyster Farming: System Accretion

Learn how to add to your software systems a small amount at a time. Other words closely related to accretion are “incremental,” “iterative,” “adaptive,” and “evolutionary.

4.Software Construction: Building Software

If you’re building a first-class software product, you
might build your own scientific functions for better speed or accuracy. You
might build your own container classes, user interface
classes and database
classes to give your system a seamless, perfectly consistent
look and feel.
Both building construction and software construction both benefit from
appropriate levels of planning. If you build software in the wrong order, it’s hard
to code, hard to test, and hard to debug. It can take longer to complete, or the
project can fall apart because everyone’s work is too complex and therefore too
confusing when it’s all combined.
The more experienced you have with the
kind of software you’re building, the more details you can take for granted. You
just want to be sure that you plan enough so that lack of planning doesn’t create
major problems later.
In software you might generally use flexible,
lightweight development approaches, but sometimes rigid, heavyweight
approaches are required to achieve safety goals and other goals.
Making structural changes in a program costs more than adding or deleting peripheral features.

5.Toolbox Metaphor

A good craftsman knows the
right tool for the job and knows how to use it correctly. Programmers do too.
The more you learn about programming, the more you fill your mental toolbox
with analytical tools and the knowledge of when to use them and how to use
them correctly.
The toolbox metaphor helps to keep all the methods, techniques, and tips in
perspective—ready for use when appropriate.

To extend the “building a house metaphor” construction workers do not use a single tool — the hammer — to build a house. Instead, they have a complete toolbox full of different implements for the varied tasks involved in construction. Likewise, in software engineering, or data science, the person who only knows one method will not get very far.You should be open to learning from others, respond constructively to feedback, and fill your toolbox with the appropriate techniques.