DEV Community: Caleb Kilemba

How to Prepare for a Technical Data Engineer Interview

Caleb Kilemba — Thu, 28 May 2026 18:21:22 +0000

Data Engineering landscape in Kenya

Data engineering has become one of the most important technical roles in modern organizations. Every company wants dashboards, AI models, automated reports, customer insights, fraud detection, credit scoring, logistics optimization, and real-time monitoring. But behind all these solutions is one key person: the data engineer.

In Kenya, the demand for data engineers is growing across banks, fintechs, insurance companies, telecoms, health-tech startups, NGOs, logistics companies, government-linked digital projects, and international organizations. Companies are no longer just looking for someone who can write SQL queries. They want someone who can design reliable data pipelines, clean messy data, manage databases, work with cloud tools, automate workflows, and support analytics and machine learning teams.

Preparing for a technical data engineer interview therefore requires more than memorizing definitions. You need to understand the role deeply, practise real technical problems, build projects, and be able to explain your thinking clearly.

This article breaks down how to prepare for a data engineering interview, especially if you are applying in the Kenyan job market.

Steps by step guidance

1. Understand the Role Clearly

Before the interview, understand what a data engineer actually does.
Common responsibilities include:

Building ETL and ELT pipelines
Designing databases and data warehouses
Ensuring data quality and security
Automating reports and workflows
Supporting dashboards and machine learning teams
Cleaning and transforming data
Connecting to APIs
Monitoring failed pipelines

In Kenya, many companies are still moving from manual Excel reporting to automated data systems. This means employers value candidates who can solve practical problems, not just explain theory.

2. Master SQL
SQL is one of the most important skills in a data engineering interview.You need to understand your logic clearly because interviewers want to know how you think as well.
Practise with questions such as:
For example, finding duplicate customers

SELECT 
    customer_id,
    SUM(amount) AS total_amount
FROM transactions
GROUP BY total_amount
ORDER BY total_amount DESC
LIMIT 5;

3. Learn Python for Data Engineering

Python is commonly used for automation, API integration, file processing, and data cleaning.
Focus on python libraries such as pandas, requests, and error handling.
For example; importing pandas libraries
import pandas as pd
import requests

4. Prepare for Cloud and Pipeline Tools

Many Kenyan companies are adopting cloud tools, especially AWS, Azure, and Google Cloud.
Also learn pipeline and orchestration tools such as airflow, dbt, prefect.
A good answer to a cloud pipeline question may be:

"I would extract data from the source API, store the raw data in cloud storage, validate and transform it, load it into a warehouse, and add monitoring to alert the team when the pipeline fails."

Here is an flowchart to help you understand various areas to prepare for an interview as a data engineer.

To understand how to land a data analyst role, kindly check the article below;
How to land a data analyst role in 6 months

Conclusion

Many organizations need people who can move them from manual reporting to automated, reliable, and scalable data systems. Do not just memorize definitions. Build projects, explain your thinking, practise real interview questions, and show how your skills solve business problems. A strong data engineer is not just someone who moves data. A strong data engineer builds systems that businesses can trust.

Understanding Git and GitHub for beginners

Caleb Kilemba — Thu, 29 Jan 2026 14:01:49 +0000

Before diving into modern software development, it’s important to understand the tools that make collaboration, version control, and code management possible. Whether you are just starting your programming journey or looking to understand how developers work together on real-world projects, Git and GitHub are foundational skills you cannot ignore. This article breaks down these concepts in a simple, practical, and beginner-friendly way, helping you build a strong base before moving into hands-on usage.

What is Git

Git is a version control system/software on your computer that tracks every change that is used to track changes. This tool is used mostly by software developers, its helps them trace easily any changes made or errors on projects. It also makes it possible for multiple people to work on the same project simultaneously using branches thus avoiding code overlapping. It can be used as a backup where historical projects can be locally saved.

What is GitHub?

Github is a web -based platform designed to help developers, collaborate and manage projects with ease. It also helps to store code. Github serves as a portfolio for coding projects. It also allows developers from all around the world to contribute to your project.

Hope we are still together, lets continue with the learning

GitHub	Git
Cloud-based hosting platform for Git repositories	Version control system
Requires an internet connection to access repositories	Operates locally on your machine
Provides collaboration and project management tools	Tracks changes in code

Prerequisites for Using Git

Before you can start using Git on your machine, you need to ensure it is properly installed based on your operating system:

Windows: Download and install Git Bash. This provides a Unix-style command-line experience which is the standard for Git operations.
macOS / Linux: Open your Terminal. Git is often pre-installed, but if it isn't, you can install it using your system's package manager (e.g., brew install git for Mac or sudo apt install git for Linux).

git --version

[Wanna know about kenya? Here is the description of Kenya](https://www.google.com/search?gs_ssp=eJzj4tTP1TcwtCxKNzVg9GLNTs2rTAQALWAFHw&q=kenya&rlz=1C1GCEA_enRO1099KE1196&oq=kenya&gs_lcrp=EgZjaHJvbWUqCggBEC4YsQMYgAQyBggAEEUYOTIKCAEQLhixAxiABDIMCAIQIxgnGIAEGIoFMhAIAxAuGIMBGLEDGIAEGIoFMg0IBBAAGIMBGLEDGIAEMg0IBRAAGIMBGLEDGIAEMhAIBhAuGMcBGLEDGNEDGIAEMg0IBxAAGIMBGLEDGIAEMhMICBAuGIMBGMcBGLEDGNEDGIAEMhAICRAuGMcBGLEDGNEDGIAE0gEMMTU2MjE5NmowajE1qAIIsAIB8QUzIP-8ONwPJQ&sourceid=chrome&ie=UTF-8

trnteurewywm

Caleb Kilemba — Thu, 15 Jan 2026 19:24:00 +0000

eg4yjrum

My First Article in the Data Engineering Series

Caleb Kilemba — Thu, 15 Jan 2026 19:22:55 +0000

title: Part 2 – Designing Data Pipelines
published: true
series: Data Engineering from Zero to Production

tags: dataengineering, etl, pipelines

Heading 1

heading 2

Today this is the first mark down class that we have done
k = 5

jskdjdld
hdid

kenya
Uganda
Nigeria

Dev.to

kenya
kenya

15 foundational concepts on Data Engineering

Caleb Kilemba — Tue, 12 Aug 2025 23:14:39 +0000

Introduction

Data engineering is the backbone in mordern analytics, AI, and business intelligence. It involves designing, building, and mantaining the systems that store, process, and make data accessible for analysis. In this article, I will explain the 15 core foundational concepts every aspiring or practicing data engineer should master.

Data Modeling

Data modeling is the process of designing how data is structured and related. Data modeling provides a blueprint for databases, ensuring that data is stored logically and efficiently. A well designed model reduces redundancy, improves query performance, and ensures data integrity.
The core aspects of data modeling include conceptual models which include entities and relationships, logical model that includes tables, columns, and data types, and Physical model that includes implementation details such as indexes and partitions.

An ER (Entity-Relationship) diagram showing customers, orders, and products.

Data Warehousing

A data warehouse is a repository that stores intergrated data from multiple sources for analysis and reporting. It plays a vital role in business intelligence and decision making processes.
characteristics of a data warehouse
It is subject oriented --> it is organized around key business subjects
Intergrated --> it combines data from different sources with consistent naming and formating.
It is non-volatile --> Data is read-only once centered and not changed.
Time-Variant --> It mantains historical data for trend analysis.

--> Data sources for a data warehouse include operational systems, external data, flat files, and external data.
--> The ETL process is an architectural component of a data warehouse in data preparation.
There are three types of data warehouses;

Enterprise Data warehouse --> this is comprehensive and organization wide.
Data Mart --> This is smaller and department specific subset
Operational Data Store --> This is Near real time data used for data reporting.

ETL (Extract, Transform, Load)

This is the process of extracting data from sources, transforming the data into a usable format and loading the data into storage.
ETL process is a foundational in data engineering as it ensures clean, and reliable data for analytics. In cloud warehouses, ELT (Extract, Load, Transform) is common. There are also modern variations of streaming ETL for real time pipelines.

Data Pipelines

A data pipeline is a system that automates the movement, transformation, and processing of data from various sources to a destination such as a data warehouse. Data pipelines ensures data flows efficiently and reliably through different stages, enabling analytics, and machine learning.
Types of data pipelines
Batch pipelines --> this processes data in scheduled chunks i.e daily updates, a good example is loading sales data into a warehouse hourly.
Streaming pipelines --> these process real time data i.e transaction data
ETL/ELT --> Transforms/loads data into destination

Directed Acyclic Graph (DAG)

Data Formats and Serialization

Data doesn’t just exist in thin air — it’s stored and transmitted in specific formats, and the choice of format has big consequences.
Common formats:
CSV (Comma-Separated Values) – A flat text file where each line represents a row and commas separate values. It’s easy for humans to read and for most systems to process, but lacks advanced features like data types or compression. Best for simple datasets and compatibility across tools.
JSON (JavaScript Object Notation) – Stores data in key-value pairs with a hierarchical structure. Flexible and ideal for web applications or APIs, but can be verbose, leading to larger file sizes.
Parquet / ORC – Columnar storage formats optimized for analytics. Instead of storing data row-by-row, they store it column-by-column, enabling efficient compression and faster queries for analytical workloads.
Avro / Protobuf – Schema-based formats that are compact and designed for efficient serialization (turning data into bytes for transmission). They enforce structure and are ideal for streaming pipelines or cross-language communication.
Why it matters:
Choosing the right format affects:
Performance – Columnar formats can make analytical queries much faster.
Storage cost – Compression in Parquet/ORC can significantly reduce storage usage.
Interoperability – Some formats work better for system integration (JSON) while others are better for internal analytics (Parquet).

Data Quality Management

Data quality is about ensuring that the data you’re using is fit for purpose. Bad data = bad decisions.

Key dimensions:

Completeness – No missing required values.
Consistency – The same data is represented in the same way across datasets.
Accuracy – Data reflects the real-world truth it represents.
Timeliness – Data is up-to-date when needed.

Why it matters:

If your analytics are based on incomplete, inconsistent, or outdated data, the resulting insights could mislead business decisions, waste resources, or even cause compliance issues.

Data Governance

Think of this as the rulebook for data. It defines who can access what, how data is documented, and how it complies with laws.

Key elements:
Metadata management – Keeping a record of what each dataset is, where it came from, and what it contains.
Access control – Using role-based or attribute-based permissions to control who sees what.
Regulatory compliance – Ensuring data handling follows laws like GDPR (privacy) or HIPAA (healthcare).
_Why it matters:
_Good governance builds trust in data, avoids legal trouble, and makes it easier for teams to collaborate without stepping on each other’s toes.

Scalability and Performance Optimization

When your dataset grows from gigabytes to terabytes, your systems need to keep up without slowing down.
Techniques:
Sharding and partitioning – Splitting data across multiple databases or files to reduce load on any single resource.
Caching – Storing frequent query results in fast-access memory instead of recalculating them.
Parallel processing – Breaking tasks into smaller chunks to be processed simultaneously (e.g., Spark, Dask).
Why it matters:
Without optimization, systems become bottlenecks, leading to delays, timeouts, and higher costs.

Cloud Data Platforms

Cloud providers now offer fully managed data warehouses that handle scaling, backups, and performance tuning for you.

Examples:
AWS Redshift – Great for heavy analytics workloads on AWS.
Google BigQuery – Serverless, pay-per-query, and fast.
Snowflake – Popular for its separation of storage and compute, allowing elastic scaling.
Azure Synapse – Integrates tightly with Microsoft’s ecosystem.
Why it matters:
They remove much of the operational burden, allowing teams to focus on data and analytics rather than infrastructure.

Data Security

Protecting data is non-negotiable — both for legal reasons and to maintain trust.

Practices:
Encryption at rest – Protects stored data.
Encryption in transit – Protects data while it’s moving across networks.
Access control – Restricts data access based on user roles.
Audit logging – Keeps a record of who accessed or modified data.

Why it matters:
A breach can cost millions in fines, damage a company’s reputation, and violate customer trust.

Workflow Orchestration

Data pipelines have many moving parts — they must run in the right order, handle failures, and restart if needed.

Tools:
Apache Airflow – The most widely used, with rich scheduling and monitoring features.
Prefect – More Python-friendly and developer-centric.
Luigi– Lightweight but effective for smaller pipelines.
Why it matters:
Without orchestration, pipelines may break silently, run in the wrong order, or fail without alerting anyone.

Monitoring and Observability

You can’t improve what you can’t measure. Monitoring ensures data systems are healthy and issues are detected early.

Metrics to track:
Data freshness – How recently the data was updated.
Throughput – Amount of data processed over time.
Failure rates – Percentage of failed jobs or queries.

Tools:
Prometheus – Open-source metrics collection.
Grafana – Visualization and alerting.
Datadog – Commercial, all-in-one monitoring.

Data Lineage

This is the “data family tree” — where it came from, how it changed, and where it ended up.

Why it matters:
Debugging – If a report looks wrong, you can trace back to the source.
Compliance – Regulations may require knowing exactly where data originated.
Trust – Users can see the full journey from source to dashboard.

Conclusion

Mastering these 15 foundational concepts gives a solid grounding in data engineering. Tools may change, but these principles guide the design of efficient, scalable, and trustworthy data systems.