DEV Community: John Otienoh

Data Engineering Described

John Otienoh — Sat, 13 Jun 2026 08:48:36 +0000

Source: Fundamentals of Data Engineering by Joe Reis and Matt Housley, published by O'Reilly Media.

This article summarizes and interprets key concepts from Chapter 1: Data Engineering Described. It is not a reproduction of the original text but a study guide and learning resource based on the chapter.

Data engineering has become one of the most critical disciplines in modern technology organizations. Every dashboard, machine learning model, business report, and analytical insight depends on reliable data pipelines built and maintained by data engineers.

What Is Data Engineering?

Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning. A data engineer manages the data engineering lifecycle, beginning with getting data from source systems and ending with serving data for use cases, such as analysis or machine learning.

Data engineering sits at the intersection of several disciplines:

Data Architecture
Data Management
Security
DataOps
Orchestration
Software Engineering

A data engineer's responsibility extends from collecting data from source systems to making that data available for analytics, reporting, and machine learning applications.

The Data Engineering Lifecycle

Data engineering is a lifecycle rather than a collection of isolated tools.

The lifecycle consists of five major stages:

Generation → Storage → Ingestion → Transformation → Serving

The Undercurrents of Data Engineering

While the lifecycle represents the flow of data, several concepts influence every stage i.e undercurrents:

Security
Data Management
DataOps
Data Architecture
Orchestration
Software Engineering

These disciplines are not separate activities; they support and shape the entire lifecycle.

For example:

Security protects data assets.
DataOps improves operational efficiency.
Architecture ensures scalability.
Software engineering principles improve reliability and maintainability.

The Evolution of the Data Engineer

The Data Warehousing Era (1980s–2000s)

The first major era of data engineering emerged with data warehousing.

Organizations began building centralized repositories for analytics using:

Enterprise Data Warehouses
Massively Parallel Processing (MPP) systems
Business Intelligence platforms

This period introduced large-scale analytical processing capabilities.

The Big Data Era (2000s–2010s)

As data volumes exploded, traditional systems struggled to scale.

New technologies emerged:

Hadoop
Spark
Distributed storage systems
Cloud-based data platforms

The role evolved into the "Big Data Engineer," focused on processing massive datasets efficiently.

The Modern Data Engineering Era

Today's data engineers work across:

Cloud platforms
Data lakes
Streaming systems
Machine learning infrastructure
Real-time analytics

The profession now emphasizes business value as much as technical implementation.

Data Engineering vs. Data Science

One of the most important distinctions made in Chapter 1 is that data engineering and data science are separate disciplines.

Data Engineers

Focus on:

Building pipelines
Managing infrastructure
Ensuring data quality
Scaling systems
Delivering reliable datasets

Data Scientists

Focus on:

Analysis
Experimentation
Statistical modeling
Machine learning
Business insights

In simple terms:

Data Engineers → Build the foundation

Data Scientists → Create value from that foundation

Data engineering sits upstream, providing the inputs necessary for data science and analytics.

Understanding Data Maturity

Data maturity reflects how effectively an organization leverages data as a strategic asset.

Importantly, maturity is not determined by company age or size.

A startup can be more data mature than a century-old enterprise if it uses data more effectively.

Stage	Primary Focus	Key Activities	Best Practices	Risks / Pitfalls
1. Starting with Data	Establish a data foundation aligned with business goals	- Define data architecture - Identify and audit relevant data sources - Build foundational data systems - Enable future analytics and ML use cases	- Secure executive sponsorship - Deliver quick wins to demonstrate value - Engage business stakeholders frequently - Use off-the-shelf solutions where possible - Build custom solutions only for competitive advantage	- Lack of visible business impact reduces support - Technical debt from rapid delivery - Working in silos without stakeholder feedback - Overengineering and unnecessary complexity
2. Scaling with Data	Create scalable, repeatable, and operational data practices	- Formalize data engineering processes - Build scalable architectures - Implement DevOps and DataOps practices - Develop ML-ready infrastructure	- Prioritize simplicity and maintainability - Focus on team productivity and scalability - Select technologies based on business value - Educate the organization on data usage	- Chasing trendy technologies without ROI - Overcomplicating infrastructure - Treating technology as the bottleneck instead of team capacity - Focusing on technical prestige instead of business outcomes
3. Leading with Data	Use data as a strategic competitive advantage across the organization	- Automate data onboarding and usage - Build proprietary data products - Implement governance, quality, and metadata management - Deploy data catalogs and lineage tools - Foster cross-functional collaboration	- Invest in DataOps and governance - Promote transparency and collaboration - Share data broadly across teams - Build custom systems only when they create measurable advantage	- Organizational complacency - Neglecting maintenance and continuous improvement - Pursuing expensive technology projects with little business value - Overengineering custom solutions without strategic benefit

The Background and Skills of a Data Engineer

Business Responsibilities of a Data Engineer

Technical ability alone is not sufficient.
Here are several critical business responsibilities.

Communication

Data engineers must communicate effectively with:

Executives
Product managers
Analysts
Data scientists
Software engineers

Trust and collaboration are essential.

Requirements Gathering

Engineers must understand:

What stakeholders need
Why they need it
How data solutions support business goals

Understanding Agile, DevOps, and DataOps

These are cultural practices as much as technical methodologies.

Successful implementation requires organizational alignment, not just tooling.

Cost Management

Data engineers should optimize:

Time-to-value
Total cost of ownership
Opportunity cost

Monitoring cloud and infrastructure spending is a core responsibility.

Continuous Learning

The data ecosystem evolves rapidly.

Strong data engineers:

Learn continuously
Evaluate new technologies critically
Distinguish innovations from hype

Technical Responsibilities of a Data Engineer

SQL

SQL remains the most important language in data engineering.

It is used for:

Querying data
Transformations
Analytics
Data warehousing
Data lake-house operations

Despite the rise of big data technologies, SQL continues to be the dominant language of data.

Python

Python serves as a bridge between data engineering and data science.

Popular tools include:

Pandas
NumPy
Airflow
PySpark
TensorFlow
PyTorch
Scikit-learn

Python excels at automation, orchestration, and integration.

Java and Scala

These languages are commonly used in large-scale distributed systems such as:

Apache Spark
Apache Hive
Apache Druid

They often provide greater performance and lower-level access than Python APIs.

Bash

Command-line skills remain valuable for:

Automation
File processing
System administration
Data pipeline operations

Tools like awk, sed, and shell scripting continue to play an important role in production environments.

Key Takeaways

Chapter 1 establishes several foundational ideas:

Data engineering transforms raw data into usable information.
The data engineering lifecycle includes generation, storage, ingestion, transformation, and serving.
Security, architecture, DataOps, and software engineering influence every stage of the lifecycle.
Data engineering and data science are complementary but distinct disciplines.
Data maturity reflects how effectively organizations use data as a strategic asset.
Successful data engineers combine technical expertise with business understanding.
SQL, Python, JVM languages, and Bash remain foundational technical skills.
The ultimate goal of data engineering is delivering business value through reliable and scalable data systems.

Reference

Reis, J., & Housley, M. (2022). Fundamentals of Data Engineering: Plan and Build Robust Data Systems. O'Reilly Media. Chapter 1: Data Engineering Described.

Me-API

John Otienoh — Fri, 17 Oct 2025 13:04:10 +0000

Simple RESTful API endpoint that returns your profile information along with a dynamic cat fact fetched from an external API.

Introduction

As part of the HNG Internship Backend track Stage 0 Task, I was challenged to build a simple yet dynamic RESTful API that returns my profile details along with a live cat fact fetched from the Cat Facts API.
This task has helped me sharpen key backend development concepts — from API integration, response structuring, time formatting, to error handling and deployment.

Task Requirements

The goal was to create a GET /me endpoint that returns:

{
  "status": "success",
  "user": {
    "email": "johndoe@gmail.com",
    "name": "John Doe",
    "stack": "Python/FastAPI"
  },
  "timestamp": "2025-10-17T12:34:56.789Z",
  "fact": "Cats sleep for 70% of their lives."
}

The cat fact must come dynamically from the public API

Tools & Technologies

Framework: FastAPI
Language: Python 3
Deployment Platform: Railway

My Implementation Process

- Project Setup
I created a new FastAPI project and installed dependencies:

pip install fastapi uvicorn requests python-dotenv

- Defined Schema with Pydantic.
To maintain clean and validated responses, I created a schema.py:

from datetime import datetime
from pydantic import BaseModel, EmailStr

class User(BaseModel):
    """Model for the response"""
    email: EmailStr
    name:str
    stack: str

class Profile(BaseModel):
    status: str
    user: User
    timestamp: datetime
    fact: str

- Get cat facts
To get the cat facts from the Cat Fact API, i created the catfact.py

import requests

def cat_fact(url: str) -> str:
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        response_dict = response.json()
        fact = response_dict["fact"]
    except Exception:
        fact = "Could not fetch cat fact at this time."

    return fact

- Built the /me Endpoint

from datetime import datetime, timezone
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware 

from app.schema import Profile
from app.catfact import cat_fact

app = FastAPI()
URL = "https://catfact.ninja/fact"

origins = ["*"]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)



my_info = {
    "status": "Success",
    "user": {
        "email": "johnteclaire@gmail.com",
        "name": "John Charles Otienoh",
        "stack": "Python/fastAPI"
    },
    "timestamp": datetime.now(timezone.utc).isoformat(),
    "fact": ""
}

@app.get("/me", response_model=Profile)
async def get_my_info():
    "Retreive Basic information"
    my_info['fact'] = cat_fact(URL)
    return my_info

- Tested the API
I used cURL to test:

curl http://localhost:8000/me

The API returned a live cat fact and a real-time timestamp every request!

What I Learned

How to consume external APIs safely using requests
How to handle timeouts, errors, and fallbacks
How to return structured JSON with Pydantic models
The importance of clean documentation and response validation
How to think about API reliability and best practices (timeouts, CORS, content-type headers, etc.)

Project Links

GitHub Repo: https://github.com/john-otienoh/Me_API
Live API: https://meapi-production.up.railway.app/me
Contact: johnteclaire@gmail.com

Recursion

John Otienoh — Fri, 17 Oct 2025 12:37:13 +0000

What is Recursion?
Any function which calls itself is called recursive. A recursive method solves a problem by calling a copy of itself to work on a smaller problem. This is called the recursion step.The recursion step can result in many more such recursive calls.
It is useful in sort, search, and traversal problems that often have simple recursive solutions.

Syntax

A beginner's guide to data engineering concepts, tools, and responsibilities.

John Otienoh — Mon, 05 Aug 2024 11:34:31 +0000

Data Engineering Concepts

Data engineering is the process of designing, building, and maintaining the infrastructure for storing, processing, and retrieving large datasets. It involves creating systems that efficiently collect, transform, and store data, making it usable for analysis and decision-making.

Tools for data engineers.

Scripting and programming language: python is the commonly used language in data engineering due to its simplicity and extensive libraries , which is used in transformation and data cleaning. There are other languages that are used such as ruby, Scala among others.
Databases: data engineers use various databases like MySQL to store and manage structured data for analytics and reporting.
Data visualization tools: to gain insights and patterns from data, a data engineer should be familiar with various tools used in visualization such as tableau and Power BI.
Data warehousing and storage tools: the commonly used tool in managing data is snowflake, snowflake is a cloud data warehouse that allow one to store and manage data, snowflake is very flexible since it works with some programming languages such as python

Responsibilities:

Designing data pipelines to collect and process data from various sources
Developing data warehouses and lakes to store structured and unstructured data.
Ensuring data quality, integrity, and security.
Building scalable and high-performance data processing systems.
Collaborating with data scientists and analysts to understand their data needs.
Implementing data governance policies and metadata management.

POST-MORTEM

John Otienoh — Fri, 28 Jun 2024 20:05:58 +0000

Database Connection

Issue Summary

Duration: 2 hours 55 minutes (14:55 UTC - 17:50 UTC).
The affected service was the website's search functionality leading to 20% of users experienced slow load times, with average page load times increasing by 7 seconds.
The root Cause was a misconfigured database connection pooling leading to connection timeouts.

Timeline

14:55 UTC: Monitoring alerts triggered for high response times on the search functionality.
15:00 UTC: Engineer on call investigates and notices high CPU usage on the database server.
15:10 UTC: Initial assumption is that the issue is related to a recent code deployment, and the team begins reviewing code changes.
15:20 UTC: Investigation reveals no issues with the code deployment, and attention turns to the database server.
15:40 UTC: Misleading investigation path: The team explores the possibility of a database query optimization issue.
16:00 UTC: Escalation to the database administration team.
16:20 UTC: Root cause identified: misconfigured database connection pooling.
16:55 UTC: Configuration changes made to the database connection pooling.
17:50 UTC: Issue resolved, page load times return to normal.

Root Cause and Resolution

The root cause of the issue was a misconfigured database connection pooling setting, which led to connection timeouts and increased CPU usage on the database server. This caused slow load times for 20% of users, affecting the website's search functionality.
The issue was resolved by adjusting the database connection pooling settings to optimize connection reuse and reduce timeouts. This change was made in collaboration with the database administration team.

Corrective and Preventative Measures

Improve database connection pooling configuration and monitoring.
Implement automated testing for database connection pooling settings.
Enhance monitoring for CPU usage on the database server.
Conduct regular reviews of database server performance and configuration.

TODO List

Patch database connection pooling configuration to optimize connection reuse.
Add monitoring for database connection timeouts.
Implement automated testing for database connection pooling settings.
Schedule regular database server performance reviews.
Develop a playbook for troubleshooting database connection issues.

This postmortem highlights the importance of thorough investigation and collaboration between teams to resolve complex issues. By identifying and addressing the root cause of the problem, we can prevent similar issues from occurring in the future and improve the overall reliability of our services.

0x00. Shell, navigation

John Otienoh — Sat, 08 Jun 2024 22:42:29 +0000

File System Organization

Like Windows, the files on a Linux system are arranged in what is called a hierarchical directory structure. This means that they are organized in a tree-like pattern of directories (called folders in other systems), which may contain files and subdirectories. The first directory in the file system is called the root directory. The root directory contains files and subdirectories, which contain more files and subdirectories and so on and so on.
The basic three commands include:

pwd (print working directory)
cd (change directory)
ls (list files and directories).

pwd

The directory we are standing in is called the working directory. To see the name of the working directory, we use the pwd command.

[me@linuxbox me]$ pwd
/home/me

When we first log on to our Linux system, the working directory is set to our home directory.

cd

To change the working directory (where we are standing in the maze) we use the cd command. To do this, we type cd followed by the pathname of the desired working directory. A pathname is the route we take along the branches of the tree to get to the directory we want.

me@linuxbox me]$ cd /usr/bin
me@linuxbox bin]$ pwd
/usr/bin

If we type cd followed by nothing, cd will change the working directory to our home directory.

me@linuxbox me]$ cd
me@linuxbox bin]$ pwd
/home/me

A related shortcut is to type cd ~user_name. In this case, cd will change the working directory to the home directory of the specified user.

me@linuxbox me]$ cd ~me
me@linuxbox bin]$ pwd
/home/me

Typing cd - or cd .. changes the working directory to the previous one.

me@linuxbox me]$ cd /usr/bin
me@linuxbox bin]$ pwd
/usr/bin
me@linuxbox me]$ cd ..
me@linuxbox bin]$ pwd
/usr

ls

It is used to list the files in the current working directory.

[me@linuxbox me]$ ls
Desktop   Download  Pictures  Music  Templates  Documents examples.desktop    Public  Videos

File names that begin with a period character are hidden. This only means that ls will not list them unless we say ls -a.

[me@linuxbox me]$ ls -la
.git/    .ssh/   .ipython/ Desktop   Download  Pictures  Music  Templates

ls -lList the files in the working directory in long format

[me@linuxbox me]$ ls -l
drwxr-xr-x 1 me 197121  0 Oct 17  2023  OneDrive/
drwxr-xr-x 1 me 197121  0 Jan 17  2023  Pictures/
drwxr-xr-x 1 me 197121  0 Mar  3  2023  Saved Games/
drwxr-xr-x 1 me 197121  0 Apr 27  2023  Searches/

Displaying file contents

There are several commands to display the content of a file in Linux.

Using cat command

$ cat filename

Using head and tail commands

The head command displays the first 10 lines of a file, while the tail command displays the last 10 lines of a file.

$ head filename   # displays the first 10 lines of a file
$ tail filename   # displays the last 10 lines of a file

You can modify the number of lines displayed by using the -n option, for example:

$ head -n 5 filename   # displays the first 5 lines of a file
$ tail -n 5 filename   # displays the last 5 lines of a file

Using less command

The less command allows you to view a file one page at a time. It allows you navigate through the file using the arrow keys or page up/down keys.

$ less filename

Using awk command

This command uses awk to print each line of the file.

$ awk '1' filename

Creating files and directories

Create a file:

Using the touch command:

$ touch filename

This will create a new empty file with the specified name.

Using a text editor:

$ nano filename # using the nano editor.
$ vi filename # using vim editor.
$ code filename # using vscode editor.

This will open a text editor where you can create and edit the file. Once you're done, save and exit the editor.

Using the echo command:

$ echo "Hello World!" > filename

This will create a new file with the specified name and add the text "Hello World!" to it.
Create a directory:

Using the mkdir command:

$ mkdir directoryname

This will create a new directory with the specified name.

Removing a file or directory

Removing a file:
To remove a file, use the rm command followed by the name of the file you want to remove:

$ rm filename

If the file is write-protected, rm will ask you to confirm the deletion. To remove the file without prompting, use the -f option:

$ rm -f filename

Removing a directory:
To remove an empty directory, use the rmdir command followed by the name of the directory:

$ rmdir directoryname

If the directory is not empty, you will get an error message. To remove a non-empty directory and all its contents, use the rm command with the -r option:

$ rm -r directoryname

Moving or Copying a file or directory

Moving a file or directory:
To move a file or directory, use the mv command followed by the source file or directory and the destination:

$ mv source destination

Renaming a file or directory:
To rename a file or directory, use the mv command with the source file or directory and the new name:

$ mv oldname newname

Copying a file or directory:
To copy a file or directory, use the cp command followed by the source file or directory and the destination:

$ cp source destination

To copy a directory and all its contents, use the -r option with the cp command:

$ cp -r source destination

This will copy the entire directory source and all its contents to destination.
Using rsync command:
The rsync command is a powerful tool for copying and synchronizing files and directories. It can be used to copy files and directories while preserving permissions, timestamps, and other attributes:

$ rsync -avz sourceDir destinationDir

This will copy the entire directory sourceDir and all its contents to the specified destination, preserving permissions, timestamps, and other attributes.

Thanks for your time! Please leave a comment and any suggestions are welcome. Follow me to get updates.