DEV Community

Edwin Njihia
Edwin Njihia

Posted on

Markdown & Data Engineering

#documentation #markdown #software

Introduction

At first markdown looks too simple to bother with, no complicated applications to install and no graphical editor to use. You write text using a small set of symbols which are rendered into as a structured document. It's a practical way of creating documentation that can live alongside code, data and technical projects.

So, what is markdown?

A lightweight markup language used to format plain text.
Instead of clicking buttons to make text bold, creating headings, or inserting code blocks, you can use simple characters.
For example:

# make text heading
*make text italic*
** make text bold**
***bold and italic***
Enter fullscreen mode Exit fullscreen mode

italic
bold
bold & italic

The markdown then rendered into a formatted document

Headings

headings help structure documents.

# main heading
## subheading
### sub-subheading
Enter fullscreen mode Exit fullscreen mode

The number of # determines the heading level:

Data Engineering

Databases

SQL

This creates a logical hierarchy for the reader

Paragraphs and Line breaks

Leave a blank line between the paragraphs. Yes, it's that easy...

Lists

we can make two types of lists Ordered & unordered lists

Unordered list 

- PSQL
- MySQL
- Git
Enter fullscreen mode Exit fullscreen mode

result:

  • PSQL
  • MySQL
  • Git ##Odered list 
1. Extract
2. Transform
3. Load
Enter fullscreen mode Exit fullscreen mode

Links

We create links using:

[link text](https://example.com)
Enter fullscreen mode Exit fullscreen mode

Acha nkutume google goggle
Square brackets get the text to output and the link in the paretheses

Tables

we can also make tables,who would have thought.

| Tool | Purpose|
|---|---|
|PSQL | Database|
|Git | Version control|
|GitHub | collaboration|
Enter fullscreen mode Exit fullscreen mode

Our output:

Tool Purpose
PSQL Database
Git Version control
GitHub Collaboration

Blockquote

created using >.

Place it before your quote, walaah

Checklists

Checklists are useful for planning and tracking progress

- [x] Install Git
- [x] Install GitHub
- [] Create SSH connect
- [] Create Repo
Enter fullscreen mode Exit fullscreen mode
  • [x] Install Git
  • [x] Install GitHub
  • [] Create SSH connect
  • [] Create Repo Checklists on articles aren't interactive though they are on other platforms where there's need

Whats the value point for Data Engineers?

Data Engineering isn't only about writing code and working with database. Real projects need to be understandable, clear documentation is as important and part of the Engineering.

Imagine building a pipeline that extracts data from an API, transforms it with python stores it on PostgreSQL and feeds a dashboard.
Assume we onboard a new dev, without clear documentation they would have to reverse engineer the whole system just to get it.

With a good README, they can:

  • Understand what the project does
  • Set up the environment
  • Understand the project structure
  • Run the pipeline
  • Understand how data flows through the system
  • Troubleshoot common issues
  • Contribute without reverse-engineering the code

That's Markdown and its part in the engineering. Markdown was our first lesson and to tell the truth, I now understand why, markdown is essential for documenting a Data Engineers journey and the systems they build.

Top comments (0)