DEV Community: hammoudi wissem

The Power of Context in Data Visualization (Part 1)

hammoudi wissem — Thu, 20 Nov 2025 15:30:43 +0000

Effective data visualization doesn’t start with charts, colors, or dashboards. It starts long before that—with context. Understanding the “why” behind your communication is the foundation of compelling data storytelling. In this article, we’ll break down what context really means, why it matters, and how you can use it to communicate insights that influence decisions.

Exploratory vs. Explanatory Analysis: Two Different Worlds

One of the biggest mistakes people make is mixing exploratory and explanatory analysis. Although they work together in the analytics process, they serve very different purposes:

Exploratory Analysis – The Search

This is your behind-the-scenes detective work. You dig into the data, test hypotheses, slice and filter metrics, and look at patterns from many angles. Think of opening 100 oysters to find a few valuable pearls.

Explanatory Analysis – The Story

This is what you share with your audience. You don’t show all the oysters—you present only the pearls. The goal is clarity, not completeness.

Too often, presenters drown audiences in exploratory material instead of focusing on the insights that matter. The key is restraint: highlight the findings your audience needs, not everything you discovered along the way.

Who, What, and How: The Core of Context

Before designing any visual or writing a single line of narrative, you must answer three questions with precision:

1. Who: Understand Your Audience

Effective communication starts with a clear picture of who you’re speaking to. Avoid vague definitions like “everyone” or “stakeholders.” Identify one specific audience or decision-maker and tailor your message around them.

Consider:

What does this audience care about?
What are their goals and constraints?
How familiar are they with the topic?
What is your relationship with them—trusted advisor, newcomer, peer?

Your understanding of these factors determines how you frame your message, what you emphasize, and even how much detail you include.

2. What: Inspire Meaningful Action

Ask yourself: What do I want my audience to know or do?

This is the heart of your communication. If you cannot articulate why your message matters, you’re not ready to present it.

As the person closest to the data, you are in the best position to guide your audience. Even if you can’t make direct recommendations, you can still suggest possible directions or pose thoughtful next-step questions.

Clear calls to action turn passive listening into meaningful response.

3. How: Choosing the Right Format and Tone

Mechanism – The Format

How will you deliver your message?

Communication exists on a spectrum.

Live presentations offer control, emphasis, and interaction.
Written reports offer depth, reviewability, and precision.

The right format depends on your audience, the complexity of your message, and the action you want them to take.

Tone – The Emotion

Tone shapes how your message is received.
Are you celebrating success? Addressing a problem? Urging change?

Your words, pacing, and style should reinforce the emotional intention of your message.

How to Build Your Story with Data

Once you know your audience and your objective, the next step is to determine what data best supports your narrative.

Data provides credibility and evidence, but it should never overshadow your story. Use data to strengthen your argument—not complicate it.

Crafting Clarity: The 3-Minute Story and the Big Idea

Two powerful frameworks can help you distill even the most complex information into a crisp narrative:

The 3-Minute Story

This approach forces you to explain your insights concisely—no slides, no distractions. It’s perfect for quick executive updates or impromptu conversations.

A strong 3-minute story includes:

The problem or opportunity
What you did
What you found
What should happen next

The Big Idea

Your entire message in one clear sentence.
Nancy Duarte describes three key elements:

A unique point of view
A sense of what’s at stake
A complete, standalone sentence

Your Big Idea guides your entire communication, ensuring consistency and focus.

Example

3-Minute Story:
A group of us in the science department were brainstorming ways to address a recurring issue: fourth-grade students often enter science classes with negative attitudes. We wondered whether giving students earlier exposure to science could change this perception. So we piloted a summer program for second- and third-graders. To measure its impact, we surveyed students before and after. Initially, 40% of students felt only “OK” about science. After the program, nearly 70% expressed positive interest. This indicates strong success, and we recommend continuing and expanding the program.

Big Idea:
The pilot summer learning program improved students’ perceptions of science, and we recommend expanding it; please approve the budget.

Storyboarding: Your Blueprint for Clear Communication

Storyboarding is one of the most valuable yet overlooked steps in data storytelling. It acts as a visual outline of your message and ensures clarity before you ever open presentation software.

Why Avoid Starting with Slides?

Tools like PowerPoint push you into “slide creation mode.” You start decorating instead of thinking. The flow becomes cluttered and unfocused.

Instead, start low-tech:

Paper
Post-it notes
Whiteboards

These give you the freedom to rearrange, refine, and rethink without emotional attachment to specific layouts.

Example Storyboard: Summer Learning Program

Problem: Students have negative perceptions of science.
Action: Introduce a summer early-exposure program.
Results: Significant improvement in perceptions.
Big Idea: Expand the program.

You might even experiment by starting with the Big Idea, especially if you want your audience to engage immediately and sharply.

Why Storyboarding Matters

Storyboarding ensures:

A logical, audience-focused flow
Purpose behind every section
Fewer revisions later
Faster content creation overall

It saves time, improves clarity, and sets you up for a more persuasive final product.

Start Smart, Not Fast

Though storyboarding may feel like a slow first step, it prevents wasted effort and confusion later. It grounds your message in intent and gives you a strong foundation for building visuals and narrative.

Note: This article is inspired by the book **Storytelling with Data* by Cole Nussbaumer Knaflic, which offers foundational principles for impactful data communication.*

Storytelling with Data: The Art of Communicating Insights Effectively (Introduction)

hammoudi wissem — Thu, 20 Nov 2025 15:25:12 +0000

Bad Graphs Are Everywhere

No one intentionally creates a bad graph—yet we see them everywhere. Across industries, organizations, and even major media outlets, poorly designed charts make their way into presentations and reports. Most of these come from well-meaning individuals who simply haven’t been taught how to visualize data effectively.

But why is this so common?

We’re Not Naturally Equipped for Data Storytelling

In school, we develop two main skill sets: language and mathematics. We learn to craft cohesive stories and analyze numerical problems. Yet these two skills—storytelling and data analysis—rarely meet.

In today’s data-driven world, the ability to tell meaningful stories with numbers has become essential. But most of us were never taught how to turn raw data into a compelling narrative, nor how to design visuals that communicate clearly.

As technology enables us to collect enormous amounts of data, making sense of it has become both a challenge and a necessity. Data visualization bridges this gap, transforming numbers into insights that support better decisions.

However, without proper training, many people rely heavily on tools like Excel or PowerPoint. While these tools make graph creation accessible, they also produce visuals overloaded with unnecessary 3D effects, excessive colors, and confusing pie charts.

The result?
Missed opportunities to inform, persuade, and inspire.

The Journey of Storytelling with Data

Effective data storytelling is a transformative skill—it allows you to communicate complex insights clearly and persuasively. This series of articles will explore the core principles from Storytelling with Data, breaking each concept down into actionable techniques you can apply immediately.

Part 1: The Importance of Context

Before creating a single chart, start with one critical question: What is the context?

Ask yourself:

Who is my audience?
What do I want them to know or do?

Your answers determine the structure, tone, and level of detail in your narrative. Executives may need a high-level summary, while technical teams may want deeper analysis.

Understanding context from the beginning reduces rework and ensures your visuals support your objectives.

Our next article will explore practical ways to uncover audience needs and define your narrative purpose.

Part 2: Choosing the Right Visual

Selecting the right visual form is essential for clarity. Common and effective visuals include:

Bar charts – for comparisons
Line graphs – for trends over time
Heatmaps – for patterns and correlations
Slopegraphs – for showing changes between two points

Equally important is knowing what not to use. Pie charts and 3D graphs often distort information and confuse audiences.

In an upcoming article, we’ll explore how to match the visual to your message—ensuring clarity, accuracy, and impact.

Part 3: Clutter Is Your Enemy

Clutter is the silent killer of good data visualization. Every extra line, color, or label makes it harder for your audience to understand your message.

To avoid this:

Apply Gestalt principles to organize visual elements
Use white space, alignment, and contrast to guide the eye

When you remove unnecessary elements, your insight becomes the center of attention. We’ll share concrete decluttering techniques in a future article.

Part 4: Focus Your Audience’s Attention

Great storytelling directs attention where it matters most.

Use techniques like:

Preattentive attributes (color, size, position)
Visual hierarchy to emphasize key points

These tools ensure your audience immediately sees what you want them to notice. Our article on this topic will include practical examples of guiding attention through design.

Part 5: Think Like a Designer

Good design is not decoration—it's functionality. In data visualization, form follows function.

Key design concepts include:

Accessibility – making visuals usable for everyone
Affordances – making important elements stand out
Aesthetics – balancing beauty with clarity

We will explore how design thinking helps create visuals that are both clear and appealing.

Part 6: Dissecting Model Visuals

What makes a visualization great? In this section, we’ll analyze exemplary charts, examining:

How graph types are selected
How data is ordered
How color and alignment strengthen the message

By breaking down strong examples, you’ll learn how thoughtful design improves understanding and decision-making.

Part 7: Lessons in Storytelling

Numbers rarely stick with people—but stories do.

This chapter introduces storytelling frameworks with clear beginnings, middles, and ends, applying them specifically to data communication. You'll learn how to use:

Narrative flow
Repetition for emphasis
Smooth transitions

These tools help transform your analysis into a compelling message that people remember.

Part 8: Pulling It All Together

The final chapter brings the entire process to life through a complete example:

Defining the context
Choosing appropriate visuals
Removing clutter
Designing for clarity and impact

The result is a coherent and persuasive data story that informs, convinces, and inspires action.

The Journey Begins

Storytelling with data is more than presenting numbers—it’s about crafting narratives that influence decisions. This series will guide you through each step of the process, helping you build the skills necessary to communicate insights powerfully and effectively.

Let’s begin this journey together—one part at a time.

Note: This article is inspired by Storytelling with Data by Cole Nussbaumer Knaflic, a highly recommended resource for mastering the principles of effective data visualization.

If you'd like, I can also:

✅ Format this as a LinkedIn article
✅ Add visuals suggestions
✅ Make it shorter / more concise
✅ Adapt it to your writing style

Just tell me!

Change Data Capture(Example using Microsoft SQL Server Log-based CDC vs Trigger-based CDC

hammoudi wissem — Thu, 20 Nov 2025 15:16:41 +0000

Definition of Change Data Capture

Change Data Capture (CDC) is a technique used to detect, record, and track changes—such as inserts, updates, and deletes—made to data in real-time or near real-time. It is widely used in databases and information systems to maintain an accurate history of modifications and synchronize data across systems.

Note: CDC is considered an ETL (Extract, Transform, Load) technique where only changed data is extracted from a source system and loaded into a target system such as a data warehouse or data lake.

Importance of Change Data Capture

1. Real-time Data Synchronization

CDC captures and propagates changes instantly, ensuring all connected systems remain synchronized without delay. This is especially important when multiple databases or systems depend on updated data.

2. Event-Driven Architectures

CDC transforms data changes into events that trigger downstream processes or workflows. This enables responsive systems capable of reacting immediately to user actions, business events, or operational changes.

3. Efficient Data Processing

Since CDC transmits only modified data, it reduces the overhead of full database scans or batch processing. This speeds up pipelines and ensures downstream systems always receive fresh data.

4. Scalability and Flexibility

CDC supports asynchronous, decoupled communication, making it easier for distributed systems to scale horizontally while maintaining reliability.

5. Enhanced Analytics and Insights

Real-time data flow enables updated dashboards, instant anomaly detection, and more proactive decision-making.

Change Data Capture (CDC) Principles

Capture: Identify inserts, updates, and deletes without impacting the source system.
Log-based Tracking: Leverage transaction logs or replication logs to detect changes.
Incremental Updates: Transfer only changed rows instead of full datasets.
Real-time or Near Real-time: Propagate changes quickly to ensure data freshness.
Idempotent Processing: Avoid duplicate processing and maintain consistency.

Use Cases of Change Data Capture

Data Warehousing – Keep analytical systems updated with the latest transactional data.
Replication – Synchronize data across distributed systems for load balancing or disaster recovery.
Data Integration – Integrate modern and legacy systems or synchronize between cloud and on-premises systems.
Real-time Analytics – Feed streaming platforms or dashboards with immediate updates.
Data Synchronization – Maintain consistency across microservices, caching layers, or distributed components.

Real-World Example: CDC at Netflix

Netflix employs a CDC pipeline built on Apache Kafka and Apache Flink to extract changes from internal databases and process them in real-time. This architecture powers:

Service Usage Monitoring – Tracking user interactions to optimize performance.
Recommendation Engine – Providing personalized content suggestions.
Fraud Detection – Identifying abnormal activities instantly.

Change Data Capture Methods

Below are the most common CDC techniques, including how they work and their pros/cons.

1. Log-Based CDC

How It Works

Changes are captured directly from the database transaction log, which records every data modification for recovery purposes.

Advantages

Minimal performance overhead
No changes required to the source schema
Highly accurate and reliable
No triggers or additional queries needed
Works with most transactional databases

Disadvantages

Some tools require a paid license
More complex to configure
Requires primary or unique keys
Logs must be retained if the target system is down

2. Trigger-Based CDC

How It Works

Database triggers are created on the source table to record inserts, updates, or deletes into a shadow table.

Advantages

Works on any database supporting triggers
Provides real-time capture
Shadow table can be customized

Disadvantages

Impacts source database performance
Adds processing overhead
More difficult to maintain, especially with schema changes

3. Timestamp-Based CDC

How It Works

Requires a LastModified or timestamp column.
Queries fetch only rows where the timestamp > last extraction time.

Advantages

Easy to implement
No triggers or log access required

Disadvantages

Deletes cannot be detected
Can miss changes if timestamps are not updated
Can slow performance due to full table scans

Conclusion on CDC Methods

Choosing the right CDC method depends on:

Real-time requirements
Database load tolerance
Schema complexity
Infrastructure capabilities
Operational constraints

Log-based CDC is generally the best option for performance-critical systems, while trigger-based CDC is useful when log access is restricted.

Log-Based CDC and Trigger-Based CDC Using SQL Server

1. Installation

SQL Server

Download and install SQL Server (Developer Edition recommended).

SQL Server Management Studio (SSMS)

Download SSMS and connect to the SQL Server instance.

Types of SQL Server Authentication

Windows Authentication

Uses the logged-in Windows user credentials.
Benefits:

No password management
More secure
Integrated with enterprise policies
Simplifies permission handling

SQL Server Authentication

Requires a username and password stored in SQL Server.

Choice: Windows Authentication was selected for its security and ease of use.

Implementation of Trigger-Based CDC

Create Source Table

CREATE TABLE person (
 person_id INT IDENTITY PRIMARY KEY,
 person_name VARCHAR(255)
);

Create History Table

CREATE TABLE person_history (
 hist_id INT PRIMARY KEY IDENTITY,
 person_id INT,
 person_name VARCHAR(255),
 status VARCHAR(255),
 UserName NVARCHAR(50),
 InsertedDateTime DATETIME
);
GO

Create Trigger

CREATE TRIGGER tr_AllOperationsTrigger
ON person
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
    -- DELETE
    IF EXISTS (SELECT * FROM deleted)
    INSERT INTO person_history (person_id, person_name, status, UserName, InsertedDateTime)
    SELECT person_id, person_name, 'deleted', SUSER_SNAME(), GETDATE()
    FROM deleted;

    -- INSERT
    IF EXISTS (SELECT * FROM inserted)
    INSERT INTO person_history (person_id, person_name, status, UserName, InsertedDateTime)
    SELECT person_id, person_name, 'inserted', SUSER_SNAME(), GETDATE()
    FROM inserted;

    -- UPDATE
    IF EXISTS (SELECT * FROM inserted) AND EXISTS (SELECT * FROM deleted)
    BEGIN
        INSERT INTO person_history
        SELECT person_id, person_name, 'updated old value', SUSER_SNAME(), GETDATE()
        FROM deleted;

        INSERT INTO person_history
        SELECT person_id, person_name, 'updated new value', SUSER_SNAME(), GETDATE()
        FROM inserted;
    END
END;
GO

Implementation of Log-Based CDC

Enable CDC for the Database

EXEC sys.sp_cdc_enable_db;
GO

Enable CDC for a Table

EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name = N'persontri',
    @role_name = NULL;
GO

Disable Cleanup Job (Optional)

Prevents cleanup from removing captured data.

EXEC sys.sp_cdc_drop_job @job_type = 'cleanup';

Verify CDC Status

SELECT is_cdc_enabled FROM sys.databases WHERE name = 'test';
SELECT * FROM cdc.change_tables;

View CDC Data

SELECT * FROM cdc.dbo_person_CT;

Operation Types in `__$operation`

Value	Meaning
1	Delete
2	Insert
3	Update (old values)
4	Update (new values)
7	Metadata operation

Adding User and Operation Details

Add User Column

ALTER TABLE cdc.dbo_person_CT
ADD userid2 nvarchar(128) DEFAULT USER_NAME();

Create Operation Table

CREATE TABLE operation (
    operation_id INT PRIMARY KEY,
    operation_name VARCHAR(255)
);

INSERT INTO operation VALUES
(1,'Delete'),(2,'Insert'),(3,'update(old)'),(4,'update(new)');

Convert LSN to Timestamp

SELECT
    *,
    CONVERT(VARCHAR(5), CONVERT(DATETIME, sys.fn_cdc_map_lsn_to_time([__start_lsn]), 108)) AS ModificationHourMinute,
    CONVERT(DATE, sys.fn_cdc_map_lsn_to_time([__start_lsn])) AS ModificationDate
FROM
    operation o
JOIN
    cdc.dbo_person_CT a ON o.operation_id = a.__$operation;

Comparison Between Log-based and Trigger-based CDC

Testing tools:

SQLQueryStress → workload generation
Performance Monitor → hardware-level performance metrics

Monitored Metrics

Disk Reads/Writes per second
Virtual memory page reads/writes
Transactions per second
Lock requests per second
Batch requests per second

Performance Summary

SQLQueryStress Results

Log-Based CDC
- ~0.0037 seconds/iteration
- ~5.672 logical reads
Trigger-Based CDC
- ~0.0017 seconds/iteration
- ~3.013 logical reads

→ Log-based CDC outperforms trigger-based CDC by 55 seconds over 100,000 iterations.

Performance Monitor Results

Log-Based CDC

Lower transactions/sec (623,995)
Lower disk writes/sec (316,574)
Much lower lock requests/sec (4,978)
Higher page writes/sec

Trigger-Based CDC

More batch requests/sec (833,331)
Very high lock requests/sec (2,548,997)
Higher disk writes/sec (370,824)

Interpretation:
Log-based CDC uses transaction logs efficiently without introducing overhead, while trigger-based CDC generates extra writes and locks, causing higher resource consumption.

Final Conclusion

Log-based CDC is the best choice for large-scale, high-performance systems.
It minimizes overhead, provides accurate change tracking, and integrates naturally with SQL Server transaction logs.
Trigger-based CDC is easier to implement and works on many databases but introduces overhead and complexity.
It is suitable when log access is restricted or custom auditing logic is required.

In performance tests, log-based CDC clearly outperforms trigger-based CDC, resulting in lower lock contention, fewer disk operations, and overall faster execution.

Change Data Capture(Example using Microsoft SQL Server Log-based CDC vs Trigger-based CDC

hammoudi wissem — Thu, 20 Nov 2025 15:16:41 +0000

Definition of Change Data Capture

Note: CDC is considered an ETL (Extract, Transform, Load) technique where only changed data is extracted from a source system and loaded into a target system such as a data warehouse or data lake.

Importance of Change Data Capture

1. Real-time Data Synchronization

2. Event-Driven Architectures

3. Efficient Data Processing

Since CDC transmits only modified data, it reduces the overhead of full database scans or batch processing. This speeds up pipelines and ensures downstream systems always receive fresh data.

4. Scalability and Flexibility

CDC supports asynchronous, decoupled communication, making it easier for distributed systems to scale horizontally while maintaining reliability.

5. Enhanced Analytics and Insights

Real-time data flow enables updated dashboards, instant anomaly detection, and more proactive decision-making.

Change Data Capture (CDC) Principles

Capture: Identify inserts, updates, and deletes without impacting the source system.
Log-based Tracking: Leverage transaction logs or replication logs to detect changes.
Incremental Updates: Transfer only changed rows instead of full datasets.
Real-time or Near Real-time: Propagate changes quickly to ensure data freshness.
Idempotent Processing: Avoid duplicate processing and maintain consistency.

Use Cases of Change Data Capture

Data Warehousing – Keep analytical systems updated with the latest transactional data.
Replication – Synchronize data across distributed systems for load balancing or disaster recovery.
Data Integration – Integrate modern and legacy systems or synchronize between cloud and on-premises systems.
Real-time Analytics – Feed streaming platforms or dashboards with immediate updates.
Data Synchronization – Maintain consistency across microservices, caching layers, or distributed components.

Real-World Example: CDC at Netflix

Netflix employs a CDC pipeline built on Apache Kafka and Apache Flink to extract changes from internal databases and process them in real-time. This architecture powers:

Service Usage Monitoring – Tracking user interactions to optimize performance.
Recommendation Engine – Providing personalized content suggestions.
Fraud Detection – Identifying abnormal activities instantly.

Change Data Capture Methods

Below are the most common CDC techniques, including how they work and their pros/cons.

1. Log-Based CDC

How It Works

Changes are captured directly from the database transaction log, which records every data modification for recovery purposes.

Advantages

Minimal performance overhead
No changes required to the source schema
Highly accurate and reliable
No triggers or additional queries needed
Works with most transactional databases

Disadvantages

Some tools require a paid license
More complex to configure
Requires primary or unique keys
Logs must be retained if the target system is down

2. Trigger-Based CDC

How It Works

Database triggers are created on the source table to record inserts, updates, or deletes into a shadow table.

Advantages

Works on any database supporting triggers
Provides real-time capture
Shadow table can be customized

Disadvantages

Impacts source database performance
Adds processing overhead
More difficult to maintain, especially with schema changes

3. Timestamp-Based CDC

How It Works

Requires a LastModified or timestamp column.
Queries fetch only rows where the timestamp > last extraction time.

Advantages

Easy to implement
No triggers or log access required

Disadvantages

Deletes cannot be detected
Can miss changes if timestamps are not updated
Can slow performance due to full table scans

Conclusion on CDC Methods

Choosing the right CDC method depends on:

Real-time requirements
Database load tolerance
Schema complexity
Infrastructure capabilities
Operational constraints

Log-based CDC is generally the best option for performance-critical systems, while trigger-based CDC is useful when log access is restricted.

Log-Based CDC and Trigger-Based CDC Using SQL Server

1. Installation

SQL Server

Download and install SQL Server (Developer Edition recommended).

SQL Server Management Studio (SSMS)

Download SSMS and connect to the SQL Server instance.

Types of SQL Server Authentication

Windows Authentication

Uses the logged-in Windows user credentials.
Benefits:

No password management
More secure
Integrated with enterprise policies
Simplifies permission handling

SQL Server Authentication

Requires a username and password stored in SQL Server.

Choice: Windows Authentication was selected for its security and ease of use.

Implementation of Trigger-Based CDC

Create Source Table

CREATE TABLE person (
 person_id INT IDENTITY PRIMARY KEY,
 person_name VARCHAR(255)
);

Create History Table

CREATE TABLE person_history (
 hist_id INT PRIMARY KEY IDENTITY,
 person_id INT,
 person_name VARCHAR(255),
 status VARCHAR(255),
 UserName NVARCHAR(50),
 InsertedDateTime DATETIME
);
GO

Create Trigger

CREATE TRIGGER tr_AllOperationsTrigger
ON person
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
    -- DELETE
    IF EXISTS (SELECT * FROM deleted)
    INSERT INTO person_history (person_id, person_name, status, UserName, InsertedDateTime)
    SELECT person_id, person_name, 'deleted', SUSER_SNAME(), GETDATE()
    FROM deleted;

    -- INSERT
    IF EXISTS (SELECT * FROM inserted)
    INSERT INTO person_history (person_id, person_name, status, UserName, InsertedDateTime)
    SELECT person_id, person_name, 'inserted', SUSER_SNAME(), GETDATE()
    FROM inserted;

    -- UPDATE
    IF EXISTS (SELECT * FROM inserted) AND EXISTS (SELECT * FROM deleted)
    BEGIN
        INSERT INTO person_history
        SELECT person_id, person_name, 'updated old value', SUSER_SNAME(), GETDATE()
        FROM deleted;

        INSERT INTO person_history
        SELECT person_id, person_name, 'updated new value', SUSER_SNAME(), GETDATE()
        FROM inserted;
    END
END;
GO

Implementation of Log-Based CDC

Enable CDC for the Database

EXEC sys.sp_cdc_enable_db;
GO

Enable CDC for a Table

EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name = N'persontri',
    @role_name = NULL;
GO

Disable Cleanup Job (Optional)

Prevents cleanup from removing captured data.

EXEC sys.sp_cdc_drop_job @job_type = 'cleanup';

Verify CDC Status

SELECT is_cdc_enabled FROM sys.databases WHERE name = 'test';
SELECT * FROM cdc.change_tables;

View CDC Data

SELECT * FROM cdc.dbo_person_CT;

Operation Types in `__$operation`

Value	Meaning
1	Delete
2	Insert
3	Update (old values)
4	Update (new values)
7	Metadata operation

Adding User and Operation Details

Add User Column

ALTER TABLE cdc.dbo_person_CT
ADD userid2 nvarchar(128) DEFAULT USER_NAME();

Create Operation Table

CREATE TABLE operation (
    operation_id INT PRIMARY KEY,
    operation_name VARCHAR(255)
);

INSERT INTO operation VALUES
(1,'Delete'),(2,'Insert'),(3,'update(old)'),(4,'update(new)');

Convert LSN to Timestamp

SELECT
    *,
    CONVERT(VARCHAR(5), CONVERT(DATETIME, sys.fn_cdc_map_lsn_to_time([__start_lsn]), 108)) AS ModificationHourMinute,
    CONVERT(DATE, sys.fn_cdc_map_lsn_to_time([__start_lsn])) AS ModificationDate
FROM
    operation o
JOIN
    cdc.dbo_person_CT a ON o.operation_id = a.__$operation;

Comparison Between Log-based and Trigger-based CDC

Testing tools:

SQLQueryStress → workload generation
Performance Monitor → hardware-level performance metrics

Monitored Metrics

Disk Reads/Writes per second
Virtual memory page reads/writes
Transactions per second
Lock requests per second
Batch requests per second

Performance Summary

SQLQueryStress Results

Log-Based CDC
- ~0.0037 seconds/iteration
- ~5.672 logical reads
Trigger-Based CDC
- ~0.0017 seconds/iteration
- ~3.013 logical reads

→ Log-based CDC outperforms trigger-based CDC by 55 seconds over 100,000 iterations.

Performance Monitor Results

Log-Based CDC

Lower transactions/sec (623,995)
Lower disk writes/sec (316,574)
Much lower lock requests/sec (4,978)
Higher page writes/sec

Trigger-Based CDC

More batch requests/sec (833,331)
Very high lock requests/sec (2,548,997)
Higher disk writes/sec (370,824)

Interpretation:
Log-based CDC uses transaction logs efficiently without introducing overhead, while trigger-based CDC generates extra writes and locks, causing higher resource consumption.

Final Conclusion

Log-based CDC is the best choice for large-scale, high-performance systems.
It minimizes overhead, provides accurate change tracking, and integrates naturally with SQL Server transaction logs.
Trigger-based CDC is easier to implement and works on many databases but introduces overhead and complexity.
It is suitable when log access is restricted or custom auditing logic is required.

In performance tests, log-based CDC clearly outperforms trigger-based CDC, resulting in lower lock contention, fewer disk operations, and overall faster execution.

Web Scraping with Python and BeautifulSoup: A Beginner-Friendly Guide

hammoudi wissem — Wed, 19 Nov 2025 08:45:00 +0000

A General Overview

When people discuss "web scrapers," they commonly refer to a process that involves:

Retrieving HTML data from a domain name
Parsing that data for target information
Storing the target information
Optionally, moving to another page to repeat the process

Introduction to BeautifulSoup

Because the BeautifulSoup library is not a default Python library, it must be installed.

!pip install beautifulsoup4

The most commonly used object in the BeautifulSoup library is, appropriately, the BeautifulSoup object. Let's take a look at it in action:

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('http://www.pythonscraping.com/pages/page1.html')
bs = BeautifulSoup(html.read(), 'html.parser')
print(bs.h1)

The output is as follows:

|<h1>Beautiful Soup Documentation<a class="headerlink" href="#module-bs4" title="Link to this heading">¶</a></h1>

Note that this returns only the first instance of the h1 tag found on the page. By con‐ vention, only one h1 tag should be used on a single page, but conventions are often broken on the web, so you should be aware that this will retrieve the first instance of the tag only, and not necessarily the one that you're looking for.

Another popular parser is lxml. This can be installed through pip:|

!pip install lxml

lxml has some advantages over html.parser in that it is generally better at parsing "messy" or malformed HTML code. It is forgiving and fixes problems like unclosed tags, tags that are improperly nested, and missing head or body tags. It is also some‐ what faster than html.parser

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('https://www.crummy.com/software/BeautifulSoup/bs4/doc/')
bs = BeautifulSoup(html, 'lxml')
print(bs.h1)

Connecting Reliably and Handling Exceptions

The web is messy. Data is poorly formatted, websites go down, and closing tags go missing. One of the most frustrating experiences in web scraping is to go to sleep with a scraper running, dreaming of all the data you'll have in your database the next day - only to find that the scraper hit an error on some unexpected data format and stopped execution shortly after you stopped looking at the screen.
Two main things can go wrong in this line:

The page is not found on the server (or there was an error in retrieving it)
The server is not found.

Of course, if the page is retrieved successfully from the server, there is still the issue of the content on the page not quite being what you expected. Every time you access a tag in a BeautifulSoup object, it's smart to add a check to make sure the tag actually exists.

from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
def getTitle(url):
    try:
        html = urlopen(url)
    except HTTPError as e:
        return None
    try:
        bs = BeautifulSoup(html.read(), 'html.parser')
        title = bs.body.h1
    except AttributeError as e:
        return None
    return title
title = getTitle('https://www.crummy.com/software/BeautifulSoup/bs4/doc/')
if title == None:
    print('Title could not be found')
else:
    print(title)

find() and find_all() with BeautifulSoup

BeautifulSoup's find() and find_all() are the two functions you will likely use the most. With them, you can easily filter HTML pages to find lists of desired tags, or a single tag, based on their various attributes. The two functions are extremely similar, as evidenced by their definitions in the BeautifulSoup documentation:

find_all(tag, attributes, recursive, text, limit, keywords)
find(tag, attributes, recursive, text, keywords)

These examples demonstrate how find_all can be used to locate elements based on different criteria such as tag names, attributes, and text content within an HTML or XML document parsed with BeautifulSoup.

bs.find_all(['h1','h2','h3','h4','h5','h6'])
bs.find_all('span', {'class':{'n'}})
nameList = bs.find_all(text='soup')
title = bs.find_all(id='searchlabel')

The find_all method searches for all occurrences of tags that match the given criteria. On the other hand, The find method is used to search for the first occurrence of a tag that matches the given criteria.

Navigating Trees

The find_all function is responsible for finding tags based on their name and attributes. But what if you need to find a tag based on its location in a document?

**Dealing with children and other descendants

For example, if you need to retrieve the contents of a table by accessing only its immediate children, you can use the .children method:

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('https://www.crummy.com/software/BeautifulSoup/bs4/doc/')
bs = BeautifulSoup(html, 'html.parser')
for child in bs.find('table').children:
    print(child)

**Dealing with siblings

Additionally, to extract data from a table beyond its immediate children, BeautifulSoup offers the next_siblings() function. This function simplifies the process by allowing us to retrieve siblings of a

tag.

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('https://www.crummy.com/software/BeautifulSoup/bs4/doc/')
bs = BeautifulSoup(html, 'html.parser')
for child in bs.find('table').tr.next_siblings:
    print(child)

Dealing with parents

hen you're scraping web pages, you often focus more on finding children or siblings of tags rather than their parents. Typically, when you begin crawling an HTML page, you start by examining the top-level tags and then work your way down to locate specific pieces of data. However, there are times when you might encounter unusual scenarios that necessitate using BeautifulSoup's parent-finding functions like .parent and .parents.

we can also get the table by find the tag td then finding his parent and printing all his siblings

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen('https://www.crummy.com/software/BeautifulSoup/bs4/doc/')
bs = BeautifulSoup(html, 'html.parser')

element = bs.find('td')  
if element:
    parent_element = element.parent
    for sibling in parent_element.next_siblings:
        print(sibling)

Regular Expressions

One classic example of regular expressions can be found in the practice of identifying email addresses. Although the exact rules governing email addresses vary slightly from mail server to mail server, we can create a few general rules.
[A-Za-z0–9._+]+@[A-Za-z]+.(com|org|edu|net)

When scraping web pages using BeautifulSoup, the synergy with regular expressions becomes invaluable, especially for tasks like extracting specific elements such as event images from complex HTML structures. For instance, consider a scenario where you need to retrieve URLs of event images from a page.

To address this, leveraging regular expressions allows you to pinpoint images by specific attributes like their src attribute, which contains the file path. This approach ensures you accurately target product images regardless of their position or surrounding elements on the page.

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

html = urlopen('http://enetcomje.com/gallerie.php')
bs = BeautifulSoup(html, 'html.parser')

# Adjust the regular expression pattern to match 'event' images
images = bs.find_all('img', {'src': re.compile('event\d+\.png')})

for image in images:
    print(image['src'])

Accessing Attributes
When web scraping, you often need to access attributes of HTML tags rather than their content. This is especially useful for tags like , where the URL is in the href attribute, or , where the image source (src) is crucial. BeautifulSoup simplifies this process with its attrs attribute, which returns a Python dictionary containing all attributes of a tag.

print(bs.img.attrs['src'])

Writing Web Crawlers

So far, you've seen single static pages . In this part, you'll start looking at real-world problems, with scrapers traversing multiple pages and even multiple sites.

Web crawlers are called such because they crawl across the web. At their core is an element of recursion. They must retrieve page contents for a URL, examine that page for another URL, and retrieve that page, ad infinitum.

this is a code that retives an arbitrary wikepideia pages and produces a list of link of that pages :

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen('http://en.wikipedia.org/wiki/Kevin_Bacon')
bs = BeautifulSoup(html, 'html.parser')
for link in bs.find_all('a'):
    if 'href' in link.attrs:
        print(link.attrs['href'])

If you look at the list of links produced, you'll notice that all the articles you'd expect are there: "Apollo 13," "Philadelphia," "Primetime Emmy Award," and so on. However, there are some things that you don't want as well: //wikimediafoundation.org/wiki/Privacy_policy //en.wikipedia.org/wiki/Wikipedia:Contact_us

If you examine the links that point to article pages (as opposed to other internal pages), you'll see that they all have three things in common:

They reside within the div with the id set to bodyContent.
The URLs do not contain colons.
The URLs begin with /wiki/.

so we can improve the last code to recivee only the desired article by using the regular expression ^(/wiki/)((?!:).)*$")

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
html = urlopen('http://en.wikipedia.org/wiki/Kevin_Bacon')
bs = BeautifulSoup(html, 'html.parser')
for link in bs.find('div', {'id':'bodyContent'}).find_all(
    'a', href=re.compile('^(/wiki/)((?!:).)*$')):
    if 'href' in link.attrs:
        print(link.attrs['href'])

Crawling an Entire Site

Web scrapers that traverse an entire site are useful for several purposes:

Generating a Site Map: Using a crawler, you can scan an entire site, gather all internal links, and organize the pages into their actual folder structure. This could help you discover hidden sections and accurately count the number of pages.
Gathering Data: If you need to collect articles (such as stories, blog posts, news articles, etc.) to create a prototype for a specialized search platform, you would need data from only a few sites but want a broad collection. Therefore, you will build crawlers that traverse each site and collect data specifically from article pages.

The general approach to an exhaustive site crawl is to start with a top-level page (such as the home page), and search for a list of all internal links on that page. Every one of those links is then crawled, and additional lists of links are found on each one of them, triggering another round of crawling.

this a code that could give me all the link available in wikipidia :

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
pages = set()
def getLinks(pageUrl):
    global pages
    html = urlopen('http://en.wikipedia.org{}'.format(pageUrl))
    bs = BeautifulSoup(html, 'html.parser')
    for link in bs.find_all('a', href=re.compile('^(/wiki/)')):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href']
                print(newPage)
                pages.add(newPage)
                getLinks(newPage)
getLinks('')

Collecting Data Across an Entire Site

Web crawlers would be fairly boring if all they did was hop from one page to the other. To make them useful, you need to be able to do something on the page while you're there. Let's look at how to build a scraper that collects the title, the first para‐ graph of content, and the link to edit the page (if available).

As always, the first step to determine how best to do this is to look at a few pages from the site and determine a pattern. By looking at a handful of Wikipedia pages (both articles and nonarticle pages such as the privacy policy page), the following things should be clear:

All titles (on all pages, regardless of their status as an article page, an edit history page, or any other page) have titles under h1 → span tags, and these are the only h1 tags on the page.
As mentioned before, all body text lives under the div#bodyContent tag. How‐ ever, if you want to get more specific and access just the first paragraph of text, you might be better off using div#mw-content-text → p (selecting the first para‐ graph tag only). This is true for all content pages except file pages (for example, https://en.wikipedia.org/wiki/File:Orbit_of_274301_Wikipedia.svg), which do not have sections of content text.
Edit links occur only on article pages. If they occur, they will be found in the li#ca-edit tag, under li#ca-edit → span → a.

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
pages = set()
def getLinks(pageUrl):
    global pages
    html = urlopen('http://en.wikipedia.org{}'.format(pageUrl))
    bs = BeautifulSoup(html, 'html.parser')
    try:
        print(bs.h1.get_text())
        print(bs.find(id ='mw-content-text').find_all('p')[0])
        print(bs.find(id='ca-edit').find('span')
         .find('a').attrs['href'])
    except AttributeError:
        print('This page is missing something! Continuing.')

    for link in bs.find_all('a', href=re.compile('^(/wiki/)')):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
            #We have encountered a new page
                newPage = link.attrs['href']
                print('-'*20)
                print(newPage)
                pages.add(newPage)
                getLinks(newPage)
getLinks('')

Note: Handling Redirects

Redirects allow a web server to point one domain name or URL to a piece of content at a different location. There are two types of redirects:

Server-side redirects, where the URL is changed before the page is loaded
Client-side redirects, sometimes seen with a "You will be redirected in 10 sec‐ onds" type of message, where the page loads before redirecting to the new one

With server-side redirects, you usually don't have to worry. If you're using the urllib library with Python 3.x, it handles redirects automatically! If you're using the requests library, make sure to set the allow-redirects flag to True:

r = requests.get('http://github.com', allow_redirects=True)

Just be aware that, occasionally, the URL of the page you're crawling might not be exactly the URL that you entered the page on.

Crawling Across the Internet

Before you start writing a crawler that follows all outbound links, you should ask yourself a few questions:

What data am I trying to gather? Can this be accomplished by scraping just a few predefined websites (almost always the easier option), or does my crawler need to be able to discover new websites I might not know about?
When my crawler reaches a particular website, will it immediately follow the next outbound link to a new website, or will it stick around for a while and drill down into the current website?
Are there any conditions under which I would not want to scrape a particular site? Am I interested in non-English content?
How am I protecting myself against legal action if my web crawler catches the attention of a webmaster on one of the sites it runs across?

import time
import random
from urllib.request import urlopen
from urllib.parse import urlparse, urljoin
from bs4 import BeautifulSoup
import re

pages = set()

# Seed random generator with current time
random.seed(time.time())

def getInternalLinks(bs, includeUrl):
    """ Retrieves internal links from a BeautifulSoup object """
    includeUrl = '{}://{}'.format(urlparse(includeUrl).scheme,
                                  urlparse(includeUrl).netloc)
    internalLinks = []
    for link in bs.find_all('a', href=re.compile('^(/|.*'+includeUrl+')')):
        if 'href' in link.attrs:
            if link.attrs['href'] is not None:
                url = urljoin(includeUrl, link.attrs['href'])
                if url not in internalLinks:
                    internalLinks.append(url)
    return internalLinks

def getExternalLinks(bs, excludeUrl):
    """ Retrieves external links from a BeautifulSoup object """
    externalLinks = []
    for link in bs.find_all('a', href=re.compile('^(http|www)((?!'+excludeUrl+').)*$')):
        if 'href' in link.attrs:
            if link.attrs['href'] is not None:
                url = link.attrs['href']
                if url not in externalLinks:
                    externalLinks.append(url)
    return externalLinks

def getRandomExternalLink(startingPage):
    """ Retrieves a random external link from the starting page """
    html = urlopen(startingPage)
    bs = BeautifulSoup(html, 'html.parser')
    externalLinks = getExternalLinks(bs, urlparse(startingPage).netloc)
    if len(externalLinks) == 0:
        print('No external links found, looking for internal links')
        domain = '{}://{}'.format(urlparse(startingPage).scheme, urlparse(startingPage).netloc)
        internalLinks = getInternalLinks(bs, domain)
        if internalLinks:
            return getRandomExternalLink(random.choice(internalLinks))
        else:
            return None
    else:
        return random.choice(externalLinks)

def followExternalOnly(startingSite):
    """ Follows external links recursively from the starting site """
    if startingSite in pages:
        return
    pages.add(startingSite)
    externalLink = getRandomExternalLink(startingSite)
    if externalLink:
        print('Random external link is: {}'.format(externalLink))
        followExternalOnly(externalLink)
    else:
        print('No external links found on {}'.format(startingSite))

# Start following external links from a specified starting site
followExternalOnly('http://oreilly.com')

Web Crawling Models

Planning and Defining Objects

One common trap of web scraping is defining the data that you want to collect based entirely on what's available in front of your eyes.

If you want to collect product data, you may first look at a clothing store and decide that each product you scrape needs to have the

following fields:

Product name
Price
Description
Sizes
Colors

Looking at another website, you find that it has SKUs (stock keeping units, used to track and order items) listed on the page. You definitely want to collect that data as well, even if it doesn't appear on the first site! You add this field:

Item SKU

Clearly, this is an unsustainable approach. Simply adding attributes to your product type every time you see a new piece of information on a website will lead to far too many fields to keep track of. Not only that, but every time you scrape a new website, you'll be forced to perform a detailed analysis of the fields the website has and the fields you've accumulated so far, and potentially add new fields (modifying your Python object type and your database structure). This will result in a messy and difficult-to-read dataset that may lead to problems using it.

One of the best things you can do when deciding which data to collect is often to ignore the websites altogether. You don't start a project that's designed to be large and scalable by looking at a single website and saying, "What exists?" but by saying "What do I need?" and then finding ways to seek the information that you need from there.

It's important to take a step back and perform a checklist for each item you consider and ask yourself the following questions:

Will this information help with the project goals? Will it be a roadblock if I don't have it, or is it just "nice to have" but won't ultimately impact anything?
If it might help in the future, but I'm unsure, how difficult will it be to go back and collect the data at a later time?
Is this data redundant to data I've already collected?
Does it make logical sense to store the data within this particular object? (Storing a description in a product doesn't make sense if that description changes from site to site for the same product.)

If you do decide that you need to collect the data, it's important to ask a few more questions to then decide how to store and handle it in code:

Is this data sparse or dense? Will it be relevant and populated in every listing, or just a handful out of the set?
How large is the data?
Especially in the case of large data, will I need to regularly retrieve it every time I run my analysis, or only on occasion?
How variable is this type of data? Will I regularly need to add new attributes, modify types, or is it set in stone (shoe sizes)?

Dealing with Di„erent Website Layouts

The most obvious approach is to write a separate web crawler or page parser for each website. Each might take in a URL, string, or BeautifulSoup object, and return a Python object for the thing that was scraped.

The following is an example of a Content class (representing a piece of content on a website, such as a news article) and two scraper functions that take in a Beauti fulSoup object and return an instance of Content:

import requests
from bs4 import BeautifulSoup
class Content:
    def __init__(self, url, title, body):
        self.url = url
        self.title = title
        self.body = body
def getPage(url):
    req = requests.get(url)
    return BeautifulSoup(req.text, 'html.parser')
def scrapetheguardian(url):
    bs = getPage(url)
    title_tag = bs.find("h1", {"class": "dcr-u0152o"})
    title = title_tag.text.strip() if title_tag else "Title not found."

    article_body_div = bs.find("div", {"class": "article-body-commercial-selector article-body-viewer-selector dcr-fp1ya"})
    if article_body_div:
        paragraphs = article_body_div.find_all("p")
        body = '\n'.join([p.text.strip() for p in paragraphs])
    else:
        body = "Body content not found."
    return Content(url, title, body)
def scrapeBrookings(url):
    bs = getPage(url)
    title_tag = bs.find("h1")
    title = title_tag.text.strip() if title_tag else "Title not found."
    body_tag = bs.find("section", {"id": "content"})
    if body_tag:
        paragraphs = body_tag.find_all("p")
        body = '\n'.join([p.text.strip() for p in paragraphs])
    else:
        body = "Body content not found."
    return Content(url, title, body)
# Example usage
url_brookings = 'https://www.brookings.edu/articles/delivering-inclusive-urban-access-3-uncomfortable-truths/'
content_brookings = scrapeBrookings(url_brookings)
print('Title: {}'.format(content_brookings.title))
print('URL: {}\n'.format(content_brookings.url))
print(content_brookings.body)
url_theguardian = 'https://www.theguardian.com/us-news/article/2024/jul/01/trump-hush-money-supreme-court-immunity'
content_theguardian = scrapetheguardian(url_theguardian)
print('\nTitle: {}'.format(content_theguardian.title))
print('URL: {}\n'.format(content_theguardian.url))
print(content_theguardian.body)

Example: Extracting Resume Using Selnuim and beautifulSoup

The Data Analytics Lifecycle

hammoudi wissem — Sun, 16 Nov 2025 20:31:10 +0000

The data analytics lifecycle is a series of steps to transform raw data into valuable and easily consumable data products. These can range from well-managed datasets to dashboards, reports, APIs, or even web applications. In other words, it describes how data is created, collected, processed, used, and analyzed to achieve a specific product or business goal.

The increasing complexity in organizational dynamics directly impacts how data is handled. Numerous people must use the same data but with different goals. While a top executive might need to know just a few top-level key performance indicators to track business performance, a middle manager might need a more granular report to support daily decisions.

This highlights the need for a governed and standardized approach to creating and maintaining data products based on the same data foundation. Given the many decisions an organization must make regarding its data governance, technologies, and management processes, following a structured approach is fundamental to documenting and continuously updating an organization's data strategy.

The data analytics lifecycle is, therefore, an essential framework for understanding and mapping the phases and processes involved in creating and maintaining an analytics solution . It is an essential concept in data science and analytics and provides a structured approach to managing the various tasks and activities required to create an effective analytics solution.

The data analytics lifecycle typically includes the following stages:

- Problem definition :
The first phase of the analytics cycle is about understanding the problem that needs to be solved. This includes identifying the business objectives, the available data, and the resources needed to solve the problem.

- Data modeling :
After the business requirements are identified, and an assessment of data sources is completed, you can begin modeling your data according to the modeling technique that best meets your needs. You can choose a diamond strategy, a star schema, a Data Vault, or even a fully denormalized technique.

- Data ingestion and transformation :
The next phase is to ingest and prepare the data that's coming from the source systems to match the models created. Depending on the overall information architecture, you can opt for a schema-on-write strategy, where you put more effort into transforming the raw data directly into your models, or a schema-on read strategy, where you ingest and store the data with minimal transformations and move heavy transformations to the downstream layers of your data platform.

- Data storage and structuring :
Once the data pipelines are designed and potentially implemented, you need to decide on the file formats to use - simple Apache Parquet or more advanced formats like Delta Lake or Apache Iceberg - as well as the partitioning strategies and storage components to use - a cloud-based object store like Amazon Simple Storage Service (S3) or a more data warehouse–like platform like Redshift, Big‐Query, or Snowflake.

- Data visualization and analysis:
Once the data is available, the next step is to explore it, visualize it, or create dashboards that directly support decision making or enable business process monitoring. This phase is very business oriented and should be created in close coordination with business stakeholders.

*- Data quality monitoring, testing, and documentation : *
Although illustrated as the final phase of the analytics lifecycle, data quality should be an end-to-end concern and ensured by design across the whole flow. It involves implementing all quality controls to ensure that stakeholders can trust your exposed data models, documenting all transformations and semantic meanings, and ensuring proper testing along the pipelines as the data continues to flow.

The analytics lifecycle is a key concept that enables organizations to approach data engineering, science, and analytics processes in a structured and consistent manner. By following a structured process, organizations can ensure they are solving the right problem, using the right data, and building data products that are accurate and reliable, ultimately leading to better decision making and better business results.

DEV Community: hammoudi wissem

The Power of Context in Data Visualization (Part 1)

Exploratory vs. Explanatory Analysis: Two Different Worlds

Exploratory Analysis – The Search

Explanatory Analysis – The Story

Who, What, and How: The Core of Context

1. Who: Understand Your Audience

2. What: Inspire Meaningful Action

3. How: Choosing the Right Format and Tone

Mechanism – The Format

Tone – The Emotion

How to Build Your Story with Data

Crafting Clarity: The 3-Minute Story and the Big Idea

The 3-Minute Story

The Big Idea

Example

Storyboarding: Your Blueprint for Clear Communication

Why Avoid Starting with Slides?

Example Storyboard: Summer Learning Program

Why Storyboarding Matters

Start Smart, Not Fast

Storytelling with Data: The Art of Communicating Insights Effectively (Introduction)

Bad Graphs Are Everywhere

We’re Not Naturally Equipped for Data Storytelling

The Journey of Storytelling with Data

Part 1: The Importance of Context

Part 2: Choosing the Right Visual

Part 3: Clutter Is Your Enemy

Part 4: Focus Your Audience’s Attention

Part 5: Think Like a Designer

Part 6: Dissecting Model Visuals

Part 7: Lessons in Storytelling

Part 8: Pulling It All Together

The Journey Begins

Change Data Capture(Example using Microsoft SQL Server Log-based CDC vs Trigger-based CDC

Definition of Change Data Capture

Importance of Change Data Capture

1. Real-time Data Synchronization

2. Event-Driven Architectures

3. Efficient Data Processing

4. Scalability and Flexibility

5. Enhanced Analytics and Insights

Change Data Capture (CDC) Principles

Use Cases of Change Data Capture

Real-World Example: CDC at Netflix

Change Data Capture Methods

1. Log-Based CDC

How It Works

Advantages

Disadvantages

2. Trigger-Based CDC

How It Works

Advantages

Disadvantages

3. Timestamp-Based CDC

How It Works

Advantages

Disadvantages

Conclusion on CDC Methods

Log-Based CDC and Trigger-Based CDC Using SQL Server

1. Installation

SQL Server

SQL Server Management Studio (SSMS)

Types of SQL Server Authentication

Windows Authentication

SQL Server Authentication

Implementation of Trigger-Based CDC

Create Source Table

Create History Table

Create Trigger

Implementation of Log-Based CDC

Enable CDC for the Database

Enable CDC for a Table

Disable Cleanup Job (Optional)

Verify CDC Status

View CDC Data

Operation Types in __$operation

Adding User and Operation Details

Add User Column

Create Operation Table

Operation Types in `__$operation`

Operation Types in `__$operation`