DEV Community: Serge Artishev

Supercharging Databricks Asset Bundles

Serge Artishev — Fri, 16 May 2025 02:20:13 +0000

Multi-Environment Workflows with CI/CD

In my previous post, I introduced the Databricks Asset Bundle (DAB) template project that helps you get started quickly with a structured approach to Databricks development. Today, I want to dive deeper into how DAB handles variables, parameterization, and CI/CD automation across multiple environments.

The Power of Parameterization

One of the most powerful features of Databricks Asset Bundles is the ability to parameterize nearly everything using variables. This allows us to define workflows once and deploy them to multiple environments with different configurations.

Variable Structure in DAB

The template organizes variables in a clear, hierarchical structure:

variables/
├── common.yml                # Variables shared across all environments
├── {workflowName}.dev.yml    # Worklow specific variables for development
├── {workflowName}.test.yml   # Worklow specific variables for testing
├── {workflowName}.prod.yml   # Worklow specific variables for production

This organization gives us several benefits:

Clear separation of concerns (common vs specific)
Environment-specific configurations (.dev, .test, .prod)
Logical grouping of related variables per workflow

Including Variables in Your Project

The main databricks.yml file includes these variable files based on the target environment:

include:
  - resources/**/*.yml
  - variables/common.yml
  - variables/*.dev.yml  # This changes based on environment

NOTE:
Unfortunately, the Databricks CLI does not support ${bundle.target} placeholder variable files yet. This is a bit of a pain, but it's a known issue and I trust this will be fixed in a future release. So for now, we need to manually update the databricks.yml file to include the correct variables/*.{environment}.yml variable file for each environment. To do that, we can use the yq command to update the databricks.yml file when running the CI/CD pipeline.

When deploying to different environments, we simply swap out which environment-specific variable files to include. For example, in our CI/CD pipeline for test deployment:

# Update include path for test environment
yq -i 'with(.include[] | select(. == "variables/*.dev.yml"); . = "variables/*.test.yml")' databricks.yml

Real-World Example: Parameterizing SharePoint Workflows

Let's look at a real example from the template. The SharePoint Excel refresh workflow connects to SharePoint, processes Excel files, and loads the data to Delta tables. Here's how we parameterize it:

Define environment-specific variables:

# In variables/sharepoint.dev.yml
variables:
  sharepoint:
    type: complex
    default:
      secret_scope: "azure"
      tenant_id_key: "azure-tenant-id"
      client_id_key: "azure-app-client-id"
      client_secret_key: "azure-app-client-secret"
      site_id_key: "sharepoint-site-id"
      drive_id_key: "sharepoint-drive-id"
      modified_in_last_hours: 240
      target_catalog: "bronze"
      target_schema: "sharepoint_dev"
      sync_schedule: "0 0 0 * * ?"
      concurrency: 10

Take note of the target_schema variable. We can use this to deploy the same workflow to different environments with different schema names. Same goes for the target_catalog variable and any other variables that are used in the workflow definition.

Reference variables in workflow definition:

# In resources/sharepoint/sharepoint_excel_refresh.yml
resources:
  jobs:
    sharepoint_excel_refresh:
      name: "${bundle.name} Sharepoint Excel Refresh"
      tasks:
        - task_key: sharepoint_excel_file_list
          notebook_task:
            notebook_path: "${workspace.file_path}/notebooks/sharepoint/excel_list_process"
            base_parameters:
              secret_scope: "${var.sharepoint.secret_scope}"
              tenant_id_key: "${var.sharepoint.tenant_id_key}"
              # More parameters...
              modified_in_last_hours: "${var.sharepoint.modified_in_last_hours}"
      schedule:
        quartz_cron_expression: "${var.sharepoint.sync_schedule}"
        timezone_id: "${var.timezone_id}"

By using this approach, we can deploy the same workflow to different environments with environment-specific configurations. For instance, in production we might have different:

Target schema names (sharepoint_prod vs sharepoint_dev)
Sync schedules (hourly in production, daily in dev)
Lookback periods (24 hours in production, 240 hours in dev)

The complex type allows us to define a complex object with multiple properties, which is perfect for our SharePoint example that allows us to define all the workflow specific variables in one place.

Automated CI/CD Pipeline

The real magic happens when we automate deployment across environments. The template includes GitHub Actions workflows that:

Validate on PRs and feature branches:
- Run unit tests
- Validate DAB bundle configuration
- Check code quality
Auto-deploy to test environment:
- Triggered on pushes to the develop branch
- Updates variable includes for test environment
- Authenticates with service principal
- Deploys the DAB bundle
Deploy to production:
- Triggered on pushes to the main branch
- Updates variable includes for production environment
- Adds approval steps for production deployment
- Deploys with production-specific settings

Authentication with Service Principals

A key part of the CI/CD automation is using service principals for authentication. In the GitHub workflows, we:

Obtain an OAuth token using the service principal credentials
Use that token for Databricks CLI authentication
Pass the service principal ID as a variable during deployment

# Get OAuth token for service principal
response=$(curl -s -X POST \
  -u "${{ secrets.SERVICE_PRINCIPAL_APP_ID }}:${{ secrets.SERVICE_PRINCIPAL_SECRET }}" \
  "$DATABRICKS_HOST/oidc/v1/token" \
  -d "grant_type=client_credentials&scope=all-apis")

# Extract token and set environment variables
token=$(echo $response | jq -r '.access_token')
export DATABRICKS_TOKEN="$token"

# Deploy with service principal ID variable
databricks bundle deploy --target test --var service_principal_id=${{ secrets.SERVICE_PRINCIPAL_APP_ID }}

Advanced Techniques

Here are some advanced techniques you can use with this setup:

1. Dynamic Configuration Based on Branch

You can make your CI/CD pipeline smarter by adjusting configuration based on the Git branch:

- name: Set environment variables based on branch
  run: |
    if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
      echo "TARGET_ENV=prod" >> $GITHUB_ENV
    elif [[ "${{ github.ref }}" == "refs/heads/develop" ]]; then
      echo "TARGET_ENV=test" >> $GITHUB_ENV
    else
      echo "TARGET_ENV=dev" >> $GITHUB_ENV
    fi

- name: Update variable includes
  run: |
    yq -i 'with(.include[] | select(. == "variables/*.dev.yml"); . = "variables/*.${{ env.TARGET_ENV }}.yml")' databricks.yml

2. Feature Flags via Variables

You can implement simple feature flags using variables:

variables:
  features:
    enable_advanced_analytics: true
    enable_real_time_processing: false

Then in your workflows:

tasks:
  - task_key: optional_analytics_step
    notebook_task:
      notebook_path: "/path/to/analytics"
    if: ${var.features.enable_advanced_analytics}

3. Template Workflows with Parameters

You can create reusable workflow templates by parameterizing common patterns:

# resources/templates/ingest_template.yml
resources:
  jobs:
    ${var.job_name}:  # Dynamic job name
      name: "Ingest ${var.source_name} Data"
      tasks:
        - task_key: ingest_data
          notebook_task:
            notebook_path: "/Shared/ingest/${var.source_type}"
            base_parameters:
              source_config: ${var.source_config}
              target_table: ${var.target_table}

Conclusion

By combining DAB's parameterization capabilities with automated CI/CD pipelines, you can create a robust, maintainable system for deploying Databricks resources across environments. This approach gives you:

Clear separation of configuration from implementation
Environment-specific settings without code duplication
Automated testing and deployment
Consistent deployment process across environments
Version-controlled infrastructure and configuration

What's your experience with Databricks Asset Bundles? Have you found other useful patterns for managing multi-environment deployments? Let me know in the comments!

Databricks Asset Bundle: A Template to Make Life Easier

Serge Artishev — Wed, 16 Apr 2025 02:01:58 +0000

If you're working with data pipelines on Databricks, you know how time-consuming it can be starting from scratch with each new project. That's why I've put together this Databricks Asset Bundle (DAB) project that packages up proven patterns and practices. Let me show you how it can transform your workflow.

The DAB Story - What Even Is It?

Before I dive into the template itself, let me tell you a bit about Databricks Asset Bundles. DAB is relatively new to the Databricks ecosystem - it was first announced in 2022 and became generally available in 2023. It's Databricks' answer to the age-old question: "How do we manage all our Databricks stuff in a sane way?"

Before DAB, we were stuck with a hodgepodge of approaches:

Manually exporting and importing notebooks (ugh)
Using the Databricks CLI to deploy workspaces (better, but still clunky)
Writing custom scripts to manage assets (time-consuming and fragile)

DAB brings everything together with a simple YAML-based approach. It lets you package up notebooks, workflows, ML models, and more into a single deployable unit. It's a bit like Terraform but specifically built for Databricks resources.

Why I Built This Project

After setting up multiple data projects, I realised I was solving the same problems over and over:

Creating a sensible project structure
Setting up common code patterns
Figuring out how to manage environments
Establishing best practices for deployment and development

Sound familiar? Yeah, I thought so. That's why I built this project - to save both of us from that groundhog day feeling.

What's in the Box?

1. Organized Structure for Your Data Projects

This project includes a clear organization for development:

databricks-dab-project/
├── databricks.yml           # Main DAB configuration file
├── resources/               # DAB resources (workflows, etc.)
│   └── sample_workflow.yml  # Sample workflow definition
├── src/                     # Source code
│   ├── common/              # Reusable Python package shared across notebooks
│   │   ├── __init__.py
│   │   ├── utils/           # Utility modules
│   │   ├── transforms/      # Data transformation modules
│   │   └── ingestion/       # Data ingestion modules
│   └── setup.py             # Package setup file
├── notebooks/               # Databricks notebooks
│   ├── bronze/              # Bronze layer notebooks
│   ├── silver/              # Silver layer notebooks
│   └── gold/                # Gold layer notebooks
└── docs/                    # Documentation

The structure is designed to scale with your project needs and keep code organized as you expand.

2. Python Package Architecture

One of my biggest pet peeves with Databricks projects is seeing massive, unwieldy notebooks with thousands of lines of code. They're impossible to test, hard to reuse, and a nightmare to maintain.

That's why this project includes a proper Python package structure in src/common/. Here's why:

Testability: You can actually unit test functions in a package
Code Reuse: No more copy-pasting between notebooks. Import what you need from the package
Version Control: Packages have proper versioning, making it easier to track changes
IDE Support: Write code in a proper IDE with linting and autocomplete

The notebooks become thin wrappers around the package functions - they handle parameters and orchestration, but the heavy lifting happens in the package.

3. Infrastructure as Code with DAB

The project includes:

Environment configs for dev/prod environments in databricks.yml
Workflow definitions with proper configurations
CI/CD setup for validation with GitHub Actions
Clear separation of code, config, and infrastructure

I've made sure the DAB configs are parameterised properly, so you can deploy to different environments without changing the actual files - just change the variables.

4. Built-in Best Practices

This project includes common patterns I've found helpful:

Proper logging pattern in the Python package
Consistent project structure
GitHub Actions workflow for validation
Code organization that promotes reusability

5. Fast Onboarding Process

One of the biggest wins with this project is how quickly new team members can get up to speed:

Comprehensive documentation in the docs/ folder with a detailed DAB development guide
Clear setup instructions in the README
Consistent structure that follows logical patterns
Easy-to-understand configuration files

Step-by-Step Guide to Getting Started

Here's a detailed walkthrough of how to start using this template for your own project:

1. Create Your Own Repository from the Template

To create your own repository from the template, follow these steps:

Go to this GitHub repository
Click the green "Use this template" button
Choose "Create a new repository"
Name your repository and set visibility (public/private)
Click "Create repository from template"

2. Set Up Your Development Environment

Now that you have your own repository, let's set up your local environment:

Install the Databricks CLI:

   curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

   # Verify installation
   databricks version

You should see version 0.244.0 or higher. If you see version 0.18.x, you've installed the old CLI - follow the migration guide.

Configure the CLI:

   # Set up a profile for development
   databricks auth login --host https://your-dev-workspace.cloud.databricks.com

   # Optionally, set up a profile for production too
   databricks auth login --host https://your-prod-workspace.cloud.databricks.com

Set Up Python Environment:

   # Create a virtual environment
   python -m venv .venv

   # Activate it (Windows)
   .venv\Scripts\activate

   # Activate it (macOS/Linux) 
   source .venv/bin/activate

   # Install the common package in development mode
   cd src
   pip install -e .

(Optional) Set Up Pre-commit Hooks:

If you want to set up pre-commit hooks to check your code before committing, run the following commands:

   pip install pre-commit
   pre-commit install

The pre-commit hooks will check your code for linting errors and secrets before committing.

3. Configure Your Project

Update the databricks.yml file:

   bundle:
     name: your-project-name  # Change this

   # ... rest of the file

   targets:
     dev:
       workspace:
         host: your-workspace.databricks.com  # Change this to your workspace URL

Add Environment Variables for Authentication:

   # For local development
   export DATABRICKS_HOST=your-workspace.cloud.databricks.com

   # Alternative: Use profiles you set up in step 2 by uncommenting in databricks.yml:
   # profile: dev

4. Validate and Deploy Your Project

Validate the configuration:

   databricks bundle validate

Deploy to development:

   databricks bundle deploy --target dev

Extending the Common Package

The src/common package is designed to be extended with your own modules. Here's how to add new functionality:

1. Add a New Module

Let's say you want to add data quality checks, and you want to add a new module called data_quality.

# Create a new module
mkdir -p src/common/data_quality

# Create necessary files
touch src/common/data_quality/__init__.py
touch src/common/data_quality/validators.py

2. Implement Your Functions

In src/common/data_quality/validators.py:

from pyspark.sql import DataFrame
from typing import List, Dict, Any, Optional

def validate_not_null(df: DataFrame, columns: List[str]) -> Dict[str, Any]:
    """
    Validates that specified columns don't contain nulls.

    Args:
        df: DataFrame to validate
        columns: List of column names to check

    Returns:
        Dictionary with validation results
    """
    results = {}

    for column in columns:
        if column not in df.columns:
            results[column] = {"valid": False, "error": f"Column {column} not found"}
            continue

        null_count = df.filter(df[column].isNull()).count()
        results[column] = {
            "valid": null_count == 0,
            "null_count": null_count,
            "total_count": df.count()
        }

    return results

3. Import in Your Notebook

Now you can use this in any notebook:

from common.data_quality.validators import validate_not_null

# Load your data
df = spark.table("bronze.my_table")

# Validate required columns
validation_results = validate_not_null(df, ["id", "name", "timestamp"])

# Take action based on results
for column, result in validation_results.items():
    if not result["valid"]:
        print(f"Validation failed for {column}: {result}")

That's it! You've added a new module to your common package. You can now keep all your common code in one place and reuse it across your notebooks.

My Closing Thoughts

Starting data engineering projects from scratch is a waste of time. We're all solving similar problems, so why not start from a solid foundation?

This project embodies patterns I've found useful in Databricks development. It's got the structure in place, but you can customize it as needed for your specific requirements.

I'd love to hear how you use this project and what improvements you make. The whole point of sharing this is to learn from each other and keep making our data work better.

Catch you in the data lake!

Got questions or improvements? I'd love to hear from you! Drop a comment below or reach out on GitHub.

Azure SQL to Storage Account Export Tool

Serge Artishev — Sun, 20 Oct 2024 23:01:19 +0000

In the world of cloud-based data management, efficiently moving data in and out of Azure SQL databases is a common challenge. Whether you're performing backups, migrating data, or setting up test environments, having reliable tools for exporting and importing data is crucial. Today, I'm excited to share a set of bash scripts I've developed to streamline this process: the Azure SQL Data Export and Import Tools.

The Problem

Working with Azure SQL databases often requires exporting large amounts of data for backup or analysis and then importing that data back into the same or different databases. While Azure provides various methods for these tasks, they can sometimes be cumbersome or require multiple steps. I needed a solution that was:

Scriptable and easy to automate
Flexible enough to handle different table structures
Capable of compressing data for efficient storage and transfer
Able to generate DDL scripts for table recreation

The Solution

To address these needs, I created two main scripts:

export_to_tsv.sh: Exports data from Azure SQL tables to TSV (Tab-Separated Values) files, optionally compresses them, and uploads them to Azure Blob Storage.
import_from_tsv.sh: Downloads TSV files from Azure Blob Storage, decompresses them if necessary, and imports the data into Azure SQL tables.

Let's dive into the key features and how to use these tools.

Export Tool (`export_to_tsv.sh`)

This script exports data from specified Azure SQL tables to TSV files, with options to compress the files and generate DDL scripts for table recreation.

Key Features:

Exports multiple tables in a single run
Compresses TSV files using gzip (optional)
Generates DDL scripts for table recreation (optional)
Uploads exported files to Azure Blob Storage
Supports overwriting existing blobs in Azure Storage (optional)
Provides detailed logging and error reporting

Usage:

./export_to_tsv.sh --schema <schema_name> --tables <table1,table2,...> --container <container-name> --path <blob-path> [--delimiter <delimiter>] [--compress] [--generate-ddl] [--overwrite] [--dry-run]

Example:

./export_to_tsv.sh --schema "archive" --tables "logs-2023-08-25" --container "sqlbackup" --path "archive/logs" --compress --generate-ddl --overwrite

This command exports the logs-2023-08-25 table from the archive schema, compresses it, generates a DDL script, and uploads both files to the specified Azure Blob Storage location.

Import Tool (`import_from_tsv.sh`)

This script downloads TSV files from Azure Blob Storage, decompresses them if necessary, and imports the data into Azure SQL tables.

Key Features:

Imports multiple tables in a single run
Handles compressed (gzip) files
Provides detailed logging and error reporting
Supports dry-run mode for testing

Usage:

./import_from_tsv.sh --schema <schema_name> --tables <table1,table2,...> --container <container-name> --path <blob-path> [--delimiter <delimiter>] [--compressed] [--dry-run]

Example:

./import_from_tsv.sh --schema archive --tables "logs-2023-08-25" --container "sqlbackup" --path "archive/logs" --compressed

This command imports the logs-2023-08-25 table into the refine schema, downloading and decompressing the TSV file from the specified Azure Blob Storage location.

Behind the Scenes

Both scripts use a combination of Azure CLI commands, sqlcmd, and bcp utilities to interact with Azure services and perform data operations. They also implement robust error handling and logging to ensure smooth operation and easy troubleshooting.

Here's a snippet from the export script that demonstrates how we're using bcp to export data:

bcp "SELECT * FROM [$SCHEMA].[$table]" queryout "$tsv_file" -c -t "$DELIMITER" \
    -S "$SOURCE_AZURE_SQL_SERVER" -d "$SOURCE_AZURE_SQL_DATABASE" \
    -U "$SOURCE_AZURE_SQL_USERNAME" -P "$SOURCE_AZURE_SQL_PASSWORD"

And here's how we're using sqlcmd to generate DDL scripts:

sqlcmd -S "$SOURCE_AZURE_SQL_SERVER" -d "$SOURCE_AZURE_SQL_DATABASE" -U "$SOURCE_AZURE_SQL_USERNAME" -P "$SOURCE_AZURE_SQL_PASSWORD" -Q "
SET NOCOUNT ON;
DECLARE @TableName NVARCHAR(128) = '$table';
DECLARE @SchemaName NVARCHAR(128) = '$SCHEMA';
DECLARE @SQL NVARCHAR(MAX) = '';

-- Generate CREATE TABLE statement
SELECT @SQL = 'CREATE TABLE [' + @SchemaName + '].[' + @TableName + '] ('

-- ... (rest of the DDL generation logic)

PRINT @SQL
" -o "$ddl_file"

Conclusion

These Azure SQL Data Export and Import Tools have significantly streamlined our data management processes. They provide a flexible, scriptable solution for moving data in and out of Azure SQL databases, with features like compression and DDL generation that save time and storage space.

While these scripts are tailored to our specific needs, they can serve as a starting point for others facing similar challenges. Feel free to adapt and extend them to fit your own requirements.

You can find the full source code and detailed documentation in my GitHub repository

I hope you find these tools useful in your Azure SQL data management tasks. Happy data wrangling!

Building a DOCX to Markdown Converter with Node.js

Serge Artishev — Sun, 30 Jun 2024 00:42:56 +0000

Welcome to a step-by-step guide on building a powerful DOCX to Markdown converter using Node.js. This project is a great way to learn about file manipulation, command-line interfaces, and converting document formats. By the end of this series, you'll have a tool that not only converts DOCX files to Markdown but also extracts images and formats tables. Let's dive in!

Introduction
Setting Up the Project
Basic DOCX to HTML Conversion
Converting HTML to Markdown
Extracting Images
Formatting Tables
Conclusion

Introduction

Markdown is a lightweight markup language with plain text formatting syntax. It's widely used for documentation due to its simplicity and readability. However, many documents are created in DOCX format, especially in corporate environments. Converting these documents to Markdown can be tedious if done manually. This is where our converter comes in handy.

Setting Up the Project

First, let's create a new directory for our project and initialize it with npm.

mkdir docx-to-md-converter
cd docx-to-md-converter
npm init -y

Next, we'll install the necessary dependencies. We'll use mammoth for converting DOCX to HTML, turndown for converting HTML to Markdown, commander for building the CLI, and uuid for unique image names.

npm install mammoth turndown commander uuid

Create a new file named index.js in your project directory. This will be the main file for our converter.

touch index.js

Basic DOCX to HTML Conversion

Let's start by writing a simple script to convert DOCX files to HTML. We'll use the mammoth library for this.

Open index.js and add the following code:

#!/usr/bin/env node

import * as fs from 'fs';
import * as path from 'path';
import * as mammoth from 'mammoth';
import { program } from 'commander';

program
  .version('1.0.0')
  .description('Convert DOCX to HTML')
  .argument('<input>', 'Input DOCX file')
  .argument('[output]', 'Output HTML file (default: same as input with .html extension)')
  .action(async (input, output) => {
    try {
      await convertDocxToHtml(input, output);
    } catch (error) {
      console.error('Error:', error);
      process.exit(1);
    }
  });

program.parse(process.argv);

async function convertDocxToHtml(inputFile, outputFile) {
  if (!outputFile) {
    outputFile = path.join(path.dirname(inputFile), `${path.basename(inputFile, '.docx')}.html`);
  }

  const result = await mammoth.convertToHtml({ path: inputFile });
  await fs.promises.writeFile(outputFile, result.value);
  console.log(`Conversion complete. Output saved to ${outputFile}`);
}

This script uses commander to parse command-line arguments, mammoth to convert DOCX to HTML, and fs to write the output to a file. To make this script executable, add the following line at the top of index.js:

#!/usr/bin/env node

Make sure the script has execute permissions:

chmod +x index.js

Now you can run the script to convert a DOCX file to HTML:

node index.js example.docx example.html

Converting HTML to Markdown

Next, we'll add the functionality to convert HTML to Markdown using turndown.

First, install turndown:

npm install turndown

Update index.js to include the HTML to Markdown conversion:

#!/usr/bin/env node

import * as fs from 'fs';
import * as path from 'path';
import * as mammoth from 'mammoth';
import TurndownService from 'turndown';
import { program } from 'commander';

program
  .version('1.0.0')
  .description('Convert DOCX to Markdown')
  .argument('<input>', 'Input DOCX file')
  .argument('[output]', 'Output Markdown file (default: same as input with .md extension)')
  .action(async (input, output) => {
    try {
      await convertDocxToMarkdown(input, output);
    } catch (error) {
      console.error('Error:', error);
      process.exit(1);
    }
  });

program.parse(process.argv);

async function convertDocxToMarkdown(inputFile, outputFile) {
  if (!outputFile) {
    outputFile = path.join(path.dirname(inputFile), `${path.basename(inputFile, '.docx')}.md`);
  }

  const result = await mammoth.convertToHtml({ path: inputFile });
  const turndownService = new TurndownService();
  const markdown = turndownService.turndown(result.value);

  await fs.promises.writeFile(outputFile, markdown);
  console.log(`Conversion complete. Output saved to ${outputFile}`);
}

Now you can convert DOCX files to Markdown:

node index.js example.docx example.md

Extracting Images

DOCX files often contain images that we need to handle. We'll extract these images and save them to a folder, updating the image links in the Markdown file.

Update index.js to include image extraction:

#!/usr/bin/env node

import * as fs from 'fs';
import * as path from 'path';
import * as mammoth from 'mammoth';
import TurndownService from 'turndown';
import { program } from 'commander';
import { v4 as uuidv4 } from 'uuid';

program
  .version('1.0.0')
  .description('Convert DOCX to Markdown with image extraction')
  .argument('<input>', 'Input DOCX file')
  .argument('[output]', 'Output Markdown file (default: same as input with .md extension)')
  .action(async (input, output) => {
    try {
      await convertDocxToMarkdown(input, output);
    } catch (error) {
      console.error('Error:', error);
      process.exit(1);
    }
  });

program.parse(process.argv);

async function convertDocxToMarkdown(inputFile, outputFile) {
  if (!outputFile) {
    outputFile = path.join(path.dirname(inputFile), `${path.basename(inputFile, '.docx')}.md`);
  }

  const imageDir = path.join(path.dirname(outputFile), 'images');
  if (!fs.existsSync(imageDir)) {
    fs.mkdirSync(imageDir, { recursive: true });
  }

  const result = await mammoth.convertToHtml({ path: inputFile }, {
    convertImage: mammoth.images.imgElement(async (image) => {
      const buffer = await image.read();
      const extension = image.contentType.split('/')[1];
      const imageName = `image-${uuidv4()}.${extension}`;
      const imagePath = path.join(imageDir, imageName);
      await fs.promises.writeFile(imagePath, buffer);
      return { src: `images/${imageName}` };
    })
  });

  const turndownService = new TurndownService();
  const markdown = turndownService.turndown(result.value);

  await fs.promises.writeFile(outputFile, markdown);
  console.log(`Conversion complete. Output saved to ${outputFile}`);
}

Now, images will be extracted and saved in an images folder, and the Markdown file will contain the correct links to these images.

Formatting Tables

The final feature we'll add is table formatting. DOCX files often contain tables that need to be correctly formatted in Markdown.

Update index.js to include table formatting:

#!/usr/bin/env node

import * as fs from 'fs';
import * as path from 'path';
import * as mammoth from 'mammoth';
import TurndownService from 'turndown';
import { program } from 'commander';
import { v4 as uuidv4 } from 'uuid';

program
  .version('1.0.0')
  .description('Convert DOCX to Markdown with image extraction and table formatting')
  .argument('<input>', 'Input DOCX file')
  .argument('[output]', 'Output Markdown file (default: same as input with .md extension)')
  .action(async (input, output) => {
    try {
      await convertDocxToMarkdown(input, output);
    } catch (error) {
      console.error('Error:', error);
      process.exit(1);
    }
  });

program.parse(process.argv);

function createMarkdownTable(table) {
  const rows = Array.from(table.rows);
  if (rows.length === 0) return '';

  const headers = Array.from(rows[0].cells).map(cell => cell.textContent?.trim() || '');
  const markdownRows = rows.slice(1).map(row => 
    Array.from(row.cells).map(cell => cell.textContent?.trim() || '')
  );

  let markdown = '| ' + headers.join(' | ') + ' |\n';
  markdown += '| ' + headers.map(() => '---').join(' | ') + ' |\n';
  markdownRows.forEach(row => {
    markdown += '| ' + row.join(' | ') + ' |\n

';
  });

  return markdown;
}

async function convertDocxToMarkdown(inputFile, outputFile) {
  if (!outputFile) {
    outputFile = path.join(path.dirname(inputFile), `${path.basename(inputFile, '.docx')}.md`);
  }

  const imageDir = path.join(path.dirname(outputFile), 'images');
  if (!fs.existsSync(imageDir)) {
    fs.mkdirSync(imageDir, { recursive: true });
  }

  const result = await mammoth.convertToHtml({ path: inputFile }, {
    convertImage: mammoth.images.imgElement(async (image) => {
      const buffer = await image.read();
      const extension = image.contentType.split('/')[1];
      const imageName = `image-${uuidv4()}.${extension}`;
      const imagePath = path.join(imageDir, imageName);
      await fs.promises.writeFile(imagePath, buffer);
      return { src: `images/${imageName}` };
    })
  });

  let html = result.value;

  const turndownService = new TurndownService();

  turndownService.addRule('table', {
    filter: 'table',
    replacement: function(content, node) {
      return '\n\n' + createMarkdownTable(node) + '\n\n';
    }
  });

  const markdown = turndownService.turndown(html);

  await fs.promises.writeFile(outputFile, markdown);
  console.log(`Conversion complete. Output saved to ${outputFile}`);
}

Conclusion

In this blog post, we built a DOCX to Markdown converter step by step, adding features like image extraction and table formatting. This tool demonstrates the power and flexibility of Node.js for handling file manipulations and conversions.

The source code for this project is available on GitHub, where you can find the latest updates, contribute to the project, and explore further enhancements.

Thank you for following along with this guide. Happy coding!

Securing Azure Functions with OAuth2 Authentication

Serge Artishev — Sat, 23 Mar 2024 08:15:24 +0000

In this blog post, we will explore how to create an Azure Function with an HTTP trigger and OAuth2 authentication using Azure Active Directory (AAD). We will also create a sample client app that authenticates and executes the function. This solution will help you understand how to secure your Azure Functions using OAuth2 authentication.

1. Creating and Deploying an Azure Function using DevContainer in VSCode

Before we dive into the details of OAuth2 authentication, let's first create and deploy a simple Azure Function using a DevContainer in Visual Studio Code (VSCode).

Introduction

Azure Functions are a great way to build serverless applications that can scale on demand. In this section, we will create an Azure Function with an HTTP trigger and deploy it using a DevContainer in Visual Studio Code (VSCode).

Prerequisites

Before we get started, make sure you have the following prerequisites installed on your machine:

Dev Container: Azure Functions (Node.js)

Installation

To create and run the Azure Function, follow these steps:

Clone this GitHub repository.
Open the repository in Visual Studio Code.
Reopen the repository in the Dev Container.
Run the following command to install the dependencies.



npm install

Start the Azure Function by running the following command.



npm start

Open your browser and navigate to http://localhost:7071/api/ping. You should see a standard "Hello, World!" message.
Create the Azure Function resource in Azure by right-clicking on the Function App group in the Azure resources view and selecting Create Function App in Azure.... Follow the instructions to create the Function in your Azure subscription.
After successful creation, you can deploy the Azure Function to Azure by clicking on the Azure tab in your VSCode and selecting the function logo in the Workspace view. Then click on Deploy to Function App... and follow the instructions to deploy the Function in your Azure subscription.
After successful deployment, you can access the Azure Function in the Azure portal.

Make sure to test the Azure Function by navigating to the URL provided in the Azure portal. You should see the same "Hello, World!" message as before.

2. Configuring the Azure Function to use Azure Active Directory (AAD) OAuth2

Now that we have created and deployed the Azure Function, let's configure it to use Azure Active Directory (AAD) OAuth2 for authentication.

Introduction to AAD OAuth2

Azure Active Directory (AAD) OAuth2 is a secure way to authenticate users and authorize access to your applications. In this section, we will configure the Azure Function to use AAD OAuth2 for authentication.

Configuring AAD OAuth2 for the Azure Function

To configure AAD OAuth2 authentication for the Azure Function, follow these steps:

Open the Azure portal.
Navigate to the Azure Function App that you have created.
In the Settings section, click on Authentication and then click on Add identity provider.
In the Choose a tenant for your application and its users section, select Workforce configuration (current tenant).
In the App registration section, you can either create a new app registration or select an existing app registration. If you want to create a new app registration, you can specify the app Name, and select the Supported account types as Current tenant - Single tenant.
In the App Service authentication settings section, make sure that the Restrict access is set to Require authentication.
In the Unauthenticated requests section, select HTTP 401 Unauthorized: recommended for APIs.
Click Next and then click Add.

Here's a screenshot of the AAD OAuth2 configuration in the Azure portal:

Testing AAD OAuth2 Authentication

After configuring AAD OAuth2 authentication, you can test the Azure Function by navigating to the URL provided in the Azure portal. As far as we selected to return 401 for unauthenticated requests, you should see an HTTP 401 Unauthorized response when accessing the Function URL without authentication.

Now, to authenticate and access the Azure Function, follow these steps:

Open up a browser, I’d recommend in incognito/in-private mode. We now need to build up a specific URL to call MS Identity and authenticate. All of the below should be on one line, but has been broken over multiple lines so it is easier to read.



https://login.microsoftonline.com/<Tenant ID>/oauth2/v2.0/authorize
    ?client_id=<Client ID>
    &response_type=code
    &redirect_uri=https%3A%2F%2F<Function Name>.azurewebsites.net%2F.auth%2Flogin%2Faad%2Fcallback
    &response_mode=fragment
    &scope=openid%20offline_access%20https%3A%2F%2Fgraph.microsoft.com%2Fuser.read
    &state=12345

Once you have the URL, paste it into the browser and hit enter. You will be redirected to the Microsoft login page. Enter your credentials and sign in. You will then be redirected to the Azure Function.

After successful authentication, you will be able to execute the Azure Function.

3. Creating a Sample Client App that Authenticates and Executes the Function

Now that we have configured the Azure Function to use AAD OAuth2, let's create a sample client app that authenticates and executes the function.

Designing the Client App

The client app will be a simple Node.js application that authenticates with AAD OAuth2 and calls the Azure Function.

Implementing Authentication

To implement authentication in the client app, follow these steps:



const axios = require('axios');

const clientId = 'YOUR_CLIENT_ID';
const clientSecret = 'YOUR_CLIENT_SECRET';
const tenantId = 'YOUR_TENANT_ID';
const functionUrl = 'YOUR_FUNCTION_URL';

const getToken = async () => {
  const tokenEndpoint = `https://login.microsoftonline.com/${tenantId}/oauth2/v2.0/token`;

  const response = await axios.post(tokenEndpoint, {
    grant_type: 'client_credentials',
    client_id: clientId,
    client_secret: clientSecret,
    scope: `api://${clientId}/.default`,
  });

  return response.data.access_token;
};

const callFunction = async () => {
  const token = await getToken();

  const response = await axios.get(functionUrl, {
    headers: {
      Authorization: `Bearer ${token}`,
    },
  });

  console.log(response.data);
};

callFunction();

Replace the placeholders YOUR_CLIENT_ID, YOUR_CLIENT_SECRET, YOUR_TENANT_ID, YOUR_RESOURCE_NAME, and YOUR_FUNCTION_URL with your actual values.

In this code snippet, we are using the axios library to make HTTP requests to authenticate with AAD OAuth2 and call the Azure Function.

Conclusion

In this blog post, we explored how to create an Azure Function with OAuth2 authentication using Azure Active Directory (AAD). We also created a sample client app that authenticates and executes the function. This solution will help you secure your Azure Functions and build secure serverless applications. For more details, you can refer to the GitHub repository

Designing RESTful APIs with OpenAPI: A Top-Down Approach

Serge Artishev — Mon, 29 Jan 2024 10:53:15 +0000

APIs are the backbone of modern software development. Whether you're building web applications, mobile apps, or microservices, having well-defined APIs is crucial for seamless integration and collaboration between different components of your system. In this blog post, we'll explore the process of designing RESTful APIs using OpenAPI, following a top-down approach.

Section 1: Getting Started with OpenAPI

What is OpenAPI?

OpenAPI is an open-source specification for designing, documenting, and defining RESTful APIs. It provides a standardized way to describe your API's endpoints, request/response schemas, authentication methods, and more. OpenAPI specifications are written in either YAML or JSON, making them human-readable and machine-understandable.

Why Use OpenAPI?

Before we dive into the details, let's understand why OpenAPI is a valuable tool in API design:

Consistency: OpenAPI ensures that your API follows a consistent structure and adheres to best practices.
Documentation: Your API specification serves as living documentation, making it easier for developers to understand and use your API.
Validation: OpenAPI allows you to define request and response schemas with data types and validation rules.
Code Generation: You can generate server-side and client-side code in various programming languages from your OpenAPI specification.

Section 2: Defining API Endpoints

In OpenAPI, defining API endpoints is at the core of designing your RESTful API. Let's explore how to define paths, HTTP methods, and operation parameters.

API Paths and Operations

An API path represents a resource or endpoint in your API. It's the URL that clients use to interact with your API. For example:

paths:
  /api/v1/users:
    get:
      summary: Get a list of users
      description: Retrieve a list of user profiles.
      responses:
        '200':
          description: Successful response.

In the example above, /api/v1/users is the API path, and GET is the HTTP method. This path allows clients to retrieve a list of user profiles.

Path Parameters

You can also define path parameters to make your API paths more dynamic. Path parameters are placeholders in the URL that allow clients to specify values dynamically. For example:

paths:
  /api/v1/users/{userId}:
    get:
      summary: Get user by ID
      description: Retrieve a user profile by their unique ID.
      parameters:
        - name: userId
          in: path
          required: true
          description: The unique ID of the user.
          schema:
            type: integer
      responses:
        '200':
          description: Successful response.

In this example, {userId} is a path parameter, and clients can provide an actual user ID in the URL to retrieve a specific user profile.

Section 3: Defining Request and Response Schemas

One of the significant advantages of using OpenAPI is the ability to define request and response schemas with precision. This ensures that data exchanged between clients and your API is well-structured and follows specific rules. Let's explore how to define these schemas.

Request Body Schemas

When clients send data to your API, it's essential to specify the structure and validation rules for the request body. Here's an example of defining a request body schema:

paths:
  /api/v1/users:
    post:
      summary: Create a new user
      description: Create a new user profile.
      requestBody:
        description: User data to be created.
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/UserRequest'
      responses:
        '201':
          description: User created successfully.

In this example, we're defining a request body schema using the $ref keyword, which refers to a component schema called UserRequest. The UserRequest schema can include properties, data types, and validation rules for creating a user.

Response Schemas

Similarly, you can define response schemas to specify the structure of data returned by your API. Response schemas help clients understand the format of the data they can expect. Here's an example:

paths:
  /api/v1/users/{userId}:
    get:
      summary: Get user by ID
      description: Retrieve a user profile by their unique ID.
      parameters:
        - name: userId
          in: path
          required: true
          description: The unique ID of the user.
          schema:
            type: integer
      responses:
        '200':
          description: Successful response.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/UserResponse'

In this case, the UserResponse schema defines the structure of the response data when retrieving a user by their ID.

Component Schemas

To keep your OpenAPI specification organized and reusable, it's a good practice to define component schemas for request and response objects. Here's how you can define a User schema as a component:

components:
  schemas:
    UserRequest:
      type: object
      properties:
        username:
          type: string
        email:
          type: string
          format: email
        password:
          type: string
          minLength: 8
      required:
        - username
        - email
        - password

    UserResponse:
      type: object
      properties:
        id:
          type: integer
        username:
          type: string
        email:
          type: string
          format: email
      required:
        - id
        - username
        - email

By defining these schemas as components, you can reuse them across multiple endpoints in your API.

Section 4: Validations and Enums

OpenAPI allows you to enforce data validation rules and use enums to restrict possible values for certain properties. This ensures that the data exchanged via your API is consistent and error-free.

Property Validations

You can specify various validation rules for properties in your schemas, such as minimum/maximum lengths, pattern matching, and more. Here's an example:

components:
  schemas:
    UserRequest:
      type: object
      properties:
        username:
          type: string
          minLength: 3
          maxLength: 20
        email:
          type: string
          format: email
        password:
          type: string
          minLength: 8

In this example, the username must be between 3 and 20 characters long, the email must match the email format, and the password must be at least 8 characters long.

Using Enums

Enums are useful for properties with a predefined set of allowed values. For example, if you have an API property like status with only a few possible values, you can use enums to restrict those values:

components:
  schemas:
    Task:
      type: object
      properties:
        title:
          type: string
        status:
          type: string
          enum:
            - todo
            - in_progress
            - done

In this case, the status property can only have values of "todo," "in_progress," or "done."

Section 5: Generating Server-Side Code (Node.js)

One of the most powerful features of OpenAPI is the ability to generate server-side code automatically based on your API specification. This can significantly speed up the development process and reduce the chances of human error. Let's see how to generate server-side code for a Node.js application.

Step 1: Install OpenAPI Generator

To generate server-side code, you'll need the OpenAPI Generator CLI. You can install it globally using npm:

npm install @openapitools/openapi-generator-cli -g

Step 2: Generate Server Code

Once you've installed the OpenAPI Generator CLI, you can use it to generate server code. Here's a command to generate a Node.js server using Express as the framework:

npx openapi-generator-cli generate -i your-api-spec.yaml -g nodejs-express-server -o server

Replace your-api-spec.yaml with the path to your OpenAPI specification file. This command generates a Node.js server in a folder called server.

Step 3: Implement Business Logic

Generated code provides the infrastructure for your API, but you'll need to implement the actual business logic. In the generated code, you'll find controllers, routes, and models that you can customize to fit your application's requirements.

Step 4: Start the Server

Once you've implemented your business logic, you can start the server:

cd server
npm install
npm start

Your Node.js server is now running and ready to accept API requests.

Section 6: Generating Client-Side Code (TypeScript)

In addition to server-side code, OpenAPI can generate client-side code to interact with your API. This makes it easier to consume your API from various platforms and programming languages. Let's see how to generate a TypeScript client.

Step 1: Generate TypeScript Client

To generate a TypeScript client, you can use the OpenAPI Generator CLI again. Here's a command to generate a TypeScript client:

npx openapi-generator-cli generate -i your-api-spec.yaml -g typescript-axios -o client

Replace your-api-spec.yaml with the path to your OpenAPI specification file. This command generates a TypeScript client in a folder called client.

Step 2: Use the TypeScript Client

Now that you have a TypeScript client, you can use it in your front-end application. Import the generated client functions and use them to make API requests. Here's an example:

import { DefaultApi } from './client';

const api = new DefaultApi();

api.getUserById(123)
  .then((response) => {
    console.log(response.data);
  })
  .catch((error) => {
    console.error(error);
  });

In this example, we import the DefaultApi class generated by OpenAPI and use it to make a GET request to retrieve a user by ID.

With server-side and client-side code generated, you can focus on building your application's features without worrying about low-level API integration details.

Conclusion

In this blog post, we've explored the top-down approach to API design using OpenAPI. We began by defining our API's structure, endpoints, request/response schemas, and validation rules in an OpenAPI specification. Then, we demonstrated how to generate server-side code for a Node.js application and client-side code in TypeScript.

By following this approach, you can streamline your API development process, ensure consistency in data exchange, and simplify client integration. OpenAPI and code generation tools are powerful allies for modern API development, enabling you to build robust and scalable applications more efficiently.

In future articles, we'll dive deeper into advanced topics, such as authentication, versioning, and documentation. Stay tuned for more insights into API development best practices!

Understanding Human Biases in Sales: A Guide for Businesses and Consumers

Serge Artishev — Tue, 26 Dec 2023 19:27:29 +0000

Introduction

In the intricate dance of commerce, understanding human psychology is not just an advantage; it's a necessity. This blog post delves into various human biases that play a crucial role in sales strategies. Both businesses aiming to refine their sales techniques and consumers seeking to make informed decisions can benefit from this insight.

Cognitive Biases in Sales

1. Anchoring Bias: First impressions matter. In sales, initial pricing sets the stage for customer expectations. Businesses use this to their advantage by setting a high initial price, making subsequent offers seem more reasonable.

2. Confirmation Bias: We love being right. Sales strategies cater to this by aligning products with the pre-existing beliefs of the target audience, reinforcing their notion of making the 'right' choice.

3. Dunning-Kruger Effect: Overconfidence sells. Complex products are often marketed as effortlessly usable, appealing to the overconfidence in consumers.

Cognitive Bias Sample: Anchoring Bias

Electronics Store Pricing Strategy

Scenario: An electronics store displays a high-end television at $3,000 as the first item at the entrance. Subsequently, other models priced at $1,500 are showcased. Customers perceive these models as significantly more affordable due to the initial high price anchor, increasing the likelihood of purchase.

Social Biases and Their Impact

1. Bandwagon Effect: There's comfort in numbers. Highlighting the popularity of a product can sway consumers to jump on the bandwagon.

2. Authority Bias: Stars sell. Celebrities or experts endorsing products leverage this bias, influencing consumer decisions through perceived authority.

3. In-group Bias: Exclusivity creates desire. By marketing products as symbols of belonging to a special group, companies tap into our need for social identity.

Social Bias Sample: Bandwagon Effect

Social Media Marketing Campaign

Scenario: A fashion brand launches a social media campaign showing popular influencers wearing their new line of clothing. Seeing these influencers, along with large numbers of people engaging with the posts, consumers are more inclined to buy the clothing, driven by the desire to be part of a popular trend.

Emotional Biases in Marketing

1. Loss Aversion: Fear of missing out is real. Limited-time offers or exclusive deals create a sense of urgency, making consumers more likely to buy to avoid potential loss.

2. Optimism Bias: Positive sells. Focusing on the beneficial outcomes of a product can be more convincing, as people tend to believe they are less likely to experience negative events.

Emotional Bias Sample: Loss Aversion

Limited Time Offer in Online Retail

Scenario: An online retailer sends an email to its customers announcing a 24-hour flash sale on select products. The fear of missing out on a good deal prompts customers to make purchases they might not have considered otherwise, motivated by the desire to avoid the loss of a bargain.

The Role of Memory in Sales

1. Recency Effect: The last one wins. Repetitive advertising ensures that a product is fresh in consumers' minds, influencing their final decision.

2. Rosy Retrospection: Nostalgia is powerful. Connecting new products with positive past experiences can make them more appealing.

Memory Bias Sample: Recency Effect

Frequent Advertising Before a Major Sale Event

Scenario: In the weeks leading up to a major holiday sale event, a department store increases its advertising frequency on various media platforms. The repeated exposure ensures that the store and its sale are fresh in the minds of consumers, influencing them to choose this store for their holiday shopping.

Choice-supportive Biases in Post-Purchase

1. Post-purchase Rationalization: Justifying decisions. Marketing efforts don't stop at purchase; they also focus on reassuring customers of the value of their choice.

2. Choice-supportive Bias: Reinforcing satisfaction. Follow-up marketing efforts aim to remind customers of the positives of their choice, enhancing customer loyalty.

Choice-supportive Bias Sample: Post-purchase Rationalization

After-Sale Service and Follow-ups

Scenario: A car dealership sends follow-up emails to customers who recently purchased vehicles, highlighting the car's advanced features, superior performance, and the benefits of their choice. This post-purchase communication helps reinforce the customers' belief that they made a good decision, enhancing brand loyalty.

Decision-making Biases and Choice

1. Paradox of Choice: Simplicity is key. Offering fewer, well-curated choices can help in avoiding customer decision paralysis.

2. Default Effect: Steering choices. Setting desirable products as default options can guide customers towards more profitable selections for the business.

Decision-making Bias Sample: Paradox of Choice

Simplified Menu in a Restaurant

Scenario: A restaurant reduces its menu options from 50 dishes to a curated list of 15 signature dishes. This simplification helps customers make quicker, less overwhelming decisions, leading to a more satisfying dining experience and potentially faster table turnover.

Conclusion

In the world of sales, understanding and leveraging human biases can create powerful strategies for businesses. However, it's equally crucial for consumers to be aware of these tactics to make choices that truly serve their interests. By bridging the gap between psychology and commerce, we can foster a marketplace that is both savvy and ethical.

This post aims to shed light on the subtle psychological factors at play in our everyday purchasing decisions. Stay informed and make choices that are best for you!

Simplify Data Cleansing with YAML Configurations

Serge Artishev — Sat, 07 Oct 2023 08:57:52 +0000

Data forms the backbone of today's business landscape. However, raw data often comes with inconsistencies and errors that can distort analytics and impede accurate decision-making. Data cleansing is the process of rectifying these discrepancies to ensure data accuracy and reliability. This article delves into a streamlined approach to data cleansing, focusing on YAML configurations.

Why is Data Cleansing Essential?

Data Accuracy: For informed decision-making, accurate data is non-negotiable. Cleansing eradicates inaccuracies and prevents misleading conclusions.
Consistency: Uniform data is more manageable. Data cleansing ensures data adheres to a standard format, making processing smoother.
Efficiency: With cleansed data, processing becomes more efficient, reducing manual corrections and enhancing processing speed.
Superior Insights: Trustworthy data paves the way for more profound insights, empowering businesses to confidently lean into data-driven decisions.

Implementing Data Cleansing Using YAML Configurations

Establish Data Cleansing Rules: Determine the specific cleansing rules required for your dataset. Common guidelines might encompass trimming whitespace, removing special characters, and formatting dates.
Document Rules in YAML: YAML (Yet Another Markup Language) is exceptionally suited for cataloging data cleansing rules due to its readability and ease of maintenance. Below is a representative YAML configuration:

cleansingRules:
  - columns:
      - 'Customer Name'
      - 'Address'
    rules:
      - name: 'trim_whitespace'
      - name: 'remove_special_characters'

This configuration directs the system to trim whitespace and discard special characters from the 'Customer Name' and 'Address' fields.

Develop a Data Cleansing Module: Design a module in your preferred programming language that can interpret the YAML configuration, apply the designated rules, and produce cleansed data.

Parsing and Applying Rules from YAML

Your module should possess the capability to read the YAML configuration, allowing it to understand and execute the rules dynamically. In JavaScript, one can employ the yaml library to seamlessly integrate YAML content. By sourcing rules from YAML, updating cleansing protocols becomes hassle-free, with no need to modify the core code.

Here's a concise illustration in JavaScript of how the rules can be applied:

import * as yaml from 'yaml';

class DataCleansingHandler {
    constructor(yamlConfig) {
        this.cleansingRules = yaml.parse(yamlConfig).cleansingRules;
    }

    applyCleansingRules(data) {
        return data.map((record) => {
            const cleansedRecord = { ...record };

            this.cleansingRules.forEach((rule) => {
                rule.columns.forEach((column) => {
                    const actions = rule.rules;
                    let value = cleansedRecord[column];

                    actions.forEach((action) => {
                        switch (action.name) {
                            case 'trim_whitespace':
                                value = value.trim();
                                break;
                            case 'remove_special_characters':
                                value = value.replace(/[^a-zA-Z0-9 ]/g, '');
                                break;
                            // Other cleansing actions can be added here as needed
                        }
                    });

                    cleansedRecord[column] = value;
                });
            });

            return cleansedRecord;
        });
    }
}

A Practical Example: YAML-Powered Data Cleansing

To elucidate further, let's adapt the previously provided code to showcase how YAML configurations can be harnessed for data cleansing:

const yamlConfig = `
cleansingRules:
  - columns:
      - 'Customer Name'
      - 'Address'
    rules:
      - name: 'trim_whitespace'
      - name: 'remove_special_characters'
`;

const sampleData = [
    {
        'Customer Name': '  John! Doe  ',
        'Address': '  1234@ Elm St!  '
    },
    // ... other data records
];

const dataCleanser = new DataCleansingHandler(yamlConfig);
const cleanedData = dataCleanser.applyCleansingRules(sampleData);

This approach allows for more accessible management of data cleansing rules, facilitating updates via the YAML configuration without the need to adjust the foundational code.

Conclusion

Data cleansing is vital for achieving accurate and trustworthy data. By using YAML configurations, the process becomes not only straightforward but also adaptable. With clean data, businesses are better equipped for informed decision-making and insightful analysis. Adopting tools like YAML ensures that your data remains consistently optimized for these tasks.

Creating an Azure File Change Listener for Real-Time Data Ingestion

Serge Artishev — Sat, 23 Sep 2023 12:03:56 +0000

In this blog post, we'll explore a practical use case for cloud integration and guide you through the step-by-step process of creating an Azure File Change Listener. This tool monitors a local folder for file changes and seamlessly integrates with Azure services to store file details and content. By the end of this tutorial, you'll have a powerful solution to automate file processing in the cloud.

Use Case Scenario: Real-Time Data Ingestion

Imagine you're running an e-commerce platform, and your business relies on real-time inventory updates. Suppliers frequently send you product availability information in the form of CSV files. To keep your customers informed and ensure smooth operations, you need to process these files as soon as they arrive.

Here's where Azure File Change Listener comes into play. By monitoring a designated folder for incoming CSV files, it can instantly send file details to an Azure storage queue and store the file content in Azure Blob Storage. This automated process ensures that your inventory data is always up-to-date, leading to improved customer satisfaction and operational efficiency.

Prerequisites

Before we begin, make sure you have the following prerequisites in place:

Azure Subscription: You'll need access to an Azure subscription to create and configure the required Azure services.
Node.js: Ensure that Node.js is installed on your local machine. You can download it from the official website.
Azure Storage Account: Create an Azure Storage Account to store queue messages and file content.
Azure Queue: Set up an Azure Queue within your storage account to receive file details.
Azure Blob Container (Optional): Create an Azure Blob Container if you want to store the file content in Azure Blob Storage.
Visual Studio Code (or any code editor of your choice): We'll use a code editor to write and run our Node.js application.

Step 1: Set Up Your Development Environment

To get started, open your code editor and create a new Node.js project and initialize a package.json file and install the necessary dependencies, including the Azure SDK libraries.

npm init -y
npm install -g oclif
oclif generate your-cli-app-name

If you are unfamiliar with building Azure-ready CLI applications, refer to my previous post where I provide a detailed walkthrough to set up an OCLIF CLI application for data integration projects on Azure.

Step 2: Define the Listener

At this stage, we'll build the foundation of our Azure File Change Listener by creating a Node.js application. We'll use the OCLIF (Open CLI Framework) to create a command-line tool that allows us to monitor a local folder for file changes and integrate with Azure services.

2.1. Set Up Your Node.js Application

Start by initializing a new Node.js project in your code editor of choice. Navigate to your project directory and open a terminal window. Run the following commands to create a package.json file and install the required dependencies:

npm install @azure/storage-blob @azure/data-tables

Here's a brief overview of the libraries we're installing:

@oclif/core: This library provides a framework for building command-line applications. It simplifies the creation of CLI tools with structured commands, flags, and arguments.
@azure/storage-blob and @azure/data-tables: These libraries are part of the Azure SDK for JavaScript and allow us to interact with Azure Blob Storage and Azure Tables. We'll use them to communicate with Azure services.

2.2. Create the Listener Command

In your project directory, create a file named listener.js. This file will define the structure of our Azure File Change Listener. We'll start by importing the necessary libraries and configuring command-line flags.

// listener.js

// Import required libraries
const { Command, Flags } = require('@oclif/core');

// Define the listener command
class ListenerCommand extends Command {
  static description = 'Listens for file changes in a local folder and sends details to Azure storage queue and a blob file or a table';

  static examples = [
    // Add usage examples here
  ];

  static flags = {
    help: Flags.help({ char: 'h' }),
    accountName: Flags.string({ char: 'a', description: 'Azure storage account name' }),
    queueName: Flags.string({ char: 'q', description: 'Azure storage queue name' }),
    tableName: Flags.string({ char: 't', description: 'Azure storage table name' }),
    containerName: Flags.string({ char: 'c', description: 'Azure storage container name (lowercase letters only)' }),
  };

  static args = {
    folderPath: Args.string({
      name: 'folderPath',
      required: true,
      description: 'Path to the folder to listen to',
    }),
  };

  // Define the run method to execute the listener
  async run() {
    const { flags, args } = this.parse(ListenerCommand);

    // Implement the listener logic here

    this.log(`Listening for file changes in folder: ${args.folderPath}`);
  }
}

// Export the listener command
module.exports = ListenerCommand;

In this code:

We import the necessary libraries: Command and Flags from OCLIF.
We define the ListenerCommand class, which represents our Azure File Change Listener. It extends the Command class provided by OCLIF.
We set the description and usage examples for the command to provide clear documentation.
We define the command-line flags that users can specify when running the listener. These flags will allow users to configure the Azure storage account, queue, table, and container they want to use.
We define a required argument named folderPath, which represents the path to the local folder that the listener will monitor.
We leave the run method empty for now, as we'll implement the listener logic in the next steps.

2.3. Test the Listener

Before we proceed further, let's test our listener to ensure it's correctly configured. In your terminal, run the following command to see if the listener description is displayed:

node listener.js --help

You should see the description and usage information for your listener command.

That's it for Step 2! We've set up the basic structure of our Azure File Change Listener, configured command-line flags, and created the ListenerCommand class. In the next steps, we'll dive into the implementation details and create the logic for monitoring file changes and interacting with Azure services.

Step 3: Monitor the Folder

In this step, we'll implement the core functionality of our Azure File Change Listener. We'll use Node.js's built-in 'fs' module to watch a folder for file changes and handle events when files are added or modified. Additionally, we'll implement a debouncing mechanism to avoid multiple events for a single file.

3.1. Using fs.watch to Monitor the Folder

Inside the run method of your ListenerCommand class (defined in listener.js), we'll incorporate the fs.watch function to monitor the specified folder for file changes. Here's how you can do it:

// Import the 'fs' module
const fs = require('fs');

// Inside the run method
async run() {
  const { flags, args } = this.parse(ListenerCommand);

  this.log(`Listening for file changes in folder: ${args.folderPath}`);

  // Use fs.watch to monitor the folder
  const watcher = fs.watch(args.folderPath, { recursive: true }, (eventType, filename) => {
    if (eventType === 'change' || eventType === 'rename') {
      // Handle file changes or additions
      this.log(`File ${filename} ${eventType}d.`);

      // Implement Azure integration logic here
    }
  });

  // Implement debouncing logic here

  // Handle errors
  watcher.on('error', (error) => {
    this.error(`Error while watching folder: ${error.message}`);
  });
}

In this code:

We import the 'fs' module, which provides file system-related functionality.
We use fs.watch to start monitoring the specified folder (args.folderPath) for changes. We set the { recursive: true } option to monitor subdirectories as well.
Inside the callback function for the watcher, we check for two event types: 'change' and 'rename'. These events are triggered when files are modified or added to the folder.
We log a message indicating which file was changed or added.

3.2. Implementing a Debouncing Mechanism

A debouncing mechanism is essential to prevent multiple events from firing rapidly for a single file change. We'll implement this debouncing logic using a simple timer. Here's how to do it:

// Inside the run method, after starting the watcher
let debounceTimer = null;

watcher.on('change', (eventType, filename) => {
  if (eventType === 'change' || eventType === 'rename') {
    // Clear the previous debounce timer (if any)
    if (debounceTimer) {
      clearTimeout(debounceTimer);
    }

    // Set a new debounce timer
    debounceTimer = setTimeout(() => {
      // Handle the file change after the debounce period
      this.log(`File ${filename} ${eventType}d (debounced).`);

      // Implement Azure integration logic here

      // Reset the debounce timer
      debounceTimer = null;
    }, 1000); // Adjust the debounce period (in milliseconds) as needed
  }
});

In this code:

We introduce a debounceTimer variable to keep track of the debounce timer.
Inside the 'change' event handler, we clear the previous debounce timer (if any) to reset it.
We set a new debounce timer using setTimeout. The timer's callback function will be executed after a specified delay (in this case, 1000 milliseconds or 1 second). This delay allows us to wait for additional file changes before processing the event.
Inside the timer's callback function, we handle the file change event and implement the Azure integration logic.
Finally, we reset the debounceTimer to null to prepare for the next event.

3.3. Handling Errors

We also handle errors that might occur while watching the folder. If an error occurs, we log an error message using the this.error function:

// Handle errors
watcher.on('error', (error) => {
  this.error(`Error while watching folder: ${error.message}`);
});

With this implementation, our Azure File Change Listener is now capable of monitoring a folder for file changes, debouncing events to avoid rapid firing, and handling errors gracefully.

In the next steps, we'll complete the listener by integrating it with Azure services to send file details and content when changes occur.

That's it for Step 3: Monitoring the Folder! We've added the core functionality to watch for file changes, and we've implemented a debouncing mechanism to ensure efficient event handling.

Step 4: Integrate with Azure

In this step, we'll integrate our Azure File Change Listener with Azure services. We'll use the Azure SDK to interact with Azure storage services, specifically Azure Queue Storage and Azure Blob Storage. Our goal is to send file details to an Azure storage queue and, optionally, store file content in Azure Blob Storage.

4.1. Initialize Azure Storage Services

Before we can interact with Azure services, we need to initialize the necessary Azure storage clients. We'll use the @azure/storage-queue and @azure/storage-blob libraries for this purpose. Here's how you can initialize these services:

// Import Azure storage libraries
const { QueueServiceClient, StorageSharedKeyCredential } = require('@azure/storage-queue');
const { BlobServiceClient } = require('@azure/storage-blob');

// Define Azure storage account details
const storageAccountName = 'your-storage-account-name';
const storageAccountKey = 'your-storage-account-key';

// Initialize Azure Queue Service Client
const queueServiceClient = new QueueServiceClient(
  `https://${storageAccountName}.queue.core.windows.net`,
  new StorageSharedKeyCredential(storageAccountName, storageAccountKey)
);

// Initialize Azure Blob Service Client (optional)
const blobServiceClient = new BlobServiceClient(
  `https://${storageAccountName}.blob.core.windows.net`,
  new StorageSharedKeyCredential(storageAccountName, storageAccountKey)
);

Replace 'your-storage-account-name' and 'your-storage-account-key' with your Azure Storage account's name and key.

4.2. Send File Details to Azure Queue

Now that we have our Azure storage clients initialized, we can send file details to an Azure storage queue. Inside the 'change' event handler in your ListenerCommand class, add the following code to send file details to the queue:

// Inside the 'change' event handler
if (eventType === 'change' || eventType === 'rename') {
  // ...

  // Implement Azure integration logic here

  // Send file details to Azure Queue
  const queueClient = queueServiceClient.getQueueClient(flags.queueName);
  const fileDetails = {
    fileId: fileId,
    fileName: filename,
    eventTime: new Date().toISOString(),
    eventType: eventType,
    fileSize: fs.statSync(filePath).size,
    fileExtension: path.extname(filename),
    contentType: mime.getType(filename) || 'application/octet-stream',
    createdTimestamp: fs.statSync(filePath).birthtime.toISOString(),
    modifiedTimestamp: fs.statSync(filePath).mtime.toISOString(),
  };

  await queueClient.sendMessage(JSON.stringify(fileDetails));
  this.log(`Sent file details to Azure Queue: ${JSON.stringify(fileDetails)}`);
}

In this code:

We create a queueClient to interact with the Azure storage queue using the queueServiceClient.
We prepare the fileDetails object containing information about the file change, such as the file ID, name, event time, file size, and more.
We use queueClient.sendMessage to send the fileDetails object to the Azure storage queue.

4.3. Store File Content in Azure Blob Storage (Optional)

If you want to store the actual file content in Azure Blob Storage, you can use the following code inside the 'change' event handler:

// Inside the 'change' event handler
if (eventType === 'change' || eventType === 'rename') {
  // ...

  // Implement Azure integration logic here

  if (flags.containerName) {
    // Store the file content in Azure Blob Storage (folder is not

 used)
    const containerClient = blobServiceClient.getContainerClient(flags.containerName);
    const blobClient = containerClient.getBlobClient(fileId);

    // Read the file content and upload it to Azure Blob Storage
    const fileContent = fs.readFileSync(filePath);
    await blobClient.uploadData(fileContent, fileContent.length);

    this.log(`Stored file content in Azure Blob Storage with ID: ${fileId}`);
  } else {
    // ... (Send to Azure Table logic, if not using Blob Storage)
  }
}

In this code:

We check if the containerName flag is provided, indicating that we want to store file content in Azure Blob Storage.
We create a containerClient and a blobClient to interact with the specified Azure Blob container and blob.
We read the file content using fs.readFileSync and then upload it to Azure Blob Storage using blobClient.uploadData.

Now, your Azure File Change Listener is integrated with Azure services, allowing you to send file details to an Azure storage queue and optionally store file content in Azure Blob Storage.

In the next steps, we'll complete the listener by adding finishing touches and handling additional scenarios.

Step 5: Run the Listener

You've reached the final step in creating your Azure File Change Listener. Now it's time to run the listener and test its functionality. Follow these steps to run your listener and observe how it seamlessly integrates with Azure services to process file changes in real-time.

5.1. Open Terminal

Open your terminal or command prompt and navigate to the directory where your Azure File Change Listener code is located.

cd /path/to/azure-file-change-listener

5.2. Run the Listener

To run your Azure File Change Listener, use the following command:

node listener.js --accountName your-storage-account-name --queueName your-queue-name --containerName your-container-name /path/to/monitored-folder

Replace the placeholders:

your-storage-account-name with your Azure Storage account name.
your-queue-name with the name of your Azure storage queue.
your-container-name with the name of your Azure Blob Storage container (if you're using Blob Storage).
/path/to/monitored-folder with the path to the local folder you want to monitor for file changes.

5.3. Test the Listener

With the listener running, start adding or modifying files in the monitored folder. You can create new files, edit existing ones, or even rename them. Your Azure File Change Listener should detect these changes and trigger the Azure integration logic.

5.4. Observe Azure Integration

As you make changes to the monitored folder, keep an eye on the terminal where your listener is running. You'll see log messages indicating that file details are being sent to the Azure storage queue, and if you configured it, that file content is being stored in Azure Blob Storage.

New file detected: example.txt
Sent file details to Azure Queue: {...}
Stored file content in Azure Blob Storage with ID: ...

5.5. Monitor Real-Time File Changes

Your Azure File Change Listener is now up and running, continuously monitoring the specified folder for file changes and seamlessly integrating with Azure services. Whether you're using it to track log files, process incoming data, or trigger workflows, your listener is ready to streamline your file-based workflows.

Congratulations! You've successfully created, configured, and run an Azure File Change Listener, empowering your applications with real-time file change detection and Azure integration capabilities.

Conclusion

In this step-by-step guide, we've created an Azure File Change Listener to automate real-time data ingestion into Azure. This powerful tool can be adapted to various use cases, from e-commerce inventory updates to financial data processing.

By harnessing the capabilities of Azure and Node.js, you can build robust and efficient cloud integration solutions that improve your business operations and enhance customer experiences. Whether you're in the retail industry or any other sector, real-time data ingestion is a game-changer, and Azure is your reliable cloud partner on this journey.

Now that you have a working Azure File Change Listener, consider expanding its functionality to suit your specific business needs. Azure's vast ecosystem of services offers endless possibilities for cloud integration, automation, and innovation.

AzCopy Chronicles: The Simple Tool that Saved the Day!

Serge Artishev — Sat, 16 Sep 2023 23:33:45 +0000

This post is a bit of an experiment, intertwining storytelling elements with the technical narrative. I'd love to hear your feedback on this approach - whether it resonated with you or not, please feel free to share your thoughts in the comments section below. Your feedback is greatly appreciated and will help in crafting future content. Thank you for reading!

Hey there, fellow data explorers!

Gather 'round, because today I've got a thrilling tale of data adventure to share - my epic journey to keep the elusive 'last modified time' intact while transferring files in Azure. Picture this: a journey peppered with trials, errors, and a glorious discovery that changed everything!

Why Keeping the 'Last Modified Time' is a Big Deal

Now, I know what you’re thinking. “Why is this timestamp even important?” Let me tell you, in the grand saga of data management, it's practically the hero of the story. Picture it as the faithful squire in the land of data, ensuring everything runs smoothly in the kingdom:

Tracking Your Data's Journey: Imagine if our data were like breadcrumbs, leading us back to the moment of inception. The 'last modified time' helps us trace the steps of our data's journey, a trustworthy guide in a world of numbers and bytes.
Keeping Your Data Safe and Sound: This unassuming timestamp is like a guardian, ensuring our data remains pristine and unaltered, guarding it valiantly against the forces of change and time.
Saving Time on Backups: It's the knight in shining armor, coming to rescue us from the daunting dragon of data backups, helping us save the damsel of time and effort.
Quick and Easy Updates: Picture a wizard who can swiftly sift through piles of data, identifying only the freshest nuggets of information. That's the magic of 'last modified time'.
Staying on the Right Side of the Rules: Even in the wild west of data engineering, there are laws to uphold. This timestamp is like the sheriff in town, helping us toe the line and avoid the wrath of the regulatory gods.

What Happens If We Don't Keep It?

Ah, but every tale has its dark side. What happens when we ignore this tiny but mighty timestamp, you ask? Here’s the potential horror movie that could unfold:

Messy Data: It's like unleashing a tornado in our neat data library. We lose the plot, the characters get jumbled, and we're left with a tragedy on our hands.
Wasted Time and Effort: Imagine being trapped in a never-ending loop of data chaos, constantly chasing our tails without this handy guide. A true data engineer's nightmare!
Trouble Spotting Mistakes: Without our trusty sidekick, identifying errors turns into a wild goose chase. It's like trying to find a needle in a haystack... but the haystack is also made of needles.
Breaking the Rules: Neglecting the 'last modified time' is akin to flirting with the dark side, potentially invoking the wrath of the compliance overlords. A risky game to play, I must say!
Tougher Decision Making: Here we are, steering our ship in the murky waters of uncertainty. Without our data compass, making informed decisions becomes a treacherous journey.

The Bumpy Start

Once upon a time, in a not-so-distant data land, I embarked on a mission to transfer files manually, with high hopes and dreams. But alas, this naive method betrayed me, resetting the precious 'last modified time' with every move, leaving me lost in the data wilderness.

Determined, I turned to scripting, hoping for a beacon of light. But yet again, my efforts were thwarted by the ever-changing, slippery 'last modified time', refusing to be tamed, laughing at my futile attempts.

The Light at the End of the Tunnel

Just when all hope seemed lost, a hero emerged from the depths of the Azure suite: the mighty azcopy. This underappreciated warrior promised not just ease of transfer but vowed to protect the sacred 'last modified time'.

With a hopeful heart and fingers crossed, I wielded this powerful command:

azcopy copy https://{accountname}.blob.core.windows.net/{container}/{folder}/* ./{targetFolder} --preserve-last-modified-time

Lo and behold, it worked like a charm! A seamless transfer, with the 'last modified time' standing strong, safeguarding the history of my files.

A Happy Ending

So, here we are, at the end of this roller-coaster journey. From despair to triumph, all thanks to a simple yet powerful tool. Remember, fellow data wanderers, when in doubt, trust in the magic of azcopy to light your path.

Until our next adventure, happy data handling and may the 'last modified time' be with you!

Streamlining Azure Storage Queue Testing with Dynamic Data Generation Using Faker.js

Serge Artishev — Wed, 13 Sep 2023 04:07:10 +0000

Introduction

In modern data engineering, the significance of using realistic data in testing environments cannot be overstated. By mimicking real-world data scenarios, engineers can not only ensure the robustness of their data pipelines but also ascertain the accuracy of their data analytics processes. It aids in identifying potential bottlenecks and errors before they manifest in production environments, thereby enhancing the reliability and efficiency of data solutions. Moreover, it helps in fine-tuning data security and privacy measures by testing them against data that closely resembles actual operational data.

Brief on Azure Storage Queue and Faker.js

Azure Storage Queue is a Microsoft Azure service that enables asynchronous message queuing between application components. It facilitates communication via messages up to 64 KB in size that can be kept in the queue for up to 7 days, enabling you to build flexible and reliable applications.

Faker.js is a powerful and flexible library that facilitates the generation of massive amounts of realistic fake data. It supports a numerous amount of data types, including names, addresses, numbers, texts, dates, and many more, making it a go-to solution for testing and developing applications that require a rich dataset.

Setting Up Your Environment

1. Dependencies and Installations

Setting up Node.js and Required Libraries

Before diving into the core implementation, it is essential to set up a conducive development environment. Begin by installing Node.js from the official website. Post installation, you can create a project directory and initiate a Node.js project using:

mkdir project_name
cd project_name
npm init -y

Next, install the necessary libraries - @azure/storage-queue, @faker-js/faker, and @oclif/core - using npm:

npm install @azure/storage-queue @faker-js/faker @oclif/core

In this project, we are also utilizing the OCLIF (Open CLI Framework) to build a Command Line Interface (CLI) application that facilitates seamless interaction with Azure services. If you are unfamiliar with building Azure-ready CLI applications, refer to my previous post where I provide a detailed walkthrough to set up an OCLIF CLI application for data integration projects on Azure.

Configuring Azure Storage Queue

To work with the Azure Storage Queue, you'll need to set up an Azure account and create a new Storage account if you haven't already. Follow the official documentation to get this done. Remember to store your accountName and queueName safely as they will be required to authenticate your application.

2. Template Structuring

Creating JSON Templates with Dynamic Data Elements

Creating dynamic JSON templates serves as the blueprint for the messages to be enqueued. Start by creating a JSON file, say sample.json, and defining the structure of the data including placeholder elements that will later be replaced by dynamic values generated by Faker.js. For instance:

{
  "uuid": "{{uniqueId}}",
  "firstName": "{{firstName}}",
  "lastName": "{{lastName}}",
  "email": "{{email}}",
  "address": {
    "city": "{{faker.address.city}}",
    "zipcode": "{{faker.address.zipCode}}",
    "streetAddress": "{{faker.address.streetAddress}}"
  }
}

Using Faker.js Expressions for Flexible Data Generation

Faker.js plays a crucial role in replacing the placeholders in your template with realistic data. In your JavaScript/TypeScript file, you can create a function to populate these templates using Faker.js methods. Here’s how you can do it:

import { faker } from '@faker-js/faker';

function populateTemplate(templateContent) {
  const replacements = {
    "{{uniqueId}}": faker.datatype.uuid(),
    "{{firstName}}": faker.name.firstName(),
    "{{lastName}}": faker.name.lastName(),
    "{{email}}": faker.internet.email(),
    // ... (include other replacements)
  };

  Object.keys(replacements).forEach((placeholder) => {
    templateContent = templateContent.replace(new RegExp(placeholder, 'g'), replacements[placeholder]);
  });

  return templateContent;
}

With these steps, you have laid a solid foundation to build a dynamic, data-driven application capable of enqueuing realistic messages into the Azure Storage Queue.

Developing the Enqueue Service

1. Class Construction

In our setup, the central component facilitating the enqueue operation is the EnqueueService class. This section dissects the construction of this class, and the initialization of the Azure Storage Queue client.

Establishing the EnqueueService Class

The EnqueueService class serves as the cornerstone of our application, encapsulating the logic needed to interact with Azure Storage Queue. It's initiated with an Azure account name, a queue name, and credentials as parameters. Here's a snippet demonstrating this:

export class EnqueueService {
  private queueClient;

  constructor(accountName: string, queueName: string, credential: ExtTokenCredential) {
    const queueServiceClient = new QueueServiceClient(`https://${accountName}.queue.core.windows.net`, credential);
    this.queueClient = queueServiceClient.getQueueClient(queueName);
  }

  // ... other methods
}

Initializing the Azure Storage Queue Client

During the initialization phase, we create an instance of QueueServiceClient from the Azure SDK. We then initialize a queueClient instance by invoking the getQueueClient method, which will be used to interact with the designated queue throughout the application. This step ensures smooth communication with the Azure Storage Queue:

async initialize() {
  await this.queueClient.createIfNotExists();
}

This method, initialize, ensures that the queue exists before any message enqueue operation, creating it if necessary, thereby preventing potential errors during runtime.

2. Message Handling

The sendMessage method plays a pivotal role in our service class, serving as the mechanism through which messages are constructed and dispatched to the queue. This section covers integrating various message templates and implementing custom logic for realistic data generation using Faker.js.

The sendMessage Method: Integrating Template Message Types

The sendMessage method in the EnqueueService class manages the sending of messages to the Azure Storage Queue. It verifies the existence of a message template and reads it if available. Below is the method in question:

async sendMessage(messageTemplate: string | undefined) {
  let message;
  const templatePath = path.resolve(process.cwd(), messageTemplate);
  if (fs.existsSync(templatePath)) {
    const templateContent = fs.readFileSync(templatePath, 'utf-8');
    message = this.populateTemplate(templateContent);
  } else {
    throw new Error('Template file not found');
  }
  await this.queueClient.sendMessage(message);
}

Implementing Custom Logic with Faker.js for Realistic Data Patterns

To create realistic and varied data patterns, we utilize Faker.js within the populateTemplate method. Initially, this method identifies and replaces Faker.js expressions within the template with actual values generated by Faker.js. Subsequently, we implement custom logic to generate a dataset with realistic relationships, as shown below:

populateTemplate(templateContent: string): string {
  // ... existing logic to replace faker expressions

  // Implementing custom logic for data patterns
  const sex = faker.person.sexType();
  const firstName = faker.person.firstName(sex);
  const lastName = faker.person.lastName();
  const email = faker.internet.email({ firstName, lastName });

  const replacements: any = {
    "{{uniqueId}}": faker.datatype.uuid(),
    "{{avatar}}": faker.image.avatar(),
    "{{birthday}}": faker.date.birthdate().toISOString(),
    "{{email}}": email,
    "{{firstName}}": firstName,
    "{{lastName}}": lastName,
    "{{sex}}": sex,
    "{{subscriptionTier}}": faker.helpers.arrayElement(['free', 'basic', 'business']),
  };

  Object.keys(replacements).forEach((placeholder: string) => {
    templateContent = templateContent.replace(new RegExp(placeholder, 'g'), replacements[placeholder]);
  });

  return templateContent;
}

In this code snippet, we further enrich the template content with a more comprehensive set of placeholders, generating a structured message with fields such as firstName, lastName, email, and others, which are populated dynamically with realistic data patterns thanks to Faker.js.

Coding the Command Line Interface

Utilizing the `@oclif/core` for CLI Development

In our solution, the OCLIF (Open CLI Framework) serves as the backbone of our CLI application. This tool enables developers to craft single or multi-command CLIs with ease, offering capabilities such as plugin support, auto-generated help, and argument/flag parsing. You'd start by importing the necessary classes and initializing the command description and flags, as seen in the enqueue.ts script.

import { Command, Flags } from '@oclif/core';

Implementing Flags for User-Defined Input

In the CLI, Flags offer a mechanism to specify various options that modify the behavior of a command. They are defined in the static flags object in the Command class. In our implementation, flags are used to define parameters such as rate, minutes, accountName, and others, providing flexibility in how the command can be executed.

static flags = {
    help: Flags.help({ char: 'h' }),
    rate: Flags.integer({ char: 'r', description: 'number of records to send per minute', default: 60, max: 100000 }),
    // ... other flags
};

Structuring the Enqueue Command

Our Enqueue class extends from the Command class provided by @oclif/core. This forms the basis of our command. Inside this class, we describe the command, define flags, and implement the run method which contains the logic for our command.

import { Command, Flags } from '@oclif/core';
import { EnqueueService } from '../utils/azure/EnqueueService';
import { errorMessages } from '../errorMessages';
import { ExtTokenCredential } from '../ExtTokenCredential';
import { Progress } from '../utils/progress';

export default class Enqueue extends Command {
  static description = 'Enqueue random strings into Azure Storage Queue';

  // Define flags 
  static flags = {
    help: Flags.help({ char: 'h' }),
    rate: Flags.integer({ char: 'r', description: 'number of records to send per minute', default: 60, max: 100000 }),
    minutes: Flags.integer({ char: 'm', description: 'Number of minutes to run the command', default: 1, max: 120 }),
    accountName: Flags.string({ char: 'a', description: 'Azure storage account name' }),
    queueName: Flags.string({ char: 'q', description: 'Azure storage queue name' }),
    messageTemplate: Flags.string({ char: 't', description: 'Message template to use for enqueueService' }),
  };

  // Command logic 
  async run() {
    // (Details in next sub-section)
  }
}

Implementing the Run Method

The run method is where the command's logic resides. Here, we parse the flags, validate the inputs, and initiate the EnqueueService. We also handle any errors that might occur during the process.

async run() {
  const { flags } = await this.parse(Enqueue);

  // Validate the inputs
  if (!flags.accountName) {
    this.error(errorMessages.MISSING_ACCOUNT_NAME.message, errorMessages.MISSING_ACCOUNT_NAME.options);
  }

  if (!flags.queueName) {
    this.error(errorMessages.MISSING_QUEUE_NAME.message, errorMessages.MISSING_QUEUE_NAME.options);
  }

  // Initialize credentials and enqueue service
  const credential = new ExtTokenCredential();
  const enqueueService = new EnqueueService(flags.accountName!, flags.queueName!, credential);

  // Setup and start the message enqueue process
  try {
    await enqueueService.initialize();

    const totalMessagesToSend = flags.rate * flags.minutes;
    const ratePerSecond = flags.rate / 60;
    let count = 0;

    const progress = new Progress(totalMessagesToSend);

    const intervalId = setInterval(async () => {
      try {
        if (count >= totalMessagesToSend) {
          clearInterval(intervalId);
          progress.stop();
          this.log('Operation completed');
          return;
        }

        await enqueueService.sendMessage(flags.messageTemplate);
        count += 1;
        progress.update(count);
      } catch (error: any) {
        clearInterval(intervalId);
        progress.stop();
        this.error(`Error sending message: ${error.message}`, { exit: 1 });
      }
    }, 1000 / ratePerSecond);
  } catch (error: any) {
    this.error(`An error occurred: ${error.message}`, { exit: 1 });
  }
}

In the above script:

We first parse and validate the command line flags.
Next, we initialize the EnqueueService with necessary credentials.
We then calculate the total messages to send and the rate per second based on the inputs.
A periodic interval is established to send messages at the defined rate, and progress is tracked using a Progress instance.

Using the CLI with the Enqueue Command

In this section, we will go through the steps and options available when using the CLI with the enqueue command. Here's a step-by-step guide:

Step 1: Setting Up the CLI

Before you begin, ensure that your CLI tool is properly installed and configured to interact with your Azure account. Connect to your Azure subscription with the following command:

your-cli-app-name login

Step 2: Understanding the Command Structure

The basic structure of the enqueue command is as follows:

your-cli-app-name enqueue -a <account_name> -q <queue_name> -r <rate> -m <minutes> -t <message_template>

Explanation of the flags used in this command:

-a, --accountName - Azure storage account name
-q, --queueName - Azure storage queue name
-r, --rate - Number of records to send per minute (default: 60, max: 100000)
-m, --minutes - Number of minutes to run the command (default: 1, max: 120)
-t, --messageTemplate - Path to the message template to use for enqueue service

Step 3: Executing the Command

Navigate to the directory where your project resides and execute the enqueue command with the desired parameters. Here’s an example:

your-cli-app-name enqueue -a "yourAccountName" -q "yourQueueName" -r 100 -m 10 -t "./path/to/your/template.json"

Step 4: Monitoring Progress

Once the command is running, you will see a progress bar indicating the number of messages enqueued in the Azure storage queue per minute (see sample output below). You can monitor this to keep track of the operation.

{
  "uuid": "b9d0dc0c-3daa-46c0-85b0-379211bf02a4",
  "sex": "male",
  "firstName": "Morris",
  "lastName": "Kautzer",
  "email": "Morris_Kautzer17@gmail.com",
  "location": {
    "city": "East Douglasboro",
    "zipcode": "70926-3491",
    "streetAddress": "29727 Braun Mountains"
  }
}

Step 5: Error Handling

In case of any errors, the CLI will output descriptive error messages. Make sure to check the error messages for clues on how to rectify any issues encountered.

Step 6: Stopping the Operation

To stop the operation at any point, you can use Ctrl+C to terminate the process.

Conclusion

In this tutorial, we have successfully navigated through the setup and utilization of a CLI tool that integrates Faker functionality into Azure's storage queue services. By following the outlined steps, data integration engineers can seamlessly generate and enqueue randomized messages into the Azure Storage Queue, enhancing data testing and validation processes.

As we have seen, the combination of a message template system along with the Faker library empowers engineers to craft realistic and complex data structures effortlessly. Moreover, with the customization offered in the message template, engineers have the flexibility to create data that suits various scenarios and requirements.

Moving forward, you might consider exploring further customization of the message template to cater to more complex data structures or integrating this CLI tool into your CI/CD pipeline for automated data testing.

Remember, the key to effectively utilizing this tool is understanding the available commands and options, and adapting the message templates to suit your project's unique needs.

I encourage you to experiment with different configurations and discover the full potential of this tool in streamlining your data integration processes on Azure.

Happy coding!

Empowering Your Azure Projects with a Secure CLI: A Step-by-Step Guide

Serge Artishev — Mon, 11 Sep 2023 06:30:13 +0000

In this easy-to-follow blog post, we're going to guide you step by step on how to build a command-line interface (CLI) application using oclif, the NodeJs Open CLI Framework. This application will be your trusted tool for securely logging into your Azure account.

What makes it even cooler? Just like other popular CLI tools such as az, our application securely stores your authenticated token in local storage. This means you can effortlessly use it to execute other CLI commands without having to log in every time.

Let's dive in and get started!

Prerequisites

Before we begin, make sure to install the oclif globally by using the following command:

npm install -g oclif

Step 1: Setting Up Your Project

Initiate your CLI project with the following command:

oclif generate your-cli-app-name

Navigate to your project folder and install necessary packages for Azure identity and HTTP client axios:

npm install @azure/identity axios

Step 2: Creating the Login Command

Generate your login command using the following:

oclif generate command login

This will create a template file for your login command. Open src/commands/login.ts and implement the login functionality using the DefaultAzureCredential which attempts to use multiple authentication methods based upon the available environment variables:

import { Command } from '@oclif/core';
import { DefaultAzureCredential } from '@azure/identity';

export default class Login extends Command {
  static description = 'Login to Azure account';

  async run() {
    const credential = new DefaultAzureCredential();

    try {
      const token = await credential.getToken('https://management.azure.com/.default');
      if (token) {
        // Storing the token securely will be implemented in the next steps
        this.log('Login successful.');
      }
    } catch (error: any) {
      this.error(`Login failed: ${error.message}`, { exit: 1 });
    }
  }
}

Step 3: Integrating Keytar for Robust and Secure Token Storage

In this step, we're going to enhance our application's security by integrating Keytar, a node module that leverages the operating system's secure services for storing sensitive information. This prevents any unauthorized access to the tokens, safeguarding your user credentials effectively.

First off, let's install Keytar to manage the secure storage of the token acquired during the login process:

npm install keytar

Next, we modify the login command to utilize Keytar. This will allow us to store the authentication token in a safe and encrypted manner, a strategy commonly adopted to prevent token theft and ensure overall application security. Here is how you can do it:

import * as keytar from 'keytar';
// ... (rest of the imports)

// ... (within the try block in the run method)
if (token) {
  await keytar.setPassword('your-cli-app-name', 'azureToken', token.token);
  this.log('Login successful.');
}
// ...

By implementing Keytar, not only are we fortifying our application's security, but we are also creating a seamless user experience. Keytar facilitates automatic and secure retrievals of tokens, which means users won't have to re-enter their credentials every time they run a command, thus aligning with best practices observed in well-established CLI tools such as az.

Remember, a secure application is a trusted application. Let's move forward, ensuring the safety of our user's data with the reliable functionalities provided by Keytar!

Step 4: Creating the Logout Command

Generate your logout command:

oclif generate command logout

Now, populate the src/commands/logout.ts with the following content to enable logging out functionality:

import { Command } from '@oclif/core';
import * as keytar from 'keytar';

export default class Logout extends Command {
  static description = 'Logout from Azure account';

  async run() {
    await keytar.deletePassword('your-cli-app-name', 'azureToken');
    this.log('Logout successful.');
  }
}

Step 5: Creating the Account Show Command with Stored Access Token

As we venture further, we'll be exploring how to utilize the stored access token in crafting our custom CLI commands. Let's start with something simple yet functional, like an 'account show' command that fetches and displays the username of the connected Azure account.

First, initiate the process by generating the account show command. Use the following command to do this:

oclif generate command account:show

Before we populate the newly created command file, we need to craft a utility function that will fetch the username of the logged-in Azure account. This utility function will leverage the stored access token we saved during the login process.

But before we proceed, ensure to install axios - a promise-based HTTP client for the browser and Node.js:

npm install axios

Next, we'll create a utility module to encapsulate the logic to fetch the logged-in username. Create a new file at src/utils/azureUtils.ts and insert the following content:

import axios from 'axios';
import * as keytar from 'keytar';

export async function getLoggedInUserName(): Promise<string> {
  try {
    // Retrieve the stored token
    const token = await keytar.getPassword('your-cli-app-name', 'azureToken');
    if (!token) throw new Error('No token found');

    // Use the token to fetch the user information from Azure
    const response = await axios.get('https://management.azure.com/me?api-version=2020-10-01', {
      headers: {
        'Authorization': `Bearer ${token}`
      }
    });

    // Return the user principal name or a default string if it's not available
    return response.data.userPrincipalName || 'Unknown User';
  } catch (error: any) {
    throw new Error(`Could not fetch user info: ${error.message}`);
  }
}

With our utility function ready, let’s now create a command that makes use of this function to display the Azure account details on the terminal. Open src/commands/account/show.ts and implement the command as follows:

import { Command } from '@oclif/core';
import { getLoggedInUserName } from '../../utils/azureUtils';

export default class AccountShow extends Command {
  static description = 'Show the connected Azure account username';

  async run() {
    try {
      // Fetch the username using the utility function
      const username = await getLoggedInUserName();
      // Log the username to the terminal
      this.log('Connected account username:', username);
    } catch (error: any) {
      // Handle any errors that occur during the process
      this.error(`Error fetching username: ${error.message}`, { exit: 1 });
    }
  }
}

This section now elucidates the usage of the stored access token and how it integrates seamlessly in creating a new CLI command. Let me know if there is anything else you'd like to add or modify!

Step 6: Understanding Scopes in Azure

When using the DefaultAzureCredential class to authenticate, it is essential to understand scopes. Scopes define the resources and permissions your token will have access to in the Azure ecosystem. Different Azure resources will require different scopes. Adjust the scopes as per your needs. For example:

const token = await credential.getToken('https://management.azure.com/.default');

Here, we are requesting a token with the scope of managing resources on Azure. Make sure to select the appropriate scope for your application needs.

Step 7: Testing Your Commands

Test your CLI application using the following commands:

./bin/dev login
./bin/dev logout
./bin/dev account show

You should see the expected outputs for each command if everything is set up correctly.

Absolutely! Here's how you can extend the guide to include building, publishing, and using the CLI application:

Step 8: Building the Final CLI Application

After thoroughly testing your commands, it's time to compile your CLI application into a version that's ready for production. Here's how you can do this:

npm run build

This command should bundle your application and prepare it for distribution. Make sure to check for any errors and fix them before proceeding to the next step.

Step 9: Publishing Your CLI Application

Now that you have built your CLI application, the next step is to publish it so that it can be easily installed and used by others. Here’s how you can publish your application to the npm registry:

Ensure that you are logged into your npm account by running:

   npm login

Publish your package using:

   npm publish

This command will upload your package to the npm registry, making it available for others to install and use.

Step 10: Sample Usage

Your CLI application is now live! Here's how users can install it and a few sample commands they can use to get started:

To install the application, users should run:

   npm install -g your-cli-app-name

Once installed, they can start using the application with commands such as:

   your-cli-app-name login
   your-cli-app-name logout
   your-cli-app-name account show

Remember to provide a comprehensive README with your package, detailing all available commands and usage examples to assist users in navigating through your application efficiently.

Conclusion

Congratulations on building a secure CLI application with oclif, integrated with Azure authentication, featuring secure login, logout, and account detail retrieval functionalities. Keep exploring the power of CLI tools with oclif!

DEV Community: Serge Artishev

Supercharging Databricks Asset Bundles

The Power of Parameterization

Variable Structure in DAB

Including Variables in Your Project

Real-World Example: Parameterizing SharePoint Workflows

Automated CI/CD Pipeline

Authentication with Service Principals

Advanced Techniques

1. Dynamic Configuration Based on Branch

2. Feature Flags via Variables

3. Template Workflows with Parameters

Conclusion

Databricks Asset Bundle: A Template to Make Life Easier

The DAB Story - What Even Is It?

Why I Built This Project

What's in the Box?

1. Organized Structure for Your Data Projects

2. Python Package Architecture

3. Infrastructure as Code with DAB

4. Built-in Best Practices

5. Fast Onboarding Process

Step-by-Step Guide to Getting Started

1. Create Your Own Repository from the Template

2. Set Up Your Development Environment

3. Configure Your Project

4. Validate and Deploy Your Project

Extending the Common Package

1. Add a New Module

2. Implement Your Functions

3. Import in Your Notebook

My Closing Thoughts

Azure SQL to Storage Account Export Tool

The Problem

The Solution

Export Tool (export_to_tsv.sh)

Key Features:

Usage:

Example:

Import Tool (import_from_tsv.sh)

Key Features:

Usage:

Example:

Behind the Scenes

Conclusion

Building a DOCX to Markdown Converter with Node.js

Table of Contents

Introduction

Setting Up the Project

Basic DOCX to HTML Conversion

Converting HTML to Markdown

Extracting Images

Formatting Tables

Conclusion

Securing Azure Functions with OAuth2 Authentication

1. Creating and Deploying an Azure Function using DevContainer in VSCode

Introduction

Prerequisites

Installation

2. Configuring the Azure Function to use Azure Active Directory (AAD) OAuth2

Introduction to AAD OAuth2

Configuring AAD OAuth2 for the Azure Function

Testing AAD OAuth2 Authentication

3. Creating a Sample Client App that Authenticates and Executes the Function

Designing the Client App

Implementing Authentication

Conclusion

Designing RESTful APIs with OpenAPI: A Top-Down Approach

Section 1: Getting Started with OpenAPI

What is OpenAPI?

Why Use OpenAPI?

Section 2: Defining API Endpoints

API Paths and Operations

Path Parameters

Section 3: Defining Request and Response Schemas

Request Body Schemas

Response Schemas

Component Schemas

Section 4: Validations and Enums

Property Validations

Export Tool (`export_to_tsv.sh`)

Import Tool (`import_from_tsv.sh`)