DEV Community: Fernando Ultremare

[Boost]

Fernando Ultremare — Sat, 14 Dec 2024 13:46:51 +0000

BigDAG: A Simple Tool for Managing BigQuery Workflows

Fernando Ultremare ・ Dec 14

BigDAG: A Simple Tool for Managing BigQuery Workflows

Fernando Ultremare — Sat, 14 Dec 2024 13:38:56 +0000

Hey everyone,

I wanted to share a tool I've been working on called BigDAG. It's born out of my own frustrations with managing data transformations in BigQuery, and I figured others might find it useful too.

The Core Problem: BigQuery Complexity

BigQuery is fantastic for data analysis, but when you start building real pipelines, things can get messy. You end up with:

SQL Files Everywhere: Queries for tables, views, and transformations scattered across different files.
Manual Dependency Tracking: Figuring out the correct order to run scripts becomes a chore.
Repetitive bq Commands: Creating datasets, tables, and views involves running similar commands repeatedly.
Deployment Headaches: Pushing changes to BigQuery can be a manual and error-prone process.

I needed a way to bring some order to this chaos, to define my workflows as code, and to automate the deployment process. That's the motivation behind BigDAG.

BigDAG: A Practical Solution

BigDAG is a Python tool that helps you manage data workflows in BigQuery. It's not trying to be a full-fledged data orchestration platform; it's a practical tool to solve a specific problem. Here's the basic idea:

Organize Your Files: You structure your SQL scripts, schema definitions, and external table definitions in a folder. This folder represents your data pipeline.
Dependency Detection: BigDAG automatically detects dependencies between your SQL scripts by looking at the queries. If a view uses another table, BigDAG knows it needs to be created first.
Manual Overrides: If automatic detection isn't enough, you can specify dependencies in a deps.yaml file.
Simple CLI: BigDAG provides a command-line interface (CLI) to execute your workflows. You can create, update, or delete objects in BigQuery with a single command.
Basic Templating: BigDAG uses simple templating to inject your project ID and dataset name into your SQL queries.

Key Features

Dependency Management: Automatically figures out the order to run your scripts.
Code-Based Workflows: Define your pipelines using files and YAML.
CLI for Deployment: Deploy changes to BigQuery with a single command.
Dry Run Option: See the commands that will be executed before running them.
Recreation Support: Easily recreate your entire dataset and all objects.

A Simple DAG Example

Let's look at a simple example of how you might structure your DAG folder:

my_dag/
├── raw/
│   └── sales.sheet.def.json
│   └── sales.sheet.schema.json
├── trusted/
│   └── sales.view.sql
└── refined/
    └── monthly_sales.table.sql

Here's what the files might contain:

my_dag/raw/sales.sheet.def.json:

{
    "sourceFormat": "GOOGLE_SHEETS",
    "sourceUris": [
      "https://docs.google.com/spreadsheets/d/FAKE_SPREADSHEET_ID"
    ],
    "googleSheetsOptions": {
        "range": "sales!A1:Z",
        "skipLeadingRows": 1
    }
}

my_dag/raw/sales.sheet.schema.json:

{
  "fields": [
    {"name": "sale_date", "type": "DATE", "mode": "REQUIRED"},
    {"name": "product_id", "type": "STRING", "mode": "REQUIRED"},
    {"name": "amount", "type": "NUMERIC", "mode": "REQUIRED"}
  ]
}

my_dag/trusted/sales.view.sql:

-- SQL script for creating the sales view
SELECT
    *
FROM
    raw_sales;

my_dag/refined/monthly_sales.table.sql:

-- SQL script for creating the monthly_sales table
SELECT
    date_trunc('month', sale_date) AS month,
    SUM(amount) AS total_sales
FROM
    `{{project_id}}.{{dataset}}.trusted_sales`
GROUP BY
    month;

In this example:

sales.sheet.def.json and sales.sheet.schema.json define an external table based on a Google Sheet.
sales.view.sql creates a view on top of the raw sales data.
monthly_sales.table.sql creates a table based on the trusted sales view.

BigDAG will automatically infer that monthly_sales.table.sql depends on trusted_sales, and that trusted_sales depends on raw_sales.

How to Use It (Quickly)

Install:

git clone https://github.com/feroult/bigdag.git
cd bigdag
pip install -e .

Set up your DAG folder: Organize your SQL scripts and definitions.

Run the CLI:

bigdag --folder path/to/your/dag --project your_project_id --dataset your_dataset_name

The Goal: Making Life Easier

BigDAG is about making data workflows more manageable. It's about:

Reducing Manual Errors: Automating repetitive tasks to avoid mistakes.
Improving Workflow Clarity: Making it easier to understand and maintain your data pipelines.
Saving Time: Spending less time on infrastructure and more time on analysis.

Check it Out

If you're dealing with similar challenges in BigQuery, you might find BigDAG useful. It's open-source, so feel free to take a look, contribute, or just use it as is.

https://github.com/feroult/bigdag

Hope it helps!

Vulcan - Exposing Eclipse JDT Programmatically

Fernando Ultremare — Wed, 22 May 2024 03:16:22 +0000

Hey Dev.to community,

I wanted to share a project I worked on a while back called Vulcan. It's a tool that lets you use Eclipse JDT (Java Development Tools) programmatically via command line or REST API, without needing to run the Eclipse workbench.

What is Vulcan?

Vulcan is designed to expose the Eclipse JDT OSGi bundle in a programmatic way. It allows you to perform various refactorings such as:

Extract Method: Extracts a block of code into a new method.
Rename Type: Renames a class or interface.
Rename Method: Renames a method.
Rename Field: Renames a field.
Rename Local Variable: Renames a local variable.
Chained Refactorings: Allows multiple refactorings to be applied in sequence.

Why I Built It

I used Vulcan to automate refactorings before AI started taking over these tasks. It was super handy for improving code quality and maintaining consistency across projects. With all the new AI advances in refactoring, Vulcan has become a bit obsolete. But I learned a ton while working on it, and it was a fun project.

How It Works

Setting Up OSGi

Setting up the OSGi environment involves selecting and finding the necessary packages from the p2 repository. Here’s how you can do it:

Create a bndrun File: This file specifies the OSGi bundles and their versions. You can find the complete bndrun file here.

# refactor/application.bndrun

index: target/index.xml;name="app"

-standalone: ${index}

-runfw: org.eclipse.osgi
-runproperties.equinox: osgi.console=, osgi.console.enable.builtin=true, osgi.ws=gtk, osgi.os=linux, osgi.arch=x86_64

-runee: JavaSE-17

-runproperties: \
    osgi.instance.area=../../../../vulcan-test-workspace,\
    jetty.home.bundle=org.eclipse.jetty.osgi.boot

-runrequires: \
    bnd.identity;id=org.eclipse.jdt.core,\
    bnd.identity;id=org.eclipse.jdt.ui,\
    bnd.identity;id=vulcan-refactor

-runbundles: \
    org.eclipse.jdt.core;version='[3.35.0,3.35.1)',\
    org.eclipse.jdt.ui;version='[3.30.0,3.30.1)',\
    vulcan-refactor;version='[1.0.0,1.0.1)'

Add Dependencies in pom.xml: Ensure all required OSGi bundles are included. You can find the complete pom.xml file here.

<dependencies>
    <dependency>
        <groupId>org.eclipse.jdt</groupId>
        <artifactId>org.eclipse.jdt.core</artifactId>
        <version>3.35.0</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jdt</groupId>
        <artifactId>org.eclipse.jdt.ui</artifactId>
        <version>3.30.0</version>
    </dependency>
    <!-- Add other dependencies as needed -->
</dependencies>

Setting Up the Workspace

To set up the workspace, you need to create or open an existing workspace and load the project. You can find the complete implementation here.

import org.eclipse.core.resources.IProject;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.JavaCore;

public class ProjectUtils {

    public static IJavaProject getJavaProject(String projectName) throws CoreException {
        IProject project = ResourcesPlugin.getWorkspace().getRoot().getProject(projectName);

        if (!project.exists()) {
            project.create(null);
        }
        project.open(null);

        IJavaProject javaProject = JavaCore.create(project);
        return javaProject;
    }
}

Accessing Project Information

Once the project is loaded, you can access information like source directories and classpaths. You can find the complete implementation here.

import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.JavaModelException;

public class ProjectInfo {

    public static void printProjectInfo(IJavaProject javaProject) throws JavaModelException {
        System.out.println("Source folders:");
        Arrays.stream(javaProject.getPackageFragmentRoots())
              .filter(root -> root.getKind() == IPackageFragmentRoot.K_SOURCE)
              .forEach(root -> System.out.println(root.getPath()));

        System.out.println("Classpath entries:");
        Arrays.stream(javaProject.getRawClasspath())
              .forEach(entry -> System.out.println(entry.getPath()));
    }
}

Handling Internals

Some refactorings do not expose a public API, and you may need to dig into internals. For example, to perform a rename refactoring, you might need to use internal classes and methods. You can find the complete implementation here.

import org.eclipse.jdt.core.IType;
import org.eclipse.jdt.core.refactoring.descriptors.RenameJavaElementDescriptor;
import org.eclipse.ltk.core.refactoring.RefactoringContribution;
import org.eclipse.ltk.core.refactoring.RefactoringCore;

public class RenameTypeRefactoring {

    public RenameJavaElementDescriptor createRenameDescriptor(IType type, String newName) {
        RefactoringContribution contribution = RefactoringCore.getRefactoringContribution(IJavaRefactorings.RENAME_TYPE);
        RenameJavaElementDescriptor descriptor = (RenameJavaElementDescriptor) contribution.createDescriptor();
        descriptor.setProject(type.getJavaProject().getElementName());
        descriptor.setJavaElement(type);
        descriptor.setNewName(newName);
        descriptor.setUpdateReferences(true);
        return descriptor;
    }
}

Conclusion

I decided to share Vulcan now in case anyone finds it interesting or useful. Check it out on GitHub: Vulcan.

Cheers!