DEV Community: anilkulkarni87

Treasure data digdag visualization

anilkulkarni87 — Fri, 25 Mar 2022 00:02:42 +0000

Treasure data is a leading CDP platform and leverages dig-dag for data pipelines within itself. digdag is similar to airflow where we define tasks inside a dag
and is available as an open source. The CDp is a powerful platform but unfortunately no easy way to visualize the pipelines which makes it hard for onboarding
when there are some complex pipelines already built. So I had the below sequence of steps to solve this problem for myself.

Solution Approach

Create Visualization
Make it colorful
Update the graph everytime there is a code change.
Make it part of CICD

Navigate to html

Installation

pip install -r requirements.txt

Usage example

Update Line 131 with the project where the graphs need to be updated.
Execute the python program to generate the graphs.

TODO

Parametrize the dir_name (Line 131)
Create github workflow to generate graphs when there are changes.
- Identify the changes at the folder level
- Execute the python script tp build graph for each folder with a change
- Commit the graphs back to the branch

References

https://github.com/y-abe/digdag-graph

Airflow Api tests

anilkulkarni87 — Mon, 09 Aug 2021 16:54:13 +0000

airflow-api-tests

This is a collection of Pytest for the 2.0 Stable Rest Apis for Apache Airflow. I have another repo where you could setup airflow locally and play around with these. I am used to RestAssured, but trying out pytest here.

Apache Airflow 2.0 Stable Rest Api calls - Python

I am used to RestAssured for api testing. Trying out python api testing with Airflow stable rest api

📝 Table of Contents

About
Uses
Getting Started
Usage
Running the tests
Github Workflow
Airflow Apis
Authors
Acknowledgments

🧐 About

This repository will contain api calls to stable Airflow Rest apis. Having worked in airflow, I felt the lack of it. Come Airflow 2.0, it resolved the issues for me.
It is also an attempt to understand api testing with python as I usually prefer Rest Assured (Java). This is a WIP repo as I continue to build and add calls to all apis.
I have another repo which can help people setup airflow locally and then play around with these api.

Uses

Listing out some reasons why I created this for myself:

Learn about airflow apis.
Learn about Python api testing.
Understand use cases of api and document for future needs.

🏁 Getting Started

Sequence of steps to be followed to be able to use this successfully:

Clone the airflow-docker repo.
Follow the instructions there to start running airflow locally.
Execute some DAGS that are part of my repo or create your own DAGS.
Start playing with the apis.

🎈 Usage

🔧 Running the tests

Here is an example of how you could run the tests. I will continue to evolve this

pytest test_dag.py

Github Workflow for running tests

I added this step for me to understand more about github workflows and how i can leverage it for this specific usecase. Essentially what I will have to do is within the workflow:

Clone airflow-docker repo
Start Airflow
Run these tests
Do something with the results

⛏️Airflow APIs

I know you could get your hands on the swagger doc for these. But I still wanted to list down here.

Get Config
- This is usually forbidden from the administrator owing to security reasons.
Connection
- Get connections : To list all connections currently available
- Post : To create a new connection Id
- Patch : To patch means to update an existing connection Id
- Delete : To delete a connection
- Get Connection Id : Get details of a specific connection based on the id
DAG
- Get Dag Source code : We can get the dag source code by passing a file token, We can get file_token by calling the get dag by dag id
- Get DAGS : Able to get all DAGS
- Get DAG Info : Get basic info about a DAG
- Patch : This is to update the dag. You can refer the example I have.
- Post - clear tasks instances of a DAG : #TODO
- Get Dag details : Get Basic info about dags
- Get Dag tasks : Get all tasks for the dag.
- Get Task details : Get a representation of the task
- Post UpdateTaskInstanceState : #TODO

✍️ Authors

@anilkulkarni87

Publish blog posts from GIT to dev.to

anilkulkarni87 — Sat, 03 Jul 2021 01:09:58 +0000

Publish blog posts from GITHUB to dev.to

This is a template repo which where I am testing and building solution for automated publishing of my articles/blog posts to dev. It's just a lot of work to post the same article across places and hence I decided to automate publishing my posts to dev and eventually any other personal blogs.

Dev exposes API through which you could do a lot of things. I am currently only creating articles which will be in your drafts.

Generated via code(Python)

Approach

Clone this repo, cleanup the files and arrange your blog posts.
Create a markdown file for your blog post. You can check the folder structure in the repo.
Setup your DEV TOKEN in your repo with the name as GIT_TO_DEV
Commit the changes made to your .md files.
Voila after the workflow is complete, you can see the blog posts created in draft.
Also, I am persisting the json payload of the article with the id populated.

Next steps

Identify new files and modified markdown files
For new files, create new articles
For modified, update the articles

Thanks

Ana María Martínez Gómez - Tracking changed files
Stefan Zweifel - Committing the changes file from a workflow
Maxime - For the triggering a thought
Chris Amico - For parsing front matter from Markdown file

Bugs/Changes

Please modify any typos or corrections and create a pull request to make this better.

How to create LogicalType in Apache Avro

anilkulkarni87 — Mon, 21 Dec 2020 20:24:37 +0000

I am writing this post to understand how to create a custom LogicalType in Apache Avro. It helps in enforcing strict schema validations for the data in transit as well as rest within their systems. Dividing this into broader sections:

Define a schema using Avro IDL
Generate Java classes using gradle avro plugin
Define Custom Logical Type
Test the Custom LogicalType
Next Steps (There is always a better way)

Define schema using Avro IDLLets go over in detail how to create a Custom LogicalType in Apache Avro.

Define a schema using Avro IDLAny simple to complex schema can be defined easily using Avro IDL. Maven and Gradle plugins which aid in generation .avsc files and java classes for the defined schema. Below is a sample schema defined by me. Read more about Avro IDL here. As a standard practice we usually define default value for our fields to accommodate for backward compatibility whenever schema changes.
Generate Java classes and schema file Gradle avro plugin to our rescue. The plugin has been updated many times. You might notice that what I have is an older version. The first task creates Avro protocol files, next task created schema files (.avsc files), The final task created the required java classes. The complete build file can be found here.
Define Custom LogicalTypeA logical type is an Avro primitive or complex type with extra attributes to represent a derived type. This basically is a custom defined type and can be leveraged for many use cases. Notice the “reversed” keyword in the .avdl file in Step 1. We need to define the LogicalType and the custom conversion for it. For example, What I am doing here is appending the data ingested as part of ‘queryAuthor‘ field with the word ‘reversed‘. Use Cases: -Create a type for encryption which encrypts the data ingested as well as decrypts the field later. -Create definition for Address (Line1, Line2, Line3, State etc ) and use that as Custom Logical type. -Choose to do a one way hash of the data ingested. You can find the LogicalType and Conversion class here: Logical Type: Click here Conversion: Click Here

Another important steps is to register this Logical Type which i am doing in the main method.
public class AvroExample { public static void main(String[] args) { LogicalTypes.register(ReversedLogicalType.REVERSED_LOGICAL_TYPE_NAME, new LogicalTypes.LogicalTypeFactory() { private final LogicalType reversedLogicalType = new ReversedLogicalType(); @Override public LogicalType fromSchema(Schema schema) { return reversedLogicalType; } });
What I am doing in the Custom conversion is self explanatory if you look at the code below.

Test the Custom LogicalTypeMy approach to test the Logical Type is: a) Write data to .avro file. b) Read the data in the file. c) Read the files using avro tools Better explained with some screenshots: In the main method I am creating an avro file and we shall see the value before the data is saved as .avro file. Then we read the data from the file and print it before conversion happens.

We also see the data in the avro file using avro-tools.

Next Steps or TODO: There is always a better way to do it. I am going to work on these and probably come up with another post – Github link is here. – Leverage latest version of plugin to register the LogicalType as part of the build. – Notice the TODO in QueryRecord.java. – Explore modifying velocity templates to include the LogicalTypes in the auto-generated code. – Some of my other learnings.

The post How to create LogicalType in Apache Avro appeared first on Anil Kulkarni | Blog.

Selenide - UI tests in minutes

anilkulkarni87 — Mon, 09 Mar 2020 22:09:48 +0000

Selenide – UI tests in minutes

Selenide, is what I discovered as part of a new assignment at work. It involved testing of a product management application, majority of it a web based UI.

I took a week to get a deeper understanding of the assignment, both in terms of technical and business. Although it said automation is out of scope for now, I wanted to make my work easier. My hunt for suitable tools ended with Selenide.

It took be around 4 days to automate considerable amount of scenarios which could be presented as ROI or effort savings. Setting up the project with right tech stack took the most time. I spent around 1.5 days in writing tests.

Selenide – For UI tests
Maven – Build tool
Junit5 – For writing tests
Allure – For reporting
Docker* – For generating an emailable report.

The intent of this post is to share the project setup so that, if anyone else is looking for something similar, they can avoid the effort to set the project up. You can find this in on my github .

Few thing that I learnt while working on this are listed here :

Ability to log custom messages in an allure report.
Ability to use excel sheet as test data input.
Java Enum
Junit5 parametrized test.
Understanding of tags in Junit

Selenide has definitely worked very well for my use cases and would definitely recommend for people to try. There are a lot more things that I would like to learn/understand.

Better way of handling Test data.
Leveraging picocli or other argument parsers for executing specific tests/modules.

The post Selenide – UI tests in minutes appeared first on Anil Kulkarni | Blog.

Json Schema and Json Validation

anilkulkarni87 — Fri, 14 Feb 2020 07:51:20 +0000

A sample json schema
A discussion in one of my office meetings, led me to think and write about Json Schema. Every post I write is an attempt to get a better understanding of the topic and keep that as a reference for my future projects.

How do you validate the complex nested Json data files?

One approach, I can think of is defining a schema and validating the JSON data against it. I had used this approach to validate a avro schema.

Define JSON Schema:

This is the most time consuming part but if done right, the JSON validation is cakewalk. Json Schema came to my rescue. After reading, I was able to understand most of the concepts and write my own schema and also validate a couple of data files. Below are some of the things you can define/achieve:

Mandatory values.
Nested objects and fields.
Data type and additional criteria:
- String can be of format:email/uuid etc.
- String can match with pattern.
- Integer – Can have max , multipleOf.

JSON Validation

I leveraged a gradle plugin to implement the validation as part of the build. So that our next steps of the build are executed only if the Validation task is successful. The errors printed are very precise and easy to understand. You can find this sample project on my github.

Validation Output

Next steps and other thoughts:

To add examples of different schemas Eg: Use definition and references.
One can also generate schema from a json and then improvise it.
For the first time, I used github actions to build the project.
I should write more about some of my other learnings.

The post Json Schema and Json Validation appeared first on Anil Kulkarni | Blog.

How to view Spark History logs locally

anilkulkarni87 — Sat, 04 Jan 2020 00:28:01 +0000

How to: View Spark History logs locally

Spark History logs are very valuable when you are trying to analyze the stats of a specific job. When in cases where you are working with a large cluster , where multiple users are executing jobs or when you have an ephemeral cluster and you want to retain your logs for analysis in future, here’s a way to do it locally.

How do you analyze the spark history logs locally?

Download logs from Spark History Server

The most important thing here and the first step is to download the spark-history logs from the UI before your cluster goes down.

Spark History Server

Setup Spark History Server Locally

Below steps would help in setting the history server locally and analyze the logs.

On a MacOs : brew install apache-spark
Create a directory for the logs.
Move the downloaded logs in the previous step to the logs directory and unpack them.
Create a file named log.properties
Inside log.properties, add spark.history.fs.logDirectory=<path to the spark-logs directory>
Navigate to /usr/local/Cellar/apache-spark/<version>/libexec/sbin
Execute sh start-history-server.sh --properties-file <path to log.properties>
Navigate to http://localhost:18080 on browser.

Now you can view and analyze the logs locally.

The post How to view Spark History logs locally appeared first on Anil Kulkarni | Blog.

Learnings from my last assignment

anilkulkarni87 — Fri, 03 Jan 2020 06:28:22 +0000

Learnings from my last assignment

The year 2019 as well as my previous work assignment, has been very interesting, challenging and educating. I happen to work on different technologies and domains.
Here I am jotting down the learnings from my last assignment. While I learnt many things, one of the most important lesson to me goes in sync with a great man’s words “All I know is, Nothing“.

Standards

Importance of Coding and Logging standards for an organization or at-least a team. By defining those standards upfront, the effort saved as the application grows bigger was quite evident to me.
Importance of defining metrics and having a clear distinction between application and business metrics.
Identifying standards for publishing events and consuming them.
Identifying data storage formats, I worked mainly on Apache Avro. The advantages it brings when there is a Schema Evolution.
Importance Encryption and Tokenization standards for the data we deal with. You know privacy is no more a wishlist, but a law. It also made me understand difference between the two terms used here.

Apache Avro

Importance of defining a schema/model for the data we deal with and how Avro enforces certain data quality checks in the pipeline.
I also understood what schema evolution is and how its needs to be a planned move. Enforcing schema compatibility checks before changing them.
Avro IDL makes it super easy to define/design schemas for the data we deal with.
Challenges involved with Union and complex union types.
Impacts of breaking schema changes on production systems and probable solutions to handle that.
Defining our custom Logical type as part of Avro. Example : Encryption and Tokenization.

Kafka

I learnt a lot about how Kafka can fit into some of the applications, especially when its event based.
Difference between System and Business events.
Leveraging Schema registry to enforce checks while producing and consuming messages from topic.
Kafka headers and how that can be leveraged in the pipeline.
Producing data with multiple schemas versus single schema to a single topic.
Importance of metadata (Data about data).
Kafka Connect and use cases around it.
A little bit about Kafka Streams.

Spark

Learnt Scala (just enough for Spark), build and execute jobs by leveraging Livy.
Understand about partitioning, re-partitioning, data shuffles.
Consuming messages from Kafka in batch mode.
Learnt few things about the executor, executor cores and memory management in Spark.
Spark History, Zeppelin notebooks for Spark.
Unit testing in Spark and its importance.

Presto/Hive

Presto is nothing but SQL, but all the processing in-memory.
Writing UDF for Presto
Hive QL.

Airflow

Writing workflow as code and understanding of how Airflow works.
Creating custom Operators.

Docker and Kubernetes

How docker can help us have $0 infrastructure cost during development and Unit testing.
Kubernetes is still a partially known area. I learnt about accessing the pods, managing secrets, YAML files.

Design Patterns – Java

From reading about Design patterns to actually to see it being used.
Importance of Unit Testing and Code Reviews.

Domains

I have listed some of the domains that I have worked on. If one cannot traverse across domains for a specific Customer, then we are hardly making use of the richness of the data.

Customer – I was able to understand the different challenges a company will have in dealing with Customer data.
Sales – All the different attributes related to a transaction and why its critical to have them available in the system Near Real time.
Preference – For people who work under marketing, the preference of customer plays a great role and with more laws around them, its important to have them updated and be available to the marketing teams.
Loyalty – The success of this program can only be measured when the company can leverage this data to its benefit.

This is a brain dump of the previous assignment I worked on but it will also serve as a reminder of all the learnings as well as the unknowns. I also plan to write other posts related to some of the topics listed here as I truly believe in “To teach is to learn twice“.

The post Learnings from my last assignment appeared first on Anil Kulkarni | Blog.

How to Create a UDF in Presto-1

anilkulkarni87 — Thu, 21 Mar 2019 08:33:25 +0000

How to Create a UDF in Presto-1 This article is going to talk about how we will create a UDF in Presto. When i started looking for creating an UDF, all resources i found spoke about Maven build. But, the one i created is via a gradle build. This Article will cover the below: Project… Read More »

The post How to Create a UDF in Presto-1 appeared first on Anil Kulkarni | Blog.