DEV Community: Ry

Testing Angular 10 Interceptors with Spectator and Jest

Ry — Wed, 23 Sep 2020 13:42:28 +0000

There are many reasons to switch from Karma and Jasmine to Jest when Testing Angular:

Jest runs faster than Karma and Jasmine
Jest supports snapshot testing
Jest runs tests in parallels
Jest does not require a browser for testing
many more...

However, what's missing are examples of how to write Angular unit tests in Jest, particularly testing Angular HTTP Interceptors.

Setting up Angular, Spectator, and Jest

For the purpose of this article, we will assume that you have an Angular project already set up with Spectator and Jest. If not, I will provide you with some links on how to setup Angular with these libraries.

Jest

While the focus of this post is NOT on how to convert Angular from Karma and Jasmine to Jest, below is a list of resources on how to do this conversion yourself. You can also use my Github project as a template. I should mention that Jest can be a bit quirky if you are used to using other testing frameworks, but these quirks are worth it.

Spectator

Spectator is an amazing library that reduces the wordy boilerplate code for setting up Angular Unit Tests to only a few lines. It has a few quirks that are absolutely worth it for the value it provides,

A Couple of things

The major thing to keep in mind when using Spectator and jest together is that Specator imports should come from the @ngneat/spectator/jest package.

import {createHttpFactory, HttpMethod, SpectatorHttp} from '@ngneat/spectator/jest';

Below is the final devDependencies section of package.json.

{

  …

   "devDependencies": {

    "@angular-devkit/build-angular": "~0.901.7",

    "@angular/cli": "~9.1.7",

    "@angular/compiler-cli": "~9.1.9",

    "@ngneat/spectator": "^5.13.3",

    "@types/jest": "^26.0.13",

    "@types/node": "^12.11.1",

    "codelyzer": "^5.1.2",

    "jest": "^26.4.2",

    "jest-preset-angular": "^8.3.1",

    "protractor": "~7.0.0",

    "ts-node": "~8.3.0",

    "tslint": "~6.1.0",

    "typescript": "~3.8.3"

  }

}

Angular 10 Interceptor Unit Test

For this example, we will be testing an Http Interceptor that logs HttpErrorResponses to the console.

import { Injectable } from '@angular/core';
import {
  HttpRequest,
  HttpHandler,
  HttpEvent,
  HttpInterceptor, HttpErrorResponse, HttpResponse
} from '@angular/common/http';
import { Observable, throwError} from 'rxjs';
import {catchError, tap} from 'rxjs/operators';

/**
 * Intercepts HttpRequests and logs any http responses of 3xx+
 * In the future we can make this a conditional retry based on the status code.
 *
 */
@Injectable({ providedIn: 'root' })
export class HttpErrorInterceptor implements HttpInterceptor {

  constructor() {}

  intercept(req: HttpRequest, next: HttpHandler): Observable> {
    return next.handle(req).pipe(tap(() => {}),
      catchError((error) => {
        if (error instanceof HttpErrorResponse) {
          if (error.error && error.error.message) {
            console.log('status: ' + error.status + '\nmessage: ' + error.error.message);
          } else {
            console.log(error);
          }
        }
        return throwError(error);
      })
    );
  }
}

What this code does is intercept an HttpRequest from the application and logs the response to the console when an HttpErrorResponse is returned. The HttpHandler is used to execute the request next.handle. Then we create a pipe in order to tap the response for processing. Note: tap is a rxjs pipe function that allows us to inspect the data without changing the actual data in the pipe.

In this case, we catch the HttpErrorResponse, allowing any non-error HttpResponse to pass through. Once the Response is caught we can inspect the error message and log it to console. Note in this case we are expecting a custom body in the HttpResponse.

The Unit Test

In this unit test, we will be checking that a response with a 2xx will pass through and that an Error Response will be thrown. For more advanced testing the console could be mocked and we can check that the console.log has been called. This is out of scope for this article.

import {HttpErrorInterceptor} from './http-error.interceptor';
 import {createHttpFactory, HttpMethod, SpectatorHttp} from '@ngneat/spectator/jest';
 import {async} from '@angular/core/testing';
 import {of, throwError} from 'rxjs';
 import {HttpErrorResponse, HttpRequest, HttpResponse} from '@angular/common/http';
 describe('HttpErrorInterceptor', () => {
 let spectator: SpectatorHttp<HttpErrorInterceptor>;
   const createHttp = createHttpFactory({
     service: HttpErrorInterceptor
   });
 beforeEach(() => {
     spectator = createHttp();
   });
 test('Http error', async(() => {
     const mockHandler = {
       handle: jest.fn(() => throwError(
         new HttpErrorResponse({status: 500, error: {message: 'This is an error'}})))
     };
     spectator.service.intercept(new HttpRequest<unknown>(HttpMethod.GET, '/thing'), mockHandler)
       .subscribe((response) => {
         fail('Expected error');
       }, (error => {
         expect(error).toBeTruthy();
       }));
 }));
 test('Http success', async(() => {
     const mockHandler = {
       handle: jest.fn(() => of(new HttpResponse({status: 500})))
     };
     spectator.service.intercept(new HttpRequest<unknown>(HttpMethod.GET, '/thing'), mockHandler)
       .subscribe((response) => {
         expect(response).toBeTruthy();
       }, (error => {
         fail('Expected Successful');
       }));
   }));
 });

The key here is 1) how the handler is mocked and 2) and how we test the interceptor response.

Mocking the HttpHandler

The first confusing thing when testing the interceptor is how to mock the HttpHandler. Since Jasmine is removed mock and SpyOn are off the table. You may notice that jest.mock exists, but it doesn't function as expected. This is one of those little Jest quirks I mentioned; jest.mock is used to mock a package and not an object. In this case, we will build an object that looks like HttpHandler interface and mock the methods expected. Below is the HttpHandler interface. As you can see it only has one method.

export declare abstract class HttpHandler {
     abstract handle(req: HttpRequest): Observable<HttpEvent<any>>;
 }

This is easily mocked with jest.fn()

const mockHandler = {
       handle: jest.fn(() => throwError(
         new HttpErrorResponse({status: 500, error: {message: 'This is an error'}})))
     };

In the error case, we will instruct the method to throw an HttpErrorResponse and create a custom object for the response body/error.

In the happy path case the mock looks like the following:

const mockHandler = {
       handle: jest.fn(() => of(new HttpResponse({status: 200})))
     };

Testing the Interceptor's Response

Now that we have the HttpHandler mocked, how do we actually test that the interceptor does anything? The key here is to specify an input on the .subscribe lambda.

spectator.service.intercept(new HttpRequest<unknownn>(HttpMethod.GET, '/thing'), mockHandler)
       .subscribe((response) => {
         expect(response).toBeTruthy();
       }, (error => {
         fail('Expected Successful');
       }));

In this case we are checking that that the interceptor passed the response through as normal, and did not throw an error.

Spectator and Unit Testing Fiddly Bits

Some might note that the code is using spectators createHttpFactory instead of createServiceFactory. In this scenario, both will work exactly the same. I'm using createHttpFactory in anticipation of adding an HTTP retry.

It is also important to note that this interceptor doesn't actually modify the Response and the tests are a bit weak. This is meant to be a basic framework to get you started with testing interceptors. If you have an interceptor that modifies the HttpRespond using map, you will be able to specify the input using the mocked HttpHandler and test the output in the subscribe portion of the interceptor call.

Summary

Using Spectator and Jest with Angular 10 is a very powerful combination. The trick is to either have a full understanding of Jest and Spectator, or have a ready source of examples to draw from. I hope this article can provide you a rough understanding of how to use Jest in concert with Spectator to test Angular HttpInterceptors. The keys here are

Using jest.fn() to mock the function of the HttpHandler
Adding the input variable to the subscribe lambda for testing

Github source: https://github.com/djchi82/angular-jest-spectator-interceptor-test

Hacking your To-Do List for More Productivity and Fulfillment

Ry — Wed, 27 May 2020 13:17:11 +0000

In the previous article of this two-part series Hacking Your Life with Task Types. I discussed the four quadrants of the Eisenhower Matrix and how to use it to determine a task's Importance and Urgency. In this article, I will introduce you to a few task management strategies that I've learned and implemented to ensure I'm doing what is most important to me. These strategies are the

Capture Sheet
Chunking
Life Categories.

Capture Sheet

The capture sheet is much like a to-do list, but expands it, allowing it to be used as a base for planning. Most to-do lists are typically short-termed or project-focused. They are things that need to be done soon and often are filled with urgent tasks. A capture sheet will contain all tasks, plans, and ideas for all your projects and timelines.

Why is having a capture sheet important?

Capture sheets keep us from forgetting tasks that we might think are important to accomplish in the future. It doesn't matter how good of a memory we think we have, we can only juggle so many thoughts at once. Keeping everything in our heads might cause us to forget about an important but less urgent task.

Keeping everything in our heads also puts unneeded stress on our brain, which uses energy. This impacts our capacity to think clearly and concentrate. The first time I did a capture sheet a sense of euphoria came over me. It was like a weight was lifted and I could be more present with the immediate tasks.

Capture sheets give us a platform for organizing, planning, and chunking our tasks.

What is a capture sheet?

A dumping ground: There are no bad thoughts or tasks on a capture sheet. All ideas are valid, because you thought of it.
Impermanent: Because the capture sheet is an idea dumping ground, it can be thrown away or altered. Expect to have many iterations of capture sheets, either weekly, monthly, or even versions of the same sheet.
Easy to access: The purpose of the capture sheet is to get the idea or task out of your head as quickly as possible. Phone apps are generally a go-to for easy access, but personally I use a journal that I keep close by. A journal allows me to easily see many items and allows me to write notes, timestamps, levels of importance, and categories. Sometime I'll use my phone to capture a thought if my journal is not nearby and I transcribe the idea to my journal when I get back home.
Confidence booster: Keeping old capture sheets and reviewing them over time allows us to see how far we come. It tracks our thoughts and shows us how we've evolved as we worked through important tasks.
Satisfying: Let's face it, physically scratching items off a list is satisfying.

Chunking

Now that we have a list of items we need a way not to feel overwhelmed with all we have to do. Chunking is a great way to handle the overwhelm. Chunking allows us to group tasks into bite-size pieces that we can plan and work with.

How to Chunk

There are many ways to chunk tasks that will work differently for different people, situations, and projects. Sometimes chunking can be a combination of approaches, and the same task can exist in different chunks. The important thing to remember when chunking is to keep it as simple as possible. Here a few ways to chunk tasks.

Project: Chunking tasks by the project or goal is the most common way to get started. This allows us to see what needs to be done in order to reach an objective. Once tasks are grouped by the project they can be ordered in sequence, importance, or urgency. A trick I employ when organizing a project chunk is to say to myself "What task can I do that will make all other tasks easier or irrelevant." This will help you make the most out of your precious time.
Importance: Sometimes chunking tasks by importance is a good way to see what task will be the most impactful on your life. Often this is a great way to organize tasks across multiple projects in order to plan your immediate schedule.
Urgency: This is a consequence-based way to look at tasks. It is also a way to organize tasks across multiple projects.
Life Categories: Life categories are areas or your life that are important to you and typically consists of multiple projects and goals. They can be thought of as a larger chunk of goals. Life categories will be discussed later in this article.

Up and Down Chunking

Up and down chunking is a way of organizing your tasks to make them detailed enough to accomplish without being overwhelmed. After grouping tasks on a project level, you may find that not all the tasks seem to be at the same level logically. For example, planning a trip to a foreign country. You may end up with a list of tasks like

Buy tickets
Buy travel clothes
Get passport renewed
Research trip
Buy guidebooks
Ask friends about the location
Research airfare
Plan activities
Book hotel

In the above list, there are a bunch of tasks that appear at different levels of the project. Items 5, 6, 7, and 8 can be seen as sub-tasks of "researching the trip" (item 4). Up chunking, these 4 items into a larger chunk make organizing the project easier. If we look at this project at larger chunks we might end up with

Research Trip
Book trip
Prepare for travel
Travel

This list of 4 items is easier to manage over the life of this travel project. Four items are less overwhelming than 9+ items. The fact that it is less overwhelming helps ensure that we will follow through with this project. As we begin to work through these four larger chunks, we can break each large chunk into small chunks to plan over the course of a day(s) or week(s).

Life Categories

Life categories is the idea that our lives are made up of many parts or categories that need to be managed. Many of these categories such as finances, career, and family, maybe obvious, but often we are not aware of all of our life categories. Often these unknown categories take the shape of multiple related goals. The fulfillment of these goals brings us a feeling of peace, joy, accomplishment, and fulfillment. Being able to identify them and plan meaningful (important) activities is the key to mitigating burnout and frustration.

For example, if you are solely focusing on career and finances and begin to feel burnt out, it might be because you may not be honoring your need for relationships or physical health. Often we can be so locked into a particular goal that we deem everything else as unimportant. So we grind. This causes us to obsess and burnout. If we step back and take care of other needs we find the obtaining the main goal will become easier and more enjoyable.

So how could this manifest? In the previous article I mentioned making time for important, non-urgent tasks. Sometimes we label a task as not important, such as playing a board game or going for a swim, but in actuality, it is important to us, because it brings us joy and fulfillment.

Summary

How we manage tasks can lead to a more fulfilled and productive life. Especially if we take a holistic approach to how we manage the tasks and our life. Utilizing the Einshower Matrix, Capture Sheets, Chunking, and Life Categories are a few great approaches. However, there are many more strategies, and often how we use and approach these strategies may change with time.

Finding Diacritics in a Postgres Table

Ry — Fri, 20 Mar 2020 13:39:05 +0000

Recently I had the task to find all the Diacritics in a database table. I had to find an easy way to select every row that has a diacritic in the name. The solution was surprisingly simple, but a bit unintuitive. In this article, I will provide you with that solution as well as additional information about how Java and Postgres treat diacritics and other characters.

What is a diacritic?

A Diacritic is a mark or an accent on a letter that indicates that the letter should be pronounced differently. Diacritics are often referred to as accents. The marked letter and the un-marked letter are logically the same letters in the language. Examples of Diacritics are

áàâüñçå

It is important to understand that some characters that we might consider diacritics are not actually diacritics. This is because, In the character's respective alphabet, they are individual letters. Examples are

Æ æ Ø ø Ł ł

When dealing with diacritics its important to understand how the system will treat certain characters.

How does Java deal with Diacritics?

Java handles diacritics in a strict fashion. For these examples, we will use the Normalizer.normalize utility in the Apache Commons-text library. We will see that "Æ æ Ø ø Ł ł", are not replaced with a similar English analog.

We will use the following method for testing

public static String stripDiacritics(String input){
    if ( !Normalizer.isNormalized(input, Normalizer.Form.NFD) ) {
        return Normalizer.normalize(input, Normalizer.Form.NFD).replaceAll("\\p{M}+", "");
    }
    return input;
}

Testing this method will result in the following

 @test

public void testStripDiacritic(){
    assertEquals("aabbsser3ssa35ds", DiacriticUtility.stripDiacritics("aabbsser3ssa35ds"));
    assertEquals("This is a funky String", DiacriticUtility.stripDiacritics("Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ"));
    assertEquals("Ø or Ł", DiacriticUtility.stripDiacritics("Ø or Ł"));
    assertEquals("the German umlauts: a, o, u", DiacriticUtility.stripDiacritics("the German umlauts: ä, ö, ü"));
    assertEquals("manana", DiacriticUtility.stripDiacritics("mañana"));
    assertEquals("Æ æ Æ æ or Æ æ", DiacriticUtility.stripDiacritics("Ǣ ǣ Ǽ ǽ or Æ æ"));
}

Notice diacritics such as "ůňķŷ" are replaced with "unky", but "Æ æ" are not replaced with "AE ae"

How does Postgres handle diacritics?

Postgres treats diacritics the same as Java but has a different effect on the other characters. In Postgres, we use the unaccent extension.

To install unccent run the following.

CREATE EXTENSION unaccent;

Now let's run a simple query to see how characters are treated.

select unaccent('Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ | łŁØø | ä, ö, ü | Ǣ ǣ Ǽ ǽ æ or Æ æ') 
FROM blog.diacritic_blog LIMIT1;

Output:

This is a funky String | lLOo | a, o, u | Ǣ ǣ Ǽ ǽ ae or AE ae

Notice that the expected diacritics are resolved, but so is "Æ æ ł Ł Ø ø". Interestingly enough is the Postgres's unaccent does not resolve "Ǣ ǣ Ǽ ǽ" to "Æ æ Æ æ" or to "AE ae AE as".

Finding diacritics in a Postgres Table?

As we saw above there are a lot of special cases regarding how Postgres will handle diacritics and special characters. This solution to finding the rows with diacritics will not take those edge cases into account but will give you a basic solution to querying the table for diacritics. You can add regex and additional where clauses to the query as needed.

This example will use the following

CREATE TABLE blog.diacritic_blog(
  id bigserial NOT NULL,
  name varchar(255) NOT NULL,
  CONSTRAINT pk_diacritic_id PRIMARY KEY (id)
);

INSERT INTO blog.diacritic_blog(name) values ('aabbsser3ssa35ds');
INSERT INTO blog.diacritic_blog(name) values ('Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ');
INSERT INTO blog.diacritic_blog(name) values ('Ø or Ł or łŁØø');
INSERT INTO blog.diacritic_blog(name) values ('the German umlauts: ä, ö, ü');
INSERT INTO blog.diacritic_blog(name) values ('mañana');
INSERT INTO blog.diacritic_blog(name) values ('Ǣ ǣ Ǽ ǽ æ or Æ æ');

The following query will return 5 rows:

SELECT name, unaccent(name) FROM blog.diacritic_blog where name != unaccent(name);

OUTPUT:

Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ	This is a funky String
Ø or Ł or łŁØø	O or L or lLOo
the German umlauts: ä, ö, ü	the German umlauts: a, o, u
mañana	manana
Ǣ ǣ Ǽ ǽ æ or Æ æ	Ǣ ǣ Ǽ ǽ ae or AE ae

Summary

Finding all the Diacritics in a Postgres table is simple if not a bit unintuitive. However, handling diacritics and non-English alphabet characters is a minefield. Hopefully, now you have a basic understanding of diacritics and how Java and Postgres treat these characters.

If you'd like to see a complete treatment of how other languages (JavaScript or Python) treats these character please reach out to me via Twitter @ryboflavin42 and let me know what other languages you'd like to see.

The examples from this article can be found on GitHub at https://github.com/djchi82/DiacriticBlogProject.

Hacking Your Life with Task Types

Ry — Tue, 03 Mar 2020 21:08:29 +0000

I want to share an idea that has changed my life by mitigating burnout and feeling more fulfilled day-to-day, month-to-month, and year-to-year. The idea is task typing.

In my next two blog posts, I will introduce you to the idea and show you a few ways to use it to improve performance, happiness, fulfillment, and productivity.

4 Types of Tasks

In essence, there are 4 categories that every task falls into. Those categories and formed by two-axis; importance and urgency. This is sometimes called the Eisenhower Matrix. In the figure below the x-axis describes a task's urgency and the y-axis describes a task's importance.

Quadrant I: Urgent - Important Tasks

A task that is Urgent and Important often consumes our life. They show up as urgent requests from our bosses, loved ones, and the unexpectedness of life. These are drop everything tasks and do it now. They cause us to lose our rhythm. Often we perceive dire consequences if these tasks aren't done immediately or in a timely matter. We often describe these tasks as "firefighting".

An example of an urgent - important task might be the production system is down due to a bug. We need to drop everything, get production running, and fix the bug. In our personal life, this might be an unexpected bill or an accident.

Quadrant II: Non-Urgent - Important Tasks

A task that is non-urgent - important is one that we deem important, often because it moves us towards our goals. However, these tasks often take a backseat to more urgent tasks. In many cases, these tasks are not done because they take energy to complete and we are tired from all the "firefighting", so we look at Quadrant III or IV task to feel good about completing something or for distraction.

An example of a non-urgent - important task, might be refactoring a component that is slowing the system down, or creating a report feature for the user to free up operations time. In our personal lives, these show up as tasks we need to complete in order to complete a goal.

Quadrant III: Non-Urgent - Non-Important Tasks

Quadrant III tasks are tasks that distract us from our objectives. These tasks are things like pointless meetings, unnecessary paperwork, playing a game. We often use these tasks for distractions or instant-gratification. Non-urgent - non-important tasks are tasks that if we don't do then they will have little impact on the overall project or vision.

An example of this task might be creating a meeting to discuss a feature and bringing in people that are not needed for the task. In our personal life, these tasks can show up as endlessly cleaning or watching tv.

Quadrant IV: Urgent - Non-Important Tasks

Urgent - non-important tasks are tasks that have a deadline but have little consequences to them if they are not completed.

An example of this task might be replying to emails or chat messages. In our personal life, they might show up as chores.

Where We Live

Most of us live in Quadrants I and III. Either we are handling things that need to happen right now, else there will be consequences, or we are living in distraction or exhaustion. The secret is to begin eliminating quadrant III and IV tasks to create space to complete more quadrant II tasks. Doing more quadrant II tasks allows us to get ahead of the things that cause quadrant I tasks, and feel more fulfilled and energized.

Understanding the 4 task types and how they may be showing up in your business or life is the first step to taking control and making changes. In the next post, I will describe some systems and philosophies to help eliminate quadrant III and IV tasks and create the space to start feeling more fulfilled and take back your time and energy.

Resources

Most of this content is adapted from third-party content.

4 Ways to Learn Object-Oriented Design

Ry — Sun, 23 Feb 2020 15:10:33 +0000

Looking back over 2019 I realized I've had the privilege to work on a diverse set of software projects:

Architecting and coding an Enterprise Transform and Load (ETL) project in Python and Django
Building an SFTP platform using AWS lambda, Node, and Vue
Adding major features and refactoring a long-term enterprise Java project

While reviewing these projects one common thing stuck out as to why these projects were considered a success, it was due to their object-oriented design.

A key feature of each of these projects was the loose nature of the specific requirements and how details of inputs and behaviors would change as we learned more about the domain. Having a good design made making changes and testing a breeze. For example, by designing the ETL pipeline to use a canonical, a requirement change in the Python ETL project that should have taken a week to implement could be made and tested in less than a day.

This got me thinking about how I learned object-oriented design. Some of the answers are obvious, but others are less so. I want to share these approaches with you.

1. Courses

My earliest object-oriented design training was from taking an employer-sponsored training course with Training Hott. The course covered the most common Gang of Four design patterns. It went on to teach how to organize a set of requirements into components and analyze them to see what design patterns could be used to build an application. To be honest, I don't remember much from this course, as I didn't have much opportunity to use the techniques and patterns in my daily job.

Today there are a ton of different online resources for learning design patterns. Going the online route is cheaper than live training and often is more hands-on, but requires self-discipline to finish the course. Lastly, the barrier to entry for creating an online course is low and sometimes the quality of the course may suffer.

2. Books

Another obvious way of learning object-oriented design is from books. Books are not only a great way to learn, but make great reference material when you've forgotten details and need a reminder. In my opinion, the secret of using books is to understand how the material is going to be presented and how you learn. The canonical design patterns book is the Gang of Four book, Design Patterns: Elements of Reusable Object-Oriented Software. However, it can be a tough read, especially for beginners.

Taken from: https://martinfowler.com/bliki/GangOfFour.html

In my experience, I found the Head First books an excellent place to start or to refresh yourself. They are broken down in digestible chunks with coding and thought exercises at the end of each section. Sort of like a Highlights for Coders. Two great books to start with are

O'Reilly also publishes great books on the subject and often teaches the subjects by building a sample application with the concepts.

3. Reading and Mimicking Code

One of my favorite methods of learning new design patterns is digging into the source code of my favorite frameworks and libraries. The Spring Framework has taught me more about design than any course or book. This method may be difficult for the beginner as some frameworks, such as Spring, are highly abstracted and it can be difficult to understand why classes are broken apart.

A great way to begin to dive into an abstract framework is to try and extend a portion of the framework. Often frameworks are set up to allow users to extend interfaces and abstracts in order to modify the behavior of a particular component. The Spring documentation is very detailed and often will walk the reader through the how and why of a component's design.

“Most of us find our own voices only after we’ve sounded like a lot of other people.”

― Neil Gaiman

Another way is to read and extend the code of your team. One of my biggest jump in understanding how to create reusable components was reading and mimicking the code of the smartest people on my team. Seeing how and why code is abstracted and asking them questions about the code will often give you insight into a different way of thinking and designing.

4. Refactoring

Picking up a refactoring task or refactoring your own code before you commit is a great way to improve your understanding of design patterns. As you read the code you can begin to see where code is duplicated, confusing, or hard to test. Refactoring causes you to remove yourself from the actual code and think abstractly about how it fits into the application. Asking questions such as

When this is refactored how can I verify it behaves the same way?
What tests are missing from the original code?
How can I componentize the code?
How can I eliminate multiple or nest ifs and loops?
How can I combine two methods or classes?
How can I test a particular piece a logic inside a long method or function with minimum set up?
If I was to support this code 3 years what would make this code easier to support?
If the requirements were to change or the business wanted to change the behavior, how can I make it configurable?

Conclusion

I believe that proper design allows developers to easily maintain, extend, change, and test code and is often overlooked in many components. On top of becoming a better coder and creating better code, I believe you also begin to appreciate coding aesthetically and get more satisfaction from the work.

Why do beginner hate Java so much

Ry — Fri, 04 Oct 2019 21:34:40 +0000

I'm getting to the point where I can start being considered an old guy. My primary language is Java. The more I engage with the community, the more I find dislike and frustration with Java. To me, it was my first serious language and I found it great as it told me everything I wanted to know about the new APIs and third-party libraries. It also didn't devastate my global environment by installing third-party libraries as would Node or Python. It's all a binary that you add to a classpath instead of installing it to the OS.

Working with JavaScript and Python I'm mostly frustrated by

magic string configurations
not knowing what is being passed into and returned by APIS
and not being able to trust third-party libraries

I'm curious

What frustrates you most about Java?
What frustrates you most about learning Java?
What is your perception about Java?

I'm really starting to think that it just needs an image overhaul on how you learn it and approach? I ask my self, would could I create to show its versatility and effectiveness as a backend language.

How to Debug Node Serverless Using JetBrains WebStorm

Ry — Thu, 03 Oct 2019 14:46:43 +0000

One of the most useful tools in a developer's quiver is the debugger. The debugger allows a developer to not only step through code and track down bugs, but it is useful as a way to profile data structures. I find the ability to profile data structures to be extremely useful when working with scripting languages such as Python and Node.

Recently I was working on a Node serverless project and had no idea what fields existed on the serverless lambda objects (event, context, callback). When I went looking how to debug serverless, I struggled to find a solution that detailed debugging serverless in JetBrains WebStorm. The following will get you started debugging node serverless using JetBrains WebStorm.

There are a ton of tutorials on how to install node, serverless, WebStorm, so I've assumed you've already taken care of that. For the purpose of this tutorial, we will be using macOS Mojave. Some locations may vary depending on your OS.

Create a new node configuration: In the toolbar click Run --> Edit Configurations...

2. Create a new node configuration by click + and then Node.js from the dropdown

Fill in the configuration details
- Name: Anything you want
- Working directory: This will default to the root of your project. Be sure it points to the directory with your serverless.js file
- JavaScript file: this should point to the serverless binary: Typically /usr/local/bin/sls If you do not know where sls is installed you can find it by typing which sls in the terminal
- Application parameters: 'offline'
  - Be sure to add any additional parameters you might need such as '-s local'

If you launch the Configuration as debug, the WebStorm debugger will automatically be hooked into the node process.

This debug configuration may be obvious to a seasoned node developer, but if you're a language transplant like me, you may need help getting started with debugging serverless using WebStorm. This configuration will definitely help you get started understanding the framework and squashing those pesky scripting bugs.

Reusability and Sustainability Using a Canonical - Enterprise Transform and Load Part 3

Ry — Thu, 11 Jul 2019 14:45:44 +0000

Overview

In the last article in my Extract Transform and Load (ETL) series, Submission Information Packages and High-level Validation, I covered the most important part of this process: the data. The data format will drive 80 percent of the design decisions for the ETL pipeline. In this article, I introduce the concept of Canonical Transformation in this context as a way to create a reusable and easy to maintain pipeline.

The term canonical may be new to most readers, so what is it?

"Authoritative or standard; conforming to an accepted rule or procedure. When referring to programming, canonical means conforming to well-established patterns or rules."
—Webopedia

In essence, it is a set of common objects that can be manipulated, submitted, and used by various components. In the context of an ETL pipeline, the canonical allows multiple data types and endpoints to converge into a set of objects that allows a common set of downstream processing logic to be applied. The consistency of downstream processing allows an enterprise to create, maintain, and monitor a single set of business rules, identification criteria, and persistence logic. The benefits of implementing the Canonical Transformation are

Reduction in maintenance
Easy business logic changes
Data monitoring
Quality control
Decoupling the database from the data formats and endpoints
Decoupling pipeline components

The main factors when designing the Canonical Transformation step are the canonical objects and mapping routines.

Canonical Objects

Most canonical objects are unremarkable. They are data holders. However, determining the canonical design can be difficult, particularly if there are multiple data formats. The key is to start with some good questions to guide the design.

What are the system’s submission input formats? (See the last article)
What is the database design?
Will the system need to disseminate data? If so what format and how?
How will data be indexed for search?
What are the systems business rules?

Most likely the first question will drive the canonical design and will be the only question covered in this article. As is often the case, the most complex or detailed input format will influence most of the canonical design. Often, these formats have been designed by a consortium of experts and data partners. These are people with domain knowledge and a good understanding of the data, so typically they know how to represent it. In other words, it is the industry-standard format. Also, it is typically easy to map simple data to a more complex format than it is to map complex data to a simpler format.

Often times when working with an industry-standard format, the schemas are complex enough to handle edge cases that most consumers will not use or need. This is why some companies design simpler formats in CSV or use a UI to service smaller data partners (See submission Input). When a simpler format is created, data structures are reduced or an array will be simplified to a single item or limited to a certain number of items. When going from simple to complex, the decision to keep data is simple. However, when going from complex to simple, a decision to truncate data needs to be made. In the case where there is no data structure, the decision may be to not map it, but if many objects (array) are converted to a single field, the question of which item should be kept needs to be answered.

Pruning the Canonical Bush

Simply creating the canonical based on the industry-standard may work, but this approach may leave the system with a messy canonical and could require pruning. The first thing to ask in such a case is: “Do I have a business case for the data being truncated?” If the business owners/domain experts agree that the data is not needed for the system's purpose, then it is okay to remove the fields from the canonical. It is important to make it clear to business owners that re-ingesting the data may be difficult, time-consuming, or in some case impossible at a later date.

Another case for pruning the canonical is unused structures. In more verbose standards, metadata envelopes or communication sequence histories are specified. These data structures typical hold data about the transaction and that data is often transient. These are good candidates for pruning. It is important to note that if the system requires a high level of data provenance and auditing, then this data may need to to be kept.

Poorly designed standards

In some cases, the industry-standard may be poorly designed. An example of this is when a data structure is overly abstracted. This typically happens when there are various types of the same data structure. For example, a digital media standard may represent video, music, podcast, and e-books. The standard may abstract all these digital media types into a structured called work. However, a video may contain fields that an e-book wouldn’t have and vice versa. If the structure was design correctly with correct sub-type inheritance this may not be an issue. However, if the design tries to collect all the different types into the abstract work object, there could be confusing on the standard. In these scenarios, there are two approaches.

Leave the abstract data structure as is and write business rules to validate each type. This would be done if the system is designed to inform submitters of any violations in their data.
Boil down the data structures into concrete types. In this case, the system is deliberately truncating data that does not make sense for the type. The one issue with this approach is that data partners may not completely understand the data standard or may conflate the meaning of two different data fields. Using the above example, a book may have an author and music may have a composer. These words have very similar meanings and a data partner may populate the composer field for a book’s author.

Once the canonical is drafted, ask “does the design accurately represent the domain?” Repeatedly asking this question throughout the design process will help ensure that the canonical stays true the system’s intention.

Mapping Routines

Mapping routines is all about how the raw data gets unmarshalled into the canonical. Often there will be at least one mapping routine per data format.

In some cases there may be multiple routines per data type. This is often seen with industry standards, particularly with more complex standards. When a standard has been around for a while, data partners will have their own way to fill them out, using some data fields, not populating others, and using nodes in different ways. In these cases, most of the data can be mapped by the parser/mapper, but will require different sub-routines to map the differences.

Another multiple-routines-for-the-same-format scenario occurs when there are multiple versions of the same industry format. In both these scenarios a factory pattern can be used to determine the routine via submitter and version. Then simple inheritance can handle submitter specific mappings.

Breaking out the mapping routines per format, version, and data partner decouples the logic for easy blackbox testing. Raw data goes in and the expected canonical comes out.

Summary

The most important step for making an ETL pipeline reusable and easy to maintain is the canonical step. A well designed canonical will decouple the raw data from the system’s business logic and internal working, allowing the code to be reused and reduce duplication. The next article in the Extract Transform and Load Series will address business rules.

Ten Mile Square has expert consultants and engineers that have a proven track record to learn new domains, analyze requirements, and design the perfect canonical to mean your business’s needs. Reach out to us if you have any questions. We love to help.

Extract Transform and Load Part 2 – Submission Information Packages and High-level Validation

Ry — Fri, 28 Jun 2019 16:45:21 +0000

The first two major pieces of an Extract Transform and Load pipeline are the Submission Information Package (SIP) and High-level validation. The major components, as well as the benefits and usages of Extract Transform and Load, were discussed in the first article in the series. It is recommended that the reader have familiarization with the overall Extract Transform and Load process before continuing on. Data is important, and in this article, the subject is Data.

Submission Information Package

The Submission Information Package, or SIP for short, is the actually data submitted from the data partner to the data ingest system. The term SIP is taken form the Open Archival Information Standard (OAIS) which defines it as

Submission Information Package (SIP): An Information Package that is delivered by the Producer to the OAIS for use in the construction or update of one or more AIPs and/or the associated Descriptive Information.

This is the saying the data provided to the system for use of creating one or more records. For the purpose of this discussion, we will expand that definition and break into two pieces Raw Data and Data End Points

Raw Data

Raw data is the data files provided to the system from a data partner to the ingest system. The data can be a single data file, multiple data files, or a package, which often contains the data files and a metadata or descriptor file. In an OAIS system, a SIP is usually

a bunch of files of various formats,
A descriptor file for each file
A manifest of all the files in the package
An authentication file or digital signature

In most ingest systems the data will just be one or more predefined data files.

The questions to ask when it comes to raw data are

Are there any particular standards used in the industry (Industry SOPS, Schemas, XSD, format best practices)?
What data types is the system ingesting (Excel, XML, JSON)
Do we need to create a standard? Or do we need to simplified an existing standard?
Who will be submitting the data? Will it be from one partner or can multiple partners submit the same data types.

Are there any particular standards used in the industry?

Typically, the first question “Are there any particular standards used in the industry?” is the best place to start. In a larger industry, it will often be the case that a data transfer standard already exists. This should be the starting point for this discussion. The best way to find this out is to google, ask industry experts, and look at digital retail sources for the data.

Data types are often easily determined from answering the industry standard question. Typically, XML is used, but it is the case that CSV and fixed-width file formats will be used. JSON is gaining popularity, but in industries where data quality is highly importing this format is avoid, due to its free form nature. However, tools such as RelaxNG are available to set up a data structure standard for JSON. The adoption of such tools is limited and resources for use in certain programming languages may not be available or incomplete.

Do we need to create a standard or a simplify standard?

The question “do we need to create a standard or a simplify standard?” often comes from one of two situations

No standards currently exist
A standard exist, but it's
- overly complicated
- it no longer fits the industry's needs
- or less sophisticated data partners want to participate

Creating a new standard requires an understanding of the domain and facilitated communication between the data partners. Best practices when creating a new standard could fill a book and such is out of scope for this article.

The second situation is when a standard exists, but it's either overly complicated, it no longer fits the industry needs, or the ingest system is trying to allow smaller data partners to contribute. As is the case with the first situation care needs to be taken when creating a new standard. However, when building a system that is trying to allow small data partners to contribute, it is often the case that some data partners do not have the technology sophistication to implement a large standard. Often the solution is to create a UI to allow partners to manually enter data, or create a CSV/Excel template to allow partners to bulk upload data. When simplifying a format, it is important the format covers or make mandatory the basic required data of the larger formats.

Who will be submitting the data?

The last question, “who will be submitting the data?” will dictate the above standards question as well as the next section Data Endpoints.

Data Endpoints

Data endpoints are how data partners send the data. There are two basic forms of data endpoints, push and pull.

Note: Data retrieval by a data partner (the system sends responses) would also be considered a data endpoint, but is outside the scope of discussion for this article.

Push

A push endpoint is where a data partner initiates the requests and sends the data to the system. Typically, these types of endpoints are web services and UI inputs. There are many articles about REST, web services, and UI design and best practice, so this article will not touch on these. However, it is recommended that the endpoints are just that endpoints, they should not dictate a whole new pipeline. Care should be taken to tie these endpoints into the data pipeline and reuse as many of the ETL components as possible. Typically this means mapping the data into a canonical. Business logic/rules are where data quality is scrutinized and keep the logic in one component makes the system easy to maintain and ensure data quality. If a UI

If a UI is offered to data partners, particularly if a CSV upload component is available it is often good to allow data partners to upload XML or JSON files via the UI. In addition, having a path to allow partners to validate the data without it being persisted to the database is useful when on-boarding new data partners.

Pull

A pull endpoint is where the ingest system reaches out to various data partners to ask for data. These pull jobs need to be scheduled and execute by the system and periodically poll the partnering system. Examples of this are when the ingest system subscribes to a data feed such as RSS or ATOM or data scrapes from a data source. Note: most data scraping endpoints will not require a formal ingest pipeline as data is often unofficial and may not be the best quality.

SFTP/Shared Directories

The one hybrid approach is SFTP or a shared directory. This requires that both the data partner and the ingest system to do pulls and pushes. In this case, the data partner will push data to a shared location. Then the ingest system will poll the shared directory for new data to ingest.

When pulling data from a shared location best practices are extremely important. things to consider are

Sub-directories: These are used to break up data partner submissions in logical chunks. A best practice is to use a unique identifier or a timestamp as the subdirectory's name. The pulling system does want to scan old data every time.
Batch complete files. The data partner needs a way to tell the pulling system that a data batch is ready for ingest. If the pulling system starts to scan when the pushing is still transferring the data it could result in a bad/incomplete batch. Often a blank file with a naming convention like 'batch.complete' is used. Be sure that the data partner knows to create the batch.complete file only after all data has been fully transferred.
Data retention policies: Over the course of time a shared directory can get large, causing performance problems for the pulling system or maintenance costs for directories host. The question of what should be done with or how long should data be stored on the directory after it is ingested. Often times the ingesting system will delete or move the data to another location after ingest is successfully completed. Note: successfully complete can contain pipeline errors if the data is invalid. But if an unknown system error occurs, such as, the database is down, then data shouldn't be removed.

High-level Validation

High-level validation deals strictly with the questions

Is the data from a trusted source (i.e. does the system know the submitter)?
Is the data in a format that is known?

Is the data from a trusted source?

Determining if the data is from a trusted is typically straight forward and solutions are already well known. Examples of solutions to this question are

Access Control Lists (ACLs)
API Keys
User login credentials
FTP credentials
SHA keys
etc

In more secure cases digital signatures and certificates can be used to ensure the data wasn't tempered with and chain of custody is maintained.

Is the data in a format that is known?

In this case we are talking about High-level validation, which does not care about business rules or fine-grained validation such as does a book include a title and an author. However, some data specifications, such as XSD, will have lower level validations included.

Some examples of high-level validations for various data formats are

XML

Is the data well-formed?
Does the XSD validate?
- It should be pointed out that not all XSD validation packages are created equal. If the XSD spec is from a later version some XSD libraries in Python and Java may not catch all the rules. Note: Most of the Java libraries will validate the spec. However, most python XML processing libraries are tied to XMLLINT, which does not handle more of the complicated validation rules.

CSV/EXCEL

Do all the expected header columns exist?
Does the column header text match the expect column header text?
Are the columns in order?
Is there at least one row of data?

File format

Does the file extension match the expect extension?
- This one is really easy to spoof. However, when dealing with CSV or Excel most users are not sophisticated enough to change the file extension, since they are using a spreadsheet application. This check ensures they didn’t accidentally upload the wrong file.
- Tools exist to inspect and determine file format regardless of the file extension, however the footprint of these tools are often large and considered overkill for most data pipelines.

Summary

When working with Extract Transform and Load systems we are focusing on a system that ingesting data. That means data is the most important part. Considering the data first can help give clarity to the downstream components. Be sure to answer the questions

What kind of will the system be ingesting?
Who will provide this data?
How will the data partners provide this data?
How will system verify the data is in a format that the system understands.

For more information about Extract Transform and Load, be sure to read Extract Transform and Load Overview article. Also be on the look out for part 3 of this series Canonical Transformation.

Extract Transform and Load Overview

Ry — Fri, 14 Jun 2019 13:24:14 +0000

Part 1 of the Data Ingest Series

The process of extract transforms and load (ETL) goes by many names:

Data acquisition
Data Ingest
Enterprise transform and load

But they all are about getting external data into the system.

The problem that most businesses face is that there are no easy to follow best practices that can guide them in creating ETL pipelines. In this 6-part series, I will address this gap by laying out a simple and flexible set of best practices for creating or evaluating an ETL system.

The importance of good quality data

First let’s discuss why ETL is important to the business. Data is frequently the most important asset in any business. In the information age it’s easy to see why. Let look at these examples:

Amazon captures buyer data to suggest new items to purchase.
Facebook tracks user data to use and sell for targeted marketing.
Google uses user search history to display google ads.
The music industry exchanges data in order to sell music, pay royalties, and exchange rights.
Governments ingest data for threat assessment in order to enhance national security.

The list goes on and on...

So how we handle these data sources and the quality of these data sources are high import for sales, governance, commerce, and in some cases life and death decision making.

Buy vs. build

This subject is deserving of its own blog article and can actually be quite controversial during the due diligence phase. I’m only going to briefly touch on the subject here in this series.

What are some of the criteria when evaluating a COTS ETL platform?

The common criteria when evaluating building something custom or buying Commercial off the Shelf (COTS) platforms are:

Price. How much license? How much is the cost of building from scratch? Are their maintenance contracts? Will I need to send my employee’s to external training? Will I need a third-party consultant on staff? What are the hosting costs?
Maintenance. How difficult is it to upgrade? Are there solid upgrade paths? Where will I host the application? Is this a stand-alone/hosted install? Or can I deploy into the supporting app.
Speed of ingest. How quickly do I need to process data? Custom built solutions are often much faster that COTS products.
Use Case. Will this application be used on one project or across the entire organization? (Be honest with this one. I’ve worked with countless COTS products bought and installed for a particular project where the organization says “Lets beta test this million-dollar product on one project and if it works, let's implement it to the rest of the organization”. Only they never do and now they are stuck with a very expensive license).
Governance. Do I need to maintain a separate set of users and roles? Can I limit who can access my data pipelines
Flexibility. How easy is it to add new input data types. Can I change the process flow?
Who will be developing it? A feature of COTS products is they often claim workflows (pipelines) can be created and managed by business users. This often requires special training of business users and a particular type of business user to maintain this. Sometimes companies will use an existing developer to work on the pipeline anyway.

There are many great products out there that will give flexible pipelines with the kitchen sink when building your data pipeline. This seems great! Buy a license, install, create a pipeline and let’s go! However, consider the hidden costs in this approach.

Components

The majority of data pipelines can be boiled down to 6 main components. Some of these components can be reordered, others may be optional in a particular use case. However, understanding and fully considering each of the components is the key to ensuring data quality and pipeline reusability.

Submission Information Package (SIP)

Submission information package is the first step in any data pipeline. It is the data itself and how the data will get into the system, also known as raw data and data endpoints.

Raw data is the data files being submitted or ingested by the system. This could be multiple formats from multiple submitters. The questions to ask are

What data types are we ingesting (Excel, XML, JSON) and are there any particular standards used in the industry (Industry SOPS, Schemas, XSD, format best practices)
Do we need to create a standard? Or do we need to simplify an existing standard?
Who will be submitting the data? Will it be from one partner or can multiple partners submit the same data types.

The endpoints discuss how the system gets the data. The questions to ask are

Will partners be submitting data to us (Push)?
Or will we need to get the data from the partners (Pull)?

SIPs and High-level validation will be discussed in depth in from Part 2 of this series.

High-level validation

High-level validation is the second step of any ETL process. High-Level validation is mainly concerned with ensuring that the data format being ingested is actually a format that the system knows how to ingest. For example, it checks that an XML file conforms to the schema, or that the CSV/Excel file has the correct columns. It is not concerned with details of the data like does the book have a title. Does the user have at least one account, etc.

SIPS and High-level validation will be discussed in-depth in Part 2 of this series.

Canonical Transform

This is the third step in any ETL process. The canonical transform is actually composed of 2 pieces. The canonical domain and the data mapping process.

The canonical form is a set of simple objects that all data formats can be mapped to. Often times the canonical will take the form of target database tables or the most complicated or standardized input data format. Industry data standards, like OMOP in healthcare, also represent canonical data models.

The data mapping process is the actual process of converting data from the input formats into the canonical. For example, taking a book title from a CSV file and putting it in the title field of the Book canonical object.

This step is important to the process because it allows for downstream processing to become reusable and easy to maintain.

Canonical Transform will be discussed in-depth in Part 3 of this series.

Business Rules

The fourth step uses the canonical from step 3 and applies detailed validation rules against the data. These rules are important to keep database integrity. These could be simple validations such as all books must have a title or more complex rules such as that a book ownership shares total to 100%. Complex business rules usually focus on data relationships or aggregated data.

Business Rules may also focus on rules that are important to the organization. Data standards are often designed to be flexible across many use cases, however, your organization may only care about a particular aspect of the data standard. Business rules are how the system will filter out the needed data and the unneeded data.

Business Rules will be discussed in-depth in Part 4 of this series.

Identification

The identification step is the fifth step in the ETL process. The identification step is how the system will reconcile new data with previously ingested data. This process will involve key identifiers to compare the data against our known data set (database, index, NO-SQL, external system, etc.). If a match (the data was identified) was found then this step may resolve how data is merged. This means if the data is known and there is a difference between the data store and incoming data how is it reconciled. Some uses case will require the data merging to happen in the persistence layer. This decision will ultimately be based on the business rules and the system’s requirements.

Depending on the needs of the business and the type of data being ingested, this step may need to be skipped. Also, in some use cases it is important to identify the data before running the business rules as the business rules may vary based on if the data is known (identified) or new (unidentified).

The identification step is discussed in-depth in part 5 of this series.

Persistence

This is the sixth step in the data ingest process. It focuses on storing the data in our datastores for downstream use or for future data identification

Persistence is discussed in-depth in part 5 of this series.

Audit/Governance/Work Queue

The audit, governance, and work queue components are not steps in the overall ingest process but are important, but optional, components of an ETL system. These are the ‘kitchen sink’ features found in most COTS products. The way the data ingest pipelines are used and managed will ultimately determine if any of these 3 components need to be implemented.

Audit

The audit component is multi-faceted and covers many areas important to:

Data provenance: Knowing the source of data stored in the data store
Business intelligence: This is buzz term for analytic about the pipeline and the data streaming through. It answers questions such as
- How much data is being ingested from a given source?
- How long are each of these steps taking to process?
- Where in process does data fail?
- Where are our bottlenecks?
- How many records are being created from each submission?
Decision Audit: Why decisions about the data were made.
- Why a record was identified?
- Why a data file did not pass a business rule?
- Why was a record persistence?

Governance

Governance covers who can access, run, and report on our data ingest pipeline? And who can change our data pipeline?

Work Queue

The work queue component is not a component that is often seen on many ingest system, but given the type of data being ingested it can greatly improve data quality, data transparency, and data throughput.

The Audit, Governance, and work queue pieces will be discussed in part 6 of this series.

Summary

The importance of data acquisition in a system is undeniably important and the quality of the data, ultimately determines the quality of the system. With these 6 basic components just about any data ingest pipeline can be designed and implemented with an assurance of data quality and pipeline reusability. In the next segment of this series we will be discussing the Submission Information Package

Gratitude in your life

Ry — Fri, 14 Jun 2019 12:54:04 +0000

As a response to @Ryland G's "What gets you out of bed in the morning?" I want to know Do you practice gratitude? and if so How do you practice gratitude?

This practice has been one of the most impactful practices in my life. I attribute this practice to helping reduce anxiety and increasing happiness. And as we know a happy person is a more productive person.

I practice by either writing or thinking about 3 things I'm grateful for in my life each day. The important thing I found through this practice is that you really need to feel emotion for each item.

How I divide the list of 3

1 thing that you are grateful for in your life
1 small thing that you are grateful for (i.e. the wind blowing, the way the grass looks in the dew, etc)
1 thing about your self you are grateful for (i.e. your intelligence, your heart, because it never stopped beating, etc)

How do you practice? What are you grateful for?