DEV Community: Oryan Moshe

The Secret Behind My Latest Blog Post: ChatGPT Did the Heavy Lifting

Oryan Moshe — Mon, 08 May 2023 14:46:40 +0000

Have you ever wondered how you could streamline your content creation process? Look no further! In this blog post, I'll reveal the secret behind my latest article: using OpenAI's ChatGPT to do the heavy lifting for me.

Mock Data & Stubs (Fake it till you make it)

Oryan Moshe ・ May 7 ・ 11 min read

#beginners #programming #career #coding

In this article, I'll explain how working iteratively with ChatGPT transformed my content creation process and made it more efficient than ever before.

Why the Iterative Method Outshines Traditional Prompting

ChatGPT, specifically GPT-4, is a powerful AI language model capable of generating human-like text. Most people use it by giving it a single prompt and waiting for the generated text. However, this "one-shot" approach often results in content that isn't quite what they were hoping for. That's where the iterative method comes in.

The iterative method is a more collaborative approach, involving working with ChatGPT step-by-step. It allows for continuous feedback and improvement, resulting in higher quality content that aligns better with your vision. In the following sections, I'll detail my journey of teaching ChatGPT my writing style, refining its understanding of the subject, and incorporating helpful prompts to create a well-structured, coherent blog post.

Teaching ChatGPT Your Writing Style

For a more personalized and natural-sounding article, it's essential to teach ChatGPT your unique writing style. To do this, you can provide the AI with examples of your previous work. The more examples you give, the better ChatGPT will be able to emulate your voice.

In our case, I instructed ChatGPT to request more example articles until it felt confident that it could mimic my writing style. I also provided the context that the AI is an engineer who loves to write technical blog posts.

Here's a simple process for teaching ChatGPT your style:

Gather a few samples of your writing that you feel accurately represent your "voice" (writing style)
Provide these samples as context to ChatGPT at the beginning of the conversation.
Instruct ChatGPT to request more examples if it needs them to better understand your writing style.
Throughout the iterative process, give feedback on the generated content, pointing out where the AI succeeded or missed the mark in capturing your style.

By consistently providing feedback and guidance, you'll help ChatGPT to fine-tune its understanding of your writing style, and the generated content will begin to sound more like you.

Instructing ChatGPT to Ask Questions for Better Understanding

Another key aspect of our iterative approach to working with ChatGPT is getting the AI to ask questions for better understanding. Instead of relying solely on your initial input, you can encourage ChatGPT to seek clarification and gather more information about the topic.

This strategy not only helps the AI produce more accurate and relevant content but also fosters a more interactive and collaborative experience between you and ChatGPT.

To instruct ChatGPT to ask questions, follow these steps:

In your initial conversation, mention that you want ChatGPT to ask questions if it needs more information or clarification.
As you work through the iterative process, provide the AI with answers to its questions and use this information to refine the generated content.
Encourage ChatGPT to continue asking questions if it's still unclear about certain aspects of the topic or your preferences.

By engaging in a back-and-forth dialogue with ChatGPT, you'll create a more comprehensive understanding of the subject matter, resulting in a higher quality article.

Orchestrating ChatGPT, Midjourney, and Grammarly Together

As with any content writing team, one person can't do it all. We leverage the power of several tools to create the perfect writing team.

Improving Prompt Clarity with Grammarly

To ensure that ChatGPT receives the best prompts possible and works more effectively, we use Grammarly, an AI-powered writing assistant, to improve the clarity of our prompts before sending them to ChatGPT. Remember, garbage in - garbage out.

Here's how to use Grammarly to improve the clarity of your prompts:

Open Grammarly's editor.
Write your prompt intended for ChatGPT directly in the editor.
Review and apply Grammarly's suggestions to enhance the clarity and quality of your prompt.
Keep a history of your prompts in Grammarly for future reference.

By refining your prompts with Grammarly, you ensure that ChatGPT has a clear understanding of your requirements and can generate better results.

Generating Midjourney Prompts

ChatGPT can be used effectively for generating prompts for other AI tools, such as Midjourney. Midjourney is an AI image generator that creates custom visuals based on text prompts.

For many tasks, ChatGPT can create content perfectly on its own. However, ChatGPT sometimes provides invalid URLs for images and cannot be trusted with these tasks. Instead, I instructed ChatGPT to generate a description of the image using a "PIC-PROMPT " whenever an image is necessary. Then, I can use Midjourney or search online to find the perfect visual based on the description provided by ChatGPT.

Practical Prompt Examples and Tips for Effective ChatGPT Collaboration

In this section, we will go through some practical prompt examples and share useful tips to make your collaboration with ChatGPT more effective.

Prompt Examples

Context: When you start out, you need to provide ChatGPT with a context of what it is and what the goals are. Use a prompt like this:

You are a hands-on, highly Pragmatic Programmer who loves to write technical blog posts about your endeavors as a <your role> of a company called <your company>.
Your blog posts are funny and light-hearted but as technical as you can get them while staying relatable.
Your goal is to help me create blog posts about different topics, trying your best to mimic the way I write blog posts.
These articles will be published on dev.to, medium.com, and LinkedIn.

I will feed you a few of my previous blog posts (in markdown format) so you can learn how I usually write.
How many posts are required?

When you're ready, ask me for the first blog-post example.

Content Styling: Now that ChatGPT has asked us for our first example, we need to provide it with as much information as possible, while making sure it remembers to keep the feedback loop going. Here is a prompt that worked for me:

I'll now provide the first blog post. After reading it, tell me that you understood it, and then (when you're ready) ask me for the next blog post. If you have learned enough, tell me you don't need another post. When we reach that point, ask me what topic I'd like my next blog post to be about.

The first blog post was published on <website and date>
It has XXX reads, XXX reactions, and XXX comments.
The article's title is: <article title here>
I tagged the article as <tags here>.
Here is the content in markdown format:
<article content here>

Setting the Foundations: Eventually, ChatGPT will tell you it has learned enough. At this point, we want it to drive the process. To do this, we instruct it to ask questions, like so:

Ask me up to 10 questions to help guide you to write my next blog posts. The questions can refer to the new blog post topic, specific info needed, or my writing style and goals. So ask me *everything you need* to write my next blog post as best as possible.

Involving ChatGPT: Now, ChatGPT will ask you several questions, probably about the topic of your next article, the target audience, format, objectives, etc. Answer these questions, and make sure you keep encouraging ChatGPT to ask you questions and get involved throughout the whole process like so:

Here are the answers to these questions:
<answers here>

Also, the title I had in mind was "<possible title here>". Feel free to suggest another title if you have a better one.

If you feel like you need more information, you can ask me up to 5 more questions (you don't have to use them all, only ask for help if you need it)
Now, we can do this in 1 of 2 ways:
1. We'll write it together incrementally - you'll send a segment of the article, we'll work on it, and then move on to the next iteratively
2. You'll send me a draft, I'll provide feedback, and then you send me a fixed draft, and so on

Which of these 2 approaches do you prefer?

Content Formatting & Triggering Generation: Usually, ChatGPT will pick approach 1, but remember you can change these prompts to fit your preference. Now we need to describe how ChatGPT formats the text it generates, and trigger the first generation like so:

Sure, we can go with approach 1.
If you have any other questions, please ask me up to 5.
If not, please provide me with the first segment to go over.

From now on, all content related to the article should be formatted as markdown and sent in a code block to allow me to read it.
Add a horizontal line ("---") between the different sections of the post to separate them better.
Whenever you want to represent a code block in the article, instead of using the markdown notation, please wrap the code block with:
^^<syntax language here>^^
...
$$<syntax language here>$$
Instead of images and gifs, add a "PIC-PROMPT <prompt for Midjourney here>" everywhere you deem fit. Midjourney prompts usually start with "/imagine <rest of prompt>".

Iterate & Improve: ChatGPT will send you the first part of your blog post, read it thoroughly (remember, ChatGPT can help with content creation, but we need to manually validate every word), send feedback for revisions, and once you're happy, tell ChatGPT to move on to the next segment. Remember to constantly remind ChatGPT it can ask you questions.

Tips & Tricks for Effective Collaboration

Be specific with your prompts: Provide clear and concise instructions to help ChatGPT understand what you want.
Iterate on the generated content: Don't hesitate to revise and refine the content generated by ChatGPT until it meets your expectations.
Leverage ChatGPT's ability to ask questions: Encourage ChatGPT to ask you questions for a better understanding of the subject matter.
Use external tools like Grammarly: Improve the clarity of your prompts using AI-powered writing assistants like Grammarly before sending them to ChatGPT.
Understand the limitations: Sometimes, some of the content generated is not up to our standards. Usually, we can ask ChatGPT to revise it, but if the fix is easy, it's usually faster to fix it yourself and then tell ChatGPT "Here's the segment I ended up using:" so it keeps learning your writing style. Don't get stuck on trying to make ChatGPT do what you want, if it's faster to just do it yourself.
Hone your skill: As with any tool, ChatGPT has a learning curve. Refine your skills and learn how to handle unexpected scenarios. For instance, when ChatGPT stops mid-answer, you could simply prompt it with Continue., but a more effective prompt might be: You got cut-off, continue from "<cut-off section>". Remember your formatting rules, and keep all content inside of a code block.

By following these tips and using the practical prompt examples, you can make your collaboration with ChatGPT more effective and efficient, ultimately creating high-quality content for your blog posts.

Addressing Common Misconceptions and Concerns

When it comes to using ChatGPT for content creation, some people might have misconceptions or concerns about the process. In this segment, we'll address some of the most common misconceptions and concerns, and help you understand how to overcome them.

Time-intensive process: Many people believe that creating blog posts with ChatGPT is time-consuming, but once you've mastered the process of working with the AI, you can actually save time compared to writing everything by yourself. The key is to be patient and invest time in learning how to effectively communicate with and instruct the AI.
Difficulty in forming coherent content: Some users might struggle with creating coherent blog posts using ChatGPT because they try to generate the entire post in a single shot, instead of working iteratively with the AI. By following the iterative process that we've outlined in this article, you can ensure that the content is well-structured and coherent.
Limited creativity: AI-generated content might be perceived as lacking creativity or a personal touch. However, by providing specific examples, context, and guidance to ChatGPT, you can infuse your writing style and personality into the generated content. Remember, the more you work with the AI, the better it becomes at understanding your preferences and mimicking your writing style.
Dependency on AI: Some people might worry about becoming too dependent on AI for content creation. While it's true that ChatGPT is a powerful tool, it's essential to strike a balance between AI-assisted writing and your own writing skills. You should always review and revise the content generated by the AI, ensuring that it meets your standards and retains your personal touch.

By addressing these misconceptions and concerns, you can make the most of ChatGPT and leverage its capabilities to create high-quality content for your blog posts.

Conclusion: Enhancing Your Blogging Process with ChatGPT

Leveraging the power of ChatGPT, you can create compelling and high-quality blog posts that resonate with your readers. By following the tips, tricks, and examples provided in this article, you'll be able to make the most of your collaboration with ChatGPT and streamline your blogging process.

Remember to:

Provide clear and concise prompts to guide the AI.
Use an iterative approach to create coherent and well-structured content.
Encourage ChatGPT to ask questions and actively participate in the content creation process.
Review and revise the generated content to ensure it meets your standards and retains your personal touch.

With patience and practice, you'll master the art of working with ChatGPT to create engaging and informative blog posts that reflect your unique writing style and voice.

Embrace the power of AI and revolutionize your blogging process with ChatGPT, happy writing!

Mock Data & Stubs (Fake it till you make it)

Oryan Moshe — Sun, 07 May 2023 12:00:00 +0000

Mock Data & Stubs (Fake it till you make it)

This is you before implementing mock data to your workflow

Developing new features can be a total blast, but it can also be a slow and frustrating process. One of the most significant contributors to this frustration is the time it takes to iterate through each development cycle. The longer it takes to develop, test, and refine each feature, the more time is wasted, and the slower progress becomes.

But what if I told you there's a way to reduce your iteration times significantly? Today, we'll explore a technique that can help you "fake it till you make it" when it comes to developing features: using mock data, stubs, and fakers. This approach will help you eliminate the dependency on external code or resources that aren't necessary at the moment and allow you to focus on the task at hand.

Why Reducing Iteration Time is Crucial

Time actually IS money

Reducing iteration time is crucial for several reasons:

Faster MVP: By quickly developing a working Minimum Viable Product (MVP), you can gather real-world feedback and make data-driven decisions about which features are essential and which are not. This way, you can adapt and change your product according to actual feedback from your experience with the feature, instead of wasting time on unnecessary additions without ensuring the core functionality works.
Increased productivity: When you spend less time waiting for external dependencies, you can focus on what really matters: writing and refining your code.
Greater flexibility: When you're not reliant on external code or resources, you can make changes and improvements more easily, allowing you to pivot when necessary and adapt to new requirements.
Improved morale: There's nothing worse than being stuck in a never-ending development cycle. By reducing iteration times, you can maintain momentum and keep your team motivated.

Now that we've covered why it's essential to reduce iteration times, let's dive into what mock data, stubs, and fakers actually are.

Mock Data, Stubs, and Fakers: The Building Blocks of Rapid Development

Let's break down these three essential concepts and clarify their differences and interactions.

Mock Data

Mock data is a set of fake data points that you can use to simulate the behavior of your application. It helps you avoid relying on real data sources, which can be slow or unavailable during development. By using mock data, you can develop and test your application's logic independently from any external dependencies.

Stubs

Stubs are simplified versions of components or functions that return a predetermined response. They help you isolate specific parts of your application and focus on the functionality you're currently developing. Stubs can be used to define the structure of your code, allowing you to move forward without getting stuck on implementing every single function. They're also helpful when collaborating with other team members who are responsible for developing certain functionalities.

Fakers

Fakers are libraries or tools that generate realistic-looking data for testing purposes. They can create data that resembles real-world information, such as names, addresses, and dates, without using actual data points.

Understanding the Interaction Between Mock Data, Stubs, and Fakers

Essentially all we're doing is connecting empty boxes to other empty boxes

Now that we know what mock data, stubs, and fakers are, let's discuss how they interact and complement each other in the development process.

Mock data represents the data structure and provides a way to simulate application behavior. It is the foundation for building test scenarios and understanding how your application should respond to various inputs.
Stubs replace parts of your application that depend on external components or services. They help you focus on the functionality you're currently developing by providing predictable behavior.
Fakers generate realistic data to populate your mock data and stubs. This allows you to create more diverse and complex test scenarios without manually crafting every single data point.

By combining these three techniques, you can create a flexible and efficient development environment that allows you to quickly iterate on your features while focusing on the actual logic at hand.

Putting It All Together: A Practical Example

Now we'll look at a practical example of how each of these techniques can be implemented. We chose JavaScript for simplicity, but you can treat this example as pseudo-code for now.

Let's assume we're building a simple application that displays a list of users and their details. We'll use mock data, stubs, and fakers to create a rapid development environment for this scenario.

Step 1: Creating Mock Data

First, we need to create some mock data to represent the user information. For this example, we'll define a simple JSON object with user details:

[
  {
    "id": 1,
    "name": "John Doe",
    "email": "john.doe@example.com",
    "address": "123 Main St, Anytown, USA"
  },
  {
    "id": 2,
    "name": "Jane Doe",
    "email": "jane.doe@example.com",
    "address": "456 Oak St, Anytown, USA"
  }
]

Step 2: Implementing Stubs

Now that we have our mock data, we can create a stub function that simulates fetching user data from a server:

function fetchUsers() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve([
        {
          "id": 1,
          "name": "John Doe",
          "email": "john.doe@example.com",
          "address": "123 Main St, Anytown, USA"
        },
        {
          "id": 2,
          "name": "Jane Doe",
          "email": "jane.doe@example.com",
          "address": "456 Oak St, Anytown, USA"
        }
      ]);
    }, 1000);
  });
}

In this example, the fetchUsers() function returns a Promise that resolves with the mock data after a 1-second delay. This simulates an asynchronous API call, allowing us to develop and test our application without relying on a real server.

Example: Using the `fetchUsers()` Stub

To demonstrate using the fetchUsers() stub function, let's create a simple function that displays the fetched user data:

async function displayUsers() {
  const users = await fetchUsers();

  users.forEach((user) => {
    console.log(`${user.name} (${user.email}): ${user.address}`);
  });
}

displayUsers();

This example calls the fetchUsers() stub function and then logs the user details to the console.

Imagine if we had to wait for a server or a database to respond in order to test our displayUsers() function, and suddenly the data source is unresponsive, or your internet connection goes out.
Good thing we don't rely on stuff like that anymore (:

Step 3: Using Fakers

To make our example more realistic, we can use a faker library to generate a larger set of user data. Here's how we can use the popular Faker.js library to create a more diverse set of mock users:

import faker from 'faker';

function generateUsers(count) {
  const users = [];

  for (let i = 0; i < count; i++) {
    users.push({
      id: i + 1,
      name: faker.name.findName(),
      email: faker.internet.email(),
      address: faker.address.streetAddress()
    });
  }

  return users;
}

const mockUsers = generateUsers(50);

Now our application has a more realistic set of user data to work with, allowing us to test various scenarios and edge cases.

By combining mock data, stubs, and fakers, we can create a rapid development environment that focuses on the functionality we're currently developing.

Best Practices and Potential Pitfalls

Let's dive into some best practices and potential pitfalls you should be aware of while working with these techniques.

Keep Your Mock Data Realistic

When creating mock data, it's essential to keep it as realistic as possible. This helps ensure that your tests and development experience closely resemble real-world scenarios. For example, if you're working on a project that deals with user profiles, create mock data that includes a variety of names, ages, and other relevant information, rather than using generic placeholders like "John Doe" or "12345."

const realisticUsers = [
  { name: 'Alice', age: 28, email: 'alice@example.com' },
  { name: 'Bob', age: 34, email: 'bob@example.com' },
  { name: 'Carol', age: 22, email: 'carol@example.com' },
];

Don't Overuse Stubs

Stubs can be incredibly useful when it comes to speeding up your development process and isolating different parts of your code. However, it's crucial not to overuse them. Overusing stubs can lead to overly rigid code that's difficult to maintain and refactor. Remember that stubs are a means to an end – they should help you focus on the most critical parts of your code and avoid getting bogged down in implementation details.

Treat stubs as "empty boxes", when the time comes you still have to fill them up with their contents.

Keep Your Fakers Consistent

When using fakers to generate random data, it's essential to maintain consistency across your tests and development environment. Inconsistent faker data can lead to false negatives or positives in your tests, making it challenging to determine whether your code is functioning correctly. To ensure consistency, consider using seed values when initializing your fakers, so the same data is generated every time you run your tests.

const faker = require('faker');

faker.seed(12345); // Seed the random number generator

const firstName = faker.name.firstName(); // This will always generate the same first name
const lastName = faker.name.lastName(); // This will always generate the same last name

Popular Libraries and Tools for Mock Data, Stubs, and Fakers

We've discussed the importance of these techniques, and now it's time to see some examples of libraries and tools you can use in different programming languages to help you achieve this. The purpose is to show how these concepts are applicable across various programming environments. However, we'll put more emphasis on C# and Unity3D since these are the main tools used by my team at Magnifica VR.

C# and Unity3D

NSubstitute: A friendly substitute for .NET mocking libraries, NSubstitute is designed as a more straightforward and more concise alternative to other mocking libraries, making it easier to create substitutes for interfaces and classes in your tests.
Moq: A popular and easy-to-use mocking library for .NET that allows you to create mock objects, set expectations on them, and verify interactions with them in your tests.
Unity Test Framework: Unity3D provides a built-in testing framework that supports unit, integration, and end-to-end testing. It allows you to create test doubles, such as stubs and fakes, to isolate the code under test and remove dependencies.

JavaScript

Faker.js: A popular library for generating fake data, such as names, addresses, and phone numbers. It can be used in combination with stubs to create realistic mock data for your tests.
Sinon.js: A library that provides standalone test spies, stubs, and mocks for JavaScript. It works with any unit testing framework and is compatible with different browsers.

Python

Faker: A Python library that generates fake data for you, similar to Faker.js for JavaScript. It's useful for creating mock data for testing or populating databases.
unittest.mock: A built-in library in Python that allows you to create stubs and mocks for your tests. It's easy to use and doesn't require any external dependencies.

Ruby

Faker: A Ruby library that generates fake data, such as names, addresses, and phone numbers. It's helpful for creating realistic mock data for your tests or seeding your database.
RSpec Mocks: A library that provides a simple and flexible mocking framework for Ruby. It's part of the RSpec testing ecosystem and can be used with other RSpec libraries or as a standalone mocking solution.

As you can see, no matter what programming language you work with, there are libraries and tools available to help you create mock data, stubs, and fakers. Utilizing these tools will save you time and make your testing process more efficient.

Now that we've explored the available tools and libraries, let's see how all these concepts can be applied in a real-world scenario.

Case Study: Implementing Mock Data, Stubs, and Fakers in a Unity3D Project

Let's explore a simple case study in a Unity3D project. We'll create a simple game that displays a list of players with their scores, and we'll use the concepts we have just discussed to speed up the development process and make it more efficient.

Setting Up the Project

To start, let's create a new Unity3D project and set up the basic structure. We'll need a script to handle the player data and another one to display it on the screen.

Create a new C# script called PlayerData.cs and add the following code:

public class PlayerData
{
    public string Name;
    public int Score;
}

This simple class will hold the player's name and score. Now, let's create another script called PlayerListDisplay.cs to display the list of players on the screen. In this script, we'll need a function that takes a list of PlayerData objects and displays them:

using UnityEngine;
using System.Collections.Generic;

public class PlayerListDisplay : MonoBehaviour
{
    public void DisplayPlayers(List<PlayerData> players)
    {
        foreach (PlayerData player in players)
        {
            Debug.Log($"{player.Name}: {player.Score}");
        }
    }
}

Creating a Stub for Player Data

At this point, we need some data to display on the screen. In a real-world scenario, we might retrieve this data from a server or a database. However, we don't want to depend on an external source while developing and testing our display function. Instead, let's create a stub that generates a list of fake players with random scores.

Create a new C# script called PlayerDataStub.cs and add the following code:

using System.Collections.Generic;

public class PlayerDataStub
{
    public List<PlayerData> GenerateFakePlayers(int count)
    {
        List<PlayerData> fakePlayers = new List<PlayerData>();

        for (int i = 0; i < count; i++)
        {
            PlayerData player = new PlayerData
            {
                Name = $"Player {i + 1}",
                Score = Random.Range(0, 100)
            };

            fakePlayers.Add(player);
        }

        return fakePlayers;
    }
}

With this stub in place, we can generate a list of fake players to test our DisplayPlayers function without relying on external data sources.

Testing the Display Function

Now, let's test the DisplayPlayers function by providing it with the fake data generated by our stub. Modify the PlayerListDisplay.cs script as follows:

using UnityEngine;
using System.Collections.Generic;

public class PlayerListDisplay : MonoBehaviour
{
    private PlayerDataStub _playerDataStub;

    private void Start()
    {
        _playerDataStub = new PlayerDataStub();

        List<PlayerData> fakePlayers = _playerDataStub.GenerateFakePlayers(10);
        DisplayPlayers(fakePlayers);
    }

    public void DisplayPlayers(List<PlayerData> players)
    {
        foreach (PlayerData player in players)
        {
            Debug.Log($"{player.Name}: {player.Score}");
        }
    }
}

With this setup, we can quickly test and iterate on our DisplayPlayers function without worrying about external dependencies.

Introducing a Faker for More Realistic Data

While our stub generates fake player data, the names and scores might not be very realistic. To improve the quality of our test data, we can introduce a faker library like Bogus. This library can generate more realistic names and other data, making our tests more representative of real-world scenarios.

First, add the Bogus package to your Unity3D project using the package manager. Then, modify the PlayerDataStub.cs script to use the Bogus library:

using System.Collections.Generic;
using Bogus;

public class PlayerDataStub
{
    private Faker _faker;

    public PlayerDataStub()
    {
        _faker = new Faker();
    }

    public List<PlayerData> GenerateFakePlayers(int count)
    {
        List<PlayerData> fakePlayers = new List<PlayerData>();

        for (int i = 0; i < count; i++)
        {
            PlayerData player = new PlayerData
            {
                Name = _faker.Name.FullName(),
                Score = _faker.Random.Number(0, 100)
            };

            fakePlayers.Add(player);
        }

        return fakePlayers;
    }
}

Now our stub generates more realistic player data, allowing us to test our DisplayPlayers function under more accurate conditions.

Conclusion

In this case study, we demonstrated how to use mock data, stubs, and fakers in a Unity3D project. By creating a stub for player data and using a faker library to generate more realistic test data, we were able to quickly iterate on our DisplayPlayers function without relying on external data sources.

Implementing these techniques in your projects can help you save time and reduce dependencies, allowing you to focus on building the core functionality of your application.

Wrapping Up

Throughout this post, we've discussed the importance of reducing iteration times when developing new features. We've looked at how using mock data, stubs, and fakers can help you achieve that by minimizing dependencies on external data sources or code that's not yet implemented.

By understanding and applying these techniques, you can focus on building the core functionality of your application, iterate faster, and adapt to real feedback sooner. Remember, it's essential to have a working MVP as fast as possible, so you can modify and improve your product based on actual experience and feedback.

Don't be afraid to fake it till you make it, and happy coding!

Remember to create your boxes and fill them later, not the other way around

Debezium Custom Converters - TimestampConverter

Oryan Moshe — Tue, 26 May 2020 09:37:29 +0000

Debezium Custom Converters

Creating custom converters using Debezium's new SPI to override value conversions

Introduction

Background about the `TimestampConverter`

Hey, my name is Oryan Moshe, and I started my own community for senior developers in Israel named in.dev.

I'm also a Lead Architect at Rivery, We are a fully managed data integration platform, and part of my job here is developing the ability to stream changes straight from the clients’ databases, to our platform, using change data capture (CDC).

The last time I coded in Java was 7 years ago, so if you have any suggestions to improve the code shown here please feel free to comment below!

You can find the converter right here:
https://github.com/oryanmoshe/debezium-timestamp-converter/

CDC — Change Data Capture

Before we talk about Debezium, we have to talk about CDC.

CDC is a way for us to get the changes happening in the database (as opposed to the actual data)

This means we can actually get every state that every record has been through in the database.

CDC is useful for a number of cases:

Compiling a log of record changes
Undoing (or reverting) a change
Tracking record deletion (which is not simply a matter of using SELECT)

What is Debezium Anyway?

Debezium is an open source platform, maintained by Red Hat, that allows developers to implement CDC into a Kafka infrastructure.

Debezium actuates CDC by configuring connections using the provided Kafka Connect data connectors. Currently there's support for MySQL, PostgreSQL, Microsoft SQL Server, MongoDB, and even some limited support for Oracle.

What are converters, and why would we need a custom one?

All messages produced by Debezium are processed before entering the designated topic.

This ensures that all fields of a given type (defined by the schema) behave the same.

In other words, all DATE fields on all of the databases will be transformed into the same format. This is, by default "Days since epoch".

But this behavior isn't always wanted, especially in this temporal example.

For our particular use case we need all temporal fields to be in the same format, whether the type is DATE, DATETIME, DATETIME2, TIME or TIMESTAMP.

The format we chose was YYYY-MM-dd'T'HH:mm:ss.SSS'Z'.

Creating a custom converter

Here's an explanation for each step needed to create our TimestampConverter.

The basics of custom converters

To allow such behavior, the Debezium SPI (Service Provider Interface) was added to Debezium Version 1.1.

This allows developers to make their own converters with Java, by creating a class that implements the io.debezium.spi.converter.CustomConverter interface.

The First Gotcha

What we didn't know when we started developing this converter, is that once we registered a custom converter to a temporal column, Debezium's behavior became sporadic. Sometimes it'll pass a DATE column as "Days since epoch", as expected, but sometimes it'll pass it as a string, matching the date format of the database it came from.

This meant we had to have all of our bases covered, both for numeric values (let's say, "Days since epoch") and for all date format databases can produce (YYYY-MM-dd, dd/MM/YYYY, YYYY-MMM-dd, etc.)

Things got a bit complicated on the logic front, but let's not get into this right now.

What's needed for our custom converter to work

Each converter has to implement at least two methods to be harnessed by Debezium:

configure

This method runs when the connector is initialised. It accepts one argument:

props
An object of type java.util.Properties, containing all of the properties we passed to our converter instance.

converterFor

This method runs once for each column defined in our schema, and its job is to define (a.k.a "register") the converter for each. It accepts two arguments:

column
An object of type io.debezium.spi.converter.RelationalColumn, containing the definition of the column we're currently handling, including its name, type, size, table, etc.

registration
An object of type io.debezium.spi.converter.CustomConverter.ConverterRegistration, an internal definition, that has one method: register.

Using the `configure` method

As stated above, we use the configure method to pass properties into our converter. This is important because we can use the same converter for multiple connectors, and change its behavior according to these properties.

For our TimestampConverter we wanted to pass four properties:

debug – Indicates whether to print debug messages. Defaults to false.
format.date – The format to convert all columns of type DATE. Defaults to YYYY-MM-dd.
format.time – The format to convert all columns of type TIME. Defaults to HH:mm:ss.
format.datetime – The format to convert all other temporal columns. Defaults to YYYY-MM-dd'T'HH:mm:ss.SSS'Z'.

All of these properties are optional and have default values associated with them.

To support them we defined each of them as a class property with the default value. Inside the configure method we assigned them with the passed value:

public class TimestampConverter implements CustomConverter<SchemaBuilder, RelationalColumn> {

    public static final String DEFAULT_DATETIME_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'";
    public static final String DEFAULT_DATE_FORMAT = "YYYY-MM-dd";
    public static final String DEFAULT_TIME_FORMAT = "HH:mm:ss.SSS";

    public String strDatetimeFormat, strDateFormat, strTimeFormat;
    public Boolean debug;

    private SimpleDateFormat simpleDatetimeFormatter, simpleDateFormatter, simpleTimeFormatter;

    @Override
    public void configure(Properties props) {
        this.strDatetimeFormat = props.getProperty("format.datetime", DEFAULT_DATETIME_FORMAT);
        this.simpleDatetimeFormatter = new SimpleDateFormat(this.strDatetimeFormat);

        this.strDateFormat = props.getProperty("format.date", DEFAULT_DATE_FORMAT);
        this.simpleDateFormatter = new SimpleDateFormat(this.strDateFormat);

        this.strTimeFormat = props.getProperty("format.time", DEFAULT_TIME_FORMAT);
        this.simpleTimeFormatter = new SimpleDateFormat(this.strTimeFormat);

        this.debug = props.getProperty("debug", "false").equals("true");

        this.simpleDatetimeFormatter.setTimeZone(TimeZone.getTimeZone("UTC"));
        this.simpleTimeFormatter.setTimeZone(TimeZone.getTimeZone("UTC"));
    }
}

Using the `converterFor` method

Now it's time for the big moment. Each column must be converted to its respective format.

First of all, we have to understand the type of the column we're currently handling. This is determined using column.typeName.

If the type is any of the temporal types (defined as a class constant) we handle it accordingly. If it's not, we do nothing, and Debezium will take control.

To tell Debezium to convert a specific column to something else, we need to use the registration passed to us. Then register it, providing a schema (create one of type string and make it optional) and a converter.

The converter is just a function, or in our case a lambda, that receives an Object. This is the source value, and returns a value matching the schema we provided. In our case, we needed to return a String (or null, because we made it optional).

@Override
public void converterFor(RelationalColumn column, ConverterRegistration<SchemaBuilder> registration) {
    if (SUPPORTED_DATA_TYPES.stream().anyMatch(s -> s.toLowerCase().equals(column.typeName().toLowerCase()))) {
        boolean isTime = "time".equals(column.typeName().toLowerCase());
        registration.register(datetimeSchema, rawValue -> {
            if (rawValue == null)
                return rawValue;

            Long millis = getMillis(rawValue.toString(), isTime);

            Instant instant = Instant.ofEpochMilli(millis);
            Date dateObject = Date.from(instant);

            switch (column.typeName().toLowerCase()) {
                case "time":
                    return simpleTimeFormatter.format(dateObject);
                case "date":
                    return simpleDateFormatter.format(dateObject);
                default:
                    return simpleDatetimeFormatter.format(dateObject);
            }
        });
    }
}

In this code snippet look at the two crucial parts we have mentioned before. These are the call to registration.register, and the return statements.

Using a Custom Converter with Debezium

Installation

Installation in our Debezium cluster is straight-forward. We just need to add the .jar file of the converter to the connector we want to use it in.

The Second Gotcha

Notice I said " ... to the connecter we want ... ", this is a thing Debezium didn't make clear in the documentation. We need to add this converter to every connector if we want to use it in.

Let's say the base folder for connectors is /kafka/connect. Then inside we'll find folders like debezium-connector-mysql, or debezium-connector-postgres.

We need to add the converter .jar file to each of those folders if we intend to use it.

Configuration

After adding the .jar file to our connector, we can configure our connectors to use it!

To do so all we need to do is add the following keys to our existing configuration:

"converters": "timestampConverter",
"timestampConverter.type": "oryanmoshe.kafka.connect.util.TimestampConverter"

If we need to customize the formats of specific data types, we can use these additional configuration keys:

"timestampConverter.format.time": "HH:mm:ss.SSS",
"timestampConverter.format.date": "YYYY-MM-dd",
"timestampConverter.format.datetime": "YYYY-MM-dd'T'HH:mm:ss.SSS'Z'",
"timestampConverter.debug": "false"

Conclusions

The addition of an SPI to Debezium brought a lot to the table in term of custom converters.

This allows us to get a tailored fit CDC connector, with the data streaming into our Kafka cluster exactly in the format we want.

I didn't include the actual logic, converting the values from their raw format into the epoch time (this part is contained in the getMillis method)

But I have published the TimestampConverter as open source, so anyone can read the code there, use the converter in an application (be it as a .jar file found in the releases section, or as a dependency found in the packages section), or contribute to its development!

Feel free to suggest contributions to this converter, and share with me what kind of converters you created using the new Debezium SPI, and which ones you wish were made!

Links

To read more about Debezium Custom Converter visit their official documentation:
https://debezium.io/documentation/reference/1.1/development/converters.html

Link to the repository of my TimestampConverter:
https://github.com/oryanmoshe/debezium-timestamp-converter

Ruby Compact Internals

Oryan Moshe — Sun, 11 Aug 2019 23:25:04 +0000

I encountered a weird phenomenon regarding ruby's compact method, and I thought I'd share my findings with you.
When working on "The Grid" dashboard here at monday.com, I got to a point where I had an array of hashes, and some of the hashes were empty.
I needed to get rid of those.

Bottom line first

To get the result I wanted I just did a.reject { |v| v.blank? } instead of a.compact.

Expectation VS Reality.

When I used compact on the array I was a bit surprised by the result.

a = [{ key: 'value' }, { key: 'value2' }, {}]

puts a.compact
#[{ key: 'value' }, { key: 'value2' }, {}]

I expected the compact method to just go over each element in the array, check if it's present? and if it is return it.
Something like this:

def compact
  select { |value| value.present? }
end

So when I had an array with empty hashes that wouldn't go away I had to find the explanation.

The implementations behind the scenes

As usual, I dug deeper for the implementations.

From past research I know ruby's Hash also has a compact method and I know it's pretty straightforward, for each key in the hash we check if it's nil, if it's not we return it.
Super easy. Similar to what I suggested above, easy to understand, although it wouldn't solve the issue (because we use nil? instead of blank? or !present?)*

activesupport/lib/active_support/core_ext/hash/compact.rb, line 12

def compact
  select { |_, value| !value.nil? }
end

In arrays however the implementation is more complex. It is implemented purely in C.

static VALUE
rb_ary_compact(VALUE ary)
{
  ary = rb_ary_dup(ary);
  rb_ary_compact_bang(ary);
  return ary;
}

C is love, C is life

If you don't have any background in C, this next part might be less interesting to you. I'll try to make it as understandable as possible, but I highly recommend learning C if you have the time!

Pointers background

In case you don't know, the way we access our program's memory (the place where the variables are stored) is by using pointers.
These pointers are basically just an address in the memory where we can find the beginning of our variable (each variable takes a different amount of memory, we just get the required number of bytes starting at the pointer's position)

Arrays work similarly (actually, an array is literally a pointer), let's say we have an array of 3 characters, each character consists of one byte, all 3 bytes are stored one after the other in memory.
We can go to the first address in the array (or the pointer to that array), and take 1 byte, that's the first character. Then we take another, that's the second, and then the third.

char arr[3] = {'m', 'i', 'p'}
/*
Behind the scenes it's stored like so:
... 6d 69 70 ...
    ^ arr[0] == *arr
... 6d 69 70 ...
       ^ arr[1] == *(arr + 1)
... 6d 69 70 ...
          ^ arr[2] == *(arr + 2)
*/

Another thing to note: in C, if we have a pointer stored in a variable, let's say ptr, this variable will contain the address to the value. To get to the value itself we have to use the "dereference" operator (an asterisk before the variable name) like so *ptr.

Back to code

Above we saw the C implementation of ruby's array compact, we can see a function that receives an array, duplicates it (using the rb_ary_dup function), compacts it using the rb_ary_compact_bang function ("bang" is the same as the "!" symbol, so it basically runs compact!), and then returns the duplicated, compacted array.

To find that rb_ary_compact_bang function I went to ruby's source code:

ruby/array.c, line 5022

static VALUE
rb_ary_compact_bang(VALUE ary)
{
  VALUE *p, *t, *end;
  long n;

  rb_ary_modify(ary);
  p = t = (VALUE *)RARRAY_CONST_PTR_TRANSIENT(ary); /* WB: no new reference */
  end = p + RARRAY_LEN(ary);

  while (t < end) {
    if (NIL_P(*t)) t++;
    else *p++ = *t++;
  }
  n = p - RARRAY_CONST_PTR_TRANSIENT(ary);
  if (RARRAY_LEN(ary) == n) {
    return Qnil;
  }
  ary_resize_smaller(ary, n);

  return ary;
}

Don't close the post! This code isn't friendly for people with no experience in C, but most of it is memory management, some games of allocating memory correctly, resizing arrays to prevent memory leaks etc. I can do a deeper dive in a seperate post if anyone wants.

The interesting part is this one:

while (t < end) {
  if (NIL_P(*t)) t++;
  else *p++ = *t++;
}

Line by line

As I explained above, an array is just a set of variables stored one after the other in our program's memory, so we can get a pointer to the first variable, and then just increase it by 1 every time to get to the next one (t++ increases t by 1)

What we have here is a loop that goes from the beginning of the array, with our current position in the t variable (a pointer to the array), and we iterate over every element until we reach the end of the array (the address of the beginning of the array + the length of the array)

Every time, we check whether the current value is nil (using the NIL_P macro defined in include/ruby/ruby.h line 482)

If it is nil we just increase the pointer, advancing one spot in our array.
If it's not nil we put the current value from the original t (remember, the * dereferences the pointer) into another array we keep stored in the pointer p, and then advance the pointer one spot.

Bottom line

In a nutshell, it's the same as doing a.reject { |v| v.nil? } but more memory aware and efficient.

Just to show the difference in efficiency I benchmarked both ways against an array of 20,000,000 elements with alternating integers and nils.

As you can see, array's compact method using C null pointer elimination is 6.6 times faster than the reject method!

I hope you enjoyed learning a bit about ruby's internal mechanisms!

* This is the activesupport implementation of Hash compact, if you'd like to see the C implementation you can go to ruby/hash.c line 4110, but it's pretty similar!

"One TV with a number on it can be better than the best of managers."

Oryan Moshe — Mon, 05 Aug 2019 23:35:15 +0000

Here at monday.com we are always excited to show off our dashboards to visitors.
No, seriously, it's like the first thing anyone that comes to our offices sees.

We have more TVs than we have employees, and we still fight on what to put on each of them.

A dashboard? Do you mean that thing in my car?

So I guess we should begin with what is a dashboard

dash·board

/ˈdaSHbôrd/

a graphical summary of various pieces of important information, typically used to give an overview of a business.

Nothing kicks off a good tech post like a bone dry dictionary definition, but the important takeaway from this definition is that it's wrong.

I mean, it's probably right if you search the dictionary, but it doesn't convey the true meaning of a dashboard.

Let me try it myself:

Important information displayed in a correct and engaging way in order to drive people to action.

Much better.

If the data is not important, then there's no reason to display it, and if it's not displayed correctly it's better not to show it at all, as incorrect data is even worse than no data, and if the data is important and correct, but doesn't engage people or drives them to action, what is the purpose of that dashboard to begin with?

All of my data is important!

So how do we define important information? How do we decide what belongs on a dashboard and what can stay in the database?

We have a term called KPI - Key Performance Indicator.
This is usually a number (or a collection of numbers) that everyone in the company understands, and are able to explain the changes in it over time (whether it improved, got worse, and what affects it in general)

We have a different KPI for every team and use case. Sometimes it's easy to define, sometimes it's hard and we have to use proxy KPIs (numbers that we know affect our main KPI in indirect way) and sometimes it's just "impossible" to find (more on that later)

KPI Engineering

Finding KPIs is a true artform, it takes time to hone that skill.
Some rules of thumb for a good KPI are:

It should be measurable
It should change frequently ("frequent" is a relative term, that varies from one business to another)
There's a clear indication on which direction is better (should we drive it up or down)
People in the company should be able to move it

Qualitaitve VS Quantitative

Sometimes it's simple to measure your KPI (quantitative), for example our Customer Success' team main KPI is their response time, and their goal is to answer a ticket under 10 minutes. Simple enough, just collect the data from Zendesk and calculate the Average response time.

But sometimes it's not as simple, sometimes your KPI is qualitative, meaning you have to deduce it from people's reaction to your work. This could be from high-touch conversations with them, reviews, or surveys they answered. This kind of KPI is not measured as easily and, as such, is harder to display on a screen. For example, my KPI is to make our marketing team work as efficiently as possible (officially we want to help them do 10 times what they were able to do without us), but it's not something we can measure or calculate, all we can do is sit with them, hear their issues and see if we were able to address them.

Proxy KPIs

Some KPIs take time to "mature" - meaning, if we change something in the system, we can only measure them after a maturity period passes.
For example, one of our main KPIs "Conversion to paying" which is basically the percentage of paying accounts out of the total number of accounts.

At first glance it seems simple enough, but the thing is, monday.com has a trial period of 14 days, and usually accounts don't pay until their trial is up.

So, if we changed something, and it hurt our conversion rate, we would only know about it after 14 days!

To solve that issue we have to find a "proxy KPI", a metric that matures faster than our main KPI, but has a high correlation to it.
One of those proxies in our specific case is the amount of content created in the system. After only 3 days we can know (with relatively high certainty) whether an account is going to convert into a paying customer or not.

Keep tuned for another post solely dedicated for KPI engineering!

Correct? Of course my data is correct!

Now, I'm sure your data is correct just like ours is! But the way you display your data is just as important as how accurate it actually is.

I can talk for hours on how to "hack" data and display it in a way where it shows something completely wrong in a very convincing way, but for now I'll settle for 2 examples of data hacking.

Before starting, it's important to reiterate that if you display your data incorrectly you're lying to yourself and your company, and it makes you think you're in a place you're not. If you don't display your data correctly, you might as well not display it at all.

Hiding data

The first, and most naive way to "hack" your data is just to hide the right stuff. Do you have a cohort of accounts that misbehaves? Just remove it from the graph!
Displaying multiple graphs of metrics and your churn graph is on the rise? Just remove that block!

Doing so will make you look better by hiding the worst things in your data. The problem is it'll hide those things from you as well, and you can't learn from your past mistakes if you bury them underground.

Axes scaling

Another, less obvious way to show your data incorrectly is by not locking your axes, the byproduct of which is increased vulnerability to graph manipulation you wouldn't even be aware of.

If you make your graph narrower, but not shorter, you will put more weight on to your Y axes, making smaller changes in your graph look much more dramatic.

Just in case I wasn't clear here are two examples, one of an incorrect graph, with scaleable axes, and another of a correct graph in a dashboard, where the axes are locked.

Here's how not to do it

And here's how to do it

Picking the correct visualization

Another part of displaying your data correctly is picking the correct visualization to display it.

Not all visualizations work on all kinds of data!
For example, a time series (continuous data where your X axis is time based) should usually be displayed in a line chart.
A stacked time series (where the Y values of multiple series can be summed up to receive a meaningful number) should be displayed in an area chart (just like line chart, but the area under the line is colored as well)

Categorical data is usually best visualized using a bar chart, but if all categories are a part of a bigger number then you should probably use a pie chart.

If you have more than 1 dimension to display (for example, displaying both the number of visits to a page, and how much they cost you) you should use a bubble chart. Or a heatmap! Speaking of heatmaps, those are also good at displaying a weekly time series and correlation matrices.

Bottom line, there are so many visualizations, each one fitting a specific use case, and if people don't understand your data at a glance it's a good signal you should rethink your chart types.

Engaging? How can numbers be engaging?

Here's the best thing about creating a dashboard. You have to make people care about it!
Creating an important dashboard, with the correct visualization just isn't enough. I created countless dashboards that aren't in use just because they weren't engaging.

Visual cues

Sometimes all a dashboard has to do to display data in an engaging way is just to find the right way to drive people forwards.

For example, one of our Customer Success' team main KPIs is having a response time of less than 10 minutes, so everytime there's a ticket in the queue for more than 10 minutes they see it on their dashboard burning in flames.

Or "The Grid", which displays both product and technical alerts for company-wide usage, that is usually completely green, but when something turns red it's immediately visible and can be easily inspected using our touch TVs.
This makes people want to keep The Grid green at all times.

Or our "Celebrations" overlay, which allows us to display a celebration with an employee picture, confetti and a short message whenever we want (for example, when one of our consultants closes a big deal with a customer.
Everyone wants to celebrate their successes, and that's a nice way to drive people to succeed.

Mobile responsiveness

Don't underestimate the importance of mobile dashboards!
Many people want to know they're current state on their way up the elevator, or between meetings.

Some people don't even have time to sit by a computer! If your dashboard displays an important metric that people would want to consume often (like "The Grid") it should look as good on mobile as it is on desktop. It also fuels the "addictive" nature of dashboards, which is awesome for you!

Sounds

Your TVs probably have speakers built in, and you can use them! Many of our dashboards make sounds whenever something important happens.
There's a "gong" sound when the consulting team closes a deal, music when a customer starts paying, notifications when someone starts deploying to production, or alerts when deployment to production fails.

It is important not to go over the edge though, as many people started muting the TVs after we started having many paying customers every minute, each making a sound. We had to decrease the frequency of those sounds.

Summary

In conclusion, dashboards are the thing that drives our company forwards, without them displaying important information in a correct and engaging way, we wouldn't be able to get to where we are right now, and I think any company that wants to succeeded should be data driven and make sure it has the right dashboards for the job!

Replicating PostgresSQL into MemSQL's columnstore

Oryan Moshe — Mon, 22 Jul 2019 23:49:18 +0000

Making the impossible hard

I… Don't think this is how it should look like

So it's this time of the year again, we need to upgrade our MemSQL cluster and expand our contract to fit the new cluster topology.

We really outdid ourselves this time, expanding to a 1TB cluster is impressive, especially when it's completely not justified.

The background

Wait. A 1TB cluster?

Yeah yeah, call us spoiled, but querying on PostgresSQL (PG from now on) is just not the same.

Sure, you can get ok speeds if you're using the correct indexes and optimize your queries, but it's not even comparable to the performance you get from the memory based rowstore, or the insanely fast aggregations of the columnstore.

A short (short) summary of Mem's different storage types

So we basically have 2 types of storage in Mem, rowstore and columnstore.

The rowstore is stored pretty much like any other database, but in the memory instead of the disk (crazy fast). This means each row is stored together with all of its columns

The columnstore is sort of a transposed rowstore, instead of storing rows we store columns (thank you captain obvious) which allows us to make aggregations stupid fast (think about it, instead of going to each row and summing the "cost" column, we can just go to the "cost" column and sum it up). The columnstore is stored on the disk.

The issue is MemSQL's license costs more as we have more memory in our cluster, not to mention the cost of the machines themselves (1TB of memory isn't exactly cheap)

"So why not store everything in the columnstore? It's cheaper both license and infrastructure wise, and it's stupid fast!" You might ask (if you talk to yourself while reading tech articles)

So here's the catch - the way the data is stored in a columnstore makes it incredibly fast in aggregated queries, and allows amazing compression, but updating a row is slow.

How slow? If we need to update some columns for rows in a specific day, it's faster for us to delete the data from this day and re-insert the updated one instead of updating the existing rows.

So how do we store our data?

Well, in my team we use 7 flavors of databases (might be more, can't really keep track these days) but the main ones are PostgresSQL, hosted and managed by RDS (for transactional processing) and MemSQL, hosted on EC2 and managed by yours truly (for analytical processing*)

Instinctively, most of our data is stored in PG (excluding some large columnstore tables containing North of 8B records)

The problem is, once you go Mem you never go back, so we created a replication service that can replicate a row from PG to Mem's rowstore in real-time. This allows us to enrich our columnstore only tables, create ETLs, and most importantly, speed up queries.

If you're here you either use Mem and thus know its performance, or just like to go around dev.to, reading random articles about niche DBs. If you're the latter let me hit you with some numbers.

A completely reasonable query, consisting of 6 joins, took 30 minutes to run on PG. After optimizing it for 2–3 hours, adding indexes, banging my head against the wall and praying for swift ending, I was able to cut it down to 3 minutes.

Taking exactly the original query (the 30 minutes one) and running it on Mem, it took 1.87 seconds.

The real deal

Problems definition, AKA what's making me lose sleep

So Mem is expensive, we're almost at our new license limit (after more than doubling it) and there's no way we can go back to query exclusively on PG.

The solution seems simple, move big tables to the columnstore, free up some memory so you don't have to increase your license and upgrade your machines.

For this article I'll use our table touch_points as an example, it's our largest (both in memory and row count) table stored in a rowstore - it has over 180M rows, and weighs more than 190GB.

Why is it in our rowstore? First, cause we replicate it from PG, and so far our service only supports replicating to rowstore tables, but, more importantly, it needs to be updated. Out of 30 columns, 2 might get updated - visitor_id and cost.

Solutions

The first solution

So this was the "correct" solution design-wise.

In short, using ActiveRecord callbacks I kept 2 tables up to date, one is the touch_points table in the columnstore, containing all columns that exist presently on touch_points except the 2 that get updated. Other than touch_points I created a table called touch_points_extra_data in the rowstore, containing the 2 missing columns and 1 ID column that allows me to connect the 2 tables.

As I said, this was the correct solution design-wise, the problem is that so much could go wrong. With so many moving parts, all dependent on rails hooks, we were sure to get out of sync sometime. Not to mention the fact that we'll have to edit all of our queries from touch_points to add that extra JOIN.

The second solution, AKA "The bruteforce"

So we realized our top priority is to keep the data correct, and we were willing to make some compromises (foreshadowing)

I decided to replicate the whole table, as is, from PG once in a while. This way we can make sure that (up to the moment of replicating) our data will be identical in both DBs.

The compromise is that we are used to having this data updated in real time, and now it'll be outdated until the next replication. This is a compromise I'm willing to take.

The technical part

Easier said than done

So apparently replicating a whole table from one DB to another isn't as straightforward as you would think. Especially when the two DBs run on different engines entirely.

The first thing I tried is using pg_dump, with the plain file format (which essentially creates a file with loads of INSERT statements) and then convert it to mysql syntax and load to Mem.

Sounds great right? I started the pg_dump, and 5 hours later it wasn't even close to finishing, while the dump file was already at 60GB. pg_dump with the plain option is the most inefficient way to store data. 5 hours delay in replication is unacceptable.

If at first you don't succeed.. Fail again

The next thing I tried was using the COPY command of PG, this command can copy (duh) a table, or a query into a FILE, a PROGRAM, or STDOUT.

First I tried using the STDOUT option (the simplest one, and it doesn't create a footprint of a huge dump file)

psql -U read_user -h very-cool-hostname.rds.amazonaws.com -p 5432 -d very_cool_db -c\
"\COPY (SELECT * FROM touch_points) TO STDOUT\
WITH(DELIMITER ',', FORMAT CSV, NULL 'NULL', QUOTE '\"');" > touch_points.csv

And it worked! I got a "dump" file from PG containing our whole touch_points table, in just under 20 minutes.

Now we just need to import it to Mem, but why do I need the file? I can just pipe the result right from PG straight into Mem!

So I needed to create the part where Mem receives this csv-like table and loads it into the db. Luckily Mem is MySQL compatible and provides us with the LOAD DATA clause!

LOAD DATA LOCAL INFILE '/dev/stdin'
  SKIP DUPLICATE KEY ERRORS
  INTO TABLE touch_points_columnstore
  FIELDS
    TERMINATED BY ','
    ENCLOSED BY '"'
    ESCAPED BY ''
  LINES
    TERMINATED BY '\n'
  MAX_ERRORS 1000000;

Now, as I said we want to pipe that data right into Mem, so we need to create a connection to our DB:

mysql -h memsql.very-cool-hostname.com -u write_user -P 3306 -D very_cool_db\
-p'4m4z1nglyS3cur3P455w0rd' -A --local-infile --default-auth=mysql_native_password -e\
"LOAD DATA LOCAL INFILE '/dev/stdin' SKIP DUPLICATE KEY ERRORS\
INTO TABLE touch_points_columnstore FIELDS TERMINATED BY ','\
ENCLOSED BY '\\\"' ESCAPED BY '' LINES TERMINATED BY '\\n' MAX_ERRORS 1000000;"

And then just pipe the data from PG to that connection!

psql -U read_user -h very-cool-hostname.rds.amazonaws.com -p 5432 -d very_cool_db -c\
"\COPY (SELECT * FROM touch_points) TO STDOUT\
WITH(DELIMITER ',', FORMAT CSV, NULL 'NULL', QUOTE '\"');" |\
mysql -h memsql.very-cool-hostname.com -u write_user -P 3306 -D very_cool_db\
-p'4m4z1nglyS3cur3P455w0rd' -A --local-infile --default-auth=mysql_native_password -e\
"LOAD DATA LOCAL INFILE '/dev/stdin' SKIP DUPLICATE KEY ERRORS\
INTO TABLE touch_points_columnstore FIELDS TERMINATED BY ','\
ENCLOSED BY '\\\"' ESCAPED BY '' LINES TERMINATED BY '\\n' MAX_ERRORS 1000000;"

And... It worked! But it took an 2 hours to complete. I'm sure we can do better than that.

Compression is your friend

So two cool things important to understand about loading data into Mem are:

When inserting a data file into Mem, it copies the file locally to the aggregator and splits the file between the nodes of the cluster, speeding up the data load significantly.
Mem supports receiving gzipped-compressed data files.

Combining these two pieces of information made me understand that creating the file in the middle maybe isn't as bad as I thought.

I can compress that file, making storage a non-issue, it'll also speed up the transfer of the file to the aggregator (before splitting) by cutting out most of the network related latency, and it'll allow Mem to split the data between the nodes.

Let's do it!

First of all I need to modify the PG part so instead of piping the content to STDIN, it pipes it to a PROGRAM, and in our case, gzip

psql -U read_user -h very-cool-hostname.rds.amazonaws.com -p 5432 -d very_cool_db -c\
"\COPY (SELECT * FROM touch_points) TO PROGRAM 'gzip > /data/tmp/replication/touch_points_columnstore.gz'\
WITH(DELIMITER ',', FORMAT CSV, NULL 'NULL', QUOTE '\"');"

After we created this tmp file we need to load it. Luckily the only thing we have to do is to change the source of the input file!
Our finished script looks like this:

psql -U read_user -h very-cool-hostname.rds.amazonaws.com -p 5432 -d very_cool_db -c\
"\COPY (SELECT * FROM touch_points) TO PROGRAM 'gzip > /data/tmp/replication/touch_points_columnstore.gz'\
WITH(DELIMITER ',', FORMAT CSV, NULL 'NULL', QUOTE '\"');" &&\
mysql -h memsql.very-cool-hostname.com -u write_user -P 3306 -D very_cool_db\
-p'4m4z1nglyS3cur3P455w0rd' -A --local-infile --default-auth=mysql_native_password -e\
"LOAD DATA LOCAL INFILE '/data/tmp/replication/touch_points_columnstore.gz' SKIP DUPLICATE KEY ERRORS\
INTO TABLE touch_points_columnstore FIELDS TERMINATED BY ','\
ENCLOSED BY '\\\"' ESCAPED BY '' LINES TERMINATED BY '\\n' MAX_ERRORS 1000000;"

And that's it!

The created file weighs 7GB, and the whole process takes less than 20 minutes, so we can run it once an hour and have semi-realtime data!

Obviously this wasn't the end, I wrapped it up in a nice rails module that allows me to replicate any query from PG to Mem easily, including truncating the old data and using 2 tables to minimize the downtime during replication.

Feel free to contact me with any questions!

*including but not limited to analytics and transactions.

I spend one hour a week optimizing my development environment.

Oryan Moshe — Sun, 14 Jul 2019 23:57:07 +0000

The story of my unreasonable efficiency obsession - and the tools I use to fuel it

If you're a productivity freak like me you probably found yourself in the following situation (presumably in the last week):

You have a task to complete, you sit down, open up your laptop and suddenly, it happens.
That annoying little thing you have to do every single time. It might happen once an hour, once a day, maybe once a week, but it drives you bananas.

It could be something small like inserting your MFA codes, closing a pop-up or even switching to "normal" mode on vim.
It could be something more time consuming like rearranging your displays layout after replugging your Mac, changing your connection string to a readonly production database, or restarting your environment when changes are not affecting it.

It doesn't matter, the point is it's something you're doing instead of what you'd rather be doing.

8 months ago I decided to stop the madness. I started reserving 1 hour a week to get rid of these little nuisances.

Let's talk tools

So there are many tools that help me work towards my goal of ultimate efficiency. Some of them are free, some of them cost some bucks, heck, I even paid 50 bucks for one of them. You don't have to use the ones I suggest, and for every paid one I'm sure you can find a free one or create one yourself.

BetterTouchTool > $7.5 - $21

BetterTouchTool is the first tool I used to customize my development experience, I installed it even before starting my current position (I received my laptop beforehand) and I can't believe how much I use it every day.
In a nutshell - it allows you to customize everything "input related" in your Mac, including your Magic Mouse, trackpad, keyboard and even your Touch Bar (if you use a compatible version)

Karabiner-Elements > Free - Open source

Karabiner is one of those tools I expect every OS to have built in. It allows you to remap any key on your keyboard to do anything you want. Why isn't it a standard??

Alfred + Powerpack > £23 - £39

Alfred is probably the first tool I'll ever go to when trying to solve a workflow issue. Alfred (a free tool, massively extended by a paid "Powerpack") has way too many functions to tell you about here, but it's essentially Spotlight on steroids. It allows countless customizations and features out-of-the-box improvements that'll make you wish you met it before.

The changes that changed my life

1. Customizing my Touch Bar

If you have a Macbook pro with a Touch Bar, sooner or later you probably thought to yourself: "Damn, that is one piece of unutilized potentiel".
And it is, Apple definitely dropped the ball on that front.
Luckily, I used BetterTouchTool to customize it and make it not suck!

As you can see I left the esc key in the top left corner (we'll talk about the esc situation later), and I put a few shortcuts to applications I frequently use (Finder, Hyper terminal, VS Code)

That red Production / green Development block indicates what is the current state of my database.yml file (in rails based applications your connection strings are kept there), this allows me to know whether I'm working against my local DB or readonly production DB in a glance.

I also added a block that shows me my current internal IP address (useful when I want to share a link to a local server with a colleague), a mute button, a DateTime block with seconds (important when wanting to get a feel about query performance) that also takes me to my calendar on tap, and a "Coffee Break" button that locks my laptop.

Obviously when Spotify or Youtube are playing I have a block appear in the middle with some useful functions.

Another cool feature I utilized is 2 fingers drag to control brightness, and 3 fingers drag to control volume.

2. Replacing my Caps Lock with `ESC`

If you know me, you probably know I have a mild case of keyboard fetish. I build them, I collect them, I try to make the ones that best suit my style.

One thing I have to do on every system I use is remap my Caps Lock key to ESC, and especially in this specific Mac. I mean, who on earth would prefer § over a physical esc key??

If you use vim you know how important it is to have the esc key as close to you as possible, and as I started using it more and more I realized I'm using it for things other than vim. Closing pop-ups, blurring inputs, opening the console in Chrome devtools. Escape is the unsung hero of keys.

I use Karabiner-Elements to remap the useless Caps Lock (it's 2019, if you use Caps Lock it's time for some re-education) into an esc. Can't get easier than that.

3. Space Cadet Shift

In case you haven't read this excellent article by Steve Losh, I highly recommend you do, but the customization I liked (and used) the most ever since I read it is the "Space Cadet Shift Parentheses".

It turns your left_shift key into a ( and your right_shift key into a ) if they're pressed briefly, and treats them as normal shift modifier keys when held and combined with another key.

This allows me to type parentheses with one keystroke, which makes much more sense than the traditional way of shift + 9, as I probably use parentheses hundreds of times a day.

4. Clipboard history

This is truly a game changer. I specifically use Alfred's clipboard from the Powerpack binded to a cmd + shift + v shortcut, but you can use the free software "Jumpcut" instead.
What this essentially does is keep a history of your copied items, including text and images.

Originally, I started using it to stop these "F*ck, I copied over that super important thing I had in my clipboard and now I have to go back and copy it again" moments I kept having, but as time went on I started using it as an actual tool.

I copy multiple items from one application, then go to another application and paste them as needed without having to switch back.

5. Rearranging the display layout

If you have a Mac and 2 displays you're probably familiar with the issue.
You disconnect your laptop to go to a meeting, and when you come back your screens are all messed up, the spaces go to wherever they feel like and you're left there dragging windows from one display to the other for 2 minutes.

What I discovered is that you can change your main display 3 times (if you do it in the correct order) and the displays will go back to where you want them!
This in itself is a huge time-saver, but I wanted to take it to the next level, so I created an Alfred workflow.

A workflow is a set of actions, these can be scripts, changes in settings, push notifications and a ton of other actions, executed in order and triggered by a keyword in Alfred or a keyboard shortcut.

Using the free cli tool displayplacer, I was able to record the current displays state after each "main display change", and after I had all 3 I chained them together in a workflow triggered by the keyword arr (short for arrange).

Now every time I sit at my desk, I plug in my displays, press ⌥ + space (my Alfred shortcut), type arr in, and the displays are fixed!

6. Changing my DB connection string

The amounts of data we work with don't allow us to pull the data to our local machine and work on it, so we sometimes have to change our connection string (our database.yml file) to connect to a read-only replica of our production DB to see how the data looks on a new dashboard or what not.

This process happens about 6-7 times a day, and every time I have to open the database.yml in my editor, uncomment / comment the lines of the production DB, save the file and restart my rails server (which we'll solve later)

I created an Alfred workflow that simply replaces my database.yml with the correct one according to the command I typed in, and saves a file stating which file is currently in use (remember that red/green block in the Touch Bar?)

Typing dbd changes my DB to development, and dbp to production.

7. Automatically restarting a local server

Anyone who developed some node.js knows nodemon, a great utility that allows you to run a node server, and restarts it whenever the files change.
But did you know nodemon can work on any cli tool and watch any file type?

Changing my rs alias (rails server) to

alias rs="nodemon --watch config -e rb,yml --exec \"rails server -b 0.0.0.0 -p 3002\"

Allows me to run my server, and whenever a file in the config directory with an rb or yml extension is changed, it restarts the server! No more manual restart on database.yml change!

8. `sudo` TouchID

Whenever I need to type the master password of my Mac I'm annoyed. Why? Because I have TouchID! Why should I type a password instead of biometrically identifying myself?

I discovered that I can add a simple line to the sudo PAM (Pluggable Authentication Module) and from now on every time I need to sudo a command I just have to put my finger on the sensor!
Just add the following line to the top of your /private/etc/pam.d/sudo file:

auth sufficient pam_tid.so

9. MFA without my phone

For this one I'll refer you to this fantastic workflow that inspired me, but in short - you can create an Alfred workflow that pastes an MFA code without you having to open your phone.

To keep my accounts secure (there's a reason for MFA after all) I changed the script permissions to "root only" and ran it as sudo.
If you remember from #8, sudoing on my Mac requires a fingerprint, and this allows me to protect my MFA tokens with a fingerprint, while not having to reach for my phone every time GitHub invalidates my session.

10. Starting my development environment

Our development environment is semi-complex, and personally I like to work in a specific way.
My terminal is Hyper, customized exactly as I like it, I have one tab split into 3 sections, one for docker-compose, one for our npm transpiler, and one for the rails server.

Other than that tab I have 2 more tabs, one for a rails console and the other for git management.

Instead of opening all of these tabs and typing all of these commands every time I'm starting my devenv, I created an Alfred workflow triggered by the keyboard shortcut cmd + shift + b that does all of that for me!

Honorable mentions

Not all of my optimizations have a story behind them, some are trivial and are still huge time savers

Aliases

Use shell aliases.
Whenever you have a long command you have to type more than once in a lifetime, throw it into an alias. You'll be surprised by how much you use it.

Terminal configuration

Apart from aliases there are a lot of customizations to do in your terminal. From shell customization including fzf and history auto-complete to better tab management and better remoting tools (like mosh)
The terminal is your friend and you spend a lot of time in it. Treat it nicely.

Calculator

You can use Spotlight / Alfred as a calculator. Apparently this isn't as known as I thought it was.

Using spaces

I know it's hard to get used to compared to Windows windows, but honestly spaces are the best way to get around a Mac. Take the time, learn to use them and learn all the gestures. Navigating spaces is way faster than dragging windows around.

Trackpad 3 fingers drag

You can drag with 3 fingers instead of click and dragging on the trackpad. Turn on this gesture, your fingers will thank you!
You can find it in Preferences > Accessibility > Mouse & Trackpad > Trackpad Options... > Enable dragging > three finger drag.

I have many more optimizations just like the ones posted here, if you'd like me to create a follow-up post just say 🙃

You spend hours in front of your computer, make sure every minute counts.

The object that misbehaved — window.performance.memory

Oryan Moshe — Thu, 11 Jul 2019 23:05:16 +0000

How I troubleshoot issues that don’t make sense

Today we wanted to start tracking some new performance metrics on our landing pages, and more specifically, the memory usage.
As usual, we got the data from our trusty window.performance object (from now on will be referred to as "the enemy"), stringified it before sending and…

Surprisingly, all we got was "{}"!

How can it be? We have an object, with properties we can access, but stringifying it returns an empty object!

Down the rabbit hole I go

I started digging deeper into it, with the goal of turning the enemy into a string without hard coding the property names.

I tried spreading it

  const result = { ...window.performance.memory };
  console.log(result); // {}

I tried getting its components

  const keys = Object.keys(window.performance.memory);
  const values = Object.values(window.performance.memory);
  console.log(keys); // []
  console.log(values); // []

I tried destructuring it

  const { jsHeapSizeLimit } = window.performance.memory;
  console.log(jsHeapSizeLimit); // 2330000000

And surprisingly — it worked!

With great confidence that I’m going to crack this I tried "rest" destructuring:

  const { ...rest } = window.performance.memory;
  console.log(rest); // {}

What is going on? Why does this specific object refuse to play nice?

A light at the end of the tunnel (that turns out to be a train)

After some more tinkering I found another thing that didn’t make sense.

  for (const prop in window.performance.memory)
    console.log(`${prop}: ${window.performance.memory[prop]}`)
  // totalJSHeapSize: 13400000
  // usedJSHeapSize: 12700000
  // jsHeapSizeLimit: 2330000000

Iterating over the enemy’s properties does work, but getting its properties names doesn’t? Even Object.getOwnPropertyNames failed!

Although this solution could work for me (remember, my original goal was "turning the enemy into a string without hard coding the property names") I wanted to find a more elegant solution, and wanted to get to the bottom of this.

The first solution, AKA "not good enough"

The next part of my trip was when I started playing around with the prototype of the enemy, called MemoryInfo. I tried changing the prototype, assigning the enemy to a new object with a different prototype, creating an array from the enemy, and eventually playing around with a combination of all the techniques mentioned above.

  Object.getOwnPropertyNames(window.performance.memory); // []
  Object.getOwnPropertyNames(window.performance.memory.__proto__) // 
  ["totalJSHeapSize", "usedJSHeapSize", "jsHeapSizeLimit", "constructor"]

Success! Well… Sort of. I got the prototype property names, and I could use this array to get what I want using the following snippet

  const success = JSON.stringify(Object.getOwnPropertyNames(window.performance.memory.__proto__).reduce((acc,key) => {
       if (key !== 'constructor')
         acc[key] = window.performance.memory[key];
       return acc;
    }, {})
  );

  console.log(success) // " {"totalJSHeapSize":20500000,"usedJSHeapSize":18200000,"jsHeapSizeLimit":2330000000}"

Not great, not terrible.

And it does work (and it is a one-liner like I love), don’t get me wrong, but it’s just not as elegant as I wanted it to be, and nothing made any sense anymore.

I am one with the code

Fast forward 1 hour and I discovered that objects have properties, and these properties have metadata (called descriptor) and this metadata defines whether we can enumerate over the property, change it, and how we get it from the object.

So it must be that the properties have some sort of a metadata tag that prevents getting them in the conventional ways. From MDN I figured that enumerable is the property that interests me:

true if and only if this property shows up during enumeration of the properties on the corresponding object.

  Object.getOwnPropertyDescriptors(window.performance.memory.__proto__);
// {..., jsHeapSizeLimit: {get: ƒ, set: undefined, enumerable: true, configurable: true}, ...}

But the properties do have the enumerable metadata-property turned on, why aren’t they showing up when using Object.keys or Object.values?

The not-so-grand finale

Finally, I succeeded in "turning on" the enumerable flags on the enemy, and my data stringified beautifly

  const finale = JSON.stringify(Object.defineProperties(window.performance.memory, Object.getOwnPropertyDescriptors(window.performance.memory.__proto__)));
  console.log(finale); // "{"totalJSHeapSize":29400000,"usedJSHeapSize":23100000,"jsHeapSizeLimit":2330000000}"

Sweet, sweet one-liner

So what’s going on here?

We already saw that the descriptors of the MemoryInfo prototype should be fine, so the only explanation is that somehow they are not set (or overridden) on the enemy itself.

Using Object.defineProperties I am able to get a copy of the enemy, but with the correct descriptors (which we can just get from the prototype using Object.getOwnPropertyDescriptors)

Then just stringify and ship!

DEV Community: Oryan Moshe

The Secret Behind My Latest Blog Post: ChatGPT Did the Heavy Lifting

Mock Data & Stubs (Fake it till you make it)

Oryan Moshe ・ May 7 ・ 11 min read

Why the Iterative Method Outshines Traditional Prompting

Teaching ChatGPT Your Writing Style

Instructing ChatGPT to Ask Questions for Better Understanding

Orchestrating ChatGPT, Midjourney, and Grammarly Together

Improving Prompt Clarity with Grammarly

Generating Midjourney Prompts

Practical Prompt Examples and Tips for Effective ChatGPT Collaboration

Prompt Examples

Tips & Tricks for Effective Collaboration

Addressing Common Misconceptions and Concerns

Conclusion: Enhancing Your Blogging Process with ChatGPT

Mock Data & Stubs (Fake it till you make it)

Mock Data & Stubs (Fake it till you make it)

Why Reducing Iteration Time is Crucial

Mock Data, Stubs, and Fakers: The Building Blocks of Rapid Development

Mock Data

Stubs

Fakers

Understanding the Interaction Between Mock Data, Stubs, and Fakers

Putting It All Together: A Practical Example

Step 1: Creating Mock Data

Step 2: Implementing Stubs

Example: Using the fetchUsers() Stub

Step 3: Using Fakers

Best Practices and Potential Pitfalls

Keep Your Mock Data Realistic

Don't Overuse Stubs

Keep Your Fakers Consistent

Popular Libraries and Tools for Mock Data, Stubs, and Fakers

C# and Unity3D

JavaScript

Python

Ruby

Case Study: Implementing Mock Data, Stubs, and Fakers in a Unity3D Project

Setting Up the Project

Creating a Stub for Player Data

Testing the Display Function

Introducing a Faker for More Realistic Data

Conclusion

Wrapping Up

Debezium Custom Converters - TimestampConverter

Debezium Custom Converters

Introduction

Background about the TimestampConverter

CDC — Change Data Capture

What is Debezium Anyway?

What are converters, and why would we need a custom one?

Creating a custom converter

The basics of custom converters

The First Gotcha

What's needed for our custom converter to work

configure

converterFor

Using the configure method

Using the converterFor method

Using a Custom Converter with Debezium

Installation

The Second Gotcha

Configuration

Conclusions

Links

Ruby Compact Internals

Bottom line first

Expectation VS Reality.

The implementations behind the scenes

C is love, C is life

Pointers background

Back to code

Line by line

Bottom line

"One TV with a number on it can be better than the best of managers."

A dashboard? Do you mean that thing in my car?

dash·board

/ˈdaSHbôrd/

All of my data is important!

KPI Engineering

Example: Using the `fetchUsers()` Stub

Background about the `TimestampConverter`

Using the `configure` method

Using the `converterFor` method

2. Replacing my Caps Lock with `ESC`

8. `sudo` TouchID