DEV Community: Lighthouse

Adopting React at Lighthouse

Kenny De Pauw — Mon, 17 Nov 2025 10:25:44 +0000

A decade with Ember.js

For over ten years, Ember.js has been the reliable engine behind Lighthouse's core platform. As a comprehensive, "batteries-included" framework, Ember provided the structural consistency and predictability necessary for a large enterprise application. This stability allowed the engineering team to focus intently on delivering product value with high productivity and minimal technical debt.

Lighthouse still considers Ember.js a quality framework built on solid principles, but a strategic decision was needed to ensure the long-term sustainability and competitiveness of the platform for the next decade. Now is the time to look forward. After extensive research and testing, a pivot is necessary to make sure we stay competitive and provide our engineers with the best possible experience.

Why we had to change

As the company scaled through significant growth, three critical, growing risks tied to continuing reliance on a smaller ecosystem were identified:

1. Declining market share

While Ember is a quality framework, its declining market share put Lighthouse at a disadvantage:

Talent pipeline: Lighthouse needs to ensure it could attract and retain engineers who want to work with the most in-demand skills. React's current unmatched market dominance and massive developer community solve this challenge, immediately expanding the talent pool and making the company more attractive to top engineers.
Ecosystem health: The team often struggled to find modern, maintained, off-the-shelf solutions for needs (like advanced table components or up-to-date tooling). They were increasingly forced to build and maintain fundamental tools themselves, diverting resources away from product innovation.

2. M&A integration and future growth

Lighthouse's growth strategy partly consists of successful mergers and acquisitions (M&A). The core framework must support a playbook for rapid integration.

React is the most prevalent frontend technology globally; standardising on it dramatically increases the probability that an acquired company's technology stack will be familiar or quickly convertible, reducing the cost and complexity of integrating new teams and codebases.

3. AI tooling and future productivity

Lighthouse is committed to integrating AI-assisted development into its workflow to boost engineering productivity. React's market dominance makes it the primary target for AI model training and tooling, ensuring teams have immediate access to the latest innovations in coding assistance and efficiency gains.

We are aware that a decision like this contributes to the very trend we are observing, making it a 'self-fulfilling prophecy' of declining market share. However, for a frontend engineering team of our size, the risk associated with being a smaller player in a shrinking field is immense. We simply do not have the internal capacity or intention to pivot resources toward the maintenance and enhancement of framework tooling. This would be a significant investment we can't make right now. Our strategic shift is therefore about mitigating that long-term business risk.

Why React over other frameworks

The decision followed an extensive evaluation, including rigorous Proofs of Concept (POCs) of both React and Angular.

Frameworks like Vue, Svelte, and others were ruled out as they presented similar, though lesser, risks concerning ecosystem size and long-term talent pool growth.

The 'batteries included' argument for Angular is a powerful one. However, the POC revealed that after its recent major upgrades Angular is currently in a period of flux. The introduction of new paradigms like Signals has created competing ways to solve problems, which can lead to a similar level of decision-making as React and a temporary inconsistency in best practices.

Lighthouse felt this made React's vastly larger, active, and more diverse ecosystem a much more compelling advantage.

The abundance of battle-tested, community-driven tools, such as the TanStack ecosystem (Query, Router, Forms), for everything from data fetching to UI components minimizes the risk of having to build or maintain their own specialized solutions.
This, paired with its focused, component-driven approach, offers superior flexibility for building complex enterprise features.

Our solution: a structured, enterprise-grade React

To ensure React apps do not become a "Wild West" of conflicting approaches, the platform team is dedicated to defining a highly opinionated set of rules and architectural standards designed specifically for Lighthouse's scale. This approach maintains the stability and consistency valued from Ember.

💡 Sneak peek at the Lighthouse React standards:

Architectural Decision	Lighthouse Standard	Rationale
Core Libraries	TanStack (Router, Query), Vite (Build Tool).	Provides enterprise-grade data handling, caching, and routing while maintaining high performance.
State Management	TanStack Query for server state; useState / Context API for local client state.	Clear rules here simplify application architecture and maintenance.
Architecture	Domain-Driven Design (DDD): We will adhere to DDD to make sure we have an easy to follow structure in all of our codebases.	Guarantees consistency, improves maintainability, and makes complex code easier to test and onboard new engineers onto.

What's next?

This transition will be strategic and phased. We are committing to a gradual approach. Each next phase is based on the previous phase's success. Lighthouse wants to be flexible, and if assumptions were wrong or expectations are not met, they can still reverse course or adapt the strategy.

Foundation Building: The platform team will finalise these architectural decisions and build the initial React design system library.
Product Pilot: Before making the plunge, one of the new applications will be built in React with the envisioned architecture decisions. This will be seen as a more in-depth trial run and is still easily reversible if the POC assumptions turn out to be wrong.
React adoption: Lighthouse will begin building all new products in React, starting with one of the acquired products already planned for a rewrite.
Future Consolidation: Once the new standard is stable and proven, a long-term plan will be developed for the gradual migration of the existing Ember applications.

This decision marks the start of an exciting new era for Lighthouse Engineering. The team is confident this move will not only secure their technical foundations but will empower teams to innovate faster than ever before.

In a next post, one of the lead engineers on this project will follow up with the detailed technical implementation of the new React architecture.

From JS Mess to TS Success: Narrow your types in Ember Templates

Andrea Scardino — Thu, 12 Jun 2025 12:26:43 +0000

A while ago, when migrating JS files to TS, I ran into an issue when having to use union types in an Ember template.

Let's imagine we have a post, where people can react in 2 ways: with an emoji or a comment

// <some cool imports>

type Comment = { author: string; comment: string };
type Emoji = string;
type ReactionDetail = Comment | Emoji;
type Reaction = { type: string; detail: ReactionDetail };


export default class PostDisplayComponent extends Component {

  @tracked reactions: Reaction[] = [
    {
      type: 'Comment',
      detail: { author: 'Alice', comment: 'Great post! Thanks for sharing.' },
    },
    { type: 'Emoji', detail: '128077' },
    {
      type: 'Comment',
      detail: { author: 'Bob', comment: 'Very informative.' },
    },
    { type: 'Emoji', detail: '128525' },
    {
      type: 'Comment',
      detail: { author: 'Charlie', comment: 'Looking forward to more!' },
    },
    { type: 'Emoji', detail: '128293' },
  ];
  // <some more cool code>
}

To show these reactions we use the following template

  <div class="reactions-list">
    {{#each this.reactions as |reaction|}}
      <div class="reaction-item">
        {{#if (eq reaction.type "Emoji")}}
          <p class="emoji-reaction">{{reaction.detail}}</p>
        {{else if (eq reaction.type "Comment")}}
          <div class="comment">
            <p class="comment-author">{{reaction.detail.author}} says:</p>
            <p class="comment-body">{{reaction.detail.comment}}</p>
          </div>
        {{/if}}
      </div>
    {{else}}
      <p class="no-reactions">No reactions yet. Be the first!</p>
    {{/each}}
  </div>

And in a JavaScript world, everything looks good:

But in a TypeScript project, we get some errors:

This is because the type ReactionDetail is defined as the union of Comment and Emoji.

Union types create a new type that tries to cover all possible properties available from the types we set in the union. Therefore, in our case, reaction.detail is a string or an object.

To fix this we have 2 options: Type Guards or Discriminated unions

Option 1: Type guards

In TypeScript, type guards are functions that perform runtime checks on a variable's type. They can signal to the TypeScript compiler that the variable has a more specific scope.

Think of them as a way to tell TypeScript, "Hey, I've checked this variable, and I can guarantee it's of this specific type inside this block of code." This allows TypeScript to narrow down a variable's type from a broader one.

Let's take a look how it works

  // The type guard function
  isEmojiReaction = (
    // We receive the value that we need to narrow
    reaction: Reaction,
  // This part makes this function a user-defined type guard.
  // Here we define the return type we expect, a boolean. 
  // If it's true we can be sure that reaction.detail is type Emoji
  ): reaction is Reaction & { detail: Emoji } => {
    return reaction.type === 'Emoji';
  };

  // We do the same for the comment's details
  isCommentReaction = (
    reaction: Reaction,
  ): reaction is Reaction & { detail: Comment } => {
    return reaction.type === 'Comment';
  };

Our template should look something like this:

  <div class="reactions-list">
    {{#each this.reactions as |reaction|}}
      <div class="reaction-item">
-        {{#if (eq reaction.type "Emoji")}}
+        {{#if (this.isEmojiReaction reaction)}}
          <p class="emoji-reaction">{{reaction.detail}}</p>
-        {{else if (eq reaction.type "Comment")}}
+        {{else if (this.isCommentReaction reaction)}}
          <div class="comment">
            <p class="comment-author">{{reaction.detail.author}} says:</p>
            <p class="comment-body">{{reaction.detail.comment}}</p>
          </div>
        {{/if}}
      </div>
    {{else}}
      <p class="no-reactions">No reactions yet. Be the first!</p>
    {{/each}}
  </div>

And this how we can narrow down our types in Ember templates using type guards.

Type guards are a great option when handling responses from external APIs, especially when you cannot control the data's structure. They are the ideal solution when the data lacks a common property to differentiate between its possible shapes, a discriminant.

Check the following example:

api/library/assets
{
  "title": "The Big Fish",
  "director": "Tim Burton",
}
{
  "title": "One Hundred Years of Solitude",
  "author": "Gabriel García Márquez",
}

interface Movie {
  title: string;
  director: string;
}
interface Book {
  title: string;
  author: string;
}

function printMedia(media: Movie | Book) {
  // There is no property available that we can use as discriminant, 
  // so we check for a unique property.
  if ('director' in media) {
    // media is now a Movie
    console.log(`Movie by ${media.director}`);
  } else {
    // media is now a Book
    console.log(`Book by ${media.author}`);
  }
}

In many real-world scenarios, we work with API responses that don't have a consistent shape. On the previous example the endpoint returns a movie object or a book object, with no single field to tell them apart. For these cases, type guards are the perfect tool. They allow us to inspect the incoming data and validate its structure before the rest of our code interacts with it.

While type guards are essential for handling such cases, a more robust and self-documenting pattern emerges when we do control the data structure. In the following section, we will explore this preferred approach: discriminated unions.

Option 2: Discriminated unions

A discriminated union is a pattern that combines several object types, where each object shares a common property with a unique literal value.

Think of it as a way to tell TypeScript, "Hey, every possible version of this variable will come with its own built-in 'kind'. You don't have to guess what's inside; just read that kind, and you'll know exactly what properties to expect."

This allows TypeScript to narrow down the variable's type to a specific shape based on a check of that "kind."

How this looks in our code:

type ReactionAsComment = {
  detail: {
    author: string;
    comment: string;
  };
+  type: 'Comment';
};
type ReactionAsEmoji = {
  detail: string;
+  type: 'Emoji';
};
type Reaction = ReactionAsComment | ReactionAsEmoji;

<div class="reactions-list">
  {{#each this.reactions as |reaction|}}
    <div class="reaction-item">
      {{#if (eq reaction.type "Emoji")}}
        <p class="emoji-reaction">{{reaction.detail}}</p>
      {{else if (eq reaction.type "Comment")}}
        <div class="comment">
          <p class="comment-author">{{reaction.detail.author}} says:</p>
          <p class="comment-body">{{reaction.detail.comment}}</p>
        </div>
      {{/if}}
    </div>
  {{else}}
    <p class="no-reactions">No reactions yet. Be the first!</p>
  {{/each}}
</div>

Have you noticed that our template is completely untouched? as it was before the TypeScript conversion. That's one of the biggest benefits of using discriminated unions: they work perfectly with Ember templates right out-of-the-box.

And just like that, after applying these changes in our component, the linting errors in the template vanished:

Conclusions

Sometimes we have scenarios where our variables have to be flexible, to be able to handle diverse data structures, avoiding the clutter of multiple, specialised variables. While having a single variable with several possible types (union types) can present challenges, it mirrors the complexities that we get to solve from the real world.

This article explored two powerful solutions for narrowing type in an Ember template: type guards and discriminated unions.

Type guards act as our versatile inspectors, providing a robust way to narrow types not only in our core logic but also within our Ember templates. They are the perfect solution when dealing with data we don't control, like a third-party API response.

Discriminated unions, on the other hand, represent a more structural approach. By designing our data with a common discriminant property, we create a contract that TypeScript and tools like Glint understand implicitly.

At the end, it isn’t about finding the single "best" approach, but about choosing the one that fits the current context. By mastering both patterns, you can not only write robust, type-safe code but also tailor your solutions specifically to the problem you're facing.

P.S. To test this proposal I created a Github project. Feel free to take a look and leave your comments on this post.

Happy coding 🌻

Supporting classic Ember asset fingerprinting in Embroider

Arnaud Weyts — Tue, 21 Jan 2025 09:10:52 +0000

TL;DR

Handling asset fingerprinting in Embroider still needs some polish. But there are ways to try out and use the RFC's that are eventually going to be productionized.

What is fingerprinting?

Assets

Let's start with the basics: assets. In the context of a web app this could mean a stylesheet, a javascript file, an image, a json file, etc. The browser caches these assets on the visitor's machine by default, which is great for performance! However when you're trying to make sure your users get the latest experience of your web app, you'll need a way to bust the cache and force a network request to updated assets.

Busting the cache

There are several ways to implement cache busting in your app. We'll be digging into the most common one: adding a unique identifier to your files. This technique usually involves appending a "contenthash" to all of your files. This unique identifier is calculated based on the file contents. If your file content changes, the hash will update.

Assets with unique identifiers shown in the Chrome devtools

Implementation in classic Ember apps

In classic Ember apps, asset fingerprinting is done using a "push-based" approach. This essentially means that all assets (build output and public folder) are inspected and a unique identifier is added for each asset. The references to these assets are subsequently updated to reflect the new name. Classic Ember apps make use of broccoli-asset-rev.

<img src="/images/bar.png" />
<script src="assets/appname.js">

background: url('/images/foo.png');

Gets translated into:

<img src="/images/bar-895d6c098476507e26bb40ecc8c1333d.png" />
<script src="assets/appname-342b0f87ea609e6d349c7925d86bd597.js">

background: url('/images/foo-735d6c098496507e26bb40ecc8c1394d.png');

This approach has worked in the past, but comes with a lot of caveats. There's no direct link between an asset and its reference because the package uses a series of regular expressions to identify an asset url. This can result in unexpected issues like the asset not getting fingerprinted when trying to refer to it using a dynamically built string. Embroider and ember-auto-import can help alleviate this problem, so let's have a look.

Implementation in Embroider apps

In Embroider, fingerprinting is delegated to the build tool (Webpack, Vite) and no longer embedded by the framework. This opens up possibilities for different approaches. The Ember community has worked on defining a recommended approach in the Asset importing spec RFC. This RFC recommends switching to a "pull-based" approach. By having direct references to assets in the code instead of plain strings, the build tool can determine the link up more easily. If an asset is not present, it can result in a build error.

While the RFC isn't truly implemented yet, we can achieve a similar "pull-based" approach ourselves by configuring the Webpack config in Embroider.

class extends Component {
  myImage = import.meta.resolve('./hello.png');
}

<img src={{this.myImage}} />

RFC proposal on how an image could be referenced.

Configuration

Webpack actually supports importing assets out of the box. Granted you configure the correct "loader". For images you can make use of the asset/resource loader. Let's take a look at how this would be configured in Embroider:

// ember-cli-build.js
const { Webpack } = require('@embroider/webpack');
const EmberApp = require('ember-cli/lib/broccoli/ember-app');

module.exports = function (defaults) {
  const app = new EmberApp(defaults, {});
  const packagerOptions = {
    webpackConfig: {
      module: {
        rules: [
          {
            test: /\.(png|jpe?g|gif|svg)$/i,
            type: 'asset/resource',
          },
        ],
      },
    },
  };

  return require('@embroider/compat').compatBuild(app, Webpack, {
    packagerOptions,
  });
};

In this code snippet, we've configured imports to .png, jp(e)g, .gif and .svg files to be handled by Webpack's asset/resource loader. This means we can now do this in our app:

import FooImage from './foo.png';

class extends Component {
  fooImage = FooImage;
}

{{!
 the string will now resolve to the file's location
 as determined by webpack!
}}
<img src={{this.fooImage}} />

Tada! Simple right? The asset/resource loader has several configuration options which can be used to customize the exact path and filename, you can refer to the docs for more info. Similar loaders exist for other assets like stylesheets or json files.

Assets referenced from css

Plain CSS

Allright, works great for assets referenced from javascript or templates, but what about assets referenced from stylesheets? If you're using plain CSS, this should just work out of the box with Embroider. Embroider's Webpack package already pre-configures the correct loaders to handle style compilation.

Sass

If you're using a package like ember-cli-sass to handle stylesheets, you'll need to move away from this package and switch to sass-loader in combination with css-loader instead.

Since the app/styles directory is reserved by Embroider for plain CSS files, you'll have to rename the directory to something else. We chose scss for now.

Once all of this is done, you can import your app.scss file in app.(js|ts) and you're all set!

// app.js
import 'app-name/scss/app.scss';

Co-location vs Separation of concerns

The Asset importing spec RFC promotes co-location of assets, which is a change we have not covered so far. In essence, it means that you should put your asset as close as possible to its reference(s) as you can. Where as previously all of the assets were stored in one root public folder, you would now store an image referenced in a component right next to the component itself.

Having a mix of both

If co-location and explicit import statements to assets might not be the ideal solution for you, there's an in-between solution available. It involves making use of a spec that's already implemented in Vite and for which there's also an Ember RFC available: Wildcard Module Import API. This RFC would introduce import.meta.glob as an alternative to import multiple files at once using a wildcard pattern:

const assets = import.meta.glob('../static/**/*.*', {
  eager: true,
});

babel-plugin-transform-vite-meta-glob

Since this RFC is not implemented yet, we'll need to set it up ourselves. This is where the babel-plugin-transform-vite-meta-glob babel plugin comes in. It will translate the import.meta.glob statement to explicit import statements under the hood:

// app-name/utils/asset-map.js
const assets = import.meta.glob('../static/**/*.*', {
  eager: true,
});

export default assets;

Becomes:

// app-name/utils/asset-map.js
import foo from "../static/images/foo.png";

const assets = {
  "../static/images/foo.png": foo,
};

export default assets;

So essentially, the babel plugin will do the explicit work for us so that in turn, Webpack can process these imports.

import-asset helper

After the asset-map is automatically generated, you can write a simple import-asset helper function on top of this to easily grab references to assets from either javascript or handlebars:

// app/helpers/import-asset
import assetMap from 'app-name/utils/asset-map';

export default function importAsset(assetPath) {
  return assetMap[`../static/${assetPath}`];
}

<img src={{import-asset 'images/foo.png'}} />

import importAsset from 'app-name/helpers/import-asset';

class extends Component {
  fooImage = importAsset('images/foo.png');
}

It should be noted that the babel-plugin mentioned is marked as "not safe to use in production". And some of the specification seems to be implemented incorrectly, for which I've opened a fix PR.

Conclusion

The final approach mentioned in this article minimizes the changes needed to adopt the new asset specification of Embroider, while keeping fingerprinting and a single asset folder around. However I must stress that co-location is the recommended way forward and as new features like single file components are hitting the Ember mainstream, supporting assets would look very similar to how it's done in other frameworks/libraries.

In general, it seems like Embroider still needs some polish in the area of fingerprinting and asset management. As Embroider is still actively being developed, anyone can contribute to getting the mentioned RFC's implemented. So I recommend everyone to read through them and try it out for themselves so we can gather feedback.

Grammar of Graphics: how it helps us to create clear visualizations and tell stories with data

Anna Varzina — Thu, 10 Aug 2023 07:47:59 +0000

Introduction

Before we begin with the description, let me ask you, the reader of this blog post, a question: Have you ever heard about the grammar of graphics (GoG)? We assume that not many people are familiar with it. And those who are, probably learned about it through the tidyverse packages in R. This is precisely how the author of this blog post discovered it as well. And it was a remarkable revelation how effortlessly one can create complex visualizations by using the concept of layered graphics. In essence, the grammar of graphics provides a unique level of flexibility in visualization creation that many other tools simply lack. Therefore, the objective of this article is to introduce the grammar of graphics and present a concise recipe for producing clean visualizations tailored to our company projects.

The history of GoG officially begins with the book written by Leland Wilkinson (first published in 1999) [1], which focussed “on rules for constructing graphs mathematically and then representing them as graphics aesthetically”. While prior research had explored clean graphics, Wilkinson's approach established a robust foundation for numerous visualization tools and frameworks that are now widely used in Data Science projects and beyond. Shortly after the publication of Wilkinson's book, the Polaris system was introduced, further expanding his approach. This system later evolved into Tableau’s VizQL technology [2].

In 2005, the concept of layered graphics within the GoG framework emerged with the introduction of the ggplot2 package in R (where gg actually stands for the grammar of graphics) [3], quickly becoming the most popular graphics package in the R ecosystem. Within our team, we use plotnine, the Python version of ggplot2, along with other visualization modules that also extend the GoG concept. These include Bokeh, Seaborn and Plotly, all of which seamlessly integrate with Pandas data frames for our visualization needs. Furthermore, the GoG concept has been implemented for visualizations in various programming languages, such as Flutter, Javascript, Matlab, and Julia.

Layered grammar

The GoG concept is the representation of a plot with layers, initially referred to as elements by Wilkinson. Imagine it as Photoshop layers, where each layer contains instructions to create a specific plot element. Furthermore, all these specifications can be organized within a single function. Unfortunately, in the basic Python Matplotlib library, an imperative approach is used for creating a figure, axis, scaling the plot, adjusting labels and title. This default approach can be less convenient and intuitive when compared to the GoG, which offers greater flexibility for updating and modifying complex graphs.

The GoG framework provides a systematic and clear way of thinking about data visualization by breaking down a graph into its fundamental components (layers). By modifying these components separately, it becomes much easier and less error-prone to improve the visualization. Combining graph specifications within a function facilitates reusability and ensures consistent style when creating multiple visualizations, such as for a dashboard or presentation.

The basic GoG layers include data, aesthetics, geometry, scale, statistics, facets (subplots), coordinates and style. When presenting visualization digitally we can also add an interactivity layer to this list. These layers also have hierarchy, which we will investigate in more details.

Data. Data is absolutely the essential part of any visualization, which comes from different formats, from simple CSV files to query connections. While we don’t aim to discuss data preparation in this context, it is crucial to emphasize that clean and preferably long-format data is very important for effective visualization.
Aesthetics. An aesthetics layer in visualization refers to the features such as position on the x and y axes, colors, shapes, sizes, labels, and more. Usually we map variables to various aesthetics to display information, provide additional context or highlight specific regions of the plot.
Geometry. A geometry layer corresponds to a visual representation of data such as points, lines, bars, shaded regions, boxplots, histograms, tiles, text, and many more. To create a plot, at least one geometry is required. However, two or more geometries can be combined together, such as lines and points, to make more comprehensive plots.
Scale. It is often necessary to rescale or transform data to adjust the data representation. This can be achieved through the use of a scale/transformation layer. For example, using log transformation when dealing with log-linear relationships. Additionally, by zooming and focusing on specific areas of the graph with most accumulated data points would help to gain more insights.
Statistics. Another type of transformation is the statistical transformation, which corresponds to a statistical layer. It allows applying statistical operations, such as calculating confidence intervals, to the data before it is mapped to the aesthetics of the plot.
Facets. Faceting is a way to divide data into subplots based on another factor present in the dataset. Rather than mapping aesthetics such as size, a more effective approach is to segregate data into subplots along the x or y axis.
Coordinates. Usually, we plot data on the familiar Cartesian x-y axes. However, you can also use polar coordinates for specific visualizations, or Mercator projection to plot geographic data, or draw other shapes such as trees or maps. Therefore, we need an additional coordinate layer to make a plot. Actually, if for some eccentric reason you want to use a pie chart, you just need to draw a bar chart in polar coordinates.
Style (theme). Finally, you would need to style your plot with color schemes, line thicknesses, tick marks, legend and font style. The style can be defined by personal preferences, a journal theme, or the organization you are creating the plot for. However, it is sometimes essential to incorporate some small style improvements driven by aesthetic sensitivity as well.
Interactivity. There are also more features that don’t fit the layers described above such as interactivity. In case if you present your plots digitally, the libraries, such as Plotly or Bokeh, or tools like Tableau offer a range of interactive capabilities such as zooming, hovering information, cross-filtering, and linking to other plots or even webpages.

The order of adding layers may vary depending on the dataset and type of analysis being performed. Nevertheless, this list of layers serves as an excellent reference to create visualizations and ensures that all essential graph components are in place.

Grammar of graphics step by step with example

In this chapter, we will guide you through the process of creating a plot in Tableau while examining the GoG layers we modified along the way. As an example, we will replicate a graph displayed on page 9 of the recently published whitepaper by OTA Insight. This whitepaper demonstrates the effectiveness of forward-looking search data in predicting and identifying early signs of market demand.

Step 0: data layer

Connecting to data (1) is an important starting point in our project. Fortunately, we have already preprocessed the dataset in Python that is now stored in CSV format. It is worth noting that researching and preparing data was an iterative process, requiring multiple attempts before reaching the optimal form. Once the data is properly formatted, it becomes much easier to work with it, understand it, explore it and add GoG layers, which will result in a beautiful visualization.

The dataset we connected to contains the following fields:

Column	Type	Description
Destination	String	City or region
Stay date	Date	Arrival date for which search or room reservation was done
Leadtime	Int	The difference between the arrival date and the reservation or search date
Category	String	This field can take 3 values: On-the-books (OTB), flight or hotel searches
Value	Float	Value of the categories above

Step 1: aesthetics and geometry layers

Once we have gained access to the dataset, we should select aesthetics for the x and y axis. In this example, our goal is to investigate the relationship between flight or hotel searches for a particular city or region (which we refer to as a destination further on) and its on-the-books (OTB). OTB refers to the percentage of rooms that is already booked and confirmed for future dates [4]. Our focus lies in examining the changes of these values over the lead time, which represents the difference between the arrival date and the reservation or search date.

To achieve this, we assign lead time to the x axis (Columns in Tableau) and use the search and OTB pickup values for the y axis (Rows in Tableau). It is important to note that Tableau automatically aggregates measures when incorporated into the graph, and we have to make adjustments to the aggregations for desired representation. Therefore, we have to adjust the lead time as a continuous axis to obtain a line graph, which fits well with time series data. At this stage, we have successfully added two fundamental aesthetics (2) and geometry (3) layers.

Step 2: aesthetics and coordinates adjustments

We make our graph clearer by introducing color aesthetics to distinguish occupancy from flight/hotel searches. This allows for easy comparison and analysis per category.

To make it easier to understand, we reverse the lead time axis. On the left side of the graph, we see points further in the past from the arrival date (lead time = 0), while the right side represents more recent data. This arrangement aligns with our natural way of perceiving time, where it progresses from left to right, resulting in a more intuitive and logical representation of the information.

Step 3: add filters

Filtering the destination and stay date actually means adding a data transformation layer (4) to our graph. It is important to note that we don’t modify the underlying dataset, but select different values interactively in Tableau, which can be further adjusted during data exploration. In ggplot or other static plot modules, you would typically need to add this step while assigning the data layer. However, Tableau offers a separate functionality dedicated to filtering. This enables a more seamless and user-friendly interaction with data.

Step 4: scale and style layers

As we progress, we fine-tune the axes of the graph. Specifically, we set the limits of x axis from 0 to 270 lead time and transform y axis format from decimal to percentage by simply adjusting data format in Tableau. Additionally, we refine the style by adjusting tick marks and removing vertical grid lines. Moreover, we increase the line thickness for better visibility.

Step 5: further adjustments

In the final stage of polishing our graph, we make style adjustments to the axis names, title, and legends. Additionally, we include the filtered destination and stay date to the subtitle. Adding final occupancy on the selected stay date to the subtitle requires an additional calculation “market final occupancy”. This can be accomplished by using the level of details (LOD) feature in Tableau. The specified final occupancy value instead of a simple average over all lead time points provides more insights to the reader.

As we can see, we didn't use all GoG layers in this work due to their lack of necessity. For example, we could add facet layer to compare values for various destinations, or we could add confidence intervals to estimate value distributions for all arrival dates per destination. However, our primary goal was to investigate the relationship between the categories. Therefore, the visualization we created fitted this purpose perfectly.

Step 6: storytelling

Now, with our visualization ready, we can tell a story about the data we have. The primary purpose of creating graphs is to achieve effective communication with others. Understanding the GoG concept assists the presenter in emphasizing the most important details and seamlessly incorporating them into the graph. However, it is important to remember that the audience may read your graph a bit differently than you would expect, as highlighted in the book “Storytelling with Data” by C.N. Knaflic [5]. Nevertheless, we can still direct their attention by such graph attributes as color, size or element position, while eliminating non-informative elements and using appropriate visual displays.

Analyzing the graph we just created, we can clearly observe that market occupancy follows the same pattern as flight and hotel searches. About 170 days prior to arrival date (lead time 0), searches started to surge, whereas market actual occupancy was still pacing at 4%. It started to increase at about 120 days before the arrival date and continued to rise until reaching its final occupancy of 94%. We can also notice that flight searches rise faster than hotel searches. It can be explained that people tend to search for a flight earlier than for a hotel.

Prior exploration notes

The steps described above allowed us to produce a visually appealing graph for the whitepaper. However, during the exploration phase, we worked with more layers (especially transformation ones) than mentioned above. For example, we explicitly compared the same increase points (referred to as "pickup" points) for OTB, flight and hotel searches. This involved creating additional calculated fields to determine the lead times for these corresponding growths.

In one of the plots, we even incorporated a statistical layer featuring an exponential fit of the data. However, this particular layer was later omitted in the final version. As shown below, we compared exponential models of the values and included supplementary information to the title for better analysis. During the research phase, we emphasized mainly on the analytical content rather than on style formatting, which we finalized at the data publication stage only.

In conclusion

The concept of grammar of graphics introduced in this article works as a valuable guide to create complex and meaningful visualizations. By breaking down the graph into its fundamental components, it facilitates easy modification and reusability of visualizations. Within our Data Science team, we leverage the grammar of graphics approach during data exploration and creating reports or research summaries. The main advantage for us lies in its time-saving capability for visualization creation and the convenience of reusability.

Nevertheless, crafting a visualization can be considered as a form of art. It often requires personalized approach based on such factors as context, audience and intent. As underscored by the author of layered GoG, Hadley Wickham, "The grammar is powerful and useful, but not all encompassing". There will be cases where we might need to exceed the scope of the defined layers to create the most fitting and insightful graphs.

References

Wilkinson, Leland. The grammar of graphics. Springer Berlin Heidelberg, 2012.
Hanrahan, Pat. Vizql: a language for query, analysis and visualization. Proceedings of the 2006 ACM SIGMOD international conference on Management of data, 2006.
Wickham H. A layered grammar of graphics. Journal of Computational and Graphical Statistics, 2010.
The ultimate guide to on-the-books data for hoteliers. OTA Insight blog, 2023.
Knaflic, Cole Nussbaumer. Storytelling with data: A data visualization guide for business professionals. John Wiley & Sons, 2015.

The road from Ember classic to Glimmer components

Ignace Maes — Thu, 02 Feb 2023 14:20:47 +0000

In late 2019, the Ember.js Octane edition was released which included a new way of writing components: Glimmer components. Components now extend the component class from the Glimmer package instead of Ember. Besides this minor difference in importing there’s a large difference in functionality. This article will go over the differences, reasons why you would want to upgrade, and an upgrade strategy to tackle this in large codebases.

Classic vs. Glimmer

Glimmer components can be seen as a slimmed down version of classic components. Most lifecycle hooks were removed. Arguments are scoped and built upon auto tracking reactivity from Glimmer. There’s no more HTML wrapping element. They use native class syntax. And a lot of classic leftovers were cleaned up.

The following example implements a component which copies the text passed as argument to the clipboard when clicking a button. When using classic Ember components, it could be implemented as follows:

// copy-to-clipboard.js
import Component from '@ember/component';
import { set } from '@ember/object';

export default Component.extend({
  isCopied: false,
  actions: {
    async copyToClipboard() {
      await navigator.clipboard.writeText(this.text);
      set(this, 'isCopied', true);
    }
  }
});

<!-- copy-to-clipboard.hbs -->
<button {{action 'copyToClipboard'}}>
  {{if isCopied 'Copied!' 'Click to copy text'}}
</button>

The same component using Glimmer would look like this:

// copy-to-clipboard.js
import Component from '@glimmer/component';
import { tracked } from '@glimmer/tracking';
import { action } from '@ember/object';

export default class CopyToClipboard extends Component {
  @tracked isCopied = false;

  @action
  async copyToClipboard() {
    await navigator.clipboard.writeText(this.args.text);
    this.isCopied = true;
  }
}

<!-- copy-to-clipboard.hbs -->
<button {{on 'click' this.copyToClipboard}}>
  {{if this.isCopied 'Copied!' 'Click to copy text'}}
</button>

The Glimmer component can make use of decorators as it uses native class syntax. There is a clear separation between arguments passed to the component (here text) and the local state (isCopied). Regular assignment expressions can be used to update state that should trigger template rerenders thanks to Glimmer auto tracking. And there's a lot more improvements which aren't illustrated in this small example.

Why migrate to Glimmer components?

Every code migration requires engineering time which can not be used to build new products to sell to customers. So for a business to invest into refactors and migrations there has to be another benefit. Classic components in Ember are still supported in the latest major version, so why upgrade? The following benefits for us made it worth the trade-off.

One way of doing things

Glimmer components for new code became the standard practice since the release of Ember Octane. This caused our codebases to contain two component types. This adds extra mental overhead when working in codebases which contain both. You have to be aware of which type of component you’re working with and make changes accordingly. For people new to Ember this can be extra confusing.

Closer to native JavaScript experience

Glimmer components contain very little Ember specific code practices compared to classic components. This makes it easier for people to get started in Ember coming from a different background. Every JavaScript developer should be able to get started in our codebase and get up to speed relatively quickly.

Rendering performance

The previous points are nice from a developers perspective. There’s, however, also a benefit for customers. Glimmer components render substantially faster than classic components.

TypeScript support

TypeScript has proven it’s here to stay in the wider JavaScript ecosystem. It has risen in interest and kept its place as the most popular JavaScript flavour.

In 2022 Ember acknowledged official TypeScript support with a dedicated core team. Glimmer components unlock the full potential of TypeScript support in Ember. Template type checking with Glint is also under active development. Exciting!

Future-proof a codebase

Ember.js development doesn’t stagnate. Progress is already being made for new improvements to the current component model. The RFC for first-class component templates has been accepted and merged in 2022 and will provide new benefits to Ember users. By first adopting Glimmer components, we’re prepared for what’s coming next.

Migration strategy

While you could jump straight in and start migrating every component one by one, we decided to go for a different strategy. For smaller codebases migrating components one by one can be a feasible approach, but this can be cumbersome for large codebases (think 100K+ lines of code). This effort is way too large for a single person and has too many side effects. This is why we broke up our migration effort into nine milestones.

1. Native JavaScript class syntax for components

Historically Ember used object syntax to define components. As class syntax matured in JavaScript in general, it also became the standard for Glimmer components. Classic components in Ember provide support for both object and class syntax. This makes switching to class syntax a great first step towards Glimmer components.

Ember provides a codemod to convert object syntax to class syntax. This has saved us a tremendous amount of time. By doing this our development experience also greatly improved.

2. No implicit this

Arguments in Glimmer components are bundled in the args object. This avoids clashes with custom defined properties in the component’s own scope and creates a clear distinction between properties defined locally and passed arguments.

Glimmer component templates reflect this by using the @ prefix when using arguments and the this. prefixes when accessing properties of the backing class. This way of working is also supported in classic components, even though arguments are in the same scope as local properties. This means the migration is non blocking, and luckily there’s a codemod available for this as well. The codemod however can’t make a distinction between arguments and local properties, and is something that will be cleaned up in a later phase.

3. Getting the simple components out of the way

By reviewing all components and checking which used none or limited of the classic component features, we were able to identify a set of components which were easily migrated to Glimmer. Examples are components which did not have any JavaScript logic, as Glimmer introduced the concept of template-only components which work without an explicit backing class. This was low hanging fruit and by getting them out of the way directly we avoided unnecessary overhead of the other phases.

4. Remove outer HTML semantics

Classic components have a wrapping HTML element which doesn’t exist in Glimmer components. A first step to prepare for this removal was to get rid of all properties that have an impact on this wrapping element. In most cases this usage was the classNames attribute which added CSS classes to the wrapping element.

Converting was done by adding these properties directly in the template of the component.

5. Making components tagless

Wrapping elements of classic components can be removed by setting the tagName to an empty string, hence the name “tagless” components. The @tagName decorator from the ember-decorators package can be used to do this. This makes it easy to spot and clean up in a later phase.

Making the component tagless in this phase still introduces breaking changes which we fixed together with adding the decorator.

A common pitfall we noticed was that attributes on an Ember component had no place to be set and were dropped. In Glimmer you explicitly need to tell where the passed attributes have to be placed. This can be done by using the ...attributes syntax. Often this caused styling bugs as classes or id’s weren’t set. Our visual tests came in useful to detect these issues. If you’re interested in how we set up visual testing, check out our talk at EmberFest 2022.

A second issue was that lifecycle hooks that depended on this wrapping element no longer got invoked. Those lifecycle events contain the Element reference, e.g. didInsertElement. To migrate these we made use of the render-modifiers package. Ever since Glimmer and Octane, there are new ways to encapsulate this logic like using the constructor and destructor, writing custom modifiers, or using resources. For the sake of limiting the scope we opted to keep this a separate effort.

6. Removing Mixins

Mixins were a way to share common code across different components. In Glimmer they’re no longer supported. We reviewed our mixins and listed a way of restructuring them as in most cases mixins could be replaced with a more specific way of sharing code.

Common cases were template formatting logic which could be made into a helper, shared constants which could be moved to a separate file, and utility functions which could be separated as they didn’t require the Ember context. For usages that didn’t fit nicely in any of the standard ways of sharing code, we opted for creating custom class decorators as described in “Do You Need EmberObject?” by Chris Garrett.

7. Removing deprecated lifecycle events

In phase 5 a subset of deprecated lifecycle hooks were already removed. There are still others left which are not bound to the wrapping element, like didRender, willUpdate and others. Removing these lifecycle events can be done using a similar strategy as used in phase 5. Generally they can also be replaced with native getters.

8. Removing observers

Usage of observers has been discouraged in Ember for a long time. They were often overused when a better alternative was available, could cause performance issues, and were hard to debug. With Glimmer components, the @observes decorator is also no longer supported.

Refactoring of observers can be non-trivial as it requires you to think of your state updates differently. Rather than reacting to changes, Glimmer introduced autotracking which allows marking of inputs which should trigger UI updates. Some usages can be replaced by working with getters and autotracking. In other cases the did-update modifier of the render-modifiers package can be used as a replacement. Writing custom modifiers is also an option here.

9. And finally … extending from Glimmer!

Now that all classic specific features have been removed, it is time to extend the Glimmer component base class instead of the Ember classic one.

By making this change, arguments will move to the args object instead of the component’s local scope. Usages in the backing class have to be adjusted to use this step in between. One edge case to take into account is that by the change of this scope, they no longer override default values in the component. This can be resolved by writing a native getter which returns the argument from the args object and falls back to a default in case the argument is not passed.

Likewise, argument usages in the template also have to be updated to indicate the difference in scope. The @ prefix has to be set for arguments as the codemod didn’t handle this like mentioned in phase 2.

Finally, the tagName decorator added in phase 5 can be removed as Glimmer components are always tagless.

Conclusion

This article provided a strategy to migrate large Ember codebases from classic to Glimmer components. Following this component migration ensures codebases don’t get stuck in the past. Even better, they unlock modern features Ember provides and new ones being worked on at the very moment!

Why we use Terraform for BigQuery

Nelis Goeminne — Tue, 24 Jan 2023 14:11:26 +0000

At OTA Insight we have been using BigQuery for many years as our main data warehouse for the massive amount of pricing data that we gather on a daily basis. Besides the hotel's pricing data and its derived datasets, BigQuery also contains other data sets such as travel search data we get from external parties and data generated by internal processes.

In addition to BigQuery we have a PostgreSQL database. This is another data warehouse hosted on Cloud SQL, where data sources such as Hubspot, Intercom, Zuora, Salesforce, static hotel data and subscription configuration data are centralised.

The difference between our BigQuery data warehouse and our PostgreSQL data warehouse is that the former contains the big volume datasets that are used in heavy batch processes that benefit significantly from the BigQuery scalable query engine.

The data in the PostgreSQL data warehouse contains smaller datasets coming from transactional databases. These are extensively modelled in our data warehouse by the data analytics team using DBT so they can easily be used for internal reporting.

Although we had been using BigQuery for a long time, in the last few years it became apparent that we needed dedicated tools to manage our BigQuery resources. This is due to several factors, two main ones being:

Scaling the team. We have scaled our engineering department considerably over the past few years (from 20 engineers in 2018 to 87 engineers in 2022). As a data company where many of the engineering teams use BigQuery in one way or another, the scaling caused ownership of specific BigQuery resources to become opaque.
Looker as a BI tool. We migrated from Chartio to Looker as our main BI tool and our implementation of Looker's access control to BigQuery data (explained below) required us to have version control on the many BigQuery views that were created. This version control is not provided by BigQuery, but can easily be achieved by having view queries in a version controlled repository in GitLab for example, which then can be established on BigQuery using continuous deployment.

What is Terraform?

Terraform is an Infrastructure as Code (IaC) tool that lets users define their cloud and on-prem resources in human-readable configuration files. These configuration files are used together with a state file keeping track of your infrastructure to create, modify or delete resources.

A big advantage of infrastructure as code is that you can version, reuse and share these configuration files. This makes infrastructure deployment and maintenance easier compared to manually deploying resources from a cloud Web UI.

Although Terraform is most often associated with deployments of low-level infrastructure such as compute and networking resources, it is also able to manage higher level infrastructure and fully managed tools, such as BigQuery. For BigQuery, projects, datasets, tables, views, routines, jobs and permissions all can be specified in the Terraform configuration files.

For datasets, the attributes "description" and "labels" can be very useful to track the ownership of datasets. For tables and views, the schema and queries can also be specified in the configuration files. Tracking these configuration files in git repositories allows users to use the common code review and versioning tools on modifications of the schemas and queries.

Terraform is not the only infrastructure as code tool that exists. Every major cloud provider has its own specific tool for its platform such as AWS Cloudformation, Azure Resource Manager or Google Cloud Deployment Manager.

Terraform however supports all prominent cloud platforms and providers in just one tool. As we were already using Terraform for managing other parts of our infrastructure and as Terraform has excellent BigQuery support, we decided to also use Terraform to manage our Looker-related BigQuery views.

Looker and the need for BQ views

When we migrated to Looker as our BI platform, we needed certain datasets from BigQuery to be incorporated in it. The problem was that we couldn’t reuse user permissions defined in BigQuery and the only way to have the data exposed was through a service account.

We made a decision to not give the Looker service account access to all raw data in BigQuery, but to work with a selection of preprocessed views. This was done for a few reasons.

Security. It is security best practice to give an application (and its users) only the least privilege access as to what is needed for its functionality.
BigQuery cost optimization. The intermediate view approach allows for optimization in BigQuery of how the data is accessed and cached by using e.g. materialized views.
Looker cost optimization. Developer accounts for LookML (Looker's "programming language") are expensive and LookML has a steep learning curve. We decided to only give the Data Analytics team LookML developer roles and all the other engineers the dashboard creator roles. This created a clear separation of responsibility and ownership concerning the data modelling process.
Avoiding bottlenecks. All engineers are able to model the data they are knowledgeable about and responsible for in a BigQuery view. This view is then simply linked with LookML by the Data Analytics team in an auto generated view and corresponding explores ready for visualisation.

In practice, we decided to organise all of our BigQuery views for Looker under one, new, central BigQuery project, to which the Looker service account has access.

For most databases, access to a view requires access to the underlying data source. In BigQuery however, one can make use of authorized views and authorised datasets to give a user or group access to just the query result. This way we are certain that the Looker service account (and the users in Looker) will not be able to access data other than what was selected for the view.

The below diagram is an adaptation of the LookML projects diagram from the LookML documentation showing the scope of data access and responsibility for our use case.

Technical details

Our repository describing the BigQuery infrastructure is hosted on GitLab and has one .tf file per dataset. This file contains the dataset definition and the definition of all views in that dataset.

An example of such a configuration file is shown below. The query of every view is stored in a separate .sql file in a 'queries' folder which allows for easier development in an IDE (use of SQL-specific tooling) and the possibility to use SQL linting (we use SQLFluff) in CI.

The Terraform state is stored within GitLab's managed Terraform HTTP backend which can be used both by developers on their local machine as well as our CI/CD pipelines on GitLab, which use the terraform plan and terraform apply commands against our infrastructure configuration.

resource "google_bigquery_dataset" "example_dataset_alias" {
 project     = "project_name"
 dataset_id  = "example_dataset"
 description = "This dataset is created as an example to demonstrate Terraform"
 location    = "EU"
}


resource "google_bigquery_table" "example_table_alias" {
 project             = "project_name"
 dataset_id          = google_bigquery_dataset.example_dataset_alias.dataset_id
 table_id            = "example_table"
 deletion_protection = false
 labels              = { user = "Name of the engineer or the engineering team responsible for the view" }
 description         = "This materialized view is created as an example to demonstrate Terraform"


 materialized_view {
   query = file("queries/example_dataset/example_query.sql")
 }


 time_partitioning {
   type  = "DAY"
   field = "partitioning_column_name"
 }


 clustering = ["clustering_column_name_0", "clustering_column_name_1"]

}

Evaluation

The use of Terraform for our Looker-related BigQuery views has made our development and deployment of those views particularly robust. Having the Terraform configuration files in GitLab allows us to use our standard code review process for the queries and makes changes easy to track and revert if necessary. Furthermore, ownership and discoverability of resources has improved by the inclusion of owner labels in the view definition.

We are happy with the benefits the Terraform setup for BigQuery brings us and have continued to add more of our BigQuery resources (such as common functions we once defined manually without version control) to be managed by Terraform.

Why we don’t use Spark

Karel Vanden Bussche — Wed, 07 Sep 2022 09:03:59 +0000

Big Data & Spark

Most people working in big data know Spark (if you don't, check out their website) as the standard tool to Extract, Transform & Load (ETL) their heaps of data. Spark, the successor of Hadoop & MapReduce, works a lot like Pandas, a data science package where you run operators over collections of data. These operators then return new data collections, which allows the chaining of operators in a functional way while keeping scalability in mind.

For most data engineers, Spark is the go-to when requiring massive scale due to the multi-language nature, the ease of distributed computing or the possibility to stream and batch. The many integrations with different persistent storages, infrastructure definitions and analytics tools make it a great solution for most companies.

Even though it has all these benefits, it is still not the holy grail. Especially if your business is built upon crunching data 24/7.

At OTA Insight, our critical view on infrastructure made us choose to go a different route, focused on our needs as a company, both from a technical, a people perspective and a long term vision angle.

Humble beginnings

Early on you have only 1 focus: building a product that solves a need and that people want to pay for, as quickly as possible. This means that spending money on things that accelerate you getting to this goal - is a good thing.

In the context of this article this means: you don’t want to spend time managing your own servers, or fine-tuning your data pipeline’s efficiency. You want to focus on making it work.

Specifically, we heavily rely on managed services from our cloud provider, Google Cloud Platform (GCP), for hosting our data in managed databases like BigTable and Spanner. For data transformations, we initially heavily relied on DataProc - a managed service from Google to manage a Spark cluster.

Managing managed services

Our first implementation was a self-hosted Spark setup, paired with a Kafka service containing our job-queue. This had clear downsides and in hindsight we don’t consider it managed. A lot of side-developments had to be done to cover all edge-cases of the deployment & its scaling needs. Things like networking, node failures and concurrency should be investigated, mitigated and modelled. This would have put a heavy strain on our development efficiency. Secondly, pricing of running a full Spark cluster with a 100% uptime was quite high and creating auto-scaling strategies for it was quite hard. Our second implementation was migrated to use the same Kafka event stream that streamed workload messages into the Spark DataProc instances instead of the initially self-hosted Spark instance.

The Kafka-Dataproc combination served us well for some time, until GCP released its own message queue implementation: Google Cloud Pub/Sub. At the time, we investigated the value of switching. There is always an inherent switching cost, but what we had underestimated with Kafka is that there is a substantial overhead in maintaining the system. This is especially true if the ingested data volume increases rapidly. As an example: the Kafka service requires you to manually shard the data streams while a managed service like Pub/Sub does the (re)sharding behind the scenes. Pub/Sub on the other hand also had some downsides, e.g. it didn’t allow for longer-term data retention which can easily be worked around by storing the data on Cloud Storage after processing. Persisting the data and keeping logs on the interesting messages made Kafka obsolete for our use case.

Now, as we had no Kafka service anymore, we found that using DataProc was also less effective when paired with Pub/Sub relative to the alternatives. After researching our options regarding our types of workloads, we chose to go a different route. It is not that DataProc was bad for our use cases, but there were some clear downsides to DataProc and some analysis taught us that there were better options.

First, DataProc, at the time, had scaling issues as it was mainly focussed on batch jobs while our main pipelines were all running on streaming data. With the introduction of Spark Streaming, this issue was alleviated a bit, though not fully for our case. Spark Streaming still works in a (micro-)batched way under the hood, which is required to conform to the exactly-once delivery pattern. This gives issues for workloads that do not have uniform running times. Our processors require fully real-time streaming, without exactly-once delivery, due to the idempotency of our services.
Secondly, the product was not very stable at the time, meaning we had to monitor quite closely what was happening and spent quite some time on alerts. Lastly, most of our orchestration & scheduling was done by custom written components, making it hard to maintain and hard to update to newer versions.

Building for the future

It was clear we needed something that was built specifically for our big-data SaaS requirements. Dataflow was our first idea, as the service is fully managed, highly scalable, fairly reliable and has a unified model for streaming & batch workloads. Sadly, the cost of this service was quite large. Secondly, at that moment in time, the service only accepted Java implementations, of which we had little knowledge within the team. This would have been a major bottleneck in developing new types of jobs, as we would either need to hire the right people, or apply the effort to dive deeper in Java. Finally, the data-point processing happens mainly in our API, making much of the benefits not weigh up against the disadvantages. Small spoiler, we didn't choose DataFlow as our main processor. We still use DataFlow within the company currently, but for fairly specific and limited jobs that require very high scalability.

None of the services we researched were an exact match, each service lacked certain properties. Each service lacks something that is a hard requirement to scale the engineering effort with the pace the company is and keeps growing with. At this point, we reached product-market fit and were ready to invest in building the pipelines of the future. Our requirements were mainly keeping the development efficiency high, keeping the structure open enough for new flows to be added, while also keeping the running costs low.

As our core business is software, keeping an eye on how much resources this software burns through is definitely a necessity. Taking into account the cost of running your software on servers can make the difference between a profit and a loss and this balance can change very quickly. We have processes in place to keep our bare-metal waste as low as possible without hindering new developments, which in turn gives us ways to optimise our bottomline. Being good custodians of resources helps us keep our profit margins high on the software we provide.

After investigating pricing of different services and machine types, we had a fairly good idea of how we could combine different services such that we had the perfect balance between maintainability and running costs. At this point, we made the decision to, for the majority of our pipelines, combine Cloud Pub/Sub & Kubernetes containers. Sometimes, the best solution is the simplest.

The reasoning behind using Kubernetes was quite simple. Kubernetes had been around a couple of years and had been used to host most of our backend microservices as well as frontend apps. As such, we had extensive knowledge on how to automate most of the manual management away from the engineers and into Kubernetes and our CI/CD. Secondly, as we already had other services using Kubernetes, this knowledge was quickly transferable to the pipelines, which made for a unified landscape between our different workloads. The ease of scaling of Kubernetes is its main selling point. Pair this with the managed autoscaling the Kubernetes Engine gives and you have a match made in heaven.

It might come as a surprise, but bare-metal Kubernetes containers are quite cheap on most cloud platforms, especially if your nodes can be pre-emptible. As all our data was stored in persistent storages or in message queues in between pipeline steps, our workloads could be exited at any time and we would still keep our consistent state. Combine the cost of Kubernetes with the very low costs of Pub/Sub as a message bus and we have our winner.

Building around simplicity

Both Kubernetes and Pub/Sub are quite barebones, without a lot of bells & whistles empowering developers. As such, we needed a simple framework to build new pipelines fast. We dedicated some engineering effort into building this pipeline framework to the right level of abstraction, where a pipeline had an input, a processor and an output. With this simple framework, we've been able to build the entire OTA Insight platform at a rapid scale, while not constricting ourselves to the boundaries of certain services or frameworks.

Secondly, as most of our product-level aggregations are done in our Go APIs, which are optimised for speed and concurrency, we can replace Spark with our own business logic which is calculated on the fly. This helps us move fast within this business logic and helps keep our ingestion simple. The combination of both the framework and the aggregations in our APIs, create an environment where Spark becomes unnecessary and complexity of business logic is spread evenly across teams.

Summary

During our growth path, from our initial Spark environment (DataProc) to our own custom pipelines, we've learned a lot about costs, engineering effort, engineering experience and growth & scale limitations.
Spark is a great tool for many big data applications that deserves to be the most common name in data engineering, but we found it limiting in our day-to-day development as well as financials. Secondly, it did not fit entirely in the architecture we envisioned for our business.
Currently, we know and own our pipelines like no framework could ever provide. This has led to rapid growth in new pipelines, new integrations and more data ingestion than ever before without having to lay awake at night pondering if this new integration would be one too many.

All in all, we are glad we took the time to investigate the entire domain of services and we encourage others to be critical in choosing their infrastructure and aligning it with their business requirements, as it can make or break your software solutions, either now, or when scaling.

Want to know more? Come talk to us!

Why we use Ember.js at OTA Insight

Kenny De Pauw — Mon, 04 Apr 2022 14:26:12 +0000

We always choose the stack that is right for the job. We often get asked why we use Ember.js. It’s not the most popular framework, or the largest community, but it is the right choice for the products we are building at OTA Insight. From our first product, Rate Insight, to the four after that, all of it is built with Ember.js. And for good reasons. We were able to create new features quickly, have a codebase that’s scalable and have a good developer experience. All of these are reasons we choose Ember.js. But let’s go in a bit more detail.

Batteries are included

It’s a phrase you’ll see floating around the Ember.js community and it’s one of the reasons why we were able to move so fast here at OTA Insight. The strong focus on convention over configuration enables you to move quickly as you don’t need to reinvent the wheel every time. It also gives developers new to your codebase a good guideline on how to write code. There’s no 100 ways of doing things.

There were times we had to make something custom, going against the conventions put forth in the framework. This can become difficult, requiring some research into the inner workings of Ember.js, but this hasn’t occurred all that much. The benefits of having strong conventions and guidelines on how code should be structured are more than worth this tradeoff.

What steep learning curve?

The thing I hear developers fear most is the steep learning curve of Ember.js. Not sure where this comes from, especially with the latest move to Ember Octane. After this milestone, Ember.js is making use of the latest javaScript features and it feels pretty close to working with standard javaScript. Gone are the days of having to call custom get/set functions, using EmberObjects, etc.. Now everything is done with class syntax, decorators etc. If you know javaScript, you’ll have no problem getting started in Ember.js.

The templates used in Ember.js are close to basic HTML with the ease of Handlebars added on top. You separate your concerns, having the template and styles separated from the logic. All of this provides a clear overview for developers starting with the framework.

Of course there are still Ember.js specific features you need to know. How is the app structured? What are the lifecycle hooks? What is the Ember.js way of doing things? But this is what you’ll see learning any framework.

Bonus is that here at OTA Insight we have a whole team of people with experience in Ember.js who will be happy to help you out. All you have to do is ask.

Modern frameworks and Ember.js

We’ll regularly check our tech stack and see if we can improve it. Our use of Ember.js is no exception. We recently built a Vue.js prototype of our application to see how it compares. We focussed on 4 factors to determine how they compared:

Developer experience
Performance
Scalability
Community

I won’t go into too much detail here, that may be a post for later (or if you want to know more, feel free to reach out).

Regarding developer experience it was clear the conventions defined in Ember.js really allowed us to develop things quickly.
Speed was comparable to Vue.js thanks to the Glimmer components recently introduced in Ember.js.
Even now with 5 products, the codebase is still scalable as well, although using a framework like Nuxt you could probably come to a similar result.
Even though there is a larger community with Vue.js, we found Ember.js had a clearer roadmap for the future.

After our research was done we had to decide, is it worth changing our framework to Vue.js? What was clear is that the two were comparable. Ember.js outperformed in some regards, and Vue.js in others as you can see above. We decided that if Ember.js can compete with one of the top three frameworks out there right now, it’s clear this was the right choice back in the day and we are still proud to be developing in Ember.js.

Paving the way

Ember.js is the right choice for our main application. But it isn’t always the right choice. For example, our design system.

Frameworks come and go in the frontend development world, but web components are forever. That’s why we have built (and are expanding) a design system using Stencil.js. Web components can be used in any framework and Stencil.js makes it easy to create new ones.

We did not build this in Ember.js on purpose. While Ember.js has its use, and while we still believe in it now, who knows what the frontend landscape will look like years from now. Having our design system, and so our most generic and most used components be framework agnostic is a huge benefit. It opens the door to have them used with other frameworks (or with no framework at all!) We have several smaller apps, some use Ember.js and some are just plain HTML, CSS and javaScript. In the future we might even have one running in React or Svelte. All of these can use our design system.

We don’t cling to Ember.js just because of a choice we made all those years ago. We do it because we still believe it’s the right choice for our main app. For the design system it wasn’t. We always choose the stack that is right for the job.