DEV Community

Eric Wang
Eric Wang

Posted on

Real time data pipeline with a single command

After working at companies big and small, I often found myself poring over logs to answer business questions for non-technical users. To be honest, a more sophisticated server-side SDK for instrumenting event data would have been ideal, with that data then streamed into a Kafka queue. This would allow me to write an ETL job to transform the data, subsequently storing it in a data warehouse, from which I could integrate with tools like Looker or Tableau so business users can create dashboards themselves! If only there were infinite time and energy for such indulgent engineering projects... It would have been marvelous!

In practice, I wrestled with messy log data until it could be condensed into a nice number suitable for a dashboard. If more data was necessary, I would simply log it in the code, then get back to building features for customers or arguing with strangers on Reddit.

After my cofounder Seb and I joined DoorDash through an acquisition from Bbot—a restaurant technology startup—we had to integrate Bbot data into DoorDash’s data pipeline. This task was intended to answer questions such as the number of completed checkouts per day, average check subtotals, and total daily sales, etc. Once again, we found ourselves in a familiar problem space, that was the last straw, we quit immediately!

In reality, it took another month to gather the courage to actually quit our day jobs, but that's only a minor detail.

We built Siege because we wanted a fast and easy way to get real time data into a format that is easily queryable using plain SQL. We spent a few months building an agent in C that you can just plop into your server(s) to mine data directly from API traffic while using negligible resources. You can then pick and choose the data fields to track with just the click of a button from a catalog of data points. We also built an user-friendly UI that allows you to visualize the data and create real-time dashboards. All in less than 10 minutes.

We have 4 criteria when choosing tools for our own use:

  1. We hate reading documentation, so it better be short.
  2. We need to be able to derive value from it within the first 10 minutes.
  3. We need to be able to play with it for free, no sales calls ever.
  4. Dark mode.

We were fully committed to these principles while building our own tool. Join our public beta for free! https://siegeai.com

Sentry blog image

How to reduce TTFB

In the past few years in the web dev world, we’ve seen a significant push towards rendering our websites on the server. Doing so is better for SEO and performs better on low-powered devices, but one thing we had to sacrifice is TTFB.

In this article, we’ll see how we can identify what makes our TTFB high so we can fix it.

Read more

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay