Hi community! Looking forward to hearing your thoughts about this.
Inngest was started out of the pain of building products that required complex integrations and background jobs — especially those driven by user interactivity. We’ve built the infrastructure so many times that we decided to open-source it for other people — so that everyone can have access to world class tooling, instead of just MANGA.
We’ve created a single API that lets you record what’s happening in your app, then lets you run serverless functions in response to these events. It’s a developer’s Zapier.
It’s more than a job queue or serverless
From our past experience, we know that building out reliable products requires more than a job queue, Kafka, or Lambda, and off-the-shelf products don't cut it for the customization us developers need.
We believe that it should be accessible to everyone — not $just 1B+ companies who have hundreds of people to configure and maintain infra.
Even for something as basic as reliably handling a webhook you’ll need some sort of queue, retries and logging which can take weeks to build, even with off the shelf systems in clouds. Here’s our plan to give this to everyone, from jamstack APIs all the way to realtime elixir webapps:
Event-driven infrastructure foundations
Using a single API to record what’s happening in your app provides rich functionality like:
- Running step functions in real time, whenever specific things happen, automatically. This includes off-the-shelf, repeatable functions, like sending emails, forwarding events to analytics, or common APIs.
- Historic replay and retries — by saving events we allow you to replay and retry past data locally on your own machine if there are errors, or retry your functions if there are errors.
- Personalized debugging — when your users do things in your app, you can assign the events that are generated to that user. This lets you see every function that ran for each of your users, making debugging and insight super easy
- Coding best practices. Sending an event describing what’s happening then reacting to the event reduces coupling and makes your critical path faster.
- No servers or job queues. You can make everything serverless by scheduling functions to run in the future when an event comes in, making your app entirely serverless
- Monitoring and visibility, allowing you to see what’s happening in your app and the functions that are running
- Complex coordinated functionality, such as “on cart added, wait 24 hours for the checkout event — and only run the next step if the user didn’t check out”. This is annoying to build manually, and usually requires complex cron jobs, messy database state, or external (expensive) services like braze.com
Local development first
We also think that local development should be a first class part of the process, without spending days installing and configuring services locally. The open source CLI already does the following:
- Makes it easy to create functions in any language, either from a blank slate or using quick starts for common use cases
- Locally run functions with a single command — using the same execution engine deployed to your cloud, with a single command. This makes sure you have local <> production parity
- Start a fully functional local test environment in a single command — which accepts events then runs functions locally
- CI/CD out of the box, using things like github actions
- Replay real production events locally for debugging & testing.
We want everyone to have world-class infrastructure to build the complex parts of their app — the stuff that’s beyond basic API routes. This includes things you’d only dream of at fortune 500s: serverless functions, event schema generation, event and function versioning, blue-green deploys, throttling, idempotency, and local debugging.
Even for basic use cases like handling webhooks, this makes your life far easier. And we want this for everyone, available locally, with the source freely available to hack on.
Status & next steps
We’ve already got our execution engine and CLI that does the above. We’re planning webassembly support, step-over debuggers, and a ton more. What would you like to see? Do you have job queues or events in your system? What would your ideal ‘programmable zapier’ do?
Feel free to comment here or on Github at https://github.com/inngest/inngest.
Top comments (2)
I won't pretend like I'm an expert in this part of the tech stack, but I do like the appeal of a fusion between cloud functions and background processes.
I don't have a use today but I've starred your project to check back on in the future. Do you know if this is something other providers might already do? I recently built out my company's first cloud function (webhook bridge to sign the request) in GCP and noticed when I was configuring that resource that I could set my function to trigger off of events.
It's something that you can do with lots of configuration, and you get the bare minimum: a trigger which can call functions. Unfortunately, clouds give you the plumbing, not the platform: you have to configure the plumbing yourself, and then build a layer on top for things like retries, replays, local testing, etc.