DEV Community: Sean Travis Taylor

So Your Boss Wants You to Use AI?

Sean Travis Taylor — Mon, 11 Aug 2025 16:27:30 +0000

Most bosses know just enough about AI to know that it can have a transformative impact on their business. Often enough, they don't know how and worse still they view AI as something like a "one weird rick", a magic spell or an efficiency hack.

Armed with enough knowledge to be dangerous, leaders make all manner of demands on their team to "use AI", to "use all of our data" in order to do...something which is never really clear.

Watch out for words and phrases like: "integrate", "collaborate" or "work with AI" or "just put it into ChatGPT". These indicate profound naivety or plain willful ignorance when it comes to the complexities of using AI to create business value.

Below are some ready replies to well-meaning but inane requests from bosses that allow you to look smart as an AI non-specialist. They may even help your team avoid a few footguns, timesinks and disaster projects in the process.

Question: Why don't we just put this into ChatGPT?
Answer: What exactly are we putting into ChatGPT? First off, I'm totally on board with using more generative AI in our workflows where that makes sense. To clarify: do you mean asking ChatGPT to create or prospect specific solution for us based on a particular prompt? Do you mean fine-tuning ChatGPT with prompting on some of our data? Can you provide a bit more color there?

Question: We have all this data. Why can't we use AI to [INSERT BUSINESS CAPABILITY OR FUNCTION]?

Answer: We certainly can explore whether a Large Language Model or another AI solution is appropriate for that use case. What data you feel is most pertinent to realizing that feature? Do we know where that data is? Is it in pristine Excel spreadsheets or is it spread across GDrive, random emails, Jira and Slack? Do we know what condition it's in? How much of the data is at the quality such that we could train a model for...what is it we're trying to accomplish again...?

Question: Can't ChatGPT already do this?

Answer: LLMs alone are general tools with a lot of general data about the world. The data that we have here at BigCo. is not only proprietary it's highly specific. No LLM, no matter how good it is, is going to have that high fidelity data. We could fine-tune ChatGPT or some other foundational model with our data to get quality outputs. Quick question: what makes you think an LLM or a chatbot is the right move here? What data from the business or our users or customers makes you believe this?

Question: I want everyone to think about how we can use AI to improve efficiency. How can we enhance the quality of operations with this technology?

Answer: Where do we think the greatest business value will emerge from efficiency improvements? We could make a lot of efficiency improvements across teams and still not move the needle on our most important metrics. Can you be specific about where in our company you see improved efficiency having measurable impact? What data do we have about current operations to measure any AI-driven efficiency efforts against?

Question: I want to see a proof of concept for an "AI approach" to [INSERT BUSINESS PROBLEM]. I won't take 'no' for an answer. How do we get started? We'll probably need to hire some data scientists soon, right?

Answer: Let's first figure out what we're trying to achieve here. Do we want predictions? Do we want classifications? For example, using AI to do sentiment analysis on customer chats or feedback messages to give us a sense of how people are feeling about our service in real time? Once we settle on that, let's find the most pertinent data and ensure that data is clean, consistent and coherent. Then we can use something like HuggingFace's AutoTrain feature to do some small experiments.

This is obviously just a smattering of the uniformed and downright asinine directives workers can expect from bosses in the throes of AI fever.

This technology isn't going anywhere, so we should absolutely embrace in the workplace. At the same time we need to ensure we remain results-focused, not matter what new tech comes down the pike and that above all we are honest with ourselves and our teams about its risks as well as its rewards.

"Get More AI Skills," They Told Me

Sean Travis Taylor — Mon, 04 Aug 2025 16:08:13 +0000

No other advice is offered more to shell-shocked workers blitzed by the lattice of layoffs, the ghosted interviews and the non-response responses than that of "focusing on AI skills".

Workers are to believe that being "skilled at AI" will make them instantly more attractive candidates to companies.

These are the same companies that, often enough, are rewarded by Wall Street for firing workers and pepper phrases like "LLM", "artificial intelligence" and "generative AI" into every earnings call, interview, think piece and forecast they create.

Workers are to believe these same companies will pursue anyone "sufficiently skilled" rather those who are most responsible for the creation of the top-tier AI models on the market.

The fiction of ‘sufficient skill’ masks the reality that AI’s biggest breakthroughs come from the few while the many are told to catch up.

If AI ends up being as transformational as its most ardent promoters claim, it is ridiculous to imagine that the same business leaders who have steered the steady cascade of layoffs over the past three years, in spite of record profits, will maintain their most consequential cost-centers rather than trim them in pursuit of improved margins.

But let's assume this isn't true.

Upskilling in AI as sound advice for today's working professional requires another assumption: that the business leaders calling for more AI skills are even arguing in good faith.

Remember, the people peddling this advice are the same people overseeing the injection of generative AI into every conceivable corner of every major product we can name, not because users and customers are clamoring for this and not because it addresses stated user needs.

Generative AI isn’t answering demand, it’s manufacturing it. One product update at a time.

The "upskilling on AI" trend is a necessary pre-condition for the mass sales and enterprise integration of AI "solutions". Without it, all the hawkers of such tools and services have is the biggest speculative bet in history, without any takers. Given the billions that have been invested and the billions more planned for AI, a narrative justifying all that spend is not only necessary but required.

Anybody remember "learn to code"? This was only the most recent major upskilling effort workers were advised to undertake.

Why?

Because software was eating the world and software-as-a-service was the knife and fork. We needed tons of engineering talent to build and scale massive SaaS platforms. The 2010s were the era when the biggest, most influential internet companies we can name were on the "come up".

In the 2010s, software ate the world and engineers set the table...

Now that the platforms have been built and the key scaling challenges resolved, armies of engineers are no longer required to maintain these multi-billion dollar businesses.

Now that the platforms have been built, where does all that engineering talent go?

Now workers are to believe that AI will be different, that we'll need all these people to build AI systems and platforms in the same way we "needed" all those engineers during the SaaS boom.

But AI and associated disciplines like Machine Learning, Deep Learning and Data Science are specialist competencies. We'll need far fewer practitioners in order to make revolutionary advances that produce outsize value for companies.

Unlike the SaaS boom that required scaling teams of engineers, AI is a high-leverage domain where a small number of specialists can generate disproportionate value. This makes broad workforce demand unlikely.

Even if most business leaders understood AI as a precision solution to a well-defined business problem rather than an efficiency hack, where would they have the willing upskiller direct their attention? Building their own models from scratch? Mastering the intricacies of fine tuning the output of pre-trained models with LoRA? Building the data pipelines that feed machine learning models? Or do they simply mean better prompt engineering?

Realistically how much of that fits into a single self-financed online course or certification program, what is the lifetime value of knowledge that has such a high degree of obsolescence and what are the competitive prospects of a holder of such certifications against a genuine specialist with a terminal degree? What really determines the amount of value either will add to a company, if any?

These are the most important and sadly unanswered questions.

Whatever else it may be, upskilling in AI will not be a vehicle for widespread or even modest employment any more than coding was before it. Cloud companies are designed for massive output relative to modest labor inputs; their ability to scale outputs infinitely is what drives their profitability.

For all the noise around "upskilling in AI", we should expect the same employment boom and bust cycle from the last decade and prepare financially, professionally and emotionally for whatever comes after the AI hype.

The Siren Song: Vibe Coding and the Build Trap

Sean Travis Taylor — Wed, 30 Jul 2025 18:02:55 +0000

The prospect of creating applications without the need of developers, designers or product managers is irresistible to anyone without the team or the technical knowledge or experience to build a modern software application. That is to say most people.

We can and should encourage the use of vibe coding as another instrument in the rapid-prototyper's toolbox, since the promise of this practice is not the creation of fully realized production apps, despite advertisements to the contrary, but to allow a person or a team to validate product ideas and features at the speed of a prompt.

The Build Trap exists when a single non-specialist can deploy an app for a few dollars after a frenzied 100 hours of vibe coding the same as it does when a full complement of designers, developers and testers spend a whole quarter to achieve the same.

The Build Trap is what happens when we mistake output for outcome—when we build features or apps simply because we can, not because anyone needs them.

Any time we prioritize output over outcome, we are in danger of creating a product that nobody wants, nobody asked for and that nobody is interested in--a product that solves no problem of any great significance to anyone.

Such products are, in a word, a waste.

We have wasted time, energy and attention on a solution that appeals to our own sense of what meets the moment over what consumers indicate an urgent appetite for.

We have wasted time, energy and attention on a course of action that will at best, do nothing and in the worst, actively destroy value: whether that's customer goodwill, earned trust or brand equity.

Does the vibe-coded shotgun approach to production apps result in less waste? Absolutely. This is not insignificant but it is hardly redemptive. Any time, energy and attention spent willing inadequate solutions into relevance is time gifted to our competitors and time for the market to move away from us.

So what should we do?

The good news is that it is the same as it ever was. Use whatever tools you have available (and generative AI is a hell of a tool) to create prototypes that you can get out quickly for the purpose of doing experiments.

Rapid experimentation matters more than easy answers—because real value comes not from how fast we build, but from how fast we learn.

What should excite the non-specialist is not the time to done, the time to a "finished" app but the time between experiments, the time between iterations: call it "cycle time".

The lower our cycle-time, the faster we can conduct experiments and get useful feedback. This was always the game. That a billion-dollar product and a pack of passionate users resulted from the experimentation is a happy accident.

Somewhere along the way we became obsessed with time-to-market as the defining element of product success, that is time-to-market with a product we can claim is "finished", instead of the arrival of a product that meets or exceeds customer expectations.

Business stakeholders and non-specialists are excited by vibe coding because it seems to confirm their perpetual suspicion that all those developers and testers and designers were really just under-miners and obstructionists.

To the untrained, vibe coding doesn't reveal hidden efficiency—it confirms a long-held fantasy: that expertise was the bottleneck all along.

"Finally," they exclaim. "Now we can just build!"

And surely they can, but to believe software is the answer is to ask the wrong question. In their glee, the non-specialist forgets that software is not the solution; it is the solution expressed in code.

Brilliant software inside a poor solution serves no one; this is hard to see and usually when we do see it, it is because the claws of the Build Trap have forced us to do so.

Vibe coding indeed opens new possibilities for creating code in greater quantity and speed than ever before--this is good--but given that, we should aim to be even wiser and act with even greater intention lest we succumb to its new perils, chief among them, the Build Trap.

Spec-Driven API Design with Artificial Intelligence

Sean Travis Taylor — Mon, 30 Jun 2025 18:20:35 +0000

The AI Risk Matrix

The greatest risk of AI-generated applications is their non-deterministic nature. The tendency to hallucinate means special care must be taken to validate LLM outputs.

The greatest advantage of AI-generated applications is speed; they are trivial to create. This means we can move really fast.

The greatest risk of existing tools is their rigidity; they struggle to adapt to emerging business requirements. They have to be fit for purpose and they must be maintained to stay useful and relevant.

The greatest advantage of tools is their precision. Their fitness for purpose ensures deterministic outcomes as long as we use the tool and they have strong reliability guarantees when requirements remain consistent.

The advantage of standards is wide interoperability, documentation and shared understanding. We save time by standing on the shoulders of giants.

How can we combine the best of standards, AI-generated code and existing toolchains to build better software faster?

The Thing That Gets You To The Thing

Code generation has emerged as one of the key use cases of generative AI. Too much time is spent thinking about how to create apps from text-based prompts. This is at the expense of discussions about the potential of AI to build reliable and interoperable systems when used to create the low-level boilerplate code that forms the lion's share of so many codebases.

Given LLMs propensity to hallucinate, we're required to create tooling and services that validate outputs. This is costly and perhaps even beside the point. Rather than use LLMs to generate apps from whole cloth, we should use them to build standards-compliant tools that can create deterministic outputs. These outputs can then be integrated into existing workflows.

AI isn't the thing. It's the thing that gets you to the thing...

We must use AI-generated code not only to produce lines of code but code that can cooperate with legacy and emerging systems, can be understood by maintainers and avoids becoming another app for the slop heap.

The New Standard

Over the next few posts we'll be sharing insights from building a tool we call Foundry.

Foundry is a CLI application that consumes an OpenAPI spec document and generates a working API service in a target language. The generated services can include a number of authentication schemes, middleware and validation mechanisms–all of which are specified within the source OpenAPI file.

The standards-based approach is the key value proposition for Foundry: wherever a known standard exists, we use it. Why the obsessive focus on standards?

Partly because we're lazy. When there is a sound proposed or fully published standard it means less work having to justify and document our approaches. It's one fewer design decision to make and guarantees that for the lifetime of our application there will be documentation and rich examples of our implementation whether we have the time or inclination to create our own documentation or not.

"Be lazy, bro. Just use a standard..."

A standards-centric also means others can create tools and services that connect and interoperate with our services without our explicit involvement.

The AI Workhorse

Lastly, we'll be using LLMs to create a lot of the boilerplate app generation code so we can focus on the best application design possible. Now that we can create code at the speed of a prompt, there is no reason to create well-crafted, standards-based code that's well-documented and delivers outsize value for end-users.

The speed of generative AI allows every team to be design-first

We'll also explore some insights on how to design codebases so that they are optimally suited to absorb AI-created code. It's imperative that we intentionally design our code so that we aren't plagued by the all-too common hallucinations and regressions that arise when using AI as a co-pilot.

Conclusion

Our project will succeed if we use AI to create a tool much faster than we otherwise could have, that itself produces dependable standards-compliant API services every single time. Stay tuned. The journey is just beginning....

Stop Hallucinating: APIs are the Missing Link For Reliable Agentic Applications

Sean Travis Taylor — Tue, 24 Dec 2024 16:46:01 +0000

In a previous post, we discussed the challenges of building agentic applications solely with large-language models. Leaving aside the immense challenge of training large-language models on an infinite array of use cases in order to function as even a rudimentary personal assistant, we declared agentic applications built on the autonomous operation of UIs designed for humans as a wrong turn.

Time and again, we hear reliability is a chief obstacle preventing the serious implementation of LLMs and agentic applications in business systems across industries.

A tool that behaves as expected only 7 out of 10 times is useless: whether it's a hammer, a light switch, an airplane or a toilet. How do we improve the reliability of a technology whose principal value propositions are: excellence at generating things (e.g. images, text, video) and with respect to text, sounding human-like?

A Tale of Two Computing Models

Applications, as we know them, are evaluated on wholly different criteria--namely: consistency, specificity and determinism. The same input produces the same output forever. This is their competitive advantage.

LLMs and other generative tools are a different story: identical inputs can produce mildly to wildly different outputs ranging from the amusing to the outright disturbing.

We call divergence from an expected outcome a bug in a traditional application and a hallucination in an LLM. This difference is more than semantic; it reflects a fundamental conflict between operating paradigms that must be reconciled before agentic applications can succeed in mass adoption.

Below we examine a novel approach to developing agentic applications by combining the best of these two powerful computing models; this solution ably competes with the current state of the art in AI agent development.

Read on to see our take on the future of agentic applications.

APIs and LLMs Compared

APIs	LLMs
Highly structured (HTTP interface)	Unstructured (natural language interface)
No tolerance for divergence	High tolerance for divergence
Fast	Slow
Established means of integrating systems	Novel means of integration systems
Does not adapt	Highly adaptable (prone to hallucinate)

The relative assets and liabilities of APIs vs. LLMs

Above is a quick breakdown of the differences between APIs and LLMs. In 2024 business systems consist of sprawling APIs designed to be accessed over HTTP. By themselves and without modification, they are incompatible with natural language interfaces like an LLM.

If we want such systems to be agentic, we have to retrofit a vector database onto them and do the hard work of vectorizing our data.

This is all in service to making that data available in an implementation of RAG so our LLM can incorporate vast proprietary information into its responses. We may have to build a custom chatbot and we may still have to train our chatbot on our data.

What an enormous investment to adapt our backend systems to what is, in essence, a new UI!

We however, take a different approach.

Use Cases

It started with use cases. LLMs are exceptional at consuming textual input and sussing out an appropriate output. We figured if an LLM could do something as simple as deducing the relevant use case associated with a natural language request, we're most of the way toward a solution for a powerful AI assistant.

A use case is anything we'd like the AI to accomplish on our behalf. We prototyped requesting a car with a ride-hailing app. We wanted to be able to speak into our phone's mic and have the assistant fetch us details on a ride somewhere, returning details like the cost and ride duration.

Just What Are Your Intentions?

We chose to use basic prompt engineering to train an LLM to create what we call an intent specification from a request. For example, with a request like, "Get me an Uber to Madison Square Garden," the LLM will return something that looks like this:

{
    context: {
      location: {
        current: {
          address: null,
          bookmarked: false,
          lat: '37.7752315',
          lng: '-122.418075',
          name: null,
        },
        destination: {
          address: null,
          bookmarked: false,
          lat: '<LAT>',
          lng: '<LNG>',
          name: 'AMC Empire 25',
        },
      },
      user: {
        displayName: 'Augustus',
        id: 'dd495dbb-5a2f-46ed-8009-ad2bf0b85fcc',
      },
    },
    domain: 'app.intents.mobility.get_ride',
    reply_id: '903496b2-b3f2-4f70-8574-16ae38403550',
  }

From unstructured to structured. The LLM derives an intent from a natural language request. Our mobile app fashions the above intent spec before sending it to our backend.

This is a template; as it is it is incomplete. However, with few-shot prompting, an LLM becomes very good at producing such templates. Now that we have an intent that has been specified along with the destination name, our mobile app gathers the other contextual data required to query our backend, such as:

display name
userId
current location data

Once our template is populated, we send the request to our backend. On our server, we use the Google Maps Geocoding API to retrieve the coordinates for the destination parameter in our intent specification.

Driving It Home...

Armed with these coordinates, we can make a call to the Uber API to see what Uber services are available.

When Uber replies, we construct an intent reply message and pass it back to our application. Our LLM knows how to convert an intent reply to natural language and from there it's a trivial matter to use a text-to-speech solution to give our app a voice.

Et voila! We now have a voice-activated valet that we can instruct to fetch us a car using natural language.

This design allows us to use structured data, which our existing APIs require, while also allowing us to explore the possibilities of natural language interfaces to provide our end users a richer experience interacting with our systems.

Thinkin' About Tomorrow

As the AI landscape continues to evolve, we will need as many strategies as possible for making the data locked away inside our systems accessible. We will want to do this with as little toil as possible. We may not have the resources or expertise in the form of AI or machine learning engineers to build bespoke agentic systems for us.

The ability to leverage our existing engineering competencies toward emerging AI applications will be critical as this highly specialized knowledge will not exist in adequate supply.

Finding ways to combine traditional APIs and advanced LLMs means we can not only get the best of both worlds when it comes to creating innovative solutions, it also ensures that, no matter what the future of LLMs holds, we will be ready to move at the speed of change.

APIs Are the Only Serious Approach to Agentic Applications

Sean Travis Taylor — Tue, 10 Dec 2024 19:17:08 +0000

The explosion of generative AI and the budding experiments with agentic applications created a slew of products touting the ability to run our digital lives autonomously. Yet these products suffer from a common flaw: they are designed to operate user interfaces that are themselves designed to be operated by human beings.

The makers of these products congratulate themselves on the ability of their offerings to use web browsers, fill out UI forms, navigate web pages and follow links.

This is waste: an AI agent without eyes doesn't need a UI and given the increasing cost in dollars and kilowatts required to run generative AI applications, we’re bound to see more waste as these products and services gain mass adoption.

Taking the Scenic Route

The result? AI agents will be slower given the need for a UI layer. Further, AI agents will run into the same obstacles human users do when using a typical web application:

counterintuitive UI design
confusing UI flows
lack of accessibility
broken web apps

Agents will be slower given the need to contact a command-and-control process to issue instructions to a peer process driving a web browser, which then completes—or fails to complete—a user request, which then sends results back to the command-and-control process, which then digests the result and finally informs a user.

This is unjustifiably wasteful irrespective of implications for increased emissions and overconsumption of electricity–to say nothing of the financial cost of a request to an AI application relative to a RESTful API.

Design Dilemma

Why build this way? Why the curious obsession with technology that accomplishes human goals with superior results but always, as the song goes: like humans do?

Solutions in AI or robotics that garner the most praise or elicit the most fear are those that mimic human behavior: humanoid robots that work in factories or AI that operates your desktop computer.

These innovations evoke such fear because of a flaw in design philosophy apparent to anyone with eyes.

Robot Dreams

These innovations are designed to replace human beings.

This leads to a troubling design approach. When the frame of reference for any AI solution is that of human beings and how they might accomplish a task, the result is androids and other Asimovian robot dreams.

Never mind that humans and their bodies make poor machines, both in practical and philosophical terms. If the design prompt, or the assumption within it, is to make a human but better, the solution will prioritize human mimicry over every other criterion.

I spent the summers of my college years working at a Toyota plant in my home state of Kentucky. It was a dazzling operation combining both human and machine innovation. Industrial robots abounded yet none of them looked like a human being.

And why should they?

Automated stamping of car bodies in the most efficient way requires unique capacities and even a super humanoid robot makes less sense than the industrial robots in use today.

Human Machines | Machine Humans

Human bodies evolved in response to unique environmental requirements irrelevant in modern industrial contexts. In such contexts, what is required is a solution sensitive to the operational challenges in question, not a superhuman substitute.

The similarities to agentic application design are striking. We see the same approach centering human mimicry: an agent autonomously clicks and navigates through your web browser or desktop UI, doing your tasks largely in the way you yourself would do them.

Humanoid robots and AI-enabled self-driving user interfaces may inspire fear or awe but if the aim is achieving the objective in the most effective efficient way possible, attempts like these will routinely fail.

The extraordinary cost to maintain AI applications demands design approaches that leverage the strength of the underlying technology and avoid the urge to mimic human UI interactions.

Alternative Approaches

What is an alternative?

It involves using LLMs (large language models) to do what they excel at: extracting user intent from natural language. Traditional APIs, on the other hand, excel at taking structured data as a request and executing business logic in response. By combining these strengths, we can build efficient and effective AI applications that avoid unnecessary complexity and waste.

In the next post, we'll explore a real-world example app where, instead of using massive sets of training data, I use basic prompt engineering and a simple REST API to create a natural language Uber app.

Stay tuned.

Bussin': Database as a Message Bus

Sean Travis Taylor — Tue, 03 Dec 2024 15:45:09 +0000

A recent post extolled the virtues of exploring reactive databases for decoupled applications. Why is loose coupling so desirable?

If you've been reading this blog for a while, you know the answer.

Decoupled applications allow us to embrace change. When we design for change every architectural decision is less consequential. It means we can reduce the switching costs associated with any given approach.

We also get the gift of freedom–to experiment, to take measured risks and to innovate.

We know we want modifiable applications. We also know we want to decouple the implementation details of communication between parts of our application. That is to say we are less concerned with whether, for example, we communicate via RESTful API or GraphQL or message brokers.

Instead we concentrate on the meaning of the messages we exchange among our services.

This week’s pattern builds on using the Publish/Subscribe capabilities of reactive databases to create a message bus. We'll see how this approach allows seamless communication between services and affords easy extensibility of the application when it's time to introduce new features.

Read on to learn more about the promise and perils of taking the bus.

The Bus? Seriously…?

At its base, a message bus is a central communication link that services can use to listen for and dispatch messages of interest to anyone or anything connected to the bus. Adding or removing capabilities is simple: either case requires only attaching or detaching from the bus. Take a look.

Set it and forget it. Application services can connect and disconnect whenever they like. Communicating with event messages offers unparalleled flexibility.

Services are unaware of what peer services are connected to the bus. They are only able to post messages of interest to the bus.

This pattern shares similarities with the classic Publish/Subscribe pattern, where each service registers its own listeners. A challenge with this approach is that it's difficult to manage all publishers and subscribers as the system grows.

A message bus, similar to a message broker, can centralize operational logic like retries, routing policies, error handling and other concerns that would be duplicated in an architecture with a host of point-to-point connections between publishers and subscribers.

A Ride Down Memory Lane…

If this sounds reminiscent of the centralized enterprise service buses of the 90s, you've got a good memory. This architectural approach was the foundation of what became known as service-oriented architecture or SOA.

This paradigm fell out of favor as more logic concentrated in the message bus, creating challenges as the system scaled in terms of connected services. In fact, it created exactly the type of obstacles such a decoupled architecture is intended to avoid with the message bus as a single point of failure.

Again, self-describing messages offer an innovative solution. The core problem with the Enterprise Service Bus is the accumulation of message processing logic within the bus itself, as well as the coordination of message schemas among connected services.

By removing the logic for managing messages from the bus and embedding data required for message processing within the messages themselves, our bus is solely concerned with carrying messages.

When we eliminate logic from the bus and store all application state and ancillary processing data inside the messages, the result is a near-stateless application.

In the example application below, notice that there is no persistence layer the application uses to do its work aside from publishing events and responding to them.

Taking the Bus for Ice Cream

Our demo app is an ice cream service. It receives orders for customers, charges the customer card and fulfills the order provided inventory is available.

We use Supabase as our database of choice because...duh; this spares us having to create an API surface just to test our app. Since our FOHService (which could be anything including a UI app) is listening for inserts into the events table, it means we have a full stack app with a REST API in seconds.

The rest of the application is driven by the flow of events.

Stackblitz is our backend in this case. Here we define our services and their subscriptions and we publish relevant events. The implementation of this pattern (the how) is less important than what is accomplished:

the app completes tasks with refreshingly little state to manage
decoupled services are easy to replace, rollback or trial
if we know how to create self-describing messages, message structure can evolve without disrupting peer services

Toward Stateless Services

This approach of putting all stateful information into streams of successive messages is known as Event-Carried State Transfer. By carrying the relevant state in the message itself, downstream services don't need to query other systems for additional context.

As in our previous exploration of self-describing messages, they once more prove an invaluable contribution to system adaptability.

As in earlier examples with self-describing messages, our connected services can evolve their message schemas without disrupting the message processing of their peer services.

Finally, as in previous posts, we see another pattern that demonstrates the possibilities of architectures that reduce the cost of experimentation, increase the freedom to innovate, and above all, allow us to author systems that can move at the speed of change.

Bulletproof Backends: A Strategy for Adaptive Services

Sean Travis Taylor — Wed, 20 Nov 2024 19:08:24 +0000

This post is a digest of a conference talk I gave at the Nordic API Summit in Stockholm, Sweden. For those who prefer video to the written word, here's a link.

For a deeper dive, read on.

Building for Change

Much of the focus of this blog is creating services that can absorb change. Why? Because change is the most reliable business condition under which software must operate.

When we create systems that are amenable to change, it means we can experiment. We get to try lots of solutions to our problem. The cost of implementing any set of changes is decreased because our systems are not designed like Fabergé eggs.

New features and capabilities are easier to add without re-architecting the whole system or significant parts of it.

All of this comes down to reduced costs: a system that is less costly to change, to build, and to maintain. It is important to note however, that we speak not only of costs that can be strictly quantified.

We should also include reduced costs of debugging and reduced costs of our engineering organization's time, energy, and attention. Such cost reductions are absolutely worth taking a look at the pattern below.

State of the Art

If you design most API integration solutions like most companies, you are probably binding directly to the internal data model of a third-party service over which you exercise no control.

It looks like the following.

Directly binding to internal data models is most unwise.

This approach is by far the most intuitive design and is effective as far as it goes.

It is also brittle.

As soon as the services change at all, everything falls apart. The only way to resolve the broken system is a manual reconciliation—that is, a meeting between teams. Meanwhile precious time and, depending on the system, money is wasted.

This cycle repeats for as long as the services are in operation.

This is the state of the art in 2024.

Acceptance Criteria

How can we break the cycle? How can we create services that do not directly bind to the internals of peer services?

We will need two key ingredients: JSON Patch and self-describing messages.

Once we learn how to apply these ingredients, we will have everything we need to create services that exchange messages without needing to bind directly to the implementation details of peer services.

First things first: the core of the integration problem is the need for services to agree exactly on the details of message structure before they can communicate. This means as soon as the message structure diverges, the communication link is broken.

What would help is a way to translate one object structure to another on the fly. JSON Patch offers a solution.

JSON Patch

JSON Patch is a formal specification for applying a series of transformations to a candidate object to produce a result object.

Since a JSON Patch document is just a JSON object, it can be serialized. Because it can be serialized, it can be sent over the wire. Because it can be sent over the wire, we can create a link to it. But what do links have to do with anything?

To understand, we need to discuss self-describing messages.

Self-describing Messages

What is a self-describing message? It is a message that contains all the information required to process the message within the message itself.

If you're looking for examples, look no further than the webpage you're viewing right now. Everything required to render the page successfully is embedded in the HTML: from fonts to stylesheets to third-party scripts, images, video, etc.

When the page updates its CSS, you don't need to update your browser. No. You get a link to a stylesheet that contains the updated style code. Your browser is none the wiser. Another example: Have you ever received a package from FedEx, UPS, or DHL?

Packages are great examples of self-describing messages.

It probably looks something like the image above.

What is significant about it? If you have ever received a package—certainly any international parcel—you will notice all kinds of metadata (i.e. labels) on the box.

These labels tell processors how to process the package, which is the message in this example. Is the package fragile? Is it perishable? Does the package contain hazardous materials? Is the package accompanied by customs paperwork that offers hints about how to further process it?

What about tracking labels?

These are links to an outside system. They allow us to query a third-party data store for additional information about where the package has been and where it is going.

We can glean all of this information without needing to know what's inside the package. This is the essence of the self-describing message.

The Adapter Schema

The last thing we need is something called an Adapter Schema. This schema is one that neither of our collaborating services depend on for their business logic. This schema is only used to adapt one internal format to another while retaining all the fields of interest. Let's take a look at an example.

// Foo Service
{
  “name”: {
    “first”: “Jane”,
    “last”: “Doe”
  }
}

// Adapter Schema
{
  “firstName”: “Jane”,
  “lastName”: “Doe”
}

// Bar Service
{
  “customer”: {
    “given_name”: “Jane”,
    “surname”: “Doe”
  }
}

When we focus on the meaning of messages rather than their structure, we give ourselves the freedom to iterate on structure as needed.

Clearly, the above three objects refer to the same bit of information and that information has equivalent meaning. Structurally they differ but semantically they are the same.

The second object is our adapter schema and the first and third objects represent the data models of two distinct services. Our cooperating services will use this adapter to capture the meaningful data fields within the messages they exchange.

In the adapter schema we have flattened everything to the top level of the object but we needn't do so. The adapter schema can be as complex as our needs require. The key is that this schema is the only information that needs to be agreed upon ahead of time by our collaborating services.

A Solution Emerges

Once we have our adapter schema, we simply need to send a link to the patch document that will allow consumers of our messages to translate them to the adapter schema. From there, the consuming service can do any needed additional translations from the reliably stable adapter schema to the consuming service's own internal model.

Above, Foo Service can provide a link to a JSON Patch document that will allow Bar Service to translate from Foo’s internal model to the Adapter Schema. That patch document looks something like this:

[ 
  { "op": "move", "from": "/name/first", "path": "/firstName" }, 
  { "op": "move", "from": "/name/last", "path": "/lastName" }, 
  { "op": "remove", "path": "/name" } 
]

Recipe for transformation: a JSON Patch document is a set of instructions for moving from one object shape to another

As an alternative, services can exchange JSON Patch links that allow message consumers to translate directly to internal object models.

Regardless of approach, so long as the adapter schema is stable and each of our outgoing messages contains the link to the translating JSON Patch document, our services will not break owing to structural changes to our messages.

Message consumers are free to change their internal structures as much as they want without fear of breaking the integration. A reply message may also contain a link for a requesting service to translate back to the adapter schema before further processing of the reply.

In this way, all of our cooperating services can modify their internal object model as much as they like while still retaining the value of integration, the possibility of innovation, and most importantly, the ability to move at the speed of change.

For the Love of God...just use Supabase

Sean Travis Taylor — Tue, 12 Nov 2024 16:43:06 +0000

I will make a bold claim: there is no reason to build CRUD APIs anymore. You know the kind. API endpoints that just ferry JSON objects into a database to make them queryable by something like /ice-creams/{id}.

Perhaps it's a SQL database. Maybe it's MongoDB. It doesn't matter. Unless you are transforming the incoming request payload, and sometimes not even then, there are few reasons to handwrite API endpoints in 2024.

Honestly, there weren't many reasons to do it in 2017 either.

A few years ago, I was in an interview for a senior back-end software engineering role. The interviewer asked how I would build the back-end for a real-time collaborative word processing app like Google Docs.

"Well, I would start with a reactive database–" I began.

"Well, let's assume we don't have one of those," she said, steering the interview down a different path.

Eventually, we did talk our way through an architecture that satisfied her and I did end up landing the job. Yet I never forgot this experience because it describes a common approach to developing backend systems then and now.

Current State

You've been in meetings that center on building this way. Maybe it's a sprint planning meeting or a call between technical and non-technical business stakeholders. It goes something like this:

"Well, first we need to create a new table in the database."

"Then we'll need to create an endpoint."

"Then we'll need to wire the two together."

"Given what we've got in our backlog already, maybe we can get to this in two or maybe three sprints."

God help you if the endpoint in question has any non-trivial business logic in it.

You'll watch helplessly as the runtime of your meeting grows, threatening to crash into...your next meeting.

If you and your team are changing an existing endpoint, the drama above still unfolds unchanged.

There are good reasons that there's often no escape from this timeless way of building. When reactive databases didn't exist, the three-tier architecture—that is, separating a backend into an API interface, a logic layer, and a persistence layer—made a lot of sense. It allows you to separate the concerns of your application while also providing flexibility for each component in the architecture.

This is a good thing.

But as database solutions matured they also became more convenient. We saw a decrease in the cost of change, which makes re-examining these solutions worthwhile.

Future State

Instead of using a traditional three-tiered architecture, you might use SupaBase or its lightweight cousin PocketBase, both of which provide a REST-like API interface for all your tables and collections out of the box.

If you need to perform business logic on incoming requests, maybe you subscribe to changes on a specific table in PocketBase or SupaBase and transform requests within that subscription, then commit the result of the transformation to the same table or an entirely new one.

If you worry about vendor lock-in, maybe you use a vanilla Postgres database and layer PostgREST on top of it to provide a rich REST-like API interface for all of your SQL tables.

Are there challenges with this architectural approach? Most certainly.

You may be uncomfortable with how this architecture gently pushes you toward an asynchronous message flow rather than request-reply, which is far more straightforward. This can be a paradigm shift that many teams are unprepared for. They may also lack the institutional support from leadership to take even calculated risks with this approach.

Having pieces of business logic spread among many subscriptions and publishers can create code management, organization, and debugging challenges.

Your DevOps or SecOps stakeholders may resist the idea of the database being directly exposed to the public internet this way. While Postgres' Row-Level Security features offer fine-grained control over visibility and access to the database rows and tables, there may be other compelling reasons why this architecture is unsuitable for your organization. It may require a complete reimagining of existing infrastructure components.

Despite these caveats, exploring today's current spate of reactive databases complete with REST APIs out of the box is a business imperative in any organization requiring agility in the face of constant change.

Helpful hint: That's every single organization.

Help Is On the Way

Below, I outline four database solutions that either allow you to transform an existing SQL database into an API or provide a database and an API custom fit for your tables or document collections. Take a look.

Supabase is a full-fledged backend-as-a-service built on PostgreSQL that offers a real-time database, authentication, storage, and API services. It automatically generates RESTful APIs based on your database schema, and integrates smoothly with PostgreSQL’s robust querying capabilities. Supabase is particularly beneficial for developers who want a quick setup with minimal configuration, but need the flexibility and power of PostgreSQL, plus real-time functionality and easy integrations for modern web and mobile applications.

Pocketbase is a lightweight, open-source backend solution that combines a real-time database with file storage and authentication services. Its key benefits include simplicity and portability, as it can be run locally or in the cloud without much overhead. Designed to be user-friendly for both small projects and rapid prototyping, Pocketbase makes it easy for developers to quickly deploy applications with built-in user authentication and admin UI for managing data.

PostgREST generates RESTful APIs directly from a PostgreSQL database, eliminating the need for an intermediary API server. Its key benefits lie in the simplicity of generating fully-featured, highly-performant APIs from the database schema itself, leveraging PostgreSQL’s robust permissions system and allowing developers to focus on database logic, rather than writing API endpoints. This ensures efficient, scalable solutions with minimal overhead for API management.

DreamFactory is an open-source REST API generator that can automatically create APIs for any database, including SQL, NoSQL or other services. Its benefits include support for multiple databases and third-party integrations, automatic API documentation and flexible API customization options. DreamFactory simplifies the process of creating secure, scalable APIs without requiring manual coding, while supporting authentication, role-based access control and real-time data streaming for diverse backend infrastructures.

I hope you’ll explore some of the solutions outlined above in your next prototypes to see if you’re finally ready to stop building APIs like it’s 1999.

Back to the Sandbox: Leveling up with Lazy Loading and Dependency Injection

Sean Travis Taylor — Wed, 23 Oct 2024 15:12:28 +0000

In the last post, we explored the Sandbox pattern for decoupled applications that we can easily modify and extend as business needs change and as new requirements emerge.

We led with the benefits of this pattern in the previous post, so now we’ll reveal its implementation:


/**
 * @summary The core of the application; registers and emits events
 */
export class Sandbox extends EventTarget {
  /**
   * @param {String[]} modules
   * @param {Function} callback
   */
  constructor(modules, callback) {
    super();

    const factories = {};

    /**
     * An object with the `core` namespaces and event-related methods available to
     * all modules; provided in the callback to the `Sandbox` constructor
     */
    const sandbox = {
      my: {},
      core: {
        generateUUID: this.generateUUID,
        logger: {
          getLoggerInstance: () => console,
        },
      },
      // Bind event methods from the EventTarget (i.e., this) to ensure proper context
      addEventListener: this.addEventListener.bind(this),
      removeEventListener: this.removeEventListener.bind(this),
      dispatchEvent: this.dispatchEvent.bind(this),
    };

    // Create factories for each module
    modules.forEach((moduleName) => {
      factories[moduleName] = () => new Sandbox.modules[moduleName](sandbox);
    });

    // Lazily initialize the modules using `Object.defineProperty`
    Object.entries(factories).forEach(([moduleName, factory]) => {
      Object.defineProperty(sandbox.my, moduleName, {
        get: () => {
          if (!sandbox.my[`__${moduleName}`]) {
            try {
              // Here we create module lazily (i.e. when the module is accessed)
              sandbox.my[`__${moduleName}`] = factory(); 
            } catch (ex) {
              console.error(
                `INTERNAL_ERROR (sandbox): Could not create module (${moduleName}); ensure this module is registered via Sandbox.modules.of. See details -> ${ex.message}`
              );
              return;
            }
          }
          return sandbox.my[`__${moduleName}`];
        },
      });
    });

    // Pass the sandbox object with `my` and `core` namespaces to the callback
    callback(sandbox);

    /**
     * Though Sandbox extends EventTarget we *only* return a `dispatchEvent` method to
     * ensure that event registrations occur inside the Sandbox. This prevents
     * "eavesdropping" on events by clients that are not sandboxed. All such clients
     * can do is notify the sandbox of an external event of interest
     */
    return {
      dispatchEvent: this.dispatchEvent.bind(this),
    };
  }

  /**
   *  Houses sandbox module definitions
   */
  static modules = {
    /**
     * @param {String} moduleName - the identifier for the module to be referenced by
     * @param {Object} moduleClass - module's constructor
     */
    of: function (moduleName, moduleClass) {
      Sandbox.modules[moduleName] = moduleClass;
    },
  };
}

The implementation of our Sandbox ensures that it does not matter how dependencies are defined or ordered. Because they are lazily loaded, it means that dependent services are not created until they are used–even if a dependency is defined after a service that relies upon it; it doesn't matter.

Without lazy loading, services must be created in reverse topological order. This means first we create all the dependencies that have no dependencies themselves, then proceed to the ones that only depend on the already-created dependencies and so on until all services are created.

The ability to define dependencies in any order is another attribute that keeps our architecture flexible.

But how does it work? Let’s break it down.

Using Factory Functions for Modules: Instead of instantiating modules immediately in the Sandbox constructor, we create factory functions for each module and store them in a factories object.
Lazy Loading via getters: We use Object.defineProperty to delay the creation of each module until it is accessed for the first time. This allows us to resolve dependencies dynamically.
Passing the Sandbox: Above, we pass the reference to the sandbox to each factory. Passing the sandbox object to each module's constructor allows modules to reference each other through the sandbox.my namespace.


 // Lazily initialize the modules using `Object.defineProperty`
    Object.entries(factories).forEach(([moduleName, factory]) => {
      Object.defineProperty(sandbox.my, moduleName, {
        get: () => {
          if (!sandbox.my[`__${moduleName}`]) {
            try {
              // Here we create module lazily (i.e. when the module is accessed)
              sandbox.my[`__${moduleName}`] = factory();
            } catch (ex) {
              console.error(
                `INTERNAL_ERROR (sandbox): Could not create module (${moduleName}); ensure this module is registered via Sandbox.modules.of. See details -> ${ex.message}`
              );
              return;
            }
          }
          return sandbox.my[`__${moduleName}`];
        },
      });
    });

In the example above our getter is actually a function that creates the module.

Lazy loading means deferring initialization of classes until they are actually needed. Our getter function, shown above, checks to see if the module has been instantiated when the module is first accessed. If not, a factory function is called–this is the class that we register with the Sandbox.modules.of method. The module is created then cached, ensuring each module is created only once.

This strategy allows us to avoid creating all our modules up front, especially if some modules may never be used. It also saves time on initialization and memory.

Dependency injection completes this architecture. All our services consume the sandbox as a constructor argument. This allows us to mock our sandbox API in unit tests and test our modules in complete isolation from the rest of our code.

At its simplest, Dependency Injection means providing a method or class all of the things it needs in order to do its job. This can be achieved by passing dependencies as function or constructor arguments or by using a framework.

Since our modules communicate with the rest of the application via the sandbox, we can place them anywhere in our code base without impacting peer modules. The Sandbox pattern, as shown above, gives us maximum flexibility when it comes to defining, executing, and organizing our code: ensuring that no matter what happens in the future, our application is ready to move at the speed of change.

Jump In: Sandboxes for Decoupled Applications

Sean Travis Taylor — Tue, 17 Sep 2024 13:50:06 +0000

A common theme of the last few posts has been containment: as a solution to invalid state changes, for managing exceptions and for unifying control flow.

This approach has another name: encapsulation. It is a fundamental aspect of program design. Why? Encapsulated programs are easier to manage; they keep things organized and they limit the scope of change required as new needs emerge.

The oft-misquoted mantra of the Single Responsibility Principle is worth repeating:

A class should have one and only one reason to change.

Think of a codebase you know. How easy is it to change a class or a larger component without changing something else? If there is an update to the PaymentService, does the OrderService have to change too?

Why are these second-order changes necessary and what is the order of difficulty in making them? The reply demonstrates how well or how poorly our systems absorb change.

Adapt Or Die

In business, our ability to absorb change impacts our ability to stay in business.

How many times has your product manager insisted a change is not only urgent but required? How many times have you bristled at such a request owing to a keen awareness of the time such a change will take?

“How long?” is invariably the next question from your product manager. They may ask about “level of effort”, which is another polite way of asking “How long?”

Frustrating as it is, how long it takes to implement a change is the difference between maintaining competitiveness or making a needed improvement and watching as our business fumbles.

What makes change difficult? Why do consequential changes take so long? A prime suspect is…dependencies.

As the number of dependencies among modules grows, the greater the requirement of cascading modifications across our application and the greater the risk of instability across modules that directly depend on each other.

It begins innocently enough...

...but as more modules are added, potential interactions and dependencies grow exponentially, not just linearly. Even a modest number of modules can be difficult to mentally model.

When this network of modules grows too gnarly to reason about, the only recourse is to break our change into pieces over a span of weeks or months.

This lead time gives competitors time to out-innovate us or for a bug in our program to persist longer than necessary; it also hampers the ability to experiment, which is the key to innovation.

How can we limit the scope of change when updates are necessary? Further, how can we de-risk that change?

Let's talk about the Sandbox pattern. For those that like details first, you can explore a detailed demo here. For a gentle introduction, read on.

Come Out and Play

This pattern is an oldie but a goodie. The exploration below is inspired by Nicholas Zakas's aging but excellent talk on creating an application framework.

The implementation is less important, as Zakas makes clear. The benefits of the Sandbox pattern include enhanced modularity, reduced coupling and streamlined communication among application components. With this pattern, we set a foundation that ensures our applications and services can move at the speed of change.

First we see create a new Sandbox, which is an isolated context for our application modules to “play” in. The callback is where all the dependencies specified in the dependencies array are made available.

// classes imported from wherever they are in the filesystem
Sandbox.modules.of('OrderService', OrderService);
Sandbox.modules.of('PaymentService', PaymentService);

const mySandbox = new Sandbox(
  ['PaymentService', 'OrderService'], function (box) {
   // Where we use the services defined on the 
   // sandbox aka where the action happens...
  });

The box argument of our callback is an EventTarget, which means we can register an event listener via addEventListener. Communication via events is key to decoupling our architecture; since our services have no knowledge of their peers they can only dispatch events to the sandbox.

import { SystemEvent, Events } from './event.js';

export class OrderService {
  constructor(sandbox) {
    this.sandbox = sandbox;
  }

  /**
   * @param {Object} event
   * @param {IEvent<Object>} event.detail
   */
  onOrderReceived({ detail: event }) {
    console.log(event);
    // do some order processing here...
    this.sandbox.dispatchEvent(new SystemEvent(Events.ORDER_FULFILLED));
  }
}

When the Sandbox is initialized it provides access to registered services. Above we send an ORDER_FULFILLED event to the Sandbox after an order is processed

Only the sandbox has knowledge of registered services. When an event of interest is dispatched, the sandbox calls an appropriate handler, which receives data about the event.


const mySandbox = new Sandbox(
  ['PaymentService', 'OrderService'], function (box) {
    try {
      box.addEventListener(Events.ORDER_RECEIVED, (event) => {
        box.my.OrderService.onOrderReceived(event);
        box.my.PaymentService.createInvoice(event);
      });

      // assuming our OrderService dispatches event
      // when the order is fulfilled  
      box.addEventListener(Events.ORDER_FULFILLED, (event) => {
        box.my.PaymentService.onOrderFulfilled(event);
      });
    } catch (ex) {
      console.error(ex.message);
    }
  }
);

This architecture allows us to write isolated classes with no dependencies and clearly defined responsibilities. We have freedom to move services anywhere at any time in our app with no changing costs. For best results we limit ourselves to importing only types, interfaces, NPM packages and files located within the same folder; this gives us maximum maneuverability.

Breaking Boundaries with Cross-Sandbox Events

The Sandbox pattern also offers another organizational unit of our code. For example, if we want to create multiple sandboxes to group services into specific domains. Since the Sandbox constructor returns an object with a dispatchEvent method, we can publish events across sandboxes without exposing the sandboxed services themselves: this means maximum modularity, maximum range of motion and a clear separation of concern for each of our services.

// here we have a sandbox instance solely focused on 
// shipping-related concerns...
const yourSandbox = new Sandbox([], function (box) {
    try {
      box.addEventListener(Events.SHIPMENT_READY, (event) => {
        // do stuff related to preparing to ship the order...
    } catch (ex) {
      console.error(ex.message);
    }
  }
);

const mySandbox = new Sandbox(
  ['PaymentService', 'OrderService'], function (box) {
    try {
      box.addEventListener(Events.ORDER_RECEIVED, (event) => {
        box.my.OrderService.onOrderReceived(event);
        box.my.PaymentService.createInvoice(event);
      });

      box.addEventListener(Events.ORDER_FULFILLED, (event) => {
        box.my.PaymentService.onOrderFulfilled(event);
        // communication across sandboxes is facilitated through events too...
        yourSandbox.dispatchEvent(new SystemEvent(Events.SHIPMENT_READY, event.detail);
      });
    } catch (ex) {
      console.error(ex.message);
    }
  }
);

Our dispatched events are instances of CustomEvent interface

If you are familiar with the classic design patterns, you will notice this implementation is a combination of the Mediator and Publish/Subscribe patterns.

As always, the exact implementation is less important than the inspiration. Whether you roll your own variety of this type of architecture or use an established framework, the principles of event-driven design and dependency injection set a solid foundation for any large application.

Through their judicious use, these principles will serve your programs well no matter what they aim to achieve–enabling them to move at the speed of change.

Stay tuned for an even deeper dive into this pattern! In the next post we get into the weeds of just how dependency injection and lazy loading enhance the flexibility of our Sandboxes.

Keeping 'em Honest: Self-Validating Objects

Sean Travis Taylor — Tue, 10 Sep 2024 14:05:56 +0000

What is a program? A program is a series of state changes that accomplishes something. It may be a game or a piece of CAD software. It may be a graphic design application or a script that automates data entry. Whatever the endeavor, a program's purpose is to do something and achieve that something over a series of changes in state.

A simple calculator offers an example. We begin with a value and through a series of operations (state changes) we produce a result: a sum, a product, a difference or a dividend.

What is a bug? A programmer's headache certainly but what is it really? It is a result of an invalid state change:

division by zero. A compiler will reject this computation
sending an HTTP POST request to create a new resource without authorization
subtracting a number from an array via the minus (-) operator

The list above presents invalid state changes within a programming language or within the business rules of an application. The programmer's mandate is to avoid such changes; when they occur, it is the programmer's job to ensure the program recovers.

Standing Guard

Strong typing helps in this regard. So do guard clauses and validation packages like Joi or Ajv. If none of these are in place in our programs, we should begin with the above. Such approaches are certainly better than nothing yet fail to aid the program at its most vulnerable: during execution.

Guard clauses require sprinkling if/else statements throughout our codebases and they create new states our program can assume. They also require immense rigor. We discussed the dangers of a gauntlet of if/else statements in a previous post.

Type-safe languages or typing packages offer confidence but serve best as we write code. They prevent introduction of invalid state changes into the program in the first place. Yet whether provided by a programming language or some third-party package, strong typing offers less protection when our program runs.

At runtime we pray we've done our best to catch exceptions whenever they occur and at least log a descriptive message that leaves us some clue. The current state of the art encourages us to catch invalid state changes through vigilance and explicit checks. We can also just wait for the program to explode.

The $64,000 Question

How can we design objects such that invalid state transitions instantly create exceptions?

How do we avoid passing invalid data around our application until a guard clause or validation exercise uncovers an invariant is violated?

Can we make invalid state transitions impossible?

We present the Self-validating Object: an object that ensures its own validity and notifies us when validation criteria are violated.

import Ajv from 'ajv';

/**
 * @param {Object} schema - a JSON Schema document to use for validation
 * @param {Object} options - valid JSON Schema configuration option see: https://ajv.js.org/options.html
 */
function Validator(schema, options = {}) {
  const ajv = new Ajv(options);

  /**
   * Runs validation of an incoming object
   * @param {Object} data - an object to be validated against the `schema`
   */
  function validate(data) {
    const valid = ajv.validate(schema, data);
    if (!valid) {
      throw new Error(
        `Validation error: ${ajv.errors
          .map((e) => JSON.stringify(e))
          .join(', ')}`
      );
    }
  }

  return {
    validate,
  };
}

/**
 * Factory function for Self-Validating Objects
 * @param {Object} schema - a JSON Schema object
 * @param {Object} initialData
 */
function SelfValidatingObject(schema, initialData = {}) {
  const validator = Validator(schema);
  let __data = { ...initialData };

  /**
   * @throws Will throw exception if data validation fails
   */
  function validate() {
    validator.validate(__data);
  }

  const proxy = new Proxy(
    {},
    {
      set: (target, prop, value) => {
        __data[prop] = value; // Set the new value
        validate(); // Validate new data state; may throw exception
        return true; // Indicate successful property set
      },
      get: (target, prop) => {
        if (prop === 'value') {
          return () => __data; // Provides access to the whole object
        }
        return __data[prop]; // Default get operation
      },
    }
  );

  validate(); // Initial validation of the provided data
  return proxy;
}

Our implementation uses JSON Schema and Ajv as a validation handler but you can adjust this pattern with your own validation library as well.

The key to this pattern is the ES6 Proxy class. We won't explore this class in this post but you can learn more here and here.

The Proxy class intercepts changes on an object. Our implementation automatically validates the object whenever a field is set or updated. This validation is executed against the JSON Schema we provide on object creation.


const schema = {
  type: 'object',
  properties: {
    foo: { type: 'integer' },
    bar: { type: 'string', minLength: 5 },
  },
  required: ['foo', 'bar'],
  additionalProperties: false,
};

try {
  const fooObject = SelfValidatingObject(schema, { foo: 1, bar: '12345' });
  console.log('Object is valid:', fooObject.value());

// This throws an exception because '123' is not a valid string for 'bar'
  fooObject.bar = '1234';
} catch (ex) {
  console.error(ex.message);
}

Staying in Touch

We learn the instant an object enters an invalid state and raise an exception. No stepping through code to learn when an object property becomes undefined. Our exception also includes the breached validation rules for our logs. This means no more logging objects to trace surprising state changes.

Since validation logic is encapsulated within our object, we know the validity of our object wherever it travels in our codebase.

When Self-Validating Objects are combined with patterns explored in previous posts, we get a robust codebase with fewer bugs owing to unpredictable state changes.

Self-Validating Objects render it impossible for objects to exist in invalid states. Whether it's the implementation shown above or something different, thinking deeply about how to prevent invalid state changes is a key activity during the design of any program.