What are integrations
The word integrations is used in different contexts. In most cases, it defines something that unifies two things into a single whole. One country can integrate another, as happened a lot in history when empires swallowed smaller states. A recipe can integrate a fresh ingredient to make a new entirely new experience.
Finally, and the topic of this article, are data integrations. These integrations combine multiple data sources into a new whole. Each integration adds to this unified data model, leveraging the scale of the data to bring forward new ideas, tools and operations.
In general, most data integrations act on data streams. Each stream goes from a "chaotic" specific representation to a unified format that is shared across all integrations. We call this a unified data model.
Such a data model can power multiple applications, while hiding the complexities of the underlying systems. If you for example query google, you can find links to PDFs, webpages, images... All of these are thrown in a uniform representation, the link. Only by following the link, and thus diving deeper into the data, can you find the actual underlying source.
The demons of a unified data model
As discussed before, each integration has a specific data model. Of course, not only the data model is important to take into account. Below we will discuss a few things that are important dimensions and attributes of integrations.
Integrations can be made via multiple ways. Focusing mainly on data ingestion, you can ingest data via:
- APIs
- Webhooks
- SFTP (files)
- Database queries
- ...
Secondly, the format plays a big role in how you will extract the data. A few examples are:
- JSON
- XML
- CSV
- Binary (Excel sheets, gzipped data, proprietary formats, ...)
- ...
Finally, the last dimension I'll expand on in this article is something that might not be important for your application, but if it is, defines largely the capabilities of your system, liveness:
- Realtime
- Query-able
- Scheduled
Each of the dimensions above adds complexity to your uniform data model. The downsides of all of the above attributes need to be incorporated in your uniform model, as otherwise, you don't have a unified whole.
Why flexibility is key
Understanding the complexities of the huge landscape of possible integrations helps with mapping your route through it. Sadly, even if you would understand the entire landscape, that still would not be enough to architect a perfect solution. Imagine if a new integration comes on the market after you have done your analysis & it uses an entire new way of working. Then, you will be scrambling to fit it into your well defined framework.
As such, I keep a simple mantra when building integrations: "Flexibility is key".
As flexibility is hard to focus on due to the definition being literally: "able to cope with chaos". In this article, I will nonetheless try to give a few pointers on how to keep an eye on your code's flexibility. Flexibility exists on a scale, as staying flexible takes effort as well. In most cases, the balance between effort and flexibility needs to be made. Business should not invest effort in flexibility that tries to reign in chaos that does not exist yet.
How to deal with flexibility
Following the last paragraph, the pointer I can give is, try to map the chaos you are dealing with. This is inherently a difficult task.
In most cases, you are not working in a system without any limits. You can in most cases put some restrictions over the possibilities you are trying to map. Each possibility will have a probability, even if you need to estimate it.
The first step in scoping where your flexibility must lie, is thus understanding the probability of each restriction to occur. This will help you understand the priority you need to assign to deal with each.
Secondly, a good way to make it easier to add new possibilities to your code is to separate your responsibilities cleanly. If you work in engineering, this statement is not going to be new to you. Separation of concern is a big topic in software engineering, rightfully so. If your components are split in logical components, adding an extra consumer or an extra interface will not be the end of the world. Keeping an eye on building your software in both logically split steps and components with well defined responsibilities will help you expand your implementation more easily.
Lastly, taking the time to fix the assumptions that have been invalidated by reality, will help keep your solution robust in the long run. Understanding that you made your analysis, but it wasn't truly wrong, just not 100% accurate, is important to stay flexible yourself. Take the time to rewrite your code slightly instead of building a quick & dirty solution on top of your previous solution might be the best way for future you.
Here, I would give the tip to use the same prioritisation as we explored before. If only a single of your integrations uses a certain way of communicating, go for the one-off quick & dirty solution. If you see more and more occurrences, maybe go over your analysis from point 1 again & see if it is still according to reality.
With these tips, I hope that you have a few more tools to reign in chaos and stay flexible in the ever changing world of software. You never know what next year will bring...
Conclusion
In the article we spoke about the nature of integrations. How they are the uniform mesh holding multiple homogenous systems together. Due to the non-uniform nature of the underlying parts, the mesh needs to be able to deal with complexities.
We looked into a few types of dimensions in which different integrations can differ & how each strains your mesh slightly more.
To deal with this, we focussed on why flexibility is key in such environment. Why it is so important & how to get a good overview of it.
Lastly, we discussed ways to lower flexibility & thus also lower hurdles to integrate with new systems. I gave a few personal tips on how I personally keep the flexibility high, while keeping the implementations simple.
My hope is that this article opens your eyes to the complexity of staying flexible in more & more complex environments. I hope it gives you some tools I've personally used in the past, to aid you in increasing flexibility such that you remain able to integrate new & innovative solutions into your applications.
Top comments (0)