DEV Community: Shantoie Vorster

Integrating Stuff - A look at why system integration is so hard and what can be done to make it easier

Shantoie Vorster — Fri, 11 Nov 2022 13:44:52 +0000

Living in an ever-connected world to have every application talk to another is the goal. I say it is the goal because the reality is that someone has to build the integration functionality. Suppose you have ever been part of an integration project or even ventured into the realm of developing something that allows two applications to share data, work together or share functionality in any way. In that case, you know it can be pretty challenging.

Fortunately, such integration projects are becoming more manageable, especially with modern applications and platforms. There is an API available for everything. Before the era of REST APIs, there was SOAP, or heaven forbid, direct data access.

If you are looking to do an integration, there are a few things to consider:

Why do I want to integrate?
How will these two (or more) systems interact with each other?
What will I use to facilitate this integration?

Of course, we can ask many more questions, but for this post, let us focus on those questions. Before we delve into those questions, it would be good to start with the basics:

What is system integration?

System integration is the process of combining different systems and software so that they can work together. It is also the process by which a system or application can interact with other systems, applications, or devices.

Some would argue that integration is all about data exchange, and although that is true to a certain extent, we must recognize sharing of system functionality. For example, if one application holds the data and another is used to send emails to clients, we need to create an integration that will allow us to access that data to send reports to our users via email. We will need a bit of both systems to do this. This is a very rudimentary example, but you get the gist of it.

An integration system is an application that focuses explicitly on handling this interaction. This can be necessary when there is no natural or built-in way for two applications to interact. Think of those legacy systems at your company with no APIs or integration features.

Since we are also moving in a direction where APIs are the norm, integration is becoming easier. APIs make it easier to access data and features of applications without having to delve directly into their data sources. Before APIs were a big thing, many integrations had to be done by accessing one application's database directly, reading files produced by that application or a custom piece of code had to be written to access the application directly.

It sounds deceptively simple, and to a certain extent, it can be. If you simply need to call an API then it is not so hard (depending of course on the unique complexity of the API in question). The tricky part comes in with orchestration and infrastructure. When should what happen, and how is data/functionality to be used? Before starting with the integration, be sure to understand your requirements and how these systems interact.

In addition to understanding the required interaction, you must choose the tool to facilitate the integration. Here you can choose code, but there is also the option to select a low-code option. Choosing a low-code option can allow you to use predefined components and functions specifically made to facilitate integration and interactions with specific systems and applications.

Why is integration challenging?

Every integration is different. Some will be difficult, and some will be easy. Depending on the system you are integrating with, how you interface with that system will differ based on its architecture, technology and even the methodology used to build that system. As mentioned, many systems have APIs that make integration and interaction much easier. However, some older and more bespoke systems may not have such luxury.

A big part of any integration project is a proper analysis of how we can retrieve data and access the system's functionality. A proper understanding of how we can interact with it is key to successful integration, especially if you need to build an integration system that needs to interact with many different systems/applications.

Understand the requirement

The first question I would ask is why we need to develop this integration. What is the end goal? This may sound strange, but I have found that people often don't understand the requirement. From personal experience, this is a recipe for disaster.

Understanding the requirement will guide you through the entire integration. It will ensure that you select the correct integration method, select the correct tool to facilitate your integration and more. Will you need to interface via an API, or will it be a direct data access scenario where you need to read from a file, database table or any combination. In such a scenario, you might benefit from a tool that will allow you to create quick little ETLs.

You don't have to fully understand all the smallest details, but a base understanding of why we are building the integration is crucial. Are we doing this to access data? Do we need to kick off some functionality from one system? What is the result of this integration?

Understand the interactions

How will one system interact with another? Your answers to the previous question will influence the answer to this question. Each integration is different. No two systems are the same, depending on when they were developed, what tools were used to develop them and what methodology and technology were popular at the time of development.

For example, let us say that we have one system or process that needs to access the data in another. If an API is available, we can simply use that to retrieve the required data. If no API is available, the interaction becomes more complex as we will have to find a way to retrieve the data from the other system. This can either be done by direct database read, direct file read or any other way, depending on what the two systems allow.

Often you will not be able to have a direct interaction between two systems, in this case, you will either need to use a tool to create an integration system or tool, or you will need to create one. This is where choosing the right tool for the job will become important.

Choose a tool

Now for the fun part: choosing the tools for the job. There are many options; you can write a bit of code to facilitate API calls, use a specialised integration tool to facilitate a specific interaction, or even use a low-code tool to build your integration system. The answers to the previous questions will guide this process. The requirement and type of interaction between the systems will dictate if there are any technical specificities to the tool you will need to choose.

I recently worked on an integration where a system had to retrieve data from Sage Intacct. I made my solution in Linx, as it is my preferred tool for developing these integrations.

What I have found is that the hardest part is to start. I will use my most recent experience, Sage Integration, as an example. Data needs to be pulled from Sage for that data to be used in reporting in another application.

I used Linx to connect to the Sage web service and pull the data. This came with a few challenges, such as figuring out how the Sage web service works. First, I had to retrieve a Session ID (their version of authentication). After the Session ID was retrieved I could call the relevant web service function to retrieve the data and write it to a database where my other process could use it. Read more on how I called the Sage Web Service here.

For this straightforward integration, I essentially needed to create a middleman process, or middleman application often referred to as middleware:
Middleware is software that connects two other software applications. It is often used in the context of computer networks to manage the flow of data between two or more programs.
With Linx, I created a very simple integration application or middleware application to facilitate the flow of data or functionality between my two processes. I have used Linx countless times to do something like this, to integrate legacy applications with newer platforms, to ensure that functionality is accessible or even just to facilitate data exchange and flow between applications or processes.
Low-Code tools as Integration builders
There is a case for using a low-code tool to facilitate integrations. Low-code tools often benefit from having pre-defined functionality or components to interact with various systems, technologies or objects.

For example, let's look at how we can use Linx to integrate different systems. Linx has components to interact with QuickBooks, Xero and other specific platforms, but we can always use a simple REST API Call component to use a REST API. Linx also allows for direct database access and easy file reads.

Let's say we have three applications or systems:

A system that captures financial data for clients that makes that data available via API
A system that dumps specific client interactions into a CSV file
A system where client data is captured via a front-end and that data, such as email addresses are recorded in a database
We need to send a report to each client via email.

Using a low code tool such as Linx, I can: Retrieve a list of clients with their email addresses from the database, read specific client interactions for the client from a CSV and then combine that data with data from the financial system to build a specific report. I can then use the low-code platform to send that email to the client and log it in a separate logging database. All of this will be done using pre-defined components and a drag-and-drop interface, making such a development much quicker than traditional development techniques such as Coding.

In Linx, I would use a file reader, a database reader and a Call REST API component to do this, and finally, compile the report and send the email using Linx components that access a mailbox.

Again, all of this is possible with code, but it will take me much longer. Low-code tools come with the benefit of quicker development, and from experience, I can also say that it makes development of systems much simpler. When using a low-code tool, especially one geared towards developers, you are developing with much larger, pre-tested and pre-defined blocks.

Final thoughts

Each integration is different and requires proper consideration and analysis before you jump into it. Choosing the right tool can make all the difference. Choose a tool that streamlines development and that allows for easier interaction with the different components of the integration. You will have more time to focus on figuring out the nuances and business rules of those interactions.

Make sure that you:

Understand your requirement.
Understand how the two systems need to interact with each other.
Choose the right tool.

Getting my API to Production

Shantoie Vorster — Mon, 12 Sep 2022 10:15:29 +0000

When you do a web search for “how to build an API”, I found that quite a few options come up, but very few are end-to-end guides. This may be due to my poor web search abilities or because it's quite hard to build an API (getting it from design to production).

Anyone that works with APIs will tell you that it can be quite the feat, and a team of people usually manages it. So building an API yourself and getting it into production so your users can start using it can be a significant challenge.

We live in a fortunate time where there are many tools that make this task much more manageable. With something like Postman, we can create the API specification, test the API specification and ultimately test our API to ensure that it is working as expected. To build the back end of our API, we will need a host of tools such as Flask, Heroku and more, or we can opt for a low-code tool such as Linx to build and host our API.

In this post, we will look at the process of building an API and how this process can be made more efficient.

API Development process

API development is complex, there is no way around it. Typically we need to design the API, code, test, debug, code again, test some more, deploy when we are ready, and then there is the seemingly endless maintenance. A typical API development life cycle will look something like this:

Each step in the process will typically be done by another tool or resource, so it can become quite an endeavour to implement an API.

For example, we might opt to use Postman to design our API (Open API Specification) and use something like flask for our code connect firebase or some database to store or retrieve our data. We might also need to make additional REST calls to other APIs and services. To test, we can use Postman again, but debugging the code and all our connectors can become troublesome. For deployment, we might opt for Heroku, but it depends on what our API requires. For monitoring, we can either create our monitoring system or use something like Splunk. And when we need to maintain our API, we need to dive back into all of that. You get what I am trying to say. API development is complicated.

Finding a better way

I wanted a way to simplify the API development lifecycle and to develop my APIs from design to production with only a hand full of tools. Thanks to low-code tools like Linx, this is possible. I was able to create an API, from design to deployment, using only three tools:

Postman
I used postman for creating my API specification (YAML based) and for testing my API
Linx
I used Linx to build my API, implement the logic, debug it and finally to host it. Quick note, you can also try your hand at building an API with this guided tutorial.
SQL Server
SQL Server was used to store data for my API. I used the pre-created AdventureWorks2019 database and its data.

The requirement

I opted for a straightforward API that will do user record maintenance. The API has five methods:

Creating the specification

I created a straightforward API specification in Postman using YAML that matches the requirement. Postman allowed me to see what I created and visually provided additional information. Creating the API definition in Postman also benefits it already sets up that API for testing. If you choose to, you can set up the testing scripts at this stage.

Building the API

Now that I have the API definition, the code can be created. Linx allows you to import an OpenAPI 3.0 specification and will automatically generate the events for each method specified. I only needed to specify the URI and then build the logic.

Creating each event's logic goes relatively quickly once the database plugin is installed. Linx does have a learning curve like every tool, but once you understand how to work with it, the pace picks up.
I added logic and functionality to each event for the API. For example, for the GetAllUsers method, we all needed to read from the SQL database and return the results via the response body.

Testing the API

Because the API is already set up on Postman, it was quite easy to test, in real-time, how the API operates now that the logic has been implemented. The GIF below shows how I used the Linx designer to debug the REST API I have created and how I am testing it in debug mode.

With the API being debugged in Linx, it's being hosted so that I can call it to see how it will behave when I deploy. This allows me to test and get an actual result back:

Of course, we can also add test scripts in Postman to automate our testing process. These scripts will ensure that you are getting the correct response.

Deploying the API

Now that the API has been designed, developed and tested, it needs to be deployed. This can be a substantial task with traditional API development as we need to come up with a deployment strategy, figure out where we will be hosting what and make sure that monitoring and logging are taken care of and more.

My deployment was quite simple. I deployed the API from the Linx Designer directly onto the Linx Server. It took about 2 minutes for the solution to be built, pushed to the server, and ready for use. The difficulty of hosting an API is removed as the Linx server handles this. It also does monitoring and logging:

I called the GetUser method with an incorrect ID to see what would happen if an unexpected error occurred. The server logs the error and indicates with red that an error occurred:

I was able to call the API from Postman again, and the server gives an indication each time that the API is called.

Admittedly, I did not add any form of security or authentication to my API, but these settings are available in the Linx designer. Another option I tried was to generate the API Documentation in swagger format. This turned out to be quite valuable because by adding /swagger to the base URI, the documentation is available and hosted with the API itself. This makes it easy to distribute API documentation when needed.

Wrapping up

When combining Linx and Postman, we can design, create, document and host APIs. It does take a bit of getting used to, much like any tool. Because Linx uses standard programming paradigms and jargon, it is easy to pick up if you are familiar with a programming language like C#. I do feel that we save the most time when deploying and hosting an API with Linx. Monitoring and hosting are done for you, meaning that a substantial headache is taken care of. If the logging and monitoring are not granular enough, you can add your functionality to the Linx solution.

If you want to try building an API with Linx, you can try it yourself here.

Another way to provide data via web-apps (an MDS alternative)

Shantoie Vorster — Mon, 01 Aug 2022 09:12:24 +0000

Making data available via a front-end can be a headache when your (back-end) team is busy with other high-priority tasks. It can be an even greater challenge when there are requests to update specific records or to allow specific users to have access to change very specific fields. MDS is always an option, but it has some limitations and availability constraints.

Let's say we need to make data available for our users. Some users need to be able to simply view it, and others need access to update certain fields. What if our users also want some bespoke functionality that we need to add in such as validations, business rules or simply notifications that fields were updated?

This bespoke functionality can go beyond just data, for example, what if our user needs to upload a file that needs to be loaded via a back-end process?

MDS Alternatives

MDS or similar alternatives will not easily suit our needs with regard to the above-mentioned scenario. We can opt to create a bespoke single-page application (SPA) via a popular web framework, but this can be costly, time intensive and come with a high skill requirement.

Another option is to turn to a low-code alternative, like Stadium, to speed up our development. There are many other options available, but I found that Stadium offers the tools that I need to make data available, add bespoke functionality and do it all with little front-end development knowledge.

Make data available fast

The goal of any MDM-like tool is to make data available for users to interact with quickly. With Stadium, I was able to do so in a few clicks. I could also make a simple editor by combining SQL scripts (generated by Stadium with some amendments) for a row of data. This highlights that it is possible to create dynamic and rich user experiences with Stadium. The below gif illustrates what I was able to do in a few clicks:

More than just data availability

A big trap that I have personally fallen into is to force a tool like MDS to do things that it is not made for, such as kick-off processes. I am a big fan of the ‘tool for every job’ mentality. Using the right tool to create a bespoke screen that can upload files, kick off processes or act as a data dashboard is very important, but finding the right tool that can integrate all these tasks can be a challenge.

This is where I feel that Stadium does a good job. It allows you to create web applications without restricting you to a specific path. You can create a simple application that will replace or enhance your MDS functionality, you can create dashboards, create pages that will assist users with business processes or create a file management system where users can upload and maintain files to be loaded into the database via your back-end processes. Stadium also works great with any back-end, but I highly recommend trying Linx with Stadium.

Conclusion

Making data available to your users via a front-end can be a bit of a headache when you do not have the tools or skillset to do so. Stadium and tools like it can speed up front-end development, helping with those tight deadlines. Just be sure that the low-code tool you select is correct for what you are trying to accomplish.

Read more on the latest Stadium release here.

A better way to import files

Shantoie Vorster — Mon, 18 Jul 2022 12:15:24 +0000

When I was a junior developer I always wanted to find new and complex ways to do things. There was a strong urge to follow the path of most resistance because it seemed to be the ‘coolest’ way to do something. At this stage, a very wise and tired senior developer told me this: ‘In your career, you are going to write 1000 processes that import a file, at some point, you will realise that seniority is not about the coolest way to do things, but the easiest and most efficient way.’ This conversation always echos through my mind, how can I make things easier?

This is why low-code is so appealing to me, it allows users to create applications much faster than pro-code. Not to say that pro-code is bad in any way, but if I only have an hour to create a process that will import a file, test it, and get it to UAT by the close of business, low-code tools are a no-brainer. (I did an article on the viability of Low-Code for developers if you want to read more)
Creating a File Reader with low-code
In a recent post, I explored just this very rudimentary task that will be done by most, if not all, IT departments: Importing a file and making the data available elsewhere. I decided I wanted to make the data available via REST API.
By using my low-code tool of choice, Linx, I was able to develop this very simple process in less than 15 minutes. Another 30 minutes and I could add in logging, and error handling and add a layer of polish. I suppose time will have to be allocated to testing and deployment as well, but you get the idea, we are saving quite a substantial amount of time.

Understanding what you are getting into
Many like to point out that low-code has many pitfalls and limitations, and that it will never truly replace pro-code or conventional coding tools. I agree with this sentiment, it's hard to think of a world without pro code. There are too many bespoke systems and specific problems that need solving. That said, we can utilize these tools to optimize some of the more ‘ordinary’ development tasks such as loading a file.
That said, we need to be very careful in picking a low code tool, it needs to fulfil our needs and we need to ensure that it does not lock us into its own environment when we need to have freedom.
What I am trying to say is that you need to choose a platform that will meet your (and your organisational) requirements and needs.

The right tool for the job
When choosing a low-code tool, make sure that you are choosing the right tool for the job. Ensure that you are not choosing a front-end tool to do back-end tasks. It might work, but why do we want to use a spoon when we need to use a ladle. For back-end applications, I like to use Linx. Not just because I am employed by the company that makes Linx, but also because I have seen it in action. I have seen it used to create intricate back-end systems in massive enterprise environments.

If you are interested in reading more about how I created the above-mentioned process so quickly, feel free to read the post here. There is also a YouTube video that has all the steps.