My take on transforming data

When I started working in development (2011) life was simpler.

Sure, we were just 3 guys trying to build websites and manage their data, still in university, barely knew what we were doing, but where we were there wasn't much competition at the time, so we were fine.

Then the change, no more custom tailored websites, let's build our product.
And things were still easy, R&D was fun, things were raw but working.

And then, the customers arrived.

And with them, all their legacy systems, all their customers' systems, and this weird idea that programs should "talk" to each other, even when the target system is in COBOL.

So we were happy the first years, every new integration was paid work, and it was kinda easy since we did have the time to create a new module for each of them.

Things are less easy today, when the numbers have risen, and the needs are both internal and external, meaning not all integrations are actually profitable directly.

So my journey began to make reusable modules, standardized APIs, FTP that use the same format as APIs, etc.

On my own, I had an idea:
What if I could have a tool that can handle converting all these formats in a standardized way.
Someting a properly trained IT person or delivery expert could use on their own.

At first, I tried JSON schemas and started building a UI for them.
The result was actually slower to use than writing code, which for me is the issue with a lot of existing graphical mapping systems (I know, I probably use them wrong, but still, that was my feeling)

I scrapped that project after a while, I wasn't liking the result, and it didn't excite me enough to put some hard work into that, as I was working on it in my free time.

Finally, when the AI era started, I had more tools to do more work in less time.
I'm very strict on how I use AI tools, I define what and how, I review code, I do lots of parts on my own or brainstorming with a model, and I'm liking the result.

So this opened new possibilities, and I started a new idea:
Lots of my support/delivery colleagues actually know how to run rudimentary database queries, and they kinda understand what they mean.

So I decided to try using queries.

The results were and still are very promising, I've made it into a command line interface, an API server with automatic OpenApi/swagger, and a js/ts library.

I'm converting CSV, XML, JSON and even some bulky stuff like edifact and fixed-length plain text.
Under the hood queries are compiled into javascript, which was my first choice for simplicity.

Just to give you a very basic example:

from xml to json
transform
set id=CID
set number=number(progressive)

this would transform and XML to JSON with id and number fields, with number casted to number.

I've made a playground of course so it's easy to compose a query based on data:

While I use it I keep adding functions and adapters when I need them.

I think this might be an interesting solution for someone else as well.

So, I wanted to share this and hopefully start some interesting discussions.
And of course, if you think this project looks interesting, let me know!