(Title photo: Ophelia photos - Creative Commons)
I stumbled upon this twitter thread and corresponding Hacker News comments today: 90% of software engineering done today is integrating poorly documented APIs | Hacker News. This take doesn’t resonate with me at all. I actually think this attitude is toxic.
Nobody says: "90% of building a bridge is making shapes out of cement of varying quality". It is well understood that civil engineering is about which cement to use, designing a sound architecture, computing structural stresses, organizing a large workforce, establishing a project plan. It is hard, challenging and rewarding. Cement will always be of poor quality, because it is not unobtainium that can magically be molded into the specified bridge.
The same goes for software engineering and our own digital cement: APIs. In fact, combining cruddy APIs to solve problems is so challenging and fulfilling that I can’t imagine doing anything else.
Two types of software engineering
These days, I am in the business of writing a CRUD e-commerce application, using PHP, wordpress and mysql of all things. In the past 20 years I've worked on anything from bare-metal embedded systems to quantum cryptography ethernet bridges to search engines and "not google-scale" data engineering. This has taught me that there are two very separate kinds of software engineering.
Software engineering as copying bytes
One is what I call "copying bytes". You get input bytes, relevant to the business in some way. As a software engineer, your job is copy these input bytes to their specified destinations.
For example, you receive an order from the customer (line items + shipping address), and you have to forward these line items to the fulfillment department, recording a few log entries along the way. You might calculate some tax or apply a coupon along the way, slightly modifying those bytes, but roughly, you are just sending them along the way god designed them.
Software engineering as “inventing” bytes
The other kind of software engineering is what I call "inventing bytes", or "transforming bytes in really complicated ways". This is the realm of funky algorithms, data science, mathematics, complicated stuff. For example: taking an input 3D model and outputting some CAM instructions or searching for a string in complicated index structures. Often, the hard thinking is done by people at universities or research labs.
Writing code is usually the tiniest part of the undertaking. One could wonder if these “byte inventors” are software engineers at all. Maybe we should call them "3D model transformation engineers", or "search index data structure engineers". Often, their code needs to be rewritten by someone who knows how to indent files (or say, port it from hacky matlab to java).
The APIs for inventing bytes are weird. Frankly, I don't really dabble in invented bytes, so I can't say too much as to the documentation of these APIs.
You might actually copy bytes your entire career
If you are not in the business of thinking clever ideas or making pretty things like video games, but in the business of making money, you probably fall into the "copying bytes" category of software engineering.
Which bytes to copy, when, where, what log entries to add along the way, how to scale the copying, how to ensure the copied bytes are transactionally aligned with other copied bytes, that stuff is complicated. But it's turtles all the way down, one API to copy bytes building upon another. Ultimately you hit the hardware, and boy are the hardware APIs to copy bytes utterly nonsense.
You don’t need to write interesting code to be an engineer
You might think it's inane: you learned how to balance trees, how to schedule processes, and how to do some differential curve geometry; how to write compilers, how design database engines and how to implement distributed consensus; why don't you get to write all those beautiful algorithms as part of your day job? Thing is, clever people have already done so, and often they made a business out of it. Now you can pay them for their clever thinking and their diligent work, hoping that they were indeed more clever than you, and go back to calling APIs to copy whatever bytes have just been computed.
People want to transfer data for all kind of different reasons, in different ways, with different constraints. That means that even the best written API documentation will have to be generic enough to cover the use case it was built for, and with some chance, what people are using it for. It will rarely exactly fit your use case. The documentation will thus not be perfect, and might not cover what you are looking for at all.
I actually think API documentation at large is miles better than it ever was. All the APIs I had to deal with lately come with copypasta for 15 languages, a snappy searchable index, a plethora of tutorials and articles and videos, an active github. This is miles better than having to guess how much whitespace padding a file placed on a PC-NFSD spool needs for the COBOL parser to be happy.
Solving problems by combining crappy APIs is the most fulfilling thing I do
But here's why I find my job copying bytes with subpar APIs singularly fulfilling:
Copying bytes is not easy, especially in today's distributed world. Figuring out how to do it reliably, securely and at scale requires a lot of thought. Figuring out which data to transfer to whom needs business understanding, knowledge of people and systems thinking. Building and maintaining a solution requires product and project planning, managing teams, dealing with changing requirements. It requires keeping a codebase healthy, mentoring other engineers, aligning with stakeholders, dealing with legacy code.
The APIs I ultimately use are the least of my problems. Some are nice, some are terrible, some are really problematic, but they don't have much to do with the "engineering" part of my job. The engineering part is all about drawing pictures with boxes and arrows. The boxes are other people's systems, the arrows are other people's APIs. What arrows I draw and where they point to and what the little bubbles along them contain, and which boxes I chose to use, that's what I do as an engineer.
A concrete example: checking out an order
As a concrete example of a hard problem, fully implemented by glueing together shoddy APIs: fulfilling an order on an e-commerce store.
- seamlessly show payment process and checkout status to the customer using a slick UX in their browser (APIs: some shoddy REST, some janky web socket or whatever, some hacked react and some dubious CSS. Ultimately just shoveling the status code from the payment processor to the user's browser, potentially adding some curly braces and escaping a string or two.)
- log the transaction and provide metrics and analytics to business and ops (some inane pixel data, some putrid tracking JS, some shoddy SQL, maybe some janky JSON going to a SQS queue)
- process the payment information, order information and display it in the customers account section / backend CRM / fedex (I'm not even going to list how subpar the APIs are here)
- transform the shipping address into a proper shipping label with barcode (have fun with zebra and ZPL)
- dispatches the line items to the required fulfillment centers, provides picking information to the warehouse workers, with tracking, metrics, inventory handling, ...
Designing something like this so that it works correctly, efficiently, with visibility, reliability, scalability, and fulfills business needs, and sparks joy for every human involved – now that's actual engineering.
Do you think glueing together APIs is engineering?
Do you think software engineering today is boring? Do you hate glueing APIs together? What do you think constitutes "real" software engineering? What is the ideal day-to-day of a software engineer? Please be sure to comment below!
Top comments (12)
I love this! I love working with data integration, and connecting different APIs and databases, and I actually think there's incredible opportunity to be artful and creative when you're "moving bits", especially once you start to get into "how do I make this configurable for future use cases?" or "what is I need to move/process a large volume in a short time on undersized hardware?"
I also want to throw out that I do still encounter a ton of "poorly documented" APIs, but they can be almost as fun as they are frustrating. I can really enjoy "black box" developing an integration sometimes, where all you can do is throw input at it and see what the get back. : )
Definitely, and you can often see why an API is "poorly documented". It takes more effort to document something well that to design it well, and as a developer you often need to take into account things that are very hard to document (performance, scaling) anyway.
What I like the most is when I can get quickly started with a starter project and easy test tokens. This often means that it will also be easy to set up tests and catch edge cases later on.
Integrating APIs is pretty fun for me, even if the APIs are not well documented or flawed. It is the next step in programming, which is the integration of different people's work with each other to create a new and better product.
Programming is a mixture between science and art: we use logic in an imaginative way to create new things.
I've always viewed programming as more art than science, and derive most pleasure from twisting it in unusual ways, or throwing together useless "toys" mainly for my own enjoyment. Sometimes these exercises yield something useful (new libraries, tools, frameworks etc.), but that's just an added bonus.
Unfortunately, this kind of intellectual play doesn't pay the bills... developmental API "plumbing" does. To me, it's mostly boring, but occasionally throws up interesting challenges. The main reason I stick with full stack is to keep it as interesting and varied as possible.
What parts of your “toys” do you find fulfilling, and why do you think this kind of software is not useful in a professional setting?
I must admit that I do enjoy software architecture a lot, and so choosing which API, when, for what reason, and how to introduce it is what really gets my creative mind going. Do you chose an event driven architecture or stick with plain old REST calls and a DB transaction? What if you had a big old codebase using node.js and mongodb? What if this thing needs to ship tomorrow?
The creativity is moved from the actual code to the design of the system, the compromises, the realities of running a system production (very worth it using a shitty API if the tooling is better, etc…)
Moving bytes IS real engineering! Would love to know what you think of pipedream.com
I don't really have an opinion. I've had success in some cases with "click here to pipe data" tools, and also absolutely terrible experiences where doing it manually from the get go would have helped tremendously.
Did you use it? Any comments?
I am a Pipedream founder so am quite biased! But Pipedream is built specifically for developers. Check out our GitHub repo.
We do provide a UI for building integrations, but you can write any Node.js, Python, Go or Bash code within a workflow. I frequently write workflows that are a mix of pre-built actions (I don't want to write the code to send a Slack message every time) and custom code steps where I implement the logic I need for the specific automation I'm building.
Give it a try and let me know what you think! We deeply value feedback from developers. Feel free to join our community, too, if you want to chat with us via Slack / Discourse.
I think the worst API I had to use was related to that PC-NFSD COBOL thing I was referring to. In fact, the form had a “Command” field, and there were about 600 commands all labeled with seemingly random strings like CFUATUU or DOFTTNI. I had 600 pages of documentation, one per command. No table of contents, no index, no page numbers. Of course nothing was in alphabetical order either. Absolute bliss.
I see it as playing with LEGO, you join some pieces together to create something, plus you can also create your own pieces if needed, which adds some fun to it.
Feel it the same way, mostly. Glue things together don't have to be boring.
API's bring any app to life.