DEV Community: Marty Pitt

Converting SOAP APIs to REST, without code 🧼🧰🪄

Marty Pitt — Tue, 05 Sep 2023 17:21:53 +0000

TLDR:

Soap is an outdated API technology, that takes LOTS of boilerplate. Sadly, sometimes you don't have a choice.
We can quickly wrap a SOAP API into REST without code, making it much easier to work with in our web apps
We can also combine multiple SOAP APIs into a single, purpose-built REST API.

If you've ever had to work with SOAP APIs, you know how unpleasant they can be.

They're cumbersome to call, have heavy amounts of code-gen required, and if you're not using Java or C#... you're already kinda screwed.

We're gonna to take an old SOAP API (which, lets face it, aren't fun to play with), and repackage it - combining multiple calls into a single REST API.

Instead of writing piles of boilerplate plumbing code, we'll automate all the integration work with Orbital (an open-source data integration platform) and Taxi to annotate the SOAP APIs.

Soap...the stuff of nightmares.

Building a Country Info Application

We're working with a public webservice at Oorsprong.org that provides Country Information.

It's a powerful API that exposes lots of information about countries, but each piece of information needs a separate call to a SOAP API.

Each API code takes an ISO Code (eg., GBP / USA / NZ), and returns a small slice of information about a country. (Flag, Currency, Capital City).

We'd like to grab the data of a few of these services, but without having to write a bunch of SOAP code.

Taxi - A quick intro

We'll be using Taxi to help modernize and slice-n-dice this API. If you haven't heard of Taxi before - you're not alone.

Taxi is a relatively new entrant in the API space. It's an open source metadata language for describing APIs.

It's goal is to let developers add simple, (but type-safe) tags into their APIs, so software can understand how different APIs relate to one another.

It's a simple way of saying "This field is the same as that field", while keeping systems decoupled.

And the same tags we add into our APIs, we can use to query for data using TaxiQL - Taxi's query language - which replaces all the integration plubming code we'd normally have to write.

Getting going

Let's go a little deeper, and break that down into three steps:

Adding Taxi metadata to the SOAP WSDL (here's the relevant docs)
Using Taxi queries to build the responses we want to return
Publishing those queries as REST APIs, with LIVE RELOAD FTW!

The code for this blog is available on Github

If you find this tutorial useful, why not give us a star on Github

Adding metadata to SOAP for fun & profit.

Taxi metadata allows us to sprinkle a few additional tags into our WSDL which let mark data attributes that are the same.

Later we can use these tags to automate all the SOAP interaction, so we don't have to write any SOAP code ourselves.

For example, our CountryInfo wsdl has a section dedicated to request / response payloads.


<!-- Snippet from our WSDL - Request / Response messages for getting a Country Name -->
<xs:element name="CountryName">
    <xs:complexType>
       <xs:sequence>
          <xs:element name="sCountryISOCode"
                 type="xs:string" 
+                taxi:type="com.demo.IsoCountryCode"/>
       </xs:sequence>
    </xs:complexType>
    </xs:element>
    <xs:element name="CountryNameResponse">
    <xs:complexType>
       <xs:sequence>
         <xs:element name="CountryNameResult" 
             type="xs:string" 
+            taxi:type="com.demo.CountryName"/>
       </xs:sequence>
    </xs:complexType>
</xs:element>

You can summarize this by saying:

I can send a CountryName request containing a IsoCountryCode, and I'll get back a message containing a CountryName

Elsewhere, we can also use a IsoCountryCode to get more information, like a CountryFlagUrl:


<xs:element name="CountryFlag">
  <xs:complexType>
     <xs:sequence>
        <xs:element name="sCountryISOCode"
            type="xs:string"
+           taxi:type="com.demo.IsoCountryCode"/>
     </xs:sequence>
  </xs:complexType>
</xs:element>
<xs:element name="CountryFlagResponse">
  <xs:complexType>
     <xs:sequence>
        <xs:element name="CountryFlagResult"
                type="xs:string"
+               taxi:type="com.demo.CountryFlagUrl"/>
     </xs:sequence>
  </xs:complexType>
</xs:element>

That's a whole lotta XML, for not much info - which is one of the problems with SOAP - it's just too darn noisy.

Using Taxi queries to get the data we want

Now that our WSDL is annotated with Taxi, we can use TaxiQL to do all the heavy lifting for us.

Rather than generating a bunch of Java classes from the WSDL, we can simply write a taxi query:

given { iso: IsoCountryCode = "NZ"}
find { 
    // Each field comes from a different SOAP service.
    // But taxi keeps this nice and succinct, composing the services
    // together we need on demand
    name: CountryName
    flag: CountryFlagUrl
    currency: CurrencyName
 }

We can send that query to Orbital, and Orbital calls all the SOAP services we need on our behalf, giving us back the data we're looking for:

{
   "name": "New Zealand",
   "flag": "http://www.oorsprong.org/WebSamples.CountryInfo/Flags/New_Zealand.jpg",
   "currency": "New Zealand Dollars",
}

Orbital's profiler shows us exactly what happened:

We can see:

Orbital called 3 different SOAP endpoints
It dealt with all the SOAP Request/Response malarchy
It plucked the data we want, and gave it back.

And we can drill into the details of each call too:

Instant REST API

Now that we're happy with the content of our payload, we can quickly turn it into a REST API.

Simply by checking a query into our project's git repo, and adding an annotation, Orbital gives us a fully working REST API:


@HttpOperation(url = '/api/q/countrydata/{countryCode}', method = 'GET')
query countrydata(@PathVariable("countryCode") countryCode :  IsoCountryCode) {
   given { countryCode }
   find {
       name : CountryName
       flag: CountryFlagUrl
       currency: CurrencyName
       capital: CapitalCityName
   }
}

By adding a @HttpOperation annotation to our query, and saving it our local Taxi project, Orbital instantly exposes a REST API for us.
Rather than hard-coding our query to information about New Zealand, we've put the country code as a part of the path.

@HttpOperation(url = '/api/q/countrydata/{countryCode}', method = 'GET')

So, if we call:

curl http://localhost:9022/api/q/countrydata/NZ

Gives us:


[
  {
    "name": "New Zealand",
    "flag": "http://www.oorsprong.org/WebSamples.CountryInfo/Flags/New_Zealand.jpg",
    "currency": "New Zealand Dollars",
    "capital": "Wellington"
  }
]

Live reload for the win

Orbital is live reloading this query, so while we're working locally and making changes to our query, changes are instantly deployed!

Once deployed, making changes is as simple as pushing to a git repository.

Check it out:

Adding the CapitalCityName into our request actually added a whole new SOAP request and integration into our query, which Orbital instantly reloaded behind the scenes.

Conclusion

In this blog, we've quickly repackaged a SOAP API into the exact API we wanted, without having to deal with any of the nastiness that
normally comes from dealing with SOAP requests.

In the process, we've touched on a few of Orbital's handy features:

Using Taxi metadata in SOAP WSDLs
Composing together multiple SOAP calls
Instant REST APIs with live reload
Viewing profiling data

The full code to run this demo locally is available on Github

This is only scratching the surface of what we can do. Orbital can also stitch this data together with other APIs (such as REST / gRPC), call serverless functions, query our Databases, or Kafka queues.

Give us star!

If you found this useful, please give our Github repo a star!

Semantic Metadata 101: Towards Automated API Orchestration

Marty Pitt — Tue, 23 May 2023 11:29:09 +0000

This post is a lightweight introduction to the concept of Semantic Metadata, and how it makes our enterprise services
automatically composable.

What is Semantic Metadata?

Semantic Metadata is a way of defining a contract around the meaning of data. It lets teams create terms and definitions they agree on, and use those terms to better describe their APIs and analytics, and make software interoperable.

Teams can use Semantic metadata to define formal definitions for fields in APIs:

"This is what a first name means",
"This is what a company name means"

Analytical platforms such as cube.dev tend to expand this to include shared definitions of aggregates.

"This is what a customer is"
"This is what we mean by 'Active Customers'", etc.

The core idea is that Semantic Metadata is a way for teams to create a shared understanding of what data means, independent of a single specific system.

Building a Taxonomy

Semantic metadata is really just a collection of terms that describe our business.

When grouped together, this is called a Taxonomy.

Taxi is a language-agnostic tool for building semantic taxonomies:

type AccountNumber inherits Int
type CreditScore inherits Decimal
type FirstName inherits String
// etc...

Semantic Metadata is designed to be shared across multiple teams. So, just like with API technologies like OpenAPI, Protobuf, etc - it's best to have Semantic Metadata designed in a platform agnostic language.

Generators can then generate bindings / SDKs / tools as required in whichever technology consuming teams are working with.

Embedding semantic metadata

On it's own, semantic metadata isn't very helpful - it's just a set of tags and definitions.

However, embedded in API specs, it becomes much more powerful.

Here's an example in OpenAPI:

openapi: 3.0.1
info:
  title: ReviewsApi
  version: 1.0.0
paths:
  https://reviews/{id}:
    get:
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          x-taxi-type:
            name: FilmId  # <-- Semantic metadata
      responses:
        "200":
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/FilmReview'
components:
  schemas:
    FilmReview:
      type: object
      properties:
        id:
          type: string
          x-taxi-type:
            name: ReviewId  # <-- Semantic metadata
        filmId:
          type: string
          x-taxi-type:
            name: FilmId  # <-- Semantic metadata
        score:
          type: integer
          format: int32
          x-taxi-type:
            name: ReviewScore # <-- Semantic metadata

As teams enrich their exisitng API specs with semantic metadata, tooling can start inferring relationships between APIs and data sources.

Let's look at a (very) simplifed API for an insurance company that provides quotes:

This takes a request payload with two inputs:

{
   "noClaimsBonus" : 0.25,
   "creditScore"  : "AAA"
}

Semantically, this can be modelled as:

type NoClaimsBonus inherits Decimal
type CreditScore inherits String

model QuoteRequest {
  noClaimsBonus : NoClaimsBonus
  creditScore : CreditScore
}

This has added a small benefit of improved clarity in the docs.

However, the real payoff is in when we're trying to get out services to work together...

The Payoff: Automating Interoperability

This is where semantic metadata really starts to shine.

With Semantic Metadata embedded in our API specs, we can start to infer relationships between APIs and data.

Looking at our previous example without field names, we required two inputs - a NoClaimsBonus and a CreditScore.

If we don't have those pieces of information, we need to look them up. Which means we need to look for services that expose this data.

As our APIs are enriched with Semantic Metadata, tooling can automatically infer relationships between systems.

So, in our Insurance Quote example, while it's unlikely we have either a NoClaimsBonus or CreditScore available from our UI, we might have something else - like a UserName or UserId.

Semantic Metadata lets us use tooling to automate the integration, linking from the Things We Know (UserName) to the Things We Want To Find Out (a Quote).

We'll look into this in more detail in the next post.

A few tips & tricks...

Semantic metadata is a simple concept, and is super easy to get going with. Here are a few tips & tricks to take along your journey

Stay small & nimble when using distributed ownership

Semantic Metadata is intended for wide collaboration, which can be tricky.

Where individual teams are responsible for defining API definitions, semantic metadata has distributed ownership.

If the idea of Distributed Ownership is giving you sweaty palms and flashbacks to Design-by-committee meetings
when your org tried to implement Canonical Domain Models, you might already be rolling your eyes.

Therefore, it's recommended that Semantic Metadata is defined on Scalar terms only - single noun-like ideas that describe exactly one idea. eg:

FirstName
LastName
DateOfBirth
PostCode

These are relatively un-contentious to define, and - unlike domain models (which evolve as systems mature) - semantics don't really change.

GitOps all the things

Semantics are a bridge between business language and software. There's no shortage of First-generation data catalog platforms that will sell you a glorified Wiki for defining your semantic terms.

Instead, consider using Open Source tooling, that aligns with GitOps.

You get peer review, audit trails, and automated workflows through all your existing Git tooling, without having to invest in expensive enterprise tooling.

Staying technology agnostic

Organisations these days aren't "Java Shops" or ".NET Shops" anymore - they're polyglot, with tech teams
choosing the tech stack that best fits the task and team.

Likewise, most API and schema technologies (eg., OpenAPI, Protobuf, etc) are language agnostic, with bindings / generators
allowing them to be consumed in whichever technology consuming teams are working with.

Taxi is an example of a Semantic Metadata language, which is also platform-agnostic.

Summary

This has been a high level introduction into some of the ideas behind Semantic Metadata.

To go deeper, take a read into Why we created Taxi, and Using Semantic Metadata for easier Integration.

In the next article, we'll take a look at building a simple application using Semantic Metadata to automate the orchestration.

Also, if you liked this article, consider giving Taxi a star