Handling Out of Process API requests

When we work with APIs, we typically work with a request-response pattern or a streaming pattern. However, another API pattern calls an out of process function initiating a long transaction.

I worked with a service that acted like a request-response API because it returned a JSON doc with metadata about the request. However, I would have to wait for the request to be fulfilled. For example, I would send a POST request with a JSON doc like this:

{ query: {
"queryFilter": {
        "startDate": "2021-10-01",
        "daysOfWeek": {
            "U": false,
            "M": true,
            "T": true,
            "W": true,
            "R": true,
            "F": true,
            "S": false
        },
        "endDate": "2021-12-31",
        "timeIntervals": [
            "0500-0900",
            "1700-2100"
        ]
    },
    "userEmail": "[e-mail]",
    "userId":88AB
}

The API would return metadata about the request.

{
    "requestId": 93434555,
    "status": "Pending",
    "userEmail": "[e-mail]",
    "userId": 88AB,
    "whenSubmitted": "2022-01-20T14:49Z"
}

At this point, I had two options. I could wait for a notification email with the download URL for the requested data or poll an endpoint to check the status. To poll the status endpoint, I would send a GET request with my userId and requestId, e.g.

GET https://longtransaction.com/88AB/93434555

If the request were completed, the service would return the download URL in the response.

{
    "requestId": 93434555,
    "status": "Completed Successfully",
    "userEmail": "[e-mail]",
    "userId": 88AB,
    "whenSubmitted": "2022-01-20T14:49Z",
    "downloadUrl": "https://longtransaction.com/88AB/93434555/data.csv.gz",
    "urlExpirationDate": "2022-02-19T16:30Z"
}

I could then parse the response for the download URL.

Automating the process

Because I had many queries, I wrote a script to send the requests and a script to download the documents. Since the final output wasn't time-critical, I opted to wait for the email notifications to run the download script.

Here's an example requests script.

import json
import csv
import requests

durations = [{"startDate": "2019-01-01","endDate": "2019-03-31"}, ..., {"startDate": "2021-09-01","endDate": "2021-12-31"}]

query = { query: {
"queryFilter": {
        "startDate": "",
        "daysOfWeek": {
            "U": false,
            "M": true,
            "T": true,
            "W": true,
            "R": true,
            "F": true,
            "S": false
        },
        "endDate": "",
        "timeIntervals": [
            "0500-0900",
            "1700-2100"
        ]
    },
    "userEmail": "[e-mail]",
    "userId":88AB
}

writer = csv.writer("request_ids.csv", w)

for d in durations:
     start = d["startDate"]
     end = d["endDate"]
     new_query = query
     new_query['queryFilter']['startDate']= start
     new_query['queryFilter']['endDate']= end
     response = requests.post(url, json=new_query)
     r = response.json()
     request_id = r["requestId"]
     writer.writerow(request_id)

request_ids.close()

The script sends the requests to the service and writes the requestsIds to a CSV file. You can then use the requestIds to drive the download script.

import json
import requests
import csv

base_url = "https://longtransaction.com/88AB/"

def download_file(url):
     local_filename = url.split('/')[-1]
     with requests.get(url, stream=True) as r:
         r.raise_for_status()
         with open(local_filename, 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192): 
                f.write(chunk)
     return local_filename

def get_download_url(request_id):
    request_id = str(id)
    response = requests.get(base_url+request_id)
    doc = response.json()
    download_url = doc["outputUrl"]
    return download_url

with open("request_ids.csv") as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    for row in csv_reader:
        id = str(row[0])
        download_url= get_download_url(id)
        download = download_file(download_url)

Summary

While this is far from an elegant solution, it does the job in the spirit of "get things done." REST and the request-response pattern have been implemented for many services; this was an interesting variation that adopted the request-response variation for a long transaction.