The Postman Rides OCaml - Part 2

Using the Twitter API with OCaml and Postman.io

The Twitter docs for this are here. Let me say that the Twitter docs are all over the place. You will find dead links, unclear explanations and circular doc references and links. This a good part of the reason why I am writing this up, so I can refer back to it.

Follow the steps at the link to get a developer account. The postman link on that page is dead. This is the link you need: tutorials/postman-getting-started.

The first thing you need to do is get a bearer token. The link for that is bearer-tokens. I will reproduce the process here.

This is straight from the docs:

How to generate a Bearer Token

Copy the following cURL request into your command line after making changes to the following consumer API keys previously obtained from your Twitter app. Note that the consumer API keys used on this page have been disabled and will not work for real requests.
API key <API key> e.g.xvz1evFS4wEEPTGEFPHBog
API secret key <API secret key> e.g. L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg

curl -u '<API key>:<API secret key>' \
 --data 'grant_type=client_credentials' \
 'https://api.twitter.com/oauth2/token'

Here's an example of how the curl request should look with your API keys entered:

curl -u 'xvz1evFS4wEEPTGEFPHBog:L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg' \
 --data 'grant_type=client_credentials' \
 'https://api.twitter.com/oauth2/token'

Here is what the response would look like. Note that this is a decommissioned Bearer Token:

{"token_type":"bearer",>"access_token":"aRjeMU9YWuP5aS8trretKALXhcjBpQ7xY0TJBwq8Dhl1sw3GkRX5VxeyzfqDtqiwAueS9rHbA5PNzr292iE6D1DMwbqLXErMVuhOOzREtZ9eL1FSFTBw6PCVnZmWUtQl3vfiWMaGOdg3gmkQjPLBWqabOcgsufJFdUID3VWrGUo4mT9Bscuq9WZ4Ey6itIMM5InH3mjb"}

Our Bearer Token used to authenticate to resources with OAuth 2.0 would be:

aRjeMU9YWuP5aS8trretKALXhcjBpQ7xY0TJBwq8Dhl1sw3GkRX5VxeyzfqDtqiwAueS9rHbA5PNzr292iE6D1DMwbqLXErMVuhOOzREtZ9eL1FSFTBw6PCVnZmWUtQl3vfiWMaGOdg3gmkQjPLBWqabOcgsufJFdUID3VWrGUo4mT9Bscuq9WZ4Ey6itIMM5InH3mjb

Once you have your token you want to open up postman (download it at postman.io if you dont have it.).

Once you have all that set up, go ahead and click the blue send button. My json response was:

{
    "data": [
        {
            "created_at": "2013-12-14T04:35:55.000Z",
            "description": "The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API.\n\nNeed help? Visit https://t.co/DVDf7qKyS9",
            "entities": {
                "url": {
                    "urls": [
                        {
                            "start": 0,
                            "end": 23,
                            "url": "https://t.co/3ZX3TNiZCY",
                            "expanded_url": "https://developer.twitter.com/en/community",
                            "display_url": "developer.twitter.com/en/community"
                        }
                    ]
                },
                "description": {
                    "urls": [
                        {
                            "start": 129,
                            "end": 152,
                            "url": "https://t.co/DVDf7qKyS9",
                            "expanded_url": "http://twittercommunity.com",
                            "display_url": "twittercommunity.com"
                        }
                    ],
                    "hashtags": [
                        {
                            "start": 23,
                            "end": 30,
                            "tag": "DevRel"
                        }
                    ]
                }
            },
            "id": "2244994945",
            "location": "127.0.0.1",
            "most_recent_tweet_id": "1214947767250030592",
            "name": "Twitter Dev",
            "pinned_tweet_id": "1214281000932593667",
            "profile_image_url": "https://pbs.twimg.com/profile_images/880136122604507136/xHrnqf1T_normal.jpg",
            "protected": false,
            "stats": {
                "followers_count": 506515,
                "following_count": 1707,
                "tweet_count": 3498,
                "listed_count": 1641
            },
            "url": "https://t.co/3ZX3TNiZCY",
            "username": "TwitterDev",
            "verified": true,
            "format": "detailed"
        }
    ]
}

Running this in OCaml

Postman has an amazing code-generation project that gets you the code for a script you ran via postman. Check it out here: postmanlabs/postman-code-generators. This list of available languages are

Look for the code button like in the bottom left of this image:

Click the button and in the popup screen scroll down to OCaml. You should see something like this:

{::options parse_block_html="false" /}

And #FTW ⁦@getpostman⁩ has #OCaml code generation. ⁦@sgrove⁩ eat your heart out. If you want to convert it to #Reason… pic.twitter.com/4KUzqnb5xK

— mandalarian (@_idkjs) January 23, 2020

Click the copy icon in the top left of the popup to copy the snippet. Then create a file for it, touch tweets_and_users.ml, and paste it in. This is the snippet I got, with scrambled variables of course. Yours will have your own token in it:

open Lwt
open Cohttp
open Cohttp_lwt_unix

let reqBody =
  let uri = Uri.of_string "https://api.twitter.com/labs/1/users?usernames=TwitterDev&format=detailed" in
  let headers = Header.init ()
    |> fun h -> Header.add h "Authorization" "Bearer aRjeMU9YWuP5aS8trretKALXhcjBpQ7xY0TJBwq8Dhl1sw3GkRX5VxeyzfqDtqiwAueS9rHbA5PNzr292iE6D1DMwbqLXErMVuhOOzREtZ9eL1FSFTBw6PCVnZmWUtQl3vfiWMaGOdg3gmkQjPLBWqabOcgsufJFdUID3VWrGUo4mT9Bscuq9WZ4Ey6itIMM5InH3mjb"
  in
  Client.call ~headers `GET uri >>= fun (resp, body) ->
  body |> Cohttp_lwt.Body.to_string >|= fun body -> body

let () =
  let respBody = Lwt_main.run reqBody in
  print_endline (respBody)

Then run touch dune and copy over our Makefile with cp ../hello-postman/Makefile .

touch dune
cp ../hello-postman/Makefile .
tree
➜  tweets-and-users [master*]tree
.
├── Makefile
├── README.md
├── code-button.png
├── scrambler.js
└── tweets_and_users.ml

0 directories, 5 files

In the snippet we see that the project requires Lwt, Cohttp and Cohttp_lwt_unix so let configure out dune file accordingly.

(executable
 (name tweets-and-users)
 (libraries cohttp lwt cohttp-lwt-unix))

Now if you remember, we run make to build our the program.

➜  tweets-and-users [master*]make
dune build
File "tweets_and_users.ml", line 1:
Error (warning 24): bad source file name: "Dune__exe__Tweets-and-users" is not a valid module name.
File "tweets_and_users.ml", line 2, characters 5-11:
2 | open Cohttp
         ^^^^^^
Error: Unbound module Cohttp
make: *** [default] Error 1

What happened here? Running make can only install the dependencies that we have installed with opam. We haven't yet installed the deps we need here so lets do that by running opam install cohttp lwt cohttp-lwt-unix.

Then lets try running make again.

➜  tweets-and-users [master*]make
dune build
File "tweets_and_users.ml", line 14, characters 41-45:
14 |   Client.call ~headers `GET uri >>= fun (resp, body) ->
                                              ^^^^
Error (warning 27): unused variable resp.
make: *** [default] Error 1

So the code generated by postman is not perfect. Looks like the compiler is requiring to use variables or ignore them. We can ignore resp by changing it to _resp to indicate we are ignoring the variable. Being a good community citizen, I made pull request on the repo to fix this so you may not run into it.

Run make again and it compiles!:

➜  tweets-and-users [master*]make
dune build
➜  tweets-and-users [master*]

Now we finally get to run dune exec ./tweets_and_users.exe`:

sh
➜ tweets-and-users [master*]dune exec ./tweets_and_users.exe
Fatal error: exception (Failure "No SSL or TLS support compiled into Conduit")


Wow. What in the world does that mean? I searched the web but came up with nothing helpful. I then searched No SSL or TLS support compiled into Conduit in the discord #ocaml channel and found this post. The response from the always helpful @_anmonteiro says you need to add lwt_ssl or tls if you want ssl support. Let try lwt_ssl first.

`sh
opam install lwt_ssl

output

➜ tweets-and-users [master*]opam install lwt_ssl

The following actions will be performed:
∗ install conf-pkg-config 1.1 [required by conf-libssl]
∗ install conf-libssl 1 [required by ssl]
∗ install ssl 0.5.9 [required by lwt_ssl]
∗ install lwt_ssl 1.1.3
↻ recompile conduit-lwt-unix 2.0.2 [uses lwt_ssl]
↻ recompile cohttp-lwt-unix 2.5.0 [uses conduit-lwt-unix]
===== ∗ 4 ↻ 2 =====
Do you want to continue? [Y/n] y

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
[cohttp-lwt-unix.2.5.0] found in cache
[conduit-lwt-unix.2.0.2] found in cache
[lwt_ssl.1.1.3] downloaded from cache at https://opam.ocaml.org/cache
[ssl.0.5.9] downloaded from cache at https://opam.ocaml.org/cache

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><> 🐫
⊘ removed cohttp-lwt-unix.2.5.0
⊘ removed conduit-lwt-unix.2.0.2
∗ installed conf-pkg-config.1.1
∗ installed conf-libssl.1
∗ installed ssl.0.5.9
∗ installed lwt_ssl.1.1.3
∗ installed conduit-lwt-unix.2.0.2
∗ installed cohttp-lwt-unix.2.5.0
Done.
➜ tweets-and-users [master*]make
dune build
`

Let try running it again:

sh
➜ dune exec ./tweets_and_users.exe


returns:

json
{"data":[{"created_at":"2013-12-14T04:35:55.000Z","description":"The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API.\n\nNeed help? Visit https://t.co/DVDf7qKyS9","entities":{"url":{"urls":[{"start":0,"end":23,"url":"https://t.co/3ZX3TNiZCY","expanded_url":"https://developer.twitter.com/en/community","display_url":"developer.twitter.com/en/community"}]},"description":{"urls":[{"start":129,"end":152,"url":"https://t.co/DVDf7qKyS9","expanded_url":"http://twittercommunity.com","display_url":"twittercommunity.com"}],"hashtags":[{"start":23,"end":30,"tag":"DevRel"}]}},"id":"2244994945","location":"127.0.0.1","most_recent_tweet_id":"1214947767250030592","name":"Twitter Dev","pinned_tweet_id":"1214281000932593667","profile_image_url":"https://pbs.twimg.com/profile_images/880136122604507136/xHrnqf1T_normal.jpg","protected":false,"stats":{"followers_count":506517,"following_count":1707,"tweet_count":3498,"listed_count":1641},"url":"https://t.co/3ZX3TNiZCY","username":"TwitterDev","verified":true,"format":"detailed"}]}


And it works!

Let's do one more thing to clean this up because it is bothering me. Let's stick our token in an enviromental variable. I am no sure how to do this relative to the specific project so I will do it globally. If you know how to do it project specifically with dune and OCaml please do leave a comment.

First we export our token. I am using zsh shell so I am putting in my ~/.zprofile file like this:

sh
echo "export TWITTER_TOKEN=aRjeMU9YWuP5aS8trretKALXhcjBpQ7xY0TJBwq8Dhl1sw3GkRX5VxeyzfqDtqiwAueS9rHbA5PNzr292iE6D1DMwbqLXErMVuhOOzREtZ9eL1FSFTBw6PCVnZmWUtQl3vfiWMaGOdg3gmkQjPLBWqabOcgsufJFdUID3VWrGUo4mT9Bscuq9WZ4Ey6itIMM5InH3mjb" >> ~/.zprofile


Then make sure its availabe by sourcing it then test it.

`sh
➜ source ~/.zprofile
➜ echo $TWITTER_TOKEN

output should be your token

aRjeMU9YWuP5aS8trretKALXhcjBpQ7xY0TJBwq8Dhl1sw3GkRX5VxeyzfqDtqiwAueS9rHbA5PNzr292iE6D1DMwbqLXErMVuhOOzREtZ9eL1FSFTBw6PCVnZmWUtQl3vfiWMaGOdg3gmkQjPLBWqabOcgsufJFdUID3VWrGUo4mT9Bscuq9WZ4Ey6itIMM5InH3mjb
`

Now go into tweets_and_users.ml and add the following under your open calls:

`ocaml
let token = Sys.getenv "TWITTER_TOKEN";;
print_endline(token)

let headers = Header.of_list[ "Authorization", Printf.sprintf "Bearer %s" token]
`

The print_endline is just so you can see it logging from the program.

let headers is constructing the header using the Cohttp.Header module. Where you see "Bearer %s" the %s is a place holder where the variable that follows, here token will be placed.

Then change your reqBody function to use the headers variable.

ocaml
let reqBody =
let uri = Uri.of_string "https://api.twitter.com/labs/1/users?usernames=TwitterDev&format=detailed" in
Client.call ~headers:headersGET uri >>= fun (_resp, body) ->
body |> Cohttp_lwt.Body.to_string >|= fun body -> body
`

Compile and run:

sh
➜ make
➜ dune exec ./tweets_and_users.exe
{"data":[{"created_at":"2013-12-14T04:35:55.000Z","description":"The voice of Twitter's #DevRel team, and your official sourc
e for updates, news, & events about Twitter's API.\n\nNeed help? Visit https://t.co/DVDf7qKyS9","entities":{"url":{"urls":[{"
start":0,"end":23,"url":"https://t.co/3ZX3TNiZCY","expanded_url":"https://developer.twitter.com/en/community","display_url":"
developer.twitter.com/en/community"}]},"description":{"urls":[{"start":129,"end":152,"url":"https://t.co/DVDf7qKyS9","expanded_url":"http://twittercommunity.com","display_url":"twittercommunity.com"}],"hashtags":[{"start":23,"end":30,"tag":"DevRel"}]}},"id":"2244994945","location":"127.0.0.1","most_recent_tweet_id":"1214947767250030592","name":"Twitter Dev","pinned_tweet_id":"1214281000932593667","profile_image_url":"https://pbs.twimg.com/profile_images/880136122604507136/xHrnqf1T_normal.jpg","protected":false,"stats":{"followers_count":506517,"following_count":1707,"tweet_count":3498,"listed_count":1641},"url":"https://t.co/3ZX3TNiZCY","username":"TwitterDev","verified":true,"format":"detailed"}]}


Well that does it for this episode of Learn In Public which @sophiabrandt turned me on to. Thanks Sophia and @swyx. Though I don't think this project fully complies with the principle, talking to you helps me learn and remember. I encourage you to teach what you are learning. Besides all that, I end up referring back to my own posts to see how I did something so have at it.

The source code for this post is here.

Thanks.