DEV Community: Jakub Roztocil

HTTPie AI assistant

Jakub Roztocil — Wed, 04 Jan 2023 12:24:56 +0000

Today we’re launching HTTPie AI, a new way to interact with APIs ✨

HTTPie AI assistant, built into HTTPie for Web & Desktop, uses state-of-the-art artificial intelligence to increase your productivity when testing and talking to APIs.

Free product preview

This initial release allows you to create requests using human language. While in preview, you can use HTTPie AI for free with an HTTPie account. Gradually, AI capabilities will expand throughout the platform.

The problem

Let’s say you want to fetch the details of the httpie/desktop repo from the GitHub API. This requires translating the intent to the corresponding HTTP request and entering it into the client’s UI. You need to have some degree of understanding of the following:

How the API or server works (e.g., hostname, endpoints, auth)
How HTTP works (e.g., headers, methods, body)
How to instruct the client to send the exact request

What if instead of creating requests manually, we could tell HTTPie to “fetch repo details of httpie/desktop”? Or even in another language like Spanish, «obtén los detalles del repositorio httpie/desktop»?

Well, now you can!

Craft requests using human language

You can type your prompt directly in HTTPie’s URL field. It’s similar to typing a search query in your browser’s address bar instead of the exact URL.

Alternatively, you can open the HTTPie AI dialog from the “+” menu in the library. In the desktop app, you can also use ⌘I. There’s a difference in that when you start in the URL field, accepting the generated request will update the one in the current tab instead of creating a new one.

Then hit “Generate” to get a suggestion.

If you like the request, save it. The saved request automatically gets a short name generated by the AI. If you don’t like it, you can hit “Generate” again, which often produces a different result. You can also try editing the prompt. If the request includes any placeholders or obvious errors, you can modify it straight from the dialog before saving it (or you can do that after).

There are two strategies for writing HTTPie prompts: describing the request and describing the intent.

Describing the request

You can ask the AI to generate an HTTP request by describing its low-level components like the URL, headers, and body:

Generate an array of 3 blog posts with title, author, and summary and post it to localhost 3000 api as user agent HTTPie AI

Describing the intent

For actions involving public APIs of popular services, you can describe the high-level tasks you want to accomplish:

List GitHub users

Here is another example, where the model picks up that we mean a GitHub repo:

Fetch last release details of httpie/desktop

Feedback and accuracy

Like many generative AI applications, the accuracy of the outcome may vary depending on the prompt and other factors. Review the generated request and iterate on the prompt as needed. Higher specificity often leads to better results. You can quickly correct errors directly in the dialog before saving the request. As you use HTTPie AI, please provide quick feedback on the suggestions via the thumb-down and thumb-up buttons to help us fine-tune accuracy.

Entering the era of AI copilots

As large language models (LLMs) gain robust generative capabilities, creative tools are evolving toward AI-powered experiences.

HTTPie is now becoming the first API development platform with AI assistance, and we’d love your feedback!

How we built it

We built this initial feature preview using OpenAI’s GPT-3 models.

In the process, we refactored the openai/openai-python library to make the default installation size ~7⨉ smaller (167M → 23M). Now it’s much more manageable to deploy OpenAI-powered applications using this library in environments with code size constraints, such as AWS Lambda.

What’s next

We’ll continue improving the assistant’s accuracy and adding more capabilities throughout the HTTPie platform.

Jump into the Web app or download the Desktop app and check it out!

✉️ Thoughts? Questions? Shoot us an email at feedback@httpie.io.
📋 You can also help us improve by completing a quick survey.
👉 Follow @httpie and join our Discord community to stay up to date.
👩‍💻 We’re looking for new colleagues in engineering and other roles.

PS: We’re looking for a more personable name for our assistant 💡

Originally published at https://httpie.io/blog/ai

Changelog #0022 — 🪄 HTTPie AI assistant, methods in tabs, and more

Jakub Roztocil — Wed, 04 Jan 2023 12:20:25 +0000

Happy new year! 🥳 Let’s start it with a new release of HTTPie for Web & Desktop. Check out what we shipped v2023.1.0 👇

🔖 Summary

🪄 HTTPie AI assistant – Introducing a new way to interact with APIs.
🗂️ Methods in tabs – Easier navigation with HTTP methods in tabs.
📚 Library context menu — New context menu for quick library actions.
🌊 cURL as URL — Type a cURL command in the address field.
✨ Other improvements — Tweaks and fixes for improved API testing.

🪄 Introducing HTTPie AI

Today we’re launching HTTPie AI, a new way to interact with APIs.

HTTPie AI assistant uses artificial intelligence to increase your productivity when testing and talking to APIs. This free feature preview allows you to create requests using human language.

Read the full announcement →

🗂️ See methods in tabs

HTTPie now shows the HTTP method next to the request name in the request tabs:

This way, you can easily distinguish between tabs for requests using different methods but with the same name or URL. The <method> <name> pair works nicely as a <verb> <noun> sentence — a request identification system we already use in the library.

📚️ Quick library actions with new context menu

There’s a lot of action behind the innocently looking “+” button in the library. To make it more accessible, you can now invoke a similar menu through a right-click anywhere in the empty space in the sidebar:

🌊 Type a cURL command in the address field

You can paste a cURL command into the URL field for quick import. In this release, we added the ability to type an AI prompt into the address field, and at the same time, we extended it to support typing cURL commands.

Support for HTTPie CLI commands is coming soon.

✨ Other improvements

Added numerous on-import clean-ups like detecting GraphQL requests based on the body to import them with the GraphQL body type or moving Bearer tokens from Authorization: Bearer … headers to the auth tab.
Added more robust interactive URL matching. HTTPie will now add interactivity to links containing an IP address, port number, or pointing to localhost.
Added RSS feeds to this blog; grab one for your feed reader to stay up-to-date: ⚡️Atom 1.0, ⚡️RSS 2.0, ⚡️JSON Feed 1.1
Fixed the auto-update system not to bother you once you choose “Install on next launch.” (#55)

🤩 Happy API testing, and see you again soon!

✉️ Thoughts? Questions? Shoot us an email at feedback@httpie.io.
📋 You can also help us improve by completing a quick survey.
👥 We’re working on collaboration features, and you can help us out.
👉 Follow @httpie and join our Discord community to stay up to date.
👩‍💻 We’re looking for new colleagues in engineering and other roles.

Originally published on https://httpie.io/blog/changelog-0022

Changelog #0021 — 📯 Advanced import, GraphQL schema support, and more

Jakub Roztocil — Thu, 08 Dec 2022 12:25:00 +0000

Hello, API World! Check out what we shipped in v2022.16.0
of HTTPie for Web & Desktop 👇

🔖 Summary

📯 Advanced import – Migrate your data from Postman and Insomnia.
🪄 GraphQL schema support – Auto-complete, validation, and more.
✨ Other improvements — Tweaks and fixes for improved API testing.

📯 Import your Postman and Insomnia data to HTTPie

Since our public beta launch a few months ago, the adoption of HTTPie for Web & Desktop has been growing. Many of our new users have migrated from other API testing clients like Postman and Insomnia, and they’ve been asking for a smoother migration path.

So in this release, we focused on streamlining the migration process, and—in addition to the existing cURL import—you can now import entire collections, environments and data dump zips from Postman and Insomnia.

🚀 Migrate to HTTPie in a few simple steps

Export your data from Postman or from Insomnia.
Open the import dialog (Library → “+” → “Import…”).
Drag & drop the exported file and click “Next…”.
Select the target space and click “Import.”
Done! Welcome to HTTPie 🤗

💡 Tip: Create a new space during the import to keep things tidy.

🛂 Import compatibility

Here’s an overview of what elements HTTPie currently imports from Postman and Insomnia and how. As we gradually add new features to HTTPie, we’ll continue to improve the import as well.

🛫️ Importing requests

Request element	Imported?	Notes
HTTP Requests	✔
GraphQL requests	✔
gRPC requests	✘
WebSockets	✘
Request method	✔	Including custom ones.
Request URL	✔
URL parameters	✔
Headers	✔
Auth	✔	Basic, bearer, API key, and inherited auth.
Body	✔	Text, raw, JSON, and form request bodies.
Body files	✘

🍱️ Importing containers

Container	Imported?	Notes
Collections	✔	See collection variables below.
Sub-folders	✘	Folder hierarchy is preserved as breadcrumbs in request names.

🖼️️ Importing environments & variables

Element	Imported?	Notes
Environments	✔
Default variables	✔
Collection variables	✔	We move non-global variables to the space level and refactor names and references to ensure uniqueness.

💧 Importing other elements

Other element	Imported?	Notes
Scripting	✘	Pre-request scripts, tests, and dynamic tags.
Meta data	✘	Schemas, documentation, field descriptions, and response examples.
Custom settings	✘
Cookies	✘

🪄 Easier GraphQL APIs testing with schema support

When talking to GraphQL APIs, HTTPie will automatically fetch the schema from the server. That means the app is now aware of all the available queries, mutations, and the hierarchy of types, which allows us to boost your productivity with auto-complete and validation.

See auto-complete suggestions as you type, and to invoke the menu manually, press ⌃Space.

HTTPie auto-fetches the schema using the current request’s URL and authentication. To refresh the schema, select “Update from server” from the new “Schema” menu. And to disable schema for a particular tab, uncheck “Auto-apply schema.”

In addition to schema support, the GraphQL client now comes with code folding and other quality-of-life improvements.

✨ Other improvements

Improved cURL import with support for ANSI-C quoting shell syntax (e.g., $'string'). This syntax is used by Chrome DevTools when exporting a request that includes a body as cURL, among others.
HTTPie for Desktop now remembers the window's position. This way, you don't have to move it every time you launch it.
On HTTPie for Windows, the title bar color now respects the system.
The request definition tables no longer jump when switching tabs.
We’ve added error boundaries to tabs and other components for more robust exception handling and recovery.
The text search had some issues with very long lines. Now it’s fixed.
Spelling and autocorrect in the filter field is now disabled.

🤩 Happy API testing, and see you again soon!

✉️ Thoughts? Questions? Shoot us an email at feedback@httpie.io.
📋 You can also help us improve by completing a quick survey.
👥 We’re working on collaboration features, and you can help us out.
👉 Follow @httpie and join our Discord community to stay up to date.
👩‍💻 We’re looking for new colleagues in engineering and other roles.

Originally published on HTTPie blog.

Changelog #0017 — ☁️ Sync, variables, incognito, more features, and public beta!

Jakub Roztocil — Fri, 05 Aug 2022 22:06:22 +0000

HTTPie is on a mission to make APIs simple and intuitive for all those building the tools of our time. And today, we’re thrilled to announce the biggest release yet and a large step in that direction. Check out what’s new 👇

Summary

🔓 Public beta — HTTPie Web & Desktop is now publicly available 🎉
🕵️‍♀️ Incognito mode — You can now use the app without an account.
☁️ Real-time sync — Your data is now real-time synchronized across devices.
⬆️ Auto-updates — A reworked, more transparent system for updates.
💲 Variables — Work smarter and don’t repeat yourself.
🏞️ Environments — A quick way to swap between contexts.
🌊 cURL import — A humble but efficient start of our import system.
🔒 SSL verification — You can now turn off certificate validation.
📱 Mobile — Now it’s easier than ever to work on touch devices.
✨ Other improvements — Various changes to improve your experience.
🪲 Bug fixes — Hopefully, we caught them before you noticed.
🔮 What’s next? — A glimpse into the future.

🔓 Public beta

After months of development, testing, and collecting feedback in private beta, we’re opening HTTPie for Web & Desktop to everyone today 🚀

You can go to the Web app or download the Desktop app and start using it right away.

💌 Thanks to our community and private beta users

Since the launch of the Terminal app ten years ago, an enthusiastic community has been forming around HTTPie. When we announced HTTPie for Web & Desktop, the interest exceeded our expectations, and our waitlist started to fill up. We opened up a private beta program and invited thousands of beta users to whom we couldn’t be more grateful.

We were submerged in hundreds of thoughtfully answered surveys, messages in our growing Discord community, emails, GitHub issues, and Twitter DMs, and we appreciated every single one!

All that feedback has translated into improvements and features, like the ones included in this release and many more on our roadmap.

In return, HTTPie swag has found its way all around the world 💫

🕵️‍♀️ Incognito mode

Now that the private beta fence no longer exists, the account-required wall can be torn down too. And that’s exactly what we’ve done.

You can now use the app without an account. When you are not logged in—when you’re incognito—you still have access to all features apart from sync and creating new spaces. When you decide to log in or sign up, we sync your incognito space into your account, and you can continue to work uninterrupted. Easy!

☁️ Real-time sync

Sync is finally here. As long as you are not incognito, your data is backed up to our secure cloud and synchronized across your devices. When you change something, it’s instantly updated on your other clients.

Start working in the Web app, switch to the Desktop app, and check some request details on your phone later. And you don’t have to worry about hitting save because we continuously auto-save your work.

You might be wondering the most important question of all, can I still test my local API on a plane trip if I need to? Sure thing! HTTPie works offline-first. When your computer is offline (or our cloud is temporarily down), you can continue to work as usual. We’ll simply resume syncing once the connection has been restored. To see the current sync status, check out your account menu:

Data security

All synced data is encrypted at transit between the HTTPie cloud and clients (SSL) and at rest in a backed-up AWS RDS storage with redundancy (AES). All sensitive data—your requests, auth credentials, variables, etc.—are additionally encrypted at the server-side before storage and decrypted on client access (AES).

⬆️ Auto-updates

You can now check for updates and see the app update status straight in the account menu. This makes it easy for you to have the greatest and latest version of HTTPie always installed.

We continuously improve the app and release frequent updates with new features and bug fixes. The desktop app downloads new versions automatically, even though you don’t have to install them right away.

The auto-update system is integrated with sync, and the update status in the menu will tell you when an update is required to maintain compatibility with the HTTPie cloud.

💲Variables

Variables were among the most requested features by our private beta users. And we agree. When it comes to building requests in a clean, don’t-repeat-yourself way, variables are a game-changer. So here they are!

Define variables inside the current space’s detail tab. Then you can reference them from any request or collection within the space via {{NAME}}.

It’s often helpful to review the full HTTP request with variables already substituted before sending it. To do that, keep the Preview panel open.

If you reference a variable that doesn’t exist, we mark the reference red and prevent the request from being sent until the variable is declared. This way, you never have to deal with confusing results.

🤫 Secret variables

Mark a variable as secret, and we’ll mask its values in the variable list so that you don’t accidentally share them on a Zoom call, for example.

See local-only environments below for other advanced ways to deal with sensitive variables.

🏞️ Environments

Environments allow you to quickly swap between different contexts. A common practice is to have an API running in different environments—on your local machine, in staging, in production, etc.

Imagine you need to talk to the same API in production but also in a staging environment. The requests are the same, but the root URL and auth, for example, differ. You can turn those values into variables, so it’s easier to change them. And with environments, you can instantly toggle between different values for those variables:

You can create as many environments as you like and switch between them using the menu at the top right of the app.

An environment is a named set of variable values. In the table, rows represent variables, columns represent environments, and inner cells are specific values for a variable in a specific environment. So once you declare a variable, it exists in every environment in the space (possibly with differing or empty values).

Every space starts with an empty environment called “Defaults.” It works as a regular environment, and for simple use-cases, you can just use it exclusively.

When you create additional environments, they use values from Defaults for variables where you don’t explicitly specify a custom value. If you don’t need a default value for a variable, you can simply leave its value in Defaults empty.

📍Local-only environments

If you want to prevent some data from syncing to the HTTPie cloud and across your devices, you can store it as values in a local-only environment.

When creating an environment, select the option to create a local-only one, and its values will never leave your device.

🚫 No environment

In some situations, you might want to temporarily prevent requests from accessing your variables altogether. By selecting the “No environment” option, you can achieve just that:

Currently, this option is most useful when it comes to debugging your requests. But because variables can contain sensitive data, the importance of this feature will grow as we continue to add various import and sharing mechanisms.

🌊 cURL import

We are working on a robust import mechanism, which will eventually support many advanced import sources. But already today, you can import cURL commands as requests into your HTTPie library:

This allows you to quickly import cURL calls from scripts or import requests from Chrome DevTools. Find the “Import…” option under the + menu in your space library.

🪄 cURL import shortcut

There’s also a quick hidden shortcut for importing cURL. Can you guess where?

Just paste a cURL command into the URL field and see what happens 👀

🔒 SSL verification

This is a smaller feature, but many of our private beta users have been waiting for it, and we’re happy to finally ship it. By default, we verify the server SSL certificate and reject self-signed, expired, and non-matching certificates. Now you can disable this validation in the settings:

When you run into an SSL error and wish to temporarily allow invalid certificates, you can do that straight from the response view:

Turn it on again from the settings when you’re finished.

📱 Mobile

We’ve also focused on streamlining our mobile experience. The app is 100% responsive (both Web and Desktop), and you can use all features on touch devices as well.

Mobile-friendliness is one of those features that you don’t need most of the time. But when you need it, you really need it. And with HTTPie, you can get work done comfortably from anywhere, anytime.

The mobile experience of the URL bar component is much improved.
You can now use the Preview & Export feature on small screens.
Keyboards are now contextual.
Spacing is more compact and comfortable on small devices.
You can now access tooltips on touch devices.

✨ Other improvements

The body type and auth type selectors now have icons. As they say, an icon is worth a thousand labels.
Error badges are now more minimal, less clunky, and more on brand.
It’s been a while since we expanded the icon set for spaces and collections. Now you have eight crisp new icons to choose from. Check them out!
The light mode previously received less love than the dark one. So we improved it. What do you think? Can you spot all the differences?
Placeholders now look more like placeholders.
Now you see a loading indicator when the app is launching.
The monospaced font now supports non-Latin characters.
Text selection is now disabled for all UI elements while keeping content selectable.
In the desktop app, you can maximize and un-maximize the window by double-clicking the title bar.
You can also drag the window using the left toolbar, where your profile image resides.

🪲 Bug fixes

When you renamed a draft, it would change its auth to inherited, as if it was added to a collection. Fixed.

🔮 What’s next?

This release marks an important milestone on our journey to provide the best experience to anyone working with APIs.

With HTTPie, you can work on APIs with or without an account. While you are online or offline. In a dedicated cross-platform desktop app, in the browser, in the terminal, and even on your phone. Your data is auto-saved and synchronized across devices. You can organize your work in spaces and collections, parametrize requests with variables and environments, work on many things simultaneously with our tabbed system, and preview requests before sending them. These and many other features are available today, wrapped in what we aim to be a delightful, human-centered interface.

There’s still a lot of work to fully unlock the potential of APIs, and our roadmap is packed with ambitious milestones. Here’s a sample of the themes we’re focusing on next:

👥 Team collaboration
⚙️ Additional protocols and formats
⚡ Automation and testing
🔗 Full interoperability with HTTPie for Terminal
🎹 Become a keyboard-first app
🎉 Refreshed website

Happy API testing, and see you again soon!

📋 You can help us improve by completing a quick survey.
💲 If you use environments, we have a variables-specific survey as well.
👉 Follow @httpie and join our Discord community to stay up to date.
👩‍💻 We’re looking for new colleagues in engineering and design roles.

Originally published on HTTPie blog.

What’s new in HTTPie for Terminal 3.2

Jakub Roztocil — Tue, 17 May 2022 13:50:47 +0000

HTTPie has always been about making interaction with APIs as human-friendly as possible. Part of that is ensuring our apps are ubiquitous and up-to-date. This release of HTTPie for Terminal brings many improvements in all these areas.

Check out what’s new 👇

Summary

🤩 Refined UI — Improved user experience and usability.

📖 Man pages — HTTPie is finally getting complete manuals available through man httpie, man http, and man https.

📦 Single binary executables — download HTTPie as a single binary file and run it without installation.

🌀 Debian releases — most recent releases will now be offered also through our custom Apt repo with new releases instantly available for Debian, Ubuntu, etc.

⬆️ Update warnings — get notified about new versions as soon as they become available for your system straight in the terminal.

⚖️ Improved persistence for headers — multiple headers with the same name are persisted in the sessions.

🪲 Bug fixes — multiple headers with the same name are persisted in the sessions.

♥️ Community contributions — lots of amazing people who have contributed to this release.

🤩 Refined UI

This release brings various UI and UX improvements. For example, here is HTTPie’s new progress bar:

Even though our original progress bar implementation was simple and elegant, we’ve decided to adopt the powerful Rich library for various parts of the UI.

By the way, the example above uses the recently-added --style=pie-dark style. We highly recommend making it your default style for the best experience and consistency with HTTPie for Web & Desktop.

📖 Colors & formatting docs →

📖 Man Pages

The long-awaited man pages are here now. You can use man http or man https to have the full man page experience with colors and pager:

$ man http

We have also introduced a shorthand for this operation, http --manual:

$ http --manual

When the man pages are available (installed by your platform’s packager), it will automatically open you the HTTPie’s manual. When they are not, this command will open the regular manual but inside of a pager.

The httpie management command now has a man page as well:

$ man httpie

📖 Help & manual docs →

📦 Single binary executables

For Linux systems, we now offer single binary executables which you can download and start using without installation. The new asset named http now included in every release of HTTPie on GitHub is all you need:

$ https -do http_standalone github.com/httpie/httpie/releases/download/3.2.1/http

$ chmod +x ./http_standalone

$ ./http_standalone --version

📖 Single binary installation docs →

🌀 Debian releases

We now host our own Debian Apt repository to provide instant updates with new features to users of Debian, Ubuntu, and other derived distros:

$ curl -SsL https://packages.httpie.io/deb/KEY.gpg | apt-key add -
$ curl -SsL -o /etc/apt/sources.list.d/httpie.list https://packages.httpie.io/deb/httpie.list
$ apt update
$ apt install httpie

📖 Debian & Ubuntu installation docs →

⬆️ Update warnings

HTTPie for Terminal is now part of a broader platform, and we iterate quickly and deliver new features frequently. To keep existing users up-to-date, this version adds a gentle new warning:

$ http pie.dev/get
http: info: A new HTTPie release (3.3.0) is available.
To see how you can update, please visit https://httpie.io/docs/cli/pypi

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: application/json

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
    },
    "url": "http://pie.dev/get"
}

You can also check the latest release by manually running check-updates:

$ httpie cli check-updates

📖 Update warning docs →

⚖️ Improved persistence for headers

Multiple headers with the same name are now correctly persisted inside sessions. E.g.:

$ http --session test pie.dev/headers X-Foo:bar X-Foo:baz

$ http --session test pie.dev/headers

For upgrading your old sessions, you can use httpie cli sessions upgrade:

$ httpie cli sessions upgrade pie.dev test

To upgrade all named sessions in one go, you can try httpie cli sessions upgrade-all.

$ httpie cli sessions upgrade-all

📖 Session upgrade docs →

✨ Other improvements

Moved httpie plugins * command to the new httpie cli plugins namespace. This provides a more scalable namespacing of the management subcommands. The original httpie plugins * commands are now hidden aliases.

🪲 Bug fixes

Removed the redundant addition of Content-Length: 0 header to body-less OPTIONS requests.
The thread observing standard input to provide a warning when no data arrives is now properly terminated when HTTPie itself exists.

🐙 Full changelog →

♥️ Community contributions

Last but not least, we’d like to thank the amazing people who’ve contributed to this release of HTTPie: Ethan Mills, Jan Verbeek, Mohamed Daahir, Nicklas Ansman Giertz, Oliver Fish, Roberto López López, luzpaz, and zhaohanqing95.

🐙 How to contribute →

How we lost 54k GitHub stars

Jakub Roztocil — Fri, 15 Apr 2022 15:43:49 +0000

Gaining 54k GitHub stars

HTTPie for Terminal is celebrating 10 years since the first commit.

If you’re unfamiliar with the project, it’s an open-source CLI HTTP client. What makes HTTPie different is that we build it from the ground up to make API interaction from the terminal as human-friendly as possible.

Starting with the first public release, published on the 25th of February 2012 from a rainy Copenhagen, we’ve hosted the project on GitHub.

I’ve been a GitHub fan ever since I became a member a couple of years earlier (the type that wears Octocat-decorated t-shirts, no less). It was back in the day when GitHub’s about page proudly proclaimed they took $0.00 in VC funding and informed you about the number of delicious beers on tap in their SF office.

So GitHub was an obvious choice when I realized that the result of scratching my own API testing itch might be of interest to the wider developer community. And of interest it was.

I remember the rush of HTTPie becoming the top link on Hacker News for the first time and seeing the GitHub community build up. Over the years as we continued to improve the project it kept attracting widespread adoption. It became the most popular API tool on the platform, and the GitHub community grew to 54k stargazers and 1k+ watchers.

There are 289M public repos so HTTPie was among the top 80 most popular public repos on GitHub overall; in the 99.99997203 percentile. In short, It was incredible to see this humble tool attract a community of that magnitude. And GitHub played an important role in that.

In the same way that we benefited from GitHub’s “social coding” features, GitHub benefited from our hosting this popular project on their platform. Over the past decade, possibly millions of developers visited our GitHub page. That’s helped reinforce GitHub (Microsoft) as a company that cares about open source and community. It was a symbiotic relationship.

Losing 54k GitHub stars

However, if you are one of our 55k stargazers and watchers, as of a few weeks ago you no longer are 💔

What happened?

Due to an unfortunate sequence of events, I accidentally made the project’s repository private for a moment. And GitHub cascade-deleted our community that took 10 years to build.

What does it mean?

If you’re a downstream maintainer or anyone previously watching httpie/httpie for notifications, for example, you’ll need to re-watch the repo. Incidentally, we’ve recently published a security release.

The same goes for stars. If you’re one of those 54K people who’ve starred the repo any time in the past decade, the repo is no longer among your starred projects.

Why did you make the repo private‽

It’s a peculiarity of GitHub, to put it mildly, that making a repo private permanently deletes all watchers and stars. I was even aware of this, and I obviously had no intention to make httpie/httpie private. So, why then?

The proximate cause was that I thought I was inside a different repo; one with no content and zero stars. What I actually intended to do was to hide the HTTPie organization’s profile README, which I had created the week before but had no opportunity to populate.

What put me on the wrong path was an otherwise completely unrelated action: I had just done the same (i.e., hidden an empty README) on my personal profile by making jakubroztocil/jakubroztocil private.

GitHub’s conceptual model treats users and organizations as very similar entities when it comes to profiles and repos. In this context, and since I just wanted to repeat the same benign action on our organization’s profile, my brain switched to auto-pilot mode.

I didn’t realize at the moment there’s an inconsistency in the naming of this special repo containing profile READMEs and that it differs for users and organizations: name/name vs. name/.github.

That’s why I proceeded to make httpie/httpie private instead of httpie/.github without realizing my mistake.

But there’s a confirmation, right‽

There’s a confirmation box. It’s designed to stop users in a situation like mine from doing something stupid. It tells you that “You will permanently lose all stars and watchers of this repository.” That’s pretty scary.

The problem is that the box looks exactly the same for repos with no commits and stars and for repos with a decade-long history and 55k stargazers and watchers. And it says “Warning: this is a potentially destructive action.”

To paraphrase, the box tells you “You’re about to demolish a house. If there are any people inside, they will all die”.

But it doesn’t include anything specific to break you out of your auto-pilot mode if you’ve confused the address and think you’re looking at an empty house.

A 54k-star question: Which one of these two dialogs is safe to confirm and which one deletes a 10-year-old community?

The dialog should be more contextual and, paraphrasing again, it should say “You’re about to kill 55,000 people.” That would’ve certainly made me pause.

So you made it private, just flip the switch!

You can imagine my confusion when I went back to the organization page and not only could I still see the empty README but our most popular repo was nowhere to be found. After a moment I realized what happened. So I went back to the repo’s settings to flip the switch. But GitHub didn’t allow me to do that—for an entire half an hour.

If you’re wondering why so long, it’s because 🥁 that’s how long it took GitHub to cascade-delete our decade of stargazers and watchers. And there was no way to stop the process. All I could do was start writing to GitHub support, refresh the page and wait for the number of stars to reach zero before I could make it public again.

Why doesn’t GitHub restore it‽

GitHub obviously has backups. And it is indeed possible to undo the damage done by accidentally making a repo private. The GitHub team themselves accidentally made the GitHub Desktop app repo private once. And they restored everything for themselves within hours. Here’s the former GitHub CEO explaining the situation:

In our case, however, they refuse to do that, citing undesirable side effects and the cost of resources. We even offered GitHub financial compensation for any resources required. But, sadly, they declined. They have other priorities than resurrecting the community of one of the oldest and most popular community projects on their platform.

So the answer to the question is, unfortunately, the following: GitHub will restore a repo damaged by making it private. But only if it’s one of their own projects, not a community one. The latter gets a tweet, at best.

Lessons learned

One should never let a good crisis go to waste. Our options are limited, but there are at least a few lessons learned that might be valuable to share.

Lesson #1: UI/UX design

Show, don’t tell. Design confirmation dialogs in a don’t-make-me-think fashion. When the user is about to destroy something, don’t describe that as a potential scenario in abstract words that the user needs to convert to mental images and put values on. Especially when cascade-deleting as a side effect of the primary action. For example, here’s how we approach that in HTTPie for Desktop:

And, of course, the dialog should reflect the severity of the side-effect. It should be quiet when there are no side effects at all. Otherwise, we risk desensitizing the user by wasting their limited attention capital:

Lesson #2: Database design

Use soft-deletes. People are human and they make mistakes. For hard-deletes, delay the process.

Lesson #3: Relationship with GitHub

It was a human error on our side and GitHub made it clear they’re not legally obliged to help us. The tone of our decade-long mutually-beneficial relationship is set by GitHub’s Terms of Service. Thinking there was more to it was naive.

After all, GitHub has a history of taking controversial actions that go against the spirit of open source and community and then reverting them only based on public outrage. And Microsoft (which now owns GitHub), despite its newly-found affection for open source, has not always had exactly a great reputation.

What’s next?

We continue to hope GitHub/Microsoft change their machine-like attitude and restore the project’s community one day. They still have all the data and the means to do that. We also hope they improve the UI and database design to prevent this from happening to other teams in the future.

In the meantime, you can help us by sharing this story and re-watching/starring the repo.

As for me, I will probably take a break from wearing Octocat-decorated t-shirts.

Epilogue

Despite our GitHub stars turning into dust, HTTPie has never been doing better. What started as a side project has recently become a company and our team is growing HTTPie into an API development platform (one as delightful as you’d expect from HTTPie). The private beta of HTTPie for Web & Desktop has been receiving great feedback, and we cannot wait to launch it to the public in the upcoming weeks.

If you’d like to stay up-to-date, you can join our Discord community or follow @httpie.

✏️ Update after four months

The interest in this story blew us away. As of August ’22, the post has been read by more than a hundred thousand people and has been shared thousands of times on Twitter and elsewhere, and was the top post on Hacker News for days.

We’ve also received a ton of empathy and public calls for support from the open source community, and we couldn’t be more grateful. As a result, HTTPie for Terminal has regained an incredible 23k stargazers within a few months.

We never heard from GitHub. But at least they’ve quietly improved the UI of the destructive “Change repository visibility” dialog shortly after this post went out.

✏️ Update II.

Since the last update, a person from Microsoft who led the acquisition of GitHub has personally reached out to me on Twitter. He even claims to have found our stars!

Sadly enough, however, we haven’t heard from him since. But we’re happy for T3, whose deleted repo GitHub successfully restored including all forks and stars.

Another proof that complete restoration is indeed possible.

We’ll keep you and this post updated.

Originally published on HTTPie blog.

What’s new in HTTPie for Terminal 3.1

Jakub Roztocil — Tue, 15 Mar 2022 00:00:00 +0000

We released HTTPie for Terminal 3.0 about a month ago. Today we’re announcing 3.1.0 that comes with a bunch of enhancements and a security-related fix.

Summary

🍪 Improved cookie management — improved security with host-bound cookies, and more.

⚙️ Session management — automatic upgrades of sessions files.

🧩 CLI parser schema export — a new httpie cli export-args command with machine-readable arguments description.

🌲 Nested JSON enhancements — formalization of the top-level array syntax, and a fix for escaping of multiple backslahses.

✨ Simplified usage — HTTPie now has an intelligent compact help.

🪲 Bug fixes — extinguished and regression-test-covered.

♥️ Community contributions — lots of amazing people who have contributed to this release.

🍪 Improved cookie management

Before 3.1.0, HTTPie didn‘t bind cookies to the host that set them. This behavior resulted in the exposure of some cookies when there are redirects originating from the actual host to a third party website, e.g:

$ http --follow --session=jar \
  pie.dev/redirect-to url==https://httpbin.org/cookies

Prior to 3.1:

{
    "cookies": {
        "x": "y"
    }
}

Starting with 3.1:

{
    "cookies": {}
}

This behavior has been corrected in this release (with taking
RFC 6265 — HTTP State Management Mechanism
into the consideration).

Security advisory

The old host-unbound handling of cookies in sessions represents a low-severity security vulnerability. We’d like to thank @glyph for responsibly disclosing the issue. It has been assigned the CVE ID CVE-2022-24737 (and CVE-2022-0430, which is a duplicate), and you can read more about it in the advisory published on GitHub.

📖 Host-based cookie policy docs →

⚙️ Automated session upgrades

The change in HTTPie’s handling of cookies triggered the need to also change the sessions file format. When you use a session file in the old format, HTTPie will ask you to upgrade it first. There are two new commands to help you upgrade your sessions:

`httpie cli sessions upgrade`

With this command, you can upgrade a single session file.

Upgrade an anonymous session:

$ httpie cli sessions upgrade pie.dev ./jar.json
Upgraded 'jar' @ 'pie.dev' to v3.1.0

Upgrade a named session:

$ httpie cli sessions upgrade --bind-cookies pie.dev api-auth
Upgraded 'api-auth' @ 'pie.dev' to v3.1.0

`httpie cli sessions upgrade-all`

This command allows you to upgrade all your named sessions in one go:

$ httpie cli sessions upgrade-all
Upgraded 'jar' @ 'pie.dev' to v3.1.0.
Upgraded 'api-auth' @ 'pie.dev' to v3.1.0.
Upgraded 'api-auth' @ 'api.myhost.com' to v3.1.0.

📖 Session upgrade docs →

🧩 Exporting CLI’s parser schema

We’ve added a new command called httpie cli export-args. The idea is to improve interoperability by exposing our CLI definition to third-party tools in a machine-readable format (not unlike an API description). This makes it easier for developers to generate auto-completion for different shells and utilities, power playgrounds, cheatsheets, etc.

HTTPie users will also benefit from the internal CLI-definition-as-data refactorings directly as they allow us to implement a more user-friendly and contextual help output, finally generate a proper man page, and more (coming very soon)!

For a very simple example, we can use export-args to list the first ten options:

$ httpie cli export-args --format=json | jq '.spec.groups[].args[].options[0]' | head

"METHOD"
"URL"
"REQUEST_ITEM"
"--json"
"--form"
"--multipart"
"--boundary"
"--raw"
"--compress"
"--pretty"

📖 httpie cli export-args docs →

🌲 Nested JSON enhancements

Top-Level arrays

If you want to use an array (instead of an object) as the top-level encapsulator on your generated JSON data, we have now formalized a syntax for it:

$ http --offline --print=B pie.dev/post \
    []:='httpie.io' \
    []:='pie.dev' \
    []:='pie.co'

[
    "httpie.io",
    "pie.dev",
    "pie.co"
]

📖 Top-level array docs →

Escaping of multiple backslashes

This release has also fixed a bug related to escaping of multiple backslashes inside a single key group.

$ http pie.dev/post 'data[\\key]=value'

{
    "json": {
        "data": {
            "\\key": "value"
        }
    }
}

📖 Nested JSON escaping docs →

✨ Simplified usage

HTTPie for Terminal grew a lot in the past decade, so did the usage section in the help messages:

$ http
usage: http
    [--json] [--form] [--multipart] [--boundary BOUNDARY]
    [--raw RAW] [--compress]
    [--pretty {all,colors,format,none}] [--style STYLE]
    [--unsorted] [--sorted] [--response-charset ENCODING]
    [--response-mime MIME_TYPE]
    [--format-options FORMAT_OPTIONS] [--print WHAT]
    [--headers] [--meta] [--body] [--verbose] [--all]
    [--history-print WHAT] [--stream] [--output FILE]
    [--download] [--continue] [--quiet]
    [--session SESSION_NAME_OR_PATH |
    --session-read-only SESSION_NAME_OR_PATH]
    [--auth USER[:PASS] | TOKEN]
    [--auth-type {basic,bearer,digest,edgegrid}]
    [--ignore-netrc] [--offline]
    [--proxy PROTOCOL:PROXY_URL]
    [--follow] [--max-redirects MAX_REDIRECTS]
    [--max-headers MAX_HEADERS] [--timeout SECONDS]
    [--check-status] [--path-as-is] [--chunked]
    [--verify VERIFY] [--ssl {ssl2.3,tls1,tls1.1,tls1.2}]
    [--ciphers CIPHERS] [--cert CERT] [--cert-key CERT_KEY]
    [--ignore-stdin] [--help] [--version] [--traceback]
    [--default-scheme DEFAULT_SCHEME] [--debug]
    [METHOD] URL [REQUEST_ITEM ...]
http: error: the following arguments are required: URL

We redesigned this output to be more simple, so it reveals what you really care about:

$ http
usage:
    http [METHOD] URL [REQUEST_ITEM ...]

error:
    the following arguments are required: URL

For more information try 'http --help'

It can also contextually highlight the usage of specific options. In the example below, it adds --pretty {all, colors, ...} to the output becuase it’s used incorrectly:

$ http --pretty

usage:
    http [--pretty {all,colors,format,none}] [METHOD] URL [REQUEST_ITEM ...]

error:
    argument --pretty: expected one argument

For more information try 'http --help'

🪲 Bug fixes

Responses without a reason phrase in the Status-Line, e.g. HTTP/1.1 200 as opposed to HTTP/1.1 200 OK, failed to render correctly with the Pie themes. This is now fixed.
Fixed redundant issuance of stdin detection warnings on some rare cases.
-qq or --quiet --quiet now correctly suppresses warnings.

🐙 Full changelog →

♥️ Community contributions

Last but not least we’d like to thank the amazing people who’ve contributed to this phenomenal release of HTTPie:

Glyph,
Daniel,
Igor Melo,
Marcos Chicote,
Matthew Moisen,
Mohamed Daahir,
Nilushan Costa,
Patrycja,
binbjz,
daurnimator.

What’s new in HTTPie for Terminal 3.0

Jakub Roztocil — Wed, 26 Jan 2022 19:20:16 +0000

It’s hard to believe it’s been almost ten years since HTTPie for Terminal’s initial commit. Seeing the result of scratching our own API itch getting adopted by developers across the world and becoming the most popular API tool GitHub with 50K+ stars has been an incredible experience.

Since then, we have been working enthusiastically to provide the best user experience for anyone working with APIs and, as of recently, growing HTTPie into a comprehensive API development platform accessible from Terminal, Web & Desktop, and Mobile. Today, we are thrilled to announce the 3.0 release of HTTPie for Terminal.

Our fantastic community has been providing thoughtful feedback over the years. And for 3.0, we focused on some of the most requested features of all time. Let us take you on a tour of the new features 👇

Summary

🌲 Nested JSON — HTTPie language extended for crafting complex JSON structures.

🔌 Plugin manager — robust handling of plugin installation.

⏱️ Response metadata — a new built-in way to display response metrics.

🚀 Speed-ups — 30–250% speed-ups over different areas.

🎨 Improved UI/UX — unified Pie theme, improved stdin handling, and better errors.

✨ Other features — bearer auth, repeated headers, prompted session passwords, headers & params from files, effortless URL-to-command conversion, JSON types with forms, and auto-streaming.

🪲 Bug fixes — extinguished and regression-test-covered.

❤️ Community contributions — dozens of amazing people have contributed to making this the best HTTPie release ever.

🌲 Nested JSON

Have you ever wondered whether the simplicity of the existing HTTPie request language could meet with the power of nesting? This was a long journey, but ultimately, we embedded our mini path nesting format into the request language.

We designed it to offer the highest functionality with the best user experience possible. For that, we took an existing format (namely, HTML JSON form), and built our specification on top of it:

$ http pie.dev/post \
  platform[name]=HTTPie \
  platform[mission]='Make APIs simple and intuitive' \
  platform[homepage]=httpie.io \
  platform[stars]:=54000 \
  platform[apps][]=Terminal \
  platform[apps][]=Desktop \
  platform[apps][]=Web \
  platform[apps][]=Mobile

{
    "platform": {
        "name": "HTTPie",
        "mission": "Make APIs simple and intuitive",
        "homepage": "httpie.io",
        "stars": 54000,
        "apps": [
            "Terminal",
            "Desktop",
            "Web",
            "Mobile"
        ]
    }
}

We focused on keeping the experience pleasant even when the user makes errors, which made us treat this as a complete language and implement small productivity boosters all around.

For example, when typing all these structures on the terminal, if you happen to make a typo, not only do we show you where it occurs, but we also tell you how to possibly fix it:

$ http pie.dev/post data[foo][baar]]baz[:=2

HTTPie Syntax Error: Expecting '['
data[foo][baar]]baz[
               ^

Another time-saver is object-level type safety, which prevents you from implicitly creating an ill-formatted JSON object:

$ http pie.dev/post data[foo][bar]=object data[0][baz]="not an array"

HTTPie Type Error: Can't perform 'index' based access on 'data' which has a type of 'object' but this operation requires a type of 'array'.
data[0][baz]
    ^^^

(If you happen to be a Python developer and these error messages look familiar, it’s because we took inspiration from improved errors in Python 3.10 / 3.11, that our very own @isidentical helped bring to life.)

There is a lot more to learn about this feature. We recommend taking a minute familiarizing yourself with it so that your API interactions become more enjoyable than ever.

📖 Nested JSON docs →

🔌 Plugin manager

HTTPie shines with plugins, and, even though we never publicized the plugin API, developers and companies have written and open-sourced dozens of them.

Their functionality ranges from adding support for different authentication methods to changing how responses are displayed on the terminal; they serve various purposes and offer infinite possibilities to expand HTTPie itself.

For example, guardian/httpie-hmac-auth and akamai/httpie-edgegrid add custom auth protocols to HTTPie, which are then available to you through the --auth-type option:

$ http \
  --auth-type=edgegrid \
  --auth=default: \
  :/diagnostic-tools/v2/ghost-locations/available

In the past, however, it was impossible to reliably install plugins for some distribution methods of HTTPie that come with a bundled Python interpreter (e.g., brew and snap).

With 3.0, we make plugins first-class citizens. It is now possible to manage plugins directly by using the new httpie management command and its httpie plugins subcommands. This interface unifies plugin installation no matter how you installed HTTPie itself:

$ httpie plugins install your-favorite-plugin
Installing your-favorite-plugin...
[snip]
Installed your-favorite-plugin==1.0.0

$ httpie plugins list
httpie-edgegrid (1.0.6)
  httpie_oauth1 (httpie.plugins.auth.v1)
httpie-image (1.0.0)
  httpie_image (httpie.plugins.converter.v1)

If you’re curious what kind of plugins exist, you can find most of them on PyPI.

📖 Plugin manager docs →

⏱️ Response metadata

We created a new meta section in our universal request/response display format.

The section currently includes the total time elapsed. It’s the number of seconds between opening the network connection and downloading the last byte of the response body.

Surprisingly, this was one of the oldest and the most 👍’ed issue in our tracker, and we finally implemented it!

You can use -vv to display everything (whole request/response + meta section) or customize this display with our --print= option.

$ http --print=hm pie.dev/delay/0.5

HTTP/1.1 200 OK
Content-Type: application/json

Elapsed time: 0.856867s

Or only display the meta section with --meta.

$ http --meta pie.dev/delay/1

Elapsed time: 0.856867s

🐰 By the way, if you use one of the new Pie theme styles (e.g., --style=pie-dark), you’ll see the time information color-coded (green/yellow/orange/red) based on how long the exchange took:

📖 Response meta docs →

🚀 Speed-ups

We believe that performance is a critical part of the overall user experience. And HTTPie is all about the user and how they feel when using our tools.

HTTPie for Terminal is designed with interactive API communication in mind. In this context, the top priority has always been making active tasks requiring user cognition, such as defining requests and understanding responses, as fast as possible. The human brain’s I/O is a more precious resource than the computer’s one, and the number of brain cells doesn’t double every two years, regrettably.

Having said that, we consider all forms of performance important and are working hard to reduce also the waiting time for developers, and this release includes multiple speed-ups over different areas:

Startup time is now ~30-50% faster (depending on the environment).
Decoding of streamed responses is ~30% faster.
Downloads (i.e., when using --download) are more than ~2.5× faster — for locally served files, the download speed went from ~320 MB/s to ~680 MB/s.

🐙 Benchmarking code →

🎨 Improved UI/UX

Pie theme

While we are continuously improving HTTPie for Terminal, it is now part of a bigger platform that brings the beauty and simplicity of HTTPie to Web & Desktop, and Mobile. Alongside consistent behavior, we work toward a unified style across these apps. This effort led us to create a new set of styles called the Pie theme:

The theme comes in three flavors:

pie-dark — for terminals with dark background
pie-light — for terminals with light background
pie — for both, but slightly compromises text legibility

$ http -vv --style=pie-dark pie.dev/post hello=world foo:=42

While auto is still the default style (which takes colors from your shell environment), we highly recommend making pie-dark your default style for the best experience:

$ cat ~/.config/httpie/config.json

{
    "default_options": [
        "--style=pie-dark"
    ]
}

🐰 Just like the dynamically colored timings, we color the HTTP method and status code to match with our apps across the platform. You can try the examples below:

Status code colors

These colors are based on the degree of error/recoverability of each status class:

# 1xx informational – blue
$ http -h pie.dev/status/101

# 2xx successful – green
$ http -h pie.dev/status/200

# 3xx redirection – yellow
$ http -h pie.dev/status/301

# 4xx client error – orange
$ http -h pie.dev/status/400

# 5xx server error – red
$ http -h pie.dev/status/500

Method colors

These colors are based on the potential severity of the impact of the request:

# HEAD, GET – green
$ http --print=H GET pie.dev/get

# POST – yellow
$ http --print=H POST pie.dev/post

# PUT, PATCH – orange
$ http --print=H PATCH pie.dev/patch

# DELETE – red
$ http --print=H DELETE pie.dev/delete

📖 Style docs →

Improved stdin handling

Something that will excite our users who like to use HTTPie for Terminal inside their scripts, workflow definitions, or cron jobs, are our new warnings about when there is no incoming data. If you happen to forget passing --ignore-stdin, instead of waiting indefinitely for a stream that will never yield anything, HTTPie will now complain and let you know about this situation.

📖 Scripting docs →

Better errors

We also focused on improving error messages. A typical example is a mistyped hostname:

$ http this.host.does.not.exist

http: error: gaierror: [Errno -2] Name or service not known
Couldn’t resolve the given hostname. Please check the URL and try again.

✨ Various enhancements

Bearer authentication

bearer is now a natively supported authentication type:

$ http -A bearer -a token pie.dev/bearer

📖 Bearer auth docs →

Non-unique header names

Sending and receiving multiple HTTP header lines with the same name is now fully supported. They’re now shown and sent exactly as specified, i.e., multiple headers lines as opposed to a single header with a comma-separated list of values:

$ http --offline pie.dev/get X-Pie:Apple X-Pie:Pumpkin

GET /get HTTP/1.1
X-Pie: Apple
X-Pie: Pumpkin

📖 Repeated headers docs →

Prompted password in sessions

Prompted passwords are now stored in local sessions. Try typing 123 to the given prompt:

$ http -a user --session=./sess.json pie.dev/basic-auth/user/123

And now try without passing the auth option:

$ http --session=./sess.json pie.dev/basic-auth/user/123

Shout out to Sebastian Czech for contributing this feature.

📖 Password prompt docs →

Query params and headers from files

There are two new operators for reading query parameters and headers from files — ==@ and :@:

$ http --offline pie.dev/headers \
  Header-From-File:@files/text.txt \
  param-from-file==@files/text.txt

📖 File loading docs →

Effortless URL-to-command conversion

Keep :// in the URL argument if you want to. Pasting URLs into HTTPie calls has never been easier: just add a space after the protocol name:

$ https ://pie.dev
# => https://pie.dev

📖 URL docs →

JSON types with forms

We added support for basic JSON types with --form and --multipart when using the otherwise JSON-only operators := and :=@, which allows you to quickly toggle between form and JSON requests. This change doesn’t alter the existing behavior, but it makes what was previously implicit/undefined/buggy explicitly supported:

$ http --form pie.dev/post id:=10 'username:="httpie"'

📖 Forms docs →

Auto-stream

If the response headers include Content-Type: text/event-stream, we now auto-stream the response body. So you no longer need to explicitly specify --stream when reading event streams.

📖 Streaming docs →

🪲 Bug fixes

An XML declaration was auto-added to the beginning of each formatted XML response, but not anymore. Now you'll only see it if it's already present in the raw response.
Broken plugins would crash the whole application, yikes! They no longer do.
Automatic syntax highlighting and formatting now works for complex Content-Type headers (e.g., when charset is specified).
Using --raw together with --chunked is now fixed.

🐙 Full changelog →

❤️ Community contributions

Last but not least we’d like to thank the amazing people who’ve contributed to this phenomenal release of HTTPie:

Aaron Miller, Alexander Bogdanov, András Czigány, Batuhan Taskaya, Brad Crittenden, Chad, Ed Rooth, Gabriel Cruz, Gaurav, Jan Brašna, Jan Verbeek, Jesper Holmberg, Marco Seguri, Mark Rosenbaum, Mickaël Schoentgen, Mike DePalatis, Miro Hrončok, Nanashi., Patrick Taylor, Paul Laffitte, Russell Shurts, Sebastian Czech, Vivaan Verma, Vladimir Berkutov, arloan, blueray453, coldcoff, hosseingt, josephworks, peterpt, stonebig, whodidthis, zoulja.

🐙 How to contribute →

Happy API testing, and see you again soon!

But in the meantime…

🎉 Install the latest version of HTTPie for Terminal.
💬 Please report any bugs and share your feedback on 3.0.
🐙️ Star & watch the GitHub repo to learn about new releases.
👉 Join our Discord community and follow @httpie to stay up-to-date.
💁🏻‍♀️ Join the private beta of HTTPie for Web & Desktop, if you haven’t yet.
👩‍💻 We’re also looking for new colleagues in engineering and design roles.

What’s new in HTTPie 2.6.0

Jakub Roztocil — Sat, 23 Oct 2021 15:58:02 +0000

This version of HTTPie introduces numerous improvements to data display and formatting.

Body charset detection

If a message’s Content-Type header doesn’t specify charset, HTTPie now auto-detects it from the body content, and then uses this information to correctly display the text on the terminal.

# Preview a Big5-encoded request w/o charset in Content-Type:
echo  '維基大典維基大典維基大典維基大典' \
    | iconv -f utf8 -t big5 \
    | http --offline pie.dev Content-Type:text/plain

Prior to HTTPie 2.6.0:

POST / HTTP/1.1
Content-Type: text/plain

����j�����j�����j�����j��

With 2.6.0:

POST / HTTP/1.1
Content-Type: text/plain

維基大典維基大典維基大典維基大典

You can learn more about displaying encoding in the docs.

Response charset overwrite

When the server includes an incorrect charset in Content-Type, you can now overwrite that value for display purposes.

For example, the following response incorrectly specifies charset=utf-8:

$ http -F https://git.io/JiOaA

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

����j��]Wikipedia�

We can now fix that with --response-charset:

$ http -F --response-charset=big5 https://git.io/JiOaA

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

維基大典（Wikipedia)

Response MIME overwrite

HTTPie looks at Content-Type to select the right syntax highlighter and formatter for each message body. If that fails (e.g., the server provides the wrong type), or you prefer a different treatment, you can now manually overwrite the mime type for the response with --response-mime. For example, here we’re formatting and colorizing a JSON response as if it were YAML:

$ http --response-mime=text/yaml pie.dev/get

XSSI prefixes no longer break JSON formatting

HTTPie can now handle extraneous data preceding a JSON body.
For example, when an API uses an XSSI prefix.

Prior to HTTPie 2.6.0:

for(;;){"status": -1, "error":"The resource requires authentication."}

With 2.6.0, we highlight the non-JSON prefix, and then correctly format and color the rest:

for(;;)
{
    "error": "The resource requires authentication.",
    "status": -1
}

Duplicate JSON keys are now preserved

JSON doesn’t disallow repeated keys. However, before HTTPie 2.6.0, we’d incorrectly show only the last occurrence as a result of formatting the JSON:

$ http --print=B --offline --raw='{"A": 1,"A":2}' pie.dev

Prior to HTTPie 2.6.0:

{
    "A": 2
}

Starting with HTTPie 2.6.0, we preserve all keys as shown below:

{
    "A": 1,
    "A": 2
}

Other improvements

Silence warnings through using --quiet, -q twice (e.g. -qq) (#1175).
List installed plugin list to --debug output (#1165).

Community contributions

We’d like to thank these amazing people for their contributions to this release: Annette Wilson, Dave, Fabio Peruzzo, Miro Hrončok, Omer Akram, Vincent van ’t Zand, dkreeft, 崔小二, and 黄海.

Originally published on HTTPie blog.

How to navigate an API from the terminal

Jakub Roztocil — Sat, 02 Oct 2021 12:16:56 +0000

Today, we'll be diving into The Movie Database and learning how to navigate its API efficiently with HTTPie, jq, and jid.

When it comes to movie and TV show data, the first thing that comes to mind is likely to be Amazon's IMDB service. IMDB lacks a public, supported REST API though. Historically people have used various URL scraping techniques and well-known paths to get data out of IMDB, but that is very fragile. The Movie Database, on the other hand, has a REST API for developers (with free access) and commercial licensing for money-making projects making it a useful resource.

Dive preparation

When you start from scratch working or playing with a new API, there are three important questions to answer up front:

1. Where's the documentation?

The first question's answer is easy enough — developers.themoviedb.org.

2. What's the root endpoint?

This should be answered by the documentation, but not always. The MovieDB docs don’t mention the root endpoint. Spoiler, the endpoint is api.themoviedb.org. You'll only find that though after you have signed up as a user and requested an API key. It’s then that you are given an example URL with the API key:

https://api.themoviedb.org/3/movie/550?api_key=7e23cee5bfb742e781fccc26b9e9009f

3. Is there authentication and how does it work?

Yes, there is authentication in the form of an API key which is passed in the query parameter api_key.

That's a fake API key we're showing so do remember to substitute it with your own API key if you are following along. You can obtain an API key by signing up for a TMDB account and requesting API access in the settings.

Ready to HTTPie

(If you haven’t already, now’s the time to install HTTPie so that you can follow along.) Let's translate that URL to an HTTPie command first. We'll use the https command to match the protocol from the URL. The first parameter, in the example and as defined in the API docs, is the endpoint and path like so. Let’s try just that, with no authentication:

$ https api.themoviedb.org/3/movie/550

If you run that, you'll get:

HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
    "status_code": 7,
    "status_message": "Invalid API key: You must be granted a valid key.",
    "success": false
}

The HTTP status reflects an unauthorised access but there’s also a dump of response headers and the JSON error response. All because we didn't include the api_key value to authorise the request. That was deliberate, because it's a good idea to see what the error responses look like early on. With this API we get both an HTTP status code for a type of error, an explicit error message and ”success”: false in the JSON body. The API’s own error codes are listed in the documentation

Now we can add in our required authentication. With HTTPie, you can list headers and form values after the URL as name value pairs. What happens to them depends on the characters that separate them.

If they are separated with a : then they go into the HTTP request headers.

If they are separated with an = then they both go into a JSON body (and the request becomes a JSON body POST).

And, this is the one we are interested in, if they are separated with an == they get added to the query string parameters. This is what we want to do with the api_key.

$ https api.themoviedb.org/3/movie/550 \
    api_key==7e23cee5bfb742e781fccc26b9e9009f

HTTP/1.1 200 OK
Content-Type: application/json

{
    "adult": false,
    "backdrop_path": "/rr7E0NoGKxvbkb89eR1GwfoYjpA.jpg",
    "belongs_to_collection": null,
    "budget": 63000000,
    "genres": [
        {
            "id": 18,
            "name": "Drama"
        }
    ],
    "homepage": "http://www.foxmovies.com/movies/fight-club",
    "id": 550,
    "imdb_id": "tt0137523",
    "original_language": "en",
    "original_title": "Fight Club",
    "overview": "A ticking-time-bomb insomniac and a slippery soap salesman channel primal male aggression into a shocking new form of therapy.",
    "popularity": 39.996,
    "poster_path": "/pB8BM7pdSp6B6Ih7QZ4DrQ3PmJK.jpg",
    "production_companies": [
        {
            "id": 711,
            "logo_path": "/tEiIH5QesdheJmDAqQwvtN60727.png",
            "name": "Fox 2000 Pictures",
            "origin_country": "US"
        },
    ],
    "production_countries": [
        {
            "iso_3166_1": "US",
            "name": "United States of America"
        }
    ],
    "release_date": "1999-10-15",
    "revenue": 100853753,
    "runtime": 139,
    "spoken_languages": [
        {
            "english_name": "English",
            "iso_639_1": "en",
            "name": "English"
        }
    ],
    "status": "Released",
    "tagline": "Mischief. Mayhem. Soap.",
    "title": "Fight Club",
    "video": false,
    "vote_average": 8.4,
    "vote_count": 21654
}

Before we move on, a quick pro-tip: never forget that your command line shell can save you typing. Export the api_key section as an environment variable like this:

$ export API_KEY="api_key==7e23cee5bfb742e781fccc26b9e9009f"

and then you can just do:

$ https api.themoviedb.org/3/movie/550 $API_KEY

Working with responses

There's a lot of JSON in the response we just got and, thanks to HTTPie, it is both formatted and syntax-coloured for easier reading. That formatted and coloured view is the default when HTTPie is outputting to a console. If you want to page through the results, you would usually pipe the output through the more or less commands.

$ https api.themoviedb.org/3/movie/550 $API_KEY | less

But if you do that, you'll see unformatted, colour-less output. That is because the HTTPie default for output to a pipe or anything that isn't a console is to leave the content untouched for other commands to consume.

If you want to force formatted output, add --pretty=format to the command:

$ https --pretty=format api.themoviedb.org/3/movie/550 \
    $API_KEY | less

Now you can read the formatted results at your own pace. If you want the colour back too, use --pretty=all and the -R flag of the less command:

$ https --pretty=all api.themoviedb.org/3/movie/550 \
    $API_KEY | less -R

You'll now get formatted and colourised output you can page through.

There is one other difference between the defaults for output to the console and redirected output. Headers are output to the console, but skipped when outputting to another program through a pipe. If you want only headers output to a pipe, use the -h option.

You can learn more about HTTPie's default options for terminal and redirected output in the Terminal Output section of the documentation. If you want to take direct control of what is output, check out the Output Options. They control which parts of the request/response exchange are output.

Digging into responses

If you're trying to work out the structure and content of an API's JSON responses, you can keep paging through the documentation and the paged output of less or you can reach for more precise JSON parsing tools such as, jq and jid.

With jq, a command line JSON processor, you can write expressions to extract and format data from JSON streams. jid is an interactive JSON digger which lets you use some of jq's expressions and auto-completion to explore JSON files. For example, if we run

$ https api.themoviedb.org/3/movie/550 $API_KEY | jid

Up comes a view of the JSON, formatted and colourised. Control-N and Control-P will let you page up and down through the returned data, but that's just the start.

Say, for example, we are looking for which country the production occurred in. Start typing a name for a field like pro, and you'll see jid offers up production_co as a potential autocomplete. Tap Tab and you'll be able to alternate between production_companies and production_countries.

Hit return and jid displays the contents of the production_companies array. Tap Tab again and you'll be able to enter an array index 1 and Tab again to display just one entry from the array. Type . and you can auto-complete through that entry’s field names. jid is great for locating data in nested structures and the queries you use work in the command line with jq.

Now with these two tools to hand, you're ready to explore more of the MovieDB API.

Finding more about a movie

While there are endpoints for every movie under the /movie/ path — they are located with an id — there are also subsections to each movie entry for release dates, keywords, ratings, credits, all with a path that comes after the id. So if you want to get the credits for a movie, you'd call:

$ https api.themoviedb.org/3/movie/550/credits $API_KEY

This does seem to be a query for, potentially, just a small snippet of data. That's where append_to_response in the MovieDB API comes in. This lets you add, for particular endpoints like /movie, the various subsections. So, if we want a movie’s credits and images embedded into our response, we do this:

$ https api.themoviedb.org/3/movie/550 \
    $API_KEY \
    append_to_response==credits,images

It only costs us one request and merges all the data together into one JSON object.

Finding a movie

What about finding a movie? For that, there's a search endpoint, /search/movies. It takes a query parameter which "must be URI-encoded". If your query value has a space in it, you can use quotes around the value and HTTPie will take care of URL-encoding the value for you. No hand-encoding spaces to &20 or similar. All we need to do is remember to put quotes around our value like so:

$ https api.themoviedb.org/3/search/movie \
    $API_KEY \
    query=="O.C. and Stiggs"

That'll return JSON data designed for paginated search results. If we wanted to look up more about the movie, we'd want the id from the first result. For that we can pipe the results to jq ".results[0].id" and we'll get the movie id. From there, it's a call to the /movie/{id} to get the movie details:

$ MOVIE_ID=$(
    https api.themoviedb.org/3/search/movie \
        $API_KEY \
        query=='O.C. and Stiggs' \
        | jq .results[0].id
)

$ https api.themoviedb.org/3/movie/$MOVIE_ID $API_KEY

Or, if you want to pack it all into one line:

$ https $(
    https api.themoviedb.org/3/search/movie \
        $API_KEY query=='O.C. and Stiggs' \
        | jq -r '.results[0] | "api.themoviedb.org/3/movie/\(.id)"'
) $API_KEY

Discovering movies

One of the richest API endpoints in The Movie DB API is /discover/movie. It lets you specify over thirty different properties for searching across films.

So say you want the most popular science fiction movies on The Movie Database. Genres are represented in the API with an integer id so we need to look up what the id for “Science Fiction” is. The full genre list is available on another endpoint, /genres/movie/list:

$ https api.themoviedb.org/3/genre/movie/list $API_KEY

HTTP/1.1 200 OK
Content-Type: application/json

{
    "genres": [
        {
            "id": 28,
            "name": "Action"
        },
        {
            "id": 12,
            "name": "Adventure"
        },
        {
            "id": 16,
            "name": "Animation"
        },
[…]

You can manually work your way through the list or use jq to find the entry we are interested in:

$ https api.themoviedb.org/3/genre/movie/list $API_KEY \
    | jq '.genres[] | select(.name=="Science Fiction")'

{
  "id": 878,
  "name": "Science Fiction"
}

So that tells us the genre id we are looking for is 878. Back at the /discover/movie endpoint, there’s a property called with_genre which we’ll set to 878. Another property available on the endpoint is sort_by which takes a field name and a sort order. We’ll set that to popularity.desc:

$ https api.themoviedb.org/3/discover/movie \
    with_genre==878 \
    sort_by==popularity.desc \
    $API_KEY

HTTP/1.1 200 OK
Content-Type: application/json


{
    "page": 1,
    "results": [
        {
            "adult": false,
            "backdrop_path": "/9yBVqNruk6Ykrwc32qrK2TIE5xw.jpg",
            "genre_ids": [
                28,
                14,
                12,
                878
            ],
            "id": 460465,
            "original_language": "en",
            "original_title": "Mortal Kombat",
            "overview": "Washed-up MMA fighter Cole Young, unaware of his heritage, and hunted by Emperor Shang Tsung's best warrior, Sub-Zero, seeks out and trains with Earth's greatest champions as he prepares to stand against the enemies of Outworld in a high stakes battle for the universe.",
            "popularity": 5817.001,
            "poster_path": "/xGuOF1T3WmPsAcQEQJfnG7Ud9f8.jpg",
            "release_date": "2021-04-07",
            "title": "Mortal Kombat",
            "video": false,
            "vote_average": 7.7,
            "vote_count": 2266
        },
…
    ],
    "total_pages": 500,
    "total_results": 10000
}

The results are paginated and this is page 1 with 20 results a page. If we want to compress this down to a simple list, we can use jq again:

$ https api.themoviedb.org/3/discover/movie \
    with_genre==878 \
    sort_by==popularity.desc \
    $API_KEY \
    | jq -r '.results[] | "\(.title) \(.popularity)"'

Mortal Kombat 5817.001
Godzilla vs. Kong 3608.866
Tom Clancy's Without Remorse 4266.181
Nobody 2993.014
Vanquish 3156.355
Zack Snyder's Justice League 1992.158
…

That snippet of jq just extracts and formats the title and popularity from the results array for each item. The -r option just stops jq wrapping all its output in quotes.

Wrapping up

Creating queries on The MovieDB API is quick thanks to HTTPie and with jq and jid, you can make sense of the results with the least amount of fuss.

In the next part of this dive, we’ll look at how TMDB’s API now handles user-generated lists and how HTTPie makes it simpler to explore.

Written by Dj Walker-Morgan and originally published on HTTPie blog.

Command-line API testing tricks

Jakub Roztocil — Wed, 15 Sep 2021 11:16:31 +0000

Whether building or consuming APIs, you're definitely testing out API calls. The command line is a developer's best friend for quick checks and iterative development. You can move even faster with HTTPie. HTTPie releases unintuitive command flags from your brain and removes repetitive parameters completely.

This post will give some quick examples to make the most of your command line API development. Use these tips and tricks to ease your API headaches and bring some fun into your workflow.

Make JSON a first-class citizen

For at least a decade, JSON has been the de facto standard response format for HTTP APIs. Strangely, a lot of tools give equal weight to other formats. HTTPie comes with sensible defaults, which assume data is expressed in JSON.

After the fast and easy installation you can retrieve JSON responses with a simple command line call:

$ http api.open-notify.org/astros.json

When you run this command you’ll get a list of the people currently in space, thanks to Open Notify. By default, HTTPie makes your API responses readable. The results of this call are displayed in syntax-highlighted JSON. And no need to pipe to another formatting tool or ask the server to pretty print the results.

Whether request or response data, HTTPie always looks for JSON first, unless told otherwise. That means that Content-type and Accept headers are set to application/json. It will even attempt to find JSON within text/plain or unknown content types.

As much as developers love JSON, we don’t always enjoy writing it, especially on the command line. Use flexible command line parameters to construct your JSON without curly brackets, quotes, and a lifetime supply of backslashes.

Here’s a sample call to send some basic JSON, without any of the obtrusive syntax:

$ http POST pie.dev/post name=Jason year:=2001

When you fire off that command, HTTPie interprets the field=value pair as a string and field:=value as a literal (in this case, a number). All of these parameter pairs turn into the JSON data POSTed to the URL:

{
  "name": "Jason",
  "year": 2001
}

For more complex data, you can include external JSON or text files. But you can stop struggling with JSON syntax, escaped quotes, and the like, just to express simple request data. There’s a natural progression of complexity, allowing HTTPie to handle one advanced field (say, an array) while maintaining a simple command line syntax. Refer to the JSON fields documentation for guidance on these approachable details.

Use natural API-native commands

Nothing against the other command line request tools, but they weren’t built for APIs. Some of these CLI relics have been around almost as long as the Web itself. In an effort to support every protocol, data format, and usage pattern, they can make it difficult to accomplish the everyday tasks of API developers. HTTPie was built alongside modern API development to be a natural extension of your work.

JSON as a first-class citizen is certainly an API-native choice. In addition, HTTPie pays homage to the primary protocol used to access APIs—HTTP. The interface design of many HTTPie commands mimic the underlying HTTP requests.

For example, this command:

$ http POST pie.dev/post \
    X-API-Key:123 \
    User-Agent:Bacon/1.0 \
    name=Jason year:=2001

Will be sent via HTTP as the following:

POST /post HTTP/1.1
Host: pie.dev
X-API-Key: 123
User-Agent: Bacon/1.0
Content-Type: application/json

{
    "name": "Jason",
    "year": 2001
}

Request headers are communicated almost identically, without the need for special command flags. This simplicity is made possible by a minimal DSL for request items that flows naturally. The : for headers comes from HTTP, as does = for data. In fact, the power of the syntax is clear with a simple tweak to a commend: add a -f flag and the exact same data fields will be sent form-encoded:

POST /post HTTP/1.1
Host: pie.dev
X-API-Key: 123
User-Agent: Bacon/1.0
Content-Type: application/x-www-form-urlencoded

name=Jason&year=2001

If you want to see how these HTTP calls look without making live calls, you can enter --offline mode. Rather than send a request to the host, HTTPie prints it to stdout.

Offline mode is a great way to see some of the other implicit choices HTTPie makes:

POST is sent automatically when request data is present
You can remove http:// and https:// from your request URLs
Or simply use : to send requests to localhost

The natural syntax keeps your head in your API development and your eyes out of man pages. That lets you focus on the bigger issues, like getting API authentication to work.

Don’t repeat your authentication

You may be familiar with the DRY principle in software development. Meant to reduce repetition, DRY stands for Don’t Repeat Yourself. Typical code-level approaches include modularization and similar abstractions. By contrast, each API request stands on its own. Every time you make an API request you must include any authentication credentials, even if you sent them in a previous request. For example, if your API requests are to a cloud storage service, every request that accesses your account needs your credentials for that service. This is repetitive and inconvenient.

As an API client, HTTPie must abide by the API server’s rules. While the server must receive credentials fresh each time, HTTPie can store them to send on your behalf.

Use the session functionality, so you don’t repeat your authentication. Session functionality is convenient when you pass API tokens in headers or as part of a request URL. In either case, these get in the way of the rest of your API call.

When you make your first call, include the authentication, headers, and whatever else you want to send with every call. Then use the --session flag to give the session a name:

$ http --session=dev -a :APIKEY pie.dev/get X-Random-Header:Included

Here we’ve included an API key via Basic Auth and a custom header. We also created a session named “dev.” Subsequent calls will only need to include the session flag to pass the authentication and header from the previous request.

This command has the same results as the more verbose command:

$ http --session=dev pie.dev/get

The API key authentication was attached to the HTTP call, even though we did not explicitly include it in the command. Similarly, the X-Random-Header that took up so much space is sent without being explicitly present.

Sessions make for much cleaner commands, especially when you start sending query parameters or request data. Once you have the session flag in place, you don’t repeat yourself. And you can be done with the copy-pasting or tedious command line editing that so often accompanies API tinkering.

An alternative to named sessions is to store session details within local files. You can share these session files amongst your team, so you always get the same results. Then call them up by file path, or use the trick in the next section—include them in your default configuration.

Create your own defaults

It can be an adventure to watch someone develop on their own machine. You’ll notice seemingly random key combinations produce extraordinary results (but pure chaos when a Vim user mistakenly thinks they're in insert mode). These sorts of customizations can help you get the most out of any dev tool. You’ll appreciate the custom defaults you can add to HTTPie to make it your own.

The latest configuration file options and location are in the HTTPie docs.

For example, you can enable sessions by default, so credentials and custom headers from the previous sessions are automatically sent with each request to the same host. To do this, you include the --session flag in default_options. A unique session file will then be created for each host, which you can then modify. Your config file will look something like this:

{
    "default_options": [
      "--session=default"
    ]
}

Whenever you need to overwrite the defaults, include another session flag explicitly or use a --no-session flag to ignore any existing session data.

Your config defaults can include any flag options as an array, like so:

{
    "default_options": [
      "--session=default",
      "--body",
      "--style=colorful",
      "--format-options=json.indent:2"
    ]
}

In this example, we have the session flag as before, but we also have:

included --body to display only the response body (no headers);
set the syntax highlighting --style to a defined color scheme;
used --format-options to set indentation equal to two spaces;
and instructed HTTPie to --follow redirects to a new location.

Using defaults, all of these flags will be included every time you use the http command, unless you specifically override them.

You can find a categorized list of command flags, which include options for downloading data, using SSL, and employing proxies, with the following HTTPie call:

$ http --help

Within the list, you’ll find a balance of intuitive defaults with the flexibility to make it your own.

Please note that changing default_options may lead to unexpected inconsistencies when sharing HTTPie commands with other developers. You can temporarily disable custom default_options and verify HTTPie’s default behavior by setting $HTTP_CONFIG_DIR, for example:

$ HTTPIE_CONFIG_DIR=/tmp http pie.dev/get

Get fewer API headaches

HTTPie started in 2012 to eliminate our own pain when building and calling APIs. Thousands of developers add it to their command line and use it to perform effortless requests with beautiful responses.

Learn more about HTTPie and follow us for more tips, news, and API tricks.

Written by Adam DuVander and originally published on HTTPie blog.