Maximizing Efficiency with Cloudflare's AI Gateway

Hugo Romano — Tue, 31 Oct 2023 16:33:01 +0000

In the contemporary realm of AI applications, efficiency and control are paramount. Cloudflare's AI Gateway emerges as a game-changer, offering remarkable advantages: observability, caching and rate-limiting.

The cost-saving aspect is excellent, serving requests directly from Cloudflare's cache rather than the original model provider (if cached). Additionally, the in-built rate limiting feature is a robust measure against potential abuse, ensuring your application scales gracefully without being overwhelmed by excessive requests and burning your wallet.

I use it to interact with OpenAI API, but this Gateway also supports Cloudflare Workers AI, Azure OpenAI, HuggingFace and Replicate. In addition, there is a Universal Endpoint that helps you set up a fallback provider when a request fails.

The Power of Caching

Caching can be set from one minute to one year, and in my case, it caches from text-embeddings to gpt-3.5. In the above screenshot of a non-production app, the savings were 50% instead of calling OpenAI directly, plus faster response speeds.

Rate-limit your Wallet

Every time you release your App into production, you question if some bad actor will run you to the ground. The Rate-limit feature will enable you to control the frequency and volume of requests made to your app, thereby avoiding any unnecessary charges or misuse of resources. Rate-limit from 1 minute to 1 hour period, fixed or sliding.

Observability

In the case of OpenAI API, observability is very limited. The Real-time logs feature on the Gateway is bliss. You can turn it on or off as you like.

Getting Started

Visit Cloudflare's AI Gateway Docs and if you use OpenAI NPM Package is a easy as setting up baseURL.

import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: 'my api key', // defaults to process.env["OPENAI_API_KEY"]
    baseURL: "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai"
});

IPFS:🦠Pandemic resilient infrastructure

Hugo Romano — Thu, 19 Mar 2020 18:30:47 +0000

This is a unique moment for Humanity, that will touch everyone. "Shoulder to shoulder" (social distanced) we will support each other.

Remote working is now mandatory for many nations and #StayAtHome 🏠 is the most important strategy against SARS-CoV-2, flattening the curve of CoViD-19 cases, but this move will stress the Internet infrastructure.

One thing we can do about this is to use IPFS (The InterPlanetary File System) and other distributed filesystems to store and safeguard relevant public health literature, e.g. situation reports, guidelines, scientific papers. In the case of network intermittence, we will all have this data available.

Here are some examples:

World Health Organization CoViD-19 Situation Reports
QmPaidwPiMKfqgdz9f39hKdM4rcgJBymRYTHtxgKG1tR2A

WHO Coping with stress during the 2019-nCoV outbreak
QmTWsDwfRvmLbG1vs2c3UncZXCGFBNTv1jcPvTNaZaNxZ5

Portugal CoViD-19 Situation Reports
QmYCoAKArCKLQAwzYQ2VdYnL4dP9tCrhD97JQuqUQCBqAb

It doesn't stop here, your creativity will find more use cases.