Rate limiting API calls in cloud

Dom — Fri, 10 Jun 2022 19:07:14 +0000

1. This post covers the process of setting up API Management (Layer 7 Load balancer) in Azure.

2. It will also cover how to setup Subscription Key rate limited endpoint which would set you up for providing API SAAS.

I wanted to document this process when I was setting up a custom rate limited search cluster that was called from My fronted App while providing 2 endpoints.

1st endpoint being free.
2nd endpoint being available only with valid API key.

This post assumes you have already created Azure resource group and also deployed your backend service as Azure app service inside it.

After you deploy your App Service and API Management service to your Resource group, it will look similar to this.

Importing and publishing your first API is explained HERE

Start by importing an Azure Web App as an API.

Which is explained HERE

After that you can click on your added API endpoint and provision your desired policies.

After that you can set if API Key is needed to Access your endpoint.

Endpoint policies exported as XML

Paid endpoint LB policy:

"set-backend-service" is set by default when we added New API attached to the App service.
"quota-by-key" here is set to track successful 200 OK responses and also to allow max 1000 API calls every 29 days (2505600 sec)
"set-header" is appended in Load balancer before request reaches backend service as this custom header is required by my service

Free endpoint LB policy:

"rate-limit-by-key" is actually IP rate limit allowing 5 requests every 300sec

CORS Policy on all endpoints:

here we set which HTTP methods are allowed
which domains are allowed to access our endpoints
"preflight-result-max-age="300"" caches our CORS preflight (for 300sec) request so that each subsequent request from Frontend App has one less network hop to do.

With this setup we would block immediately on Load balancer endpoint (even before it reaches App service endpoints).

Beware by default APIM puts CORS policy on Load balancer endpoint(effects all API Endpoints)… so check and update default if needed.

Policies documentation:

Cross domain policies
Advanced Policies

CORS caching : Learn more

Preflight OPTIONS request example

Managing API key subscriptions

MORE About it HERE.

Contact

Twitter
Linkedin

Generating HMTL and MD files from .TXT in GO

Dom — Sun, 13 Mar 2022 16:53:09 +0000

In this post I'll be describing the code which I wrote to generate HTML (which can be hosted anywhere) and Markdown files from simple .txt file.

Idea for project

Idea for this project came from a need to quickly generate and share some basic documentation about task or bug needed to be solved by team member.

Working mostly as backend engineer, naturally my 1st idea was to build CLI tool that could easily do this by inputting file path and letting the code do the rest.

Why GO

I never wrote Go code before but I watched few Go conf's and talks that sparked interest so this was me trying it out.

I really loved how simple it is to execute code in parallel in GO with goroutines, channels & wait groups, so that was the reason for doing this in GO.

Main idea

Main idea I followed for this project was to read from .txt file line by line and at the same time write to MD & HTML files using Go's goroutines & channels.
This worked out fine for the most part, but I had to make some workarounds for 3rd party libraries which required code blocks and tables syntax to be passed all at the same time instead of line by line.
So I had to come up with a way to detect code blocks and tables, than concatenate them into single block using Go's strings.Builder package.

3rd party libraries:
goldmark
for converting MD syntax to HTML

quick
for generating Html and syntax highlighting code blocks

Code

Entry point to CLI tool:

In above code I'm declaring map/dictionary of all available custom mappings I will be exposing and parsing from .txt file to MD & HTML format.

Than I'm validating user input Args from CLI in 'shouldExit' func.

Than I'm validating file path and extension.

After that I'm creating 2 channels.

One 'writerChannel' that I'm reading/writing from.
And 'filesCreated' that I'm using to signal when reading/writing is done to main goroutine so it can stop blocking and continue executing.

Reading/writing to files

In separate goroutines

Detecting Table and Code blocks

This is the code that handles writing to MD & html formats.

'htmlStleBuilder' function handles what theme and styling to construct from static files based on passed CLI Args.

err := quick.Highlight(...)

line handles Html syntax highlighting generation for code blocks.

And finally signaling to main goroutine by reading 'done' from 'filesCreated' channel that it can proceed executing after writing and reading is complete.

<-filesCreated

Full method on Github

Source code

Github Repo

Running code locally: (with test files form ./testData directory)

Generate Html/md files to %HOMEDIR%/Downloads directory:

go run . ./testData/test.txt

selecting custom theme:

go run . ./testData/test.txt mid

Getting all currently available mappings:

go run . -options (or -o)

Note: Currently line break/newline is expected between each mapping element to be parsed into MD/HTML correctly!

Running CLI command globally:

If you want to run this tool globally you can place txtToMD.exe file to 'Path' in windows environment variables.
that way you can use it from anywhere in your system

txtToMD {txt File Path} [options:light(default),dark,mid]

txtToMD D:\Desktop\test.txt dark

Getting all currently available mappings:

txtToMD -options

Comments

Feel free to leave comments if you see some critical antipattern or mistake.
As I said this was my 1st GO project made for fun so it can probably be improved.

About me

I currently work as backend/fullstack engineer mostly in C#,JS,TS.
With focus on distributed systems area at the moment.

In free time I have been exploring and experimenting with Go and Rust because they seemed fun and enjoyable to work with.

Contact

Twitter
Linkedin

DEV Community: Dom

Rate limiting API calls in cloud

Importing and publishing your first API is explained HERE

Start by importing an Azure Web App as an API.

Endpoint policies exported as XML

Paid endpoint LB policy:

Free endpoint LB policy:

CORS Policy on all endpoints:

With this setup we would block immediately on Load balancer endpoint (even before it reaches App service endpoints).

Policies documentation:

CORS caching : Learn more

Preflight OPTIONS request example

Managing API key subscriptions

Contact

Generating HMTL and MD files from .TXT in GO

Idea for project

Why GO

Main idea

Code

Reading/writing to files

Detecting Table and Code blocks

Full method on Github

Source code

Running code locally: (with test files form ./testData directory)

Running CLI command globally:

Comments

About me

Contact