DEV Community

loading...

Unbearably slow API gateway calls between Azure App Services

buinauskas profile image Evaldas ・2 min read

Hello everyone,

I'd like to ask for help debugging ASP.NET Core Web API.

That's the situation:

We've got an authorization service that exposes multiple RESTful APIs to check if authenticated user can access certain business objects. In my case the business is object is a concept of market and they are authorized by project. So API call would look as follows:

GET https://authorization.service.com/markets?project=123456
Enter fullscreen mode Exit fullscreen mode

Successful response will return array of markets, unauthorized response will return a 403 status code.

It all works great when I call it through Swagger, call it from Postman - responses are quick and come back in tens of milliseconds.

Now my other service is calling this service in authorization filter and based on response either continues or responds with Forbidden.

I've implemented a reusable client for HTTP calls:

public class ApiClient : IApiClient
{
    private readonly HttpClient _client;
    private readonly JsonSerializer _jsonSerializer;

    public ApiClient(HttpClient httpClient, JsonSerializer jsonSerializer)
    {
        _client = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
        _jsonSerializer = jsonSerializer ?? throw new ArgumentNullException(nameof(jsonSerializer));
    }

    public async Task<T> GetDataAsync<T>(string uri, string accessToken)
        where T : class
    {
        var request = CreateRequest(uri, accessToken);
        var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

        if (!response.IsSuccessStatusCode)
        {
            return default(T);
        }

        using (var responseStream = await response.Content.ReadAsStreamAsync())
        using (var streamReader = new StreamReader(responseStream))
        using (var jsonTextReader = new JsonTextReader(streamReader))
        {
            return _jsonSerializer.Deserialize<T>(jsonTextReader);
        }
    }

    private HttpRequestMessage CreateRequest(string uri, string accessToken)
    {
        var locationUri = new Uri(uri);

        var message = new HttpRequestMessage(HttpMethod.Get, locationUri);

        message.Headers.Add("Authorization", $"Bearer {accessToken}");

        return message;
    }
}
Enter fullscreen mode Exit fullscreen mode

My authorization filter looks as follows:

public class AuthorizeProjectFilter : IAsyncActionFilter
{
    private readonly IApiClient _apiClient;
    private readonly IMarketsContext _marketsContext;

    public AuthorizeProjectFilter(IApiClient apiClient, IMarketsContext marketsContext)
    {
        _apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
        _marketsContext = marketsContext ?? throw new ArgumentNullException(nameof(marketsContext));
    }

    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        if (
            context.HttpContext.Request.Query.TryGetValue(nameof(RequestBase.Project), out var value) &&
            context.HttpContext.Request.Headers.TryGetValue("Authorization", out var authorizationHeader)
            )
        {
            var project = value.ToString();
            var accessToken = authorizationHeader.ToString().Split(" ")[1];

            var markets = await _apiClient.GetDataAsync<IEnumerable<Market>>($"{_authorizationServiceOptions.Authorization}/markets?project={project}", accessToken);

            var authorized = markets.Any();

            if (!authorized)
            {
                context.Result = new ForbidResult();
            }
            else
            {
                _marketsContext.Markets = markets;
                await next();
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

And this works great locally, requests are quick when I call this service, however when I deploy this to Azure App Service - responses take 20+ seconds, sometimes more than a minute. App plan isn't also the cheapest one and it should be fast.

That doesn't seem like a cold start issue because I can keep on calling the service - it still remains slow.

I've run a trace on Azure diagnostics and got the following:
Alt Text

Seems like most of the time is spent on BLOCKED_TIME within system.net.http and I'm literally lost here.

Tracing has also caught some exceptions with the following message:

An attempt was made to access a socket in a way forbidden by its access permissions
Enter fullscreen mode Exit fullscreen mode

Also my HttpClient is added using IHttpClientFactory:

services
    .AddHttpClient<IApiClient, ApiClient>(client =>
    {
        client.DefaultRequestHeaders.Add("Accept", "application/json");
    })
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
    {
        UseProxy = false
    });
Enter fullscreen mode Exit fullscreen mode

I'm literally lost now, I couldn't find anything on Google that would've helped me.

Please help :)

Discussion (8)

pic
Editor guide
Collapse
buinauskas profile image
Evaldas Author

Took ages, but I finally managed to find that the issue is corporate VNet that application is connected to. 🙂

Collapse
vantubbe profile image
Mike Van Tubbergen

Any possibility you could provide the details as to what was wrong with the VNet? We are currently experiencing this exact issue.

Collapse
buinauskas profile image
Evaldas Author

Hey, yes.

So the actual issue was VNet and that it used our own DNS.

Any outgoing external request would be routed through a service that would check if address is not blacklisted by our IT department. Can't remember how it was called, but it was this exact service that caused it.

Hope that helps. If you need more details, let me know.

Thread Thread
vantubbe profile image
Mike Van Tubbergen

I tried removing the route table from the Subnet the App Service was running on, but still encountered the same issue. Was there another change needing to be made, perhaps on the VNET/subnet itself, or the way the company created it internally?

Thread Thread
buinauskas profile image
Evaldas Author

To be honest we went with a totally different approach.

The only solution I was told was not to check these outgoing addresses at the company level, which we obviously didn't do.

The service we called was internal and we implemented event driven architecture to not rely on it and to be more fault tolerant.

Collapse
devloco profile image
devloco • Edited

I think the "socket access" error is the big clue. It's probably not your code (although full disclosure, it's late here, I can't sleep... so your code hit my TL;DR filter). :-)

But it definitely feels like an OS level issue and not an issue with your code. Assuming you're running some form of *nix (including MacOS), most users and OS processes are not allowed to run on any port lower than 1024, without human intervention on the config files.

Other alternative ideas:

  • there is another service running on the same port that you're try to dev on
  • some form of firewall is trying to be "helpful" by blocking whatever port you're trying to run the dev server on.
  • simple folder/file permissions. Whatever user your WWW server is running as... that user doesn't have proper access to the folder/files

Since the error specifically mentions ports, I'd start there. You're using an already occupied port, or you have a firewall setting blocking the port... etc.

Collapse
devloco profile image
devloco

And, of course, I missed the fact you're running in Azure. In that case, definitely check the firewall. Azure VMs are pretty strict about the ports that they can use. (I'm assuming you're in a VM of some sort and not an Azure Serverless Function type thing).

You can open ports from the Azure portal.... I don't remember the exact nav-path in the Azure portal, but Googling "open port on azure firewall" or some such should get you there.

Collapse
buinauskas profile image
Evaldas Author

It's an azure app service, not a virtual machine. It really is IIS as service for a single application to run. I'll see what possible network restrictions are there.