DEV Community: Dan P

AWS ALB and ECS Service Connect unexpected behaviour

Dan P — Mon, 11 Mar 2024 21:41:03 +0000

We had an issue, an issue that took many hours of diagnosing... I'm hoping this will help others so i'll keep it simple.

If you are using Kong and ECS Service Connect and see strange issues relating to the Request headers not being as you expected....

TL;DR

If you are using Service Connect in ECS, an ALB that is routing traffic to your service, WILL USE SERVICE CONNECT to transmit traffic to your service (there is no mention of this in the ALB and the ALB is not registered in any Service Connect namespaces). You need to handle traffic that is now coming from localhost and not the ALB as you'd expect.

Background (details have been simplified)

ECS Service running on port 8000 (a reverse proxy) - Kong Api Gateway
An ALB routing traffic to a Ruby on Rails app on port 8000
Kong "trusts" requests from its own VPC (containing ECS and ALB) i.e. trusted_ips=10.0.0.0/16,<IPs if trusted infrastructure, CloudFlare et al>)
Kong proxies requests to Rails via an internal ALB
Kong "sees" request coming from a private VPC IP address of the load balancer (i.e. 10.0.54.147)

all works fine.

Then you want to use Service Connect...

If you enable Service Connect, so that Kong sends traffic to Rails (which behind the scenes routes traffic into Kongs envoy sidecar container and comes out Railss envoy container; everything works and the Rails ALB is no longer required, awesome.

BUT...

We began to see ActionController::InvalidAuthenticityToken errors from the Rails app, as Kong isn't sending the correct client IP headers (X-Forwarded-Proto was incorrect in our case)

We trust the CIDR address of our ALB (and the rest of the VPC) which allows Kong to correctly set all the requiredX-Forwarded- headers, but this RELIES on traffic coming from this trusted CIDR.
https://docs.konghq.com/gateway/latest/how-kong-works/routing-traffic/#proxying-and-upstream-timeouts

In the case where $realip_remote_addr is one of the trusted addresses, the request header with the same name gets forwarded if provided. Otherwise, the value of the $scheme variable provided by ngx_http_core_module will be used."

Suddenly incorrect header values were getting to Rails and causing the above error.

Kong also starts seeing requests from the ALB to be actually from "127.0.0.1 (localhost)". The only way requests can be coming OUT of the Envoy sidecar at Kong is if the ALB is using Envoy too!? which isn't documented.

Because we trust the VPC CIDR, we did not trust 127.0.0.1 and request began to fail in very unexpected ways. As soon as we started trusting 127.0.0.1 everything was good again

Clues

After adding 127.0.0.1 to the trusted IPs, I wanted to find HOW this would fix it.

This comment from AWS gave a huge clue...

"you are able to have the ALB traffic 'bypass' the Service connect agent by setting a different port for the Service Connect Agent (ingressPortOverride defined here) which will allow non-Service Connect traffic (like Load balancer traffic) to not be intercepted by the Service Connect Agent.." https://github.com/aws/containers-roadmap/issues/1958#issuecomment-1654650077

Service Connect "intercepts" traffic, even if the service (in our case an ALB) is not configured in any way to use Service Connect (because this just isn't an option in the AWS console).

ingressPortOverride is really well documented here. With no mention of ALBs using the Envoy Tunnels ...https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceConnectService.html

"The port number for the Service Connect proxy to listen on.

Use the value of this field to bypass the proxy for traffic on the port number specified in the named portMapping in the task definition of this application, and then use it in your VPC security groups to allow traffic into the proxy for this Amazon ECS service."

No mention of ALBs self inviting themselves to the Service Connect party.

Unit Testing Complex Jsonnet Objects and Arrays

Dan P — Mon, 01 Jun 2020 17:25:34 +0000

A Problem

As part of our deployment strategy we generate kubernetes manifest json files from templates. For this we use Jsonnet. We need confidence that the "rendered" json files are correct depending on configuration that has been given to it. Therefore, we unit test them. We use the really helpful JsonnetUnit to achieve this. It's a really handy library that behaves like a unit testing runner, but is written entirely in jsonnet.

One issue we had was that JsonnetUnit compares objects in their entirety. For example, to assert the state of pod environment variables in a kubernetes manifest, you need to compare the entire list of env vars.

testEnvVars: {
    actual: deploy.spec.template.spec.containers[0].env,
    expect: [
            { name: 'ENVVAR1', value: 'VALUE1' },
            { name: 'FEATURE_A', value: 'true' },
        ]
  },

Now this is easy enough if you have a small list of env vars, but if the intention of your unit test is to check if some extra values are provided when a feature is turned on, you dont want to have to compare the entire list of environment variable for the sake of a checking a couple of new ones exist.

A Solution

The following jsonnet code behaves like an "array contains" function. It returns a boolean response depending if the array (haystack) contains the required objects (needles).

As you might expect, when working in Json as a testing language (🤨), it's not a simple as calling a lookup function on the array. Kubernetes doesn't make life easier either, as env vars are stored as an array of objects rather than the more traditional form of a map of key, values.

Because of this, the function needs to know the structure of the object it is looking for in advance. Which is fine until you want to do a similar test for a slightly different shaped object (SecretEnvVars et al.), which requires the code to be copy pasted and amended.

{
  // containsEnvVars checks that all needles (k8s env variable "name" and "value" objs) exist in the haystack (pod env block) object
  // i.e. asserts array contains n objects
  containsEnvVars(haystack, needles):: (
    // create an array, 1 element per needle
    local results = [
      // check the needle is valid (it contains the two required fields)
      if (std.objectHas(needle, 'name') && std.objectHas(needle, 'value')) then
        // needle is valid

        // create a sub array to contain result of every haystack item for the current needle
        local h = [
          // if the haystack name and value both match the needle, add 'true' to the sub array
          if (needle.name == haystackItem.name) && needle.value == haystackItem.value then true
          for haystackItem in haystack
        ];
        // strip out the nulls, leaving only a true for the items where the needle and haystack match
        local prunedResults = std.prune(h);

        if std.length(prunedResults) == 0 then
           error "needle: " + needle + " was not found in the haystack"
        else
          true
      else
        // needle is not value, raise an error
        error 'needle ' + needle + ': doesnt contain the fields "name" and "value"'
      // body of the loop is the above logic
      for needle in needles
    ];

    // remove the empty sub arrays, these are where a needle was not matched to a haystackItem
    // therefore indicate a needle wasn't found
    local prunedResults = std.prune(results);

    //the number of results (bool true) remaining should equal the number of needles, indicating they were all found
    if std.count(prunedResults, true) < std.length(needles) then
      false
    else
      true
  )
}

An example usage. Here we simply want to test that an additional environment variable is present when a config flag is turned on.

local test = import 'lib/jsonnetunit/jsonnetunit/test.libsonnet';
local utils = import 'lib/testhelpers.libsonnet';
local deploy = import 'templates/deployment-app.libsonnet';

local results = test.suite({
// Make sure the envVars contain ADDITIONAL_ENV: ADDITIONAL_VALUE without having to compare the entire map
testAdditionalEnvVars: {
  // In a real case, this vars block would be from the unit under test, in our case a rendered k8s manifest.
  local exampleVars: [
    { "name": "DB_HOST",        "value": "thedb.domain.com"          },
    { "name": "DB_PORT",        "value": 3306                        },
    { "name": "REDIS_HOST",     "value": "redis.anotherdomain.local" },
    { "name": "REDIS_PORT",     "value": 6379                        },
    { "name": "ADDITIONAL_ENV", "value": "ADDITIONAL_VALUE"          }
]

  actual: utils.containsEnvVars(exampleVars, [{ name: 'ADDITIONAL_ENV', value: 'ADDITIONAL_VALUE' }]),
  expect: true,
},

And of course, this test helper also needed testing itself.

local test = import 'lib/jsonnetunit/jsonnetunit/test.libsonnet';
local utils = import 'lib/testhelpers.libsonnet';

local mockEnvs = [{ name: 'n1', value: 'v1' }, { name: 'n2', value: 'v2' }, { name: 'n3', value: 'v3' }];

test.suite({
  testContainsSingleEnvVar: {
    expect: utils.containsEnvVars(mockEnvs, [{ name: 'n1', value: 'v1' }]),
    actual: true,
  },

  testContainsTwoEnvVars: {
    actual: utils.containsEnvVars(mockEnvs, [{ name: 'n2', value: 'v2' }, { name: 'n3', value: 'v3' }]),
    expect: true,
  },

  testDoesntContainsTwoEnvVars: {
    actual: utils.containsEnvVars(mockEnvs, [{ name: 'n5', value: 'v5' }, { name: 'n6', value: 'v6' }]),
    expect: false,
  },

  ... etc
})

A Bonus Feature

If we WERE only looking for key:value pairs in a map, the following would suffice

// containsObject will assert that the k:v's provided exist in the haystack object
// i.e assert map contains n entries
containsObject(haystack, needles):: (

// create a local array containing the element 'true' if the key and value of each needle exists in haystack
local results = [
    if std.objectHas(haystack, need) && (needles[need] == haystack[need]) then
      true
    else
      false
    for need in std.objectFields(needles)
];

// results will contain 'true' or 'null' if a needle was matched. strip out the 'nulls'
local prunedResults = std.prune(results);

// if the number of 'true's left is the same length as the needles, then assume everything was found and return true
if std.count(prunedResults, true) < std.length(needles) then
    false
else
    true
),

and in use...

testCantFindSingleNeedleInALargerObject: {
    local haystack = { k1: 'v1', k2: 'v2', k3: 'v3' },

    local needle = { k4: 'v4' },

    actual: utils.containsObjects(haystack, needle),
    expect: false,
  },