DEV Community: chuac

Assigning static IP to AWS Glue jobs

chuac — Tue, 05 Dec 2023 15:06:57 +0000

Our situation was a desire to run AWS Glue ETL (extract, transform, load) jobs where the data source is SQL Server in Azure that has an IP-whitelist firewall.

As AWS Glue is a serverless data integration service, it wasn't reasonable to maintain a range of public IP addresses that we should whitelist in the Azure firewall.

Inspired by this StackOverflow answer, it's possible to create a Glue connection where all traffic to external resources (like Azure) will use an assigned Elastic IP address.

In this post, I'll expand on their workaround with detailed steps and screenshots for the AWS console.

Head to the VPC console and hit Create VPC in your desired region.
Look for and select the VPC and more option which will help us to also create necessary subnets, route tables, and network connections - you can see these under the Preview section.
Configure other options the way you prefer but most importantly select 1 public subnet and 1 private subnet. Hit Create VPC.
With the VPC and other essentials created, take note that the subnets, and route tables created for us have a similar name to the VPC name we provided in Step 2 (e.g, glue-demo).
Head to create a NAT gateway. Select the public subnet in which to create the NAT gateway, keep connectivity type as Public. Here we will allocate an Elastic IP to the NAT gateway - this will be the IP inserted into our whitelist in Azure!
Navigate to Route tables, and at this point it's handy to filter resources by VPC.
Select the route table that is associated with your private subnet - the route table conveniently should've included private in its name tag.
Edit routes for the private route table and add a route with all destinations (AKA 0.0.0.0/0) with target of our NAT gateway created in Step 5.
We're now ready to head into AWS Glue and create a connection. We'll select Azure SQL DB as our data source here, but this should work for any other source (e.g, Snowflake).
I won't get into details of the data source connection (Azure SQL URL, etc) however we'll pay attention the the Network options section.
Here we'll choose the VPC we created above, and most importantly select the private subnet for this Glue connection. Continue and create the Glue connection for your data source.
Noting the public IP we allocated in Step 5, we can add it to our whitelist in Azure.
We're done! Now all AWS Glue jobs using the connection we created above will use the allocated public IP when communicating to external (outside of AWS) resources.

GA4 Data API with .NET

chuac — Tue, 25 Jul 2023 08:59:17 +0000

Our .NET 7 backend was querying Universal Analytics (UA) using Google.Apis.Analytics.v3 but as we drew closer to UA's shutdown date on July 1st 2023, we had to make the change to start querying our Google Analytics 4 (GA4) properties.

The package you'll need is Google.Analytics.Data.V1Beta - Not to be confused with Google.Apis.AnalyticsData.v1beta

Authenticating your client

Head here to follow the steps. After Step 1, you will have a credentials.json file which we will reference in our code soon.

Complete Step 2, to add the the service account that looks similar to quickstart@PROJECT-ID.iam.gserviceaccount.com to your GA4 property.

Now in your code, we will build the client and authenticate it using the credentials.json you created above. You are able to simply reference the JSON file from your code by setting CredentialsPath (and swap it out when deploying to different environments) however, we prefer swapping our environment dependent application settings (or secrets) as strings.

Luckily, the library accepts the JSON file as a JSON string. Simply JSON stringify the contents of your credentials.json and pass it to the client builder like so:

var analyticsDataClient = new BetaAnalyticsDataClientBuilder
{
    JsonCredentials = "JSON_STRINGIFY_YOUR_CREDENTIALS",
}
.Build();

Using the client to get data

The possibilities are endless as to what dimensions/metrics/filters/date ranges you can do, with documentation of all schema very good.

Here's a simple example to get screen page views for particular page paths between "yesterday" and "today". You will need your GA4 Property ID (you can find this in "Property Settings" in GA, remember this is not your Measurement ID)

RunReportRequest request = new ()
{
    Property = "properties/" + "YOUR_GA4_PROPERTY_ID",
    Dimensions = { new Dimension { Name = "pagePath" }, },
    Metrics = { new Metric { Name = "screenPageViews" }, },
    Limit = 10000,
    DateRanges = { new DateRange { StartDate = "yesterday", EndDate = "today", }, },
    DimensionFilter = new FilterExpression()
    {
        Filter = new Filter()
        {
            FieldName = "pagePath",
            StringFilter = new StringFilter()
            {
                MatchType = StringFilter.Types.MatchType.Contains,
                Value = "deals/",
            },
        },
    },
};

var response = await analyticsDataClient.RunReportAsync(request);

Note: Limitations with Realtime reports

There isn't 100% feature parity between UA Data API and this newer GA4 Data API. One dimension we were dependent on for Realtime reports was pagePath. You can recall above that it exists for normal reports but is missing for Realtime. There is an existing issue logged but with most recent activity being in 2022, I wouldn't count on that feature coming in anytime soon.

Updating Angular routes at runtime

chuac — Tue, 07 Feb 2023 06:34:42 +0000

There may be times where you need to update your Angular routes dynamically depending on the result of an API call, cookies, etc.

Realistically, this should be done during Angular app initialization (especially so if the routes are dependent on an API call) but we simply choose the AppModule constructor for this quick example.

Say you have a bunch of routes that have been registered:

const routes: Routes = [
    { path: 'about-us', component: AboutUsComponent },
    { path: '', component: HomeComponent },
];

and depending on some condition at runtime, you would like to swap the base path '' route to point to another component like NewHomeComponent, you can do like below and access resetConfig() on the Angular Router:

export class AppModule {
    constructor(
        private readonly router: Router,
    ) {
        if (someCondition) {
            const configRoutes = this.router.config;

            const index = configRoutes.findIndex((route: Route) => route.path === '');
            configRoutes[index] = { path: '', component: NewHomeComponent };

            this.router.resetConfig(configRoutes);
        }
    }
}

And you're done!

Specify Geolocation for Cypress tests - LambdaTest

chuac — Thu, 03 Nov 2022 00:00:07 +0000

If you're using a tool like LambdaTest Cypress Automation to execute your Cypress tests as part of your pipelines, you may eventually require the environment your tests are running on to mock certain Geolocations which your tests are dependent on.

Luckily, with LambdaTest, this is possible however it is not clear in their documentation on how to turn this on!

After consulting with their handy tech support, the following is how we can specify Australia as the Geolocation of the machine LambdaTest runs our Cypress tests on:

Inside lambdatest-config.json:

"run_settings": {
    ...
    "geo_location": "AU",
    ...
}

It is unclear what ISO standard that LambdaTest uses for their geo_location field, but ISO 3166-1 (Alpha-2) is a good bet.