DEV Community: Alan Barr

AI / LLM Testing Podcast

Alan Barr — Mon, 08 Jan 2024 17:05:03 +0000

How are we using AI?
AI Products and Testing
AI Testing

What do you think?

Solo, pair program, or mob ?

Alan Barr — Tue, 05 Sep 2023 13:57:28 +0000

Do you program, as a duo, a mob or by yourself?

When I worked at marketing companies, I usually programmed by myself. I've seen others try pair programming and at times programming as a mob. Either of these could work, but I am curious to have you tried or disliked any in particular.

What do you think? Would you try pair programming or mobbing, or would you rather work by yourself?

Is Domain-Driven Design relevant?

Alan Barr — Wed, 30 Aug 2023 16:55:31 +0000

I found a FREE guide on Domain-Driven Design that I found helpful. How is it relevant anymore?

It seems like writing code is less significant than modeling your system.

Have you found value in domain-driven design techniques? Or did you find that modeling is not worth the effort?

What do you think?

Down for me?

Alan Barr — Wed, 23 Aug 2023 15:48:41 +0000

I was trying to post and dev.to was down for me.

When you make a change, how do you know when an update is a bad break?

Have you ever had a bad deployment?

Should you ALWAYS write unit tests?

Alan Barr — Mon, 21 Aug 2023 21:24:40 +0000

I was reading an article about unit tests. Honestly, it seems like a lot of effort. Do you receive a lot of value from your unit tests? Personally, I'm still trying to understand domain-driven design. I feel like with supple design, we need to use several techniques to pin software on the wall. Constraints such as revealing interfaces, side-effects-free functions, and many assertions should make your software better.

What do you think? Is it always better to write unit tests?

Surprising Apps in Testing

Alan Barr — Sat, 19 Aug 2023 13:57:16 +0000

Do you have any applications that you test? I was wondering how to explain a common app. I decided to log in to my grocery account, and I was surprised to see...

I could guess. I'm an Android user. It's probably a hybrid app, and they probably know it. Well, that's life as a software tester. Play around and you'll find something new anytime.

What have you found?

How should I start?

Alan Barr — Thu, 17 Aug 2023 16:59:05 +0000

If you were going to look at an application with fresh eyes, how would you test?

There is always a risk of being static and stale with your testing. Mind maps, heuristics, and mnemonics like SFDIPOT.

Imagine yourself a beginner in software testing. Where could you start to investigate an application and find a problem?

Security and Observability Platforms

Alan Barr — Tue, 07 Sep 2021 19:11:07 +0000

Where do you see the future of software development platforms going with security and observability? What does security look like? What does observability look like?

I am curious because this is brand new territory for me. Where do you learn about these topics?

How do you deal with incidents?

Alan Barr — Wed, 13 Jan 2021 22:52:17 +0000

It's near the end of the workday and something is blowing up. People are raging in slack. How do you manage the chaos? I'm no expert at emergencies I've been through many but I'd like to know how you manage problems at scale? How do you keep a handle on issues?

When should you build an internal developer platform on Kubernetes?

Alan Barr — Wed, 13 Jan 2021 21:02:58 +0000

What do you all think? I built mine, its been a great six+ months, and I'm a little regretful I could not buy and leverage one of the options on the market. When do you think it makes sense to build an internal developer platform?

Who even needs Kubernetes?

Alan Barr — Thu, 01 Aug 2019 12:34:25 +0000

Is Kubernetes even necessary at all?

Export ServiceNow Report CSV to Airtable

Alan Barr — Fri, 19 Jul 2019 03:00:00 +0000

ServiceNow report data to Airtable

I manage a platform product roadmap and the majority of work lives in ServiceNow. ServiceNow began as a type of help desk ticketing utility. It appears they are positioning themselves as a general business workflow engine. I do not have API access as far as I know but I can generate reports and download them as CSV's. There appears to be a module in ServiceNow to do Agile Software development workflows but I lack the time or the interest to discover how to use this functionality. In the mean time we are using a flexible board that will place cards into our Visual Task Board based on the assignment group. This enables our team to work in a Kanban style flow. Unfortunately the burden is on me to generate metrics and charts based on their progress.

ServiceNow reporting interface

ServiceNow has a decent reporting interface that makes it easy to find data in tables and connect data structures together. I access the vtb_card table and populate specific columns based on the assignment group of my team. Specifically Task.Number, Lane, Task.Short Description, Task.State, Label 1, Label 2, Label 3, Label 4, Label 5, Label 6, Label 7. The most annoying thing about the labels are they are specific to a particular board and do not retain their names. However, it is not terrible to work around it. From there right click on the column name and export to CSV.

Preprocessing the data before uploading to AirTable

The one thing I had to do first with the exported CSV was to convert it to UTF8. The default encoding was not UTF8 and I used powershell to help achieve this. I also use the Google Cloud powershell library to upload my converted csv to cloud storage. Where I can trigger a serverless compute function to process the upload to airtable. This powershell is triggered by a usual batch script with the command. There's probably a way to do it in python but PowerShell was easy enough.

 PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& './upload\_backlog.ps1'"

$srcRoot = "dailybacklogreport\"

$utf8 = New-Object System.Text.UTF8Encoding $false

Get-ChildItem $srcRoot\*  -recurse -Include *.csv | ForEach-Object {
$content = $_ | Get-Content -Raw

Set-Content -Value $utf8.GetBytes($content) -Encoding Byte -Path $_.FullName

}

$fileList = Get-ChildItem -Path $srcRoot -File -Force -Recurse
$file = $fileList | Select-Object -First 1

New-GcsObject -Bucket "alanmbarrdatasyncing" -File $file.FullName -ObjectName "platform_engineering/backlog.csv" -Force
Remove-Item -Path "$srcRoot\*" -Include "*.csv" -Force

Once uploaded it triggers the Google Cloud Function to run on file creation. I use pandas to do the data filtering. I only want to send new records or records that changed due to AirTable Limits. What is missing from the script is keeping the labels in sync over time. It's possible but I haven't found it worth the effort to write a custom check.

The requirements.txt for the GC Function needs to contain the libraries for data processing.

airtable-python-wrapper==0.12.0
pandas==0.24.2
gcsfs==0.2.3

def parse_report(data):
    import pandas as pd
    def filterColumn(row):
        return list(filter(lambda row: row != None, row.values))

    startWithPandas = pd.read_csv('gs://' + data['bucket'] + '/' + data['name'])
    startWithPandas["label_1"] = startWithPandas["label_1"].apply(lambda x: "Defect" if x == True else None )
    startWithPandas["label_2"] = startWithPandas["label_2"].apply(lambda x: "High Priority" if x == True else None )
    startWithPandas["label_3"] = startWithPandas["label_3"].apply(lambda x: "Feature" if x == True else None )
    startWithPandas["label_4"] = startWithPandas["label_4"].apply(lambda x: "Idea" if x == True else None )
    startWithPandas["label_5"] = startWithPandas["label_5"].apply(lambda x: "Request" if x == True else None )
    startWithPandas["label_6"] = startWithPandas["label_6"].apply(lambda x: "Story" if x == True else None )
    startWithPandas["label_7"] = startWithPandas["label_7"].apply(lambda x: "Low Priority" if x == True else None )
    startWithPandas["Labels"] = startWithPandas[["label_1", "label_2", "label_3", "label_4", "label_5", "label_6", "label_7"]].apply(filterColumn,axis=1)
    return startWithPandas.drop(columns=["label_1", "label_2", "label_3", "label_4", "label_5", "label_6", "label_7"])

def parse_airtable_data():
    from airtable import Airtable
    import os, pandas as pd
    base_key = 'app5beEUuaFQatE2T'
    table_name = 'Features & Requests'
    airtable = Airtable(base_key, table_name, api_key=os.environ['AIRTABLE_KEY'])
    res = airtable.get_all(fields=["task.number","task.state","task.short_description","lane"])
    data = []
    for r in res:
        f = r["fields"]
        data.append(
            {"task.number":f["task.number"],
            "task.state":f["task.state"],
            "lane":f["lane"],
            "task.short_description":f["task.short_description"],
            "recordId":r.get("id")
            })

    airtable_data = pd.DataFrame(data)
    return airtable_data

def process_differences(servicenow,airtable_data):
    m = servicenow.merge(airtable_data, on=["task.number","task.state","task.short_description","lane"], how='outer', suffixes=['', '_'], indicator=True)
    differences = m[(m["_merge"]!= "both") ]
    return differences

def upload_to_airtable(differences):
    import os
    from airtable import Airtable
    base_key = 'app5beEUuaFQatE2T'
    table_name = 'Features & Requests'
    airtable = Airtable(base_key, table_name, api_key=os.environ['AIRTABLE_KEY'])
    unique_ids = set(differences["task.number"].values)

    for item in unique_ids:
        row = differences[differences["task.number"] == item]
        if len(row) == 2:
            recordId = row[row["recordId"].notnull()]["recordId"].values[0]
            f = row[~row["recordId"].notnull()]
            fields = f.to_dict(orient='records')[0]
            del fields["recordId"]
            del fields["_merge"]
            if len(fields.get("Labels")) == 1 and fields.get("Labels")[0] == "":
                del fields["Labels"]
            airtable.update(recordId,fields = fields)
        else:
            fields = row.to_dict(orient='records')[0]
            del fields["recordId"]
            del fields["_merge"]
            if isinstance(object, list):
                fields["Labels"]=list(filter(lambda x:x!="",fields["Labels"]))
            else:
                del fields["Labels"]
            airtable.insert(fields)


def upload_airtable_data(event, context):
    """Triggered by a change to a Cloud Storage bucket.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    file = event

    print(f"Processing file: {file['name']}.")
    if file['name'] and file['name'].startswith('platform_engineering/'):
        servicenow_frame = parse_report(file)
        airtable_frame = parse_airtable_data()
        diff = process_differences(servicenow_frame,airtable_frame)
        upload_to_airtable(diff)
    print(f"Processing Complete.")