Asked ChatGPT and other models to write a simple script

Hello! I continue my journey in the world of deep learning, ai and language models.

I was once tasked to write a simple PowerShell script, that gets all groups from Azure Active Directory and extracts all the members from the groups. The trick here, is that there might be nested groups!

For an average system administrator or coder it's a trivial task: just write a recursive function, then get all the groups and process via that function. I was interested in how well our language models can accomplish this basic task.

For all models I was using default settings and the same prompt. I removed all the comments generated by model and added mine, evaluating the solution. The results are not so obvious I must say! Let me present THE PROMPT:

Please write a PowerShell script which gets all groups from Azure AD, then iterates over all of the groups and extracts all members. Some of the groups can have nested groups. Members of those should be extracted as well.

Let's now check the outputs of our models:

GPT-4

Import-Module AzureAD

# That's good, a separate function to get group members recursively is nice
Function Get-GroupMembers ($GroupId) {
    $members = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "User"} # Correct! ObjectType is the property we are looking for, though no -All $true! Bad!

    foreach($member in $members){
        Write-Host "Member DisplayName: $($member.DisplayName), UserPrincipalName: $($member.UserPrincipalName)"
    }

    $nestedGroups = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "Group"}

    foreach($nestedGroup in $nestedGroups){
        Write-Host "Nested Group: $($nestedGroup.DisplayName)"
        Get-GroupMembers -GroupId $nestedGroup.ObjectId
    }
}

# Although usually I'm already connected, it's nice that GPT-4 suggested to handle the exception. Yet it could make it better because right now it gives you this which is funny:
# Get-AzureADUser : You must call the Connect-AzureAD cmdlet before calling any other cmdlets.
try {
    if(-not (Get-AzureADUser)) { Connect-AzureAD }
} catch {
    Write-Host "Failed to connect to Azure AD" -ForegroundColor Red
    exit
}

$groups = Get-AzureADGroup # It forgot to add -All $true, without it we get only first 100. Bad!

# Iterate over each group
foreach($group in $groups){
    Write-Host "Group: $($group.DisplayName)"
    Get-GroupMembers -GroupId $group.ObjectId
}

Overall, very good! Almost a working solution from the very first attempt.

Pros: the logic and cmdlets are correct

Cons: it didn't get ALL groups and ALL members although I specified this in the prompt. Also it just printed out the info. It could be nice to have an array which I could turn into CSV later, but that's fine, didn't ask for it in the prompt. Actually, it did tell me after writing the script: This script simply writes the details to the console. If you want to save the details to a file or do something else with them, you will need to modify the script accordingly.

GPT-3.5

# Nice! That's a simplier solution comparing to one provided by gpt-4. At least it works!
Connect-AzureAD

# It specified all! Already better than gpt-4
$groups = Get-AzureADGroup -All $true

# I hate this param stuff in PS but the logic is correct
function Get-NestedGroupMembers {
    param (
        [Parameter(Mandatory=$true)]
        [string]$GroupId
    )

    $nestedGroupMembers = Get-AzureADGroupMember -ObjectId $GroupId # forgot all here

    foreach ($nestedGroupMember in $nestedGroupMembers) {
        if ($nestedGroupMember.ObjectType -eq "Group") {
            Get-NestedGroupMembers -GroupId $nestedGroupMember.ObjectId # yep, that's okay
        } else {
            $nestedGroupMember
        }
    }
}


foreach ($group in $groups) {
    Write-Host "Group Name: $($group.DisplayName)"
    Write-Host "Members:"

    $groupMembers = Get-NestedGroupMembers -GroupId $group.ObjectId

    foreach ($groupMember in $groupMembers) {
        Write-Host "- $($groupMember.UserPrincipalName)"
    }

    Write-Host
}

Disconnect-AzureAD

I must say, it's already quite better that gpt-4 for some reason. At least we got ALL groups (but not all members)

Pros: it connects you to azure ad correctly and the logic is correct

Cons: again, not writing to any array and not getting all the members

Koala 13B

# Wrong from the start. We are in cloud dude, this cmdlet is for on-prem active directory
$groups = Get-ADGroup -Filter *

foreach ($group in $groups) {
    $members = Get-ADGroupMember -Identity $group.Name

    foreach ($member in $members) {
        # why do you need to check if the user is external or not?
        if ($member.MembershipType -eq "External") {
            $email = $member.EmailAddress

            # I don't understand the logic at all. It creates an array, then add something weird and on line 15, it checks whether the array is empty or not. What?
            $isNested = @()
            $isNested += $member.GroupName -like "$($group.Name)*"
            if ($isNested) {
                $nestedGroups = Get-ADGroupMember -Identity $member.GroupName | Where-Object {$_.MembershipType -eq "External"}
                foreach ($nestedGroup in $nestedGroups) {
                    $email += ",$($nestedGroup.EmailAddress)"
                }
            }

            Write-Output "$email"
        }
    }
}

Very bad in general. I will not write pros and cons as I really don't understand, what it's trying to do.

Nous-Hermes 13B

$groups = Get-ADGroup -Filter * | Select-Object Name | Sort-Object Name 

    foreach ($group in $groups) {
        $members = Get-ADGroupMember -Identity $group.Name -Recursive | Where-Object {($_.ObjectClass -eq 'user')} | Select-Object SamAccountName

        Write-Output "$group.Name has the following members: $(($members -join ', '))"
    }

Again, Active Directory is used not Azure Active Directory. But I checked, Get-ADGroupMember has a parameter -Recursive, I can't try it out but probably it works. Anyway, not for my needs

StarChat-beta

# good, getting aaaaal the groups
$groups = Get-AzureADGroup -All $true 

foreach ($group in $groups) {
    Write-Host "Processing group: $($group.DisplayName)"

    $members = (Get-AzureADGroupMember -ObjectId $group.ObjectID).Value # I don't know what it does, the Value is empty

    foreach($member in $members){
        if(-not $processedMembers.Contains($member)){
            $processedMembers += $member # this variable was never declared nor initialized, so I assume it's just null, not sure

            $isGroupMember = ((Get-AzureADUser -ObjectId $member).memberof | Where-Object{$_ -like "*$($group.displayName)*"} ) # so it assumes that nested group should have a similar name? 
            if($isGroupMember){
                Process-NestedGroups -GroupId $member
            }else{

                $outputMembers += $member
            }
        }
    }
}


function Process-NestedGroups([string]$groupId){
    $nestedGroups = (Get-AzureADGroupMember -ObjectId $groupId ).value 

    foreach ($ngroup in $nestedGroups) {
        if (-not $processedGroups.contains($ngroup)) {
            $processedGroups += $ngroup
            Process-NestedGroups -GroupId $ngroup
        }
    }

    $nestedGroupMembes= get-azureadgroupmember -objectid $ngroup.objectId | select -ExpandProperty id
    $outputMembers+=$nestedGroupMemebrs
}

Write-Output "Total Number of Groups:" $groups.count 
Write-Output "Number of unique members:" ($outputMembers| Measure-Object -Unique).Count # the script fails here, no unique property
write-host "List of Unique Members"
$outputMembers | sort-object -unique # even if you get there, you will get emptiness

Overall, the logic seems to be almost there, but the script is overcomplicated and doesn't actually achieve what it's supposed to

Anthropic Claude

# why in the world you forget to add -all
$groups = Get-AzureADGroup

foreach ($group in $groups) {
    $members = Get-AzureADGroupMember -ObjectId $group.ObjectId

    foreach ($member in $members) {
        if ($member.ObjectType -eq "Group") {
            $nestedMembers = Get-AzureADGroupMember -ObjectId $member.ObjectId
            $members += $nestedMembers
        }
    }

    Write-Host "Group: $($group.DisplayName)"
    $members | Format-Table

    $members = $null # why?
}

A very neat solution! It worked from the very first run.

Pros: less code, easy-to-read

Cons: not all groups, it didn't write to the variable, just printed to the screen

Conclusion

First of all, commercial models are way better at this. Secondly, sometimes even smaller and less powerful models can produce a more readable and better code.

Thank you for reading! If you like this post, please check my other two posts on how to work with LLaMa-based models. See you!

Comments

[Philip John Basile]

Can you rank them from favorite to least favorite? What if you rephrased it? Did any of the models do better? For me, ChatGPT 4 Is really good, but I also need to do a bit of back-and-forth with it to refine.

[Zakhar Smirnoff]

Okay, just a simple subjective ranking based only on this task:

1. claude
2. gpt-3.5
3. gpt-4
4. starchat
5. nous-hermes
6. koala

I tried slightly different prompts for almost each model, GPT-4 output can be improved a lot but other models seem to produce almost the same output.

Yes, GPT-4 so far is the best, surpassing even the models specialized in coding, hence I use gpt-4 at my daily work. Yes, it needs some refining but if you understand the code and know the language, it should be no problem. As a result, I spend almost 70-90% less time on writing the code

[Ben Halpern]

Can you clarify, GPT-4 is so far the best, but Claude is your favorite?

[Zakhar Smirnoff]

Like I wrote in the comment “based only on this task”. I don’t know why but for this script GPT-4 seemed to produce a less reliable result. I like Claude and GPT both, but mostly I use the latter.

It is the best for coding and other tasks (not for this particular one for some reason, even 3.5 was better), and the best to interact with. It has a website, API, and a mobile app. Claude is available only via slack for me currently. For new users it’s not available at all.

I believe, Claude should be the main competitor of GPT-4, Anthropic would be better working on public release, Claude is very nice, very comparable to gpt

[Ben Halpern]

Thanks for the extra context.

I have similar feelings around Claude and hope it makes it to accessible API soon.