DEV Community: Sheldon

RSS Feed from Hugo Site With Broken Shortcodes

Sheldon — Mon, 12 Jul 2021 18:09:06 +0000

I don't keep up with dev.to for publishing as almost every single post has broken imports from my hugo blog.
For example, gists never work, and require me to convert to the jekyll syntax.
Images often have issues as well.

Is there some way I can control my dev.to posting, and even better automatically publish from my hugo blog so I don't have to manually do this each time?
(I'm guessing perhaps I'd have to build out a custom api call with github actions or something)

Cool Way to Compare Features in Note Apps

Sheldon — Tue, 18 May 2021 18:28:03 +0000

Nice little site I came across for comparing various popular knowledge & note taking tools with ability to select features most important to you.

NoteApps

Note typically a fan of compare based tools, as they often feel like marketing junk, but this was a a pretty cool little implementation.

Git Workflow With Git Town

Sheldon — Tue, 23 Feb 2021 22:23:00 +0000

Resources

Git-Town

Painful But Powerful

Let’s get this out of the way.

Git isn’t intuitive.

It has quite a bit of a learning curve.

However, with this flexibility comes great flexibility. This tool has powered so much of modern open-source development.

Optimize for the Pain

To improve the development experience some tools can help provide structure.

This won’t be an attempt to compare every git GUI, or push any specific tooling. It’s more sharing my experience and what I’ve found helps accelerate my usage.

Tools I’ve Relied On

I’m not going to go into full detail on each, but check these out to help expedite your workflow.

The Challenge In Keeping Up To Date With Main

I use what’s normally called trunk-based development. This entails regularly moving commits from branches into the main branch, often rebasing while maintaining it in a functional state.

I’ll create a feature branch, bug fix, or refactor branch and then merge this to main as soon as functional.

I prefer a rebase approach on my branches, and when many ci/fix type commits, to squash this into a single unit of work as the results of the PR. This can result in “merge hell” as you try rebase on a busy repo.

Enter Git Town

This tool solves so many of the basic workflow issues, that it’s become one of the most impactful tools to my daily work.

Enable Aliases

The examples that follow use git sync, git hack feat/new-feature, etc as examples because I’ve run the command git-town alias true which enables the alias configuration for git town, reducing verbosity. Instead of git town sync, you can run git sync.

Example 1: Create a Branch for a New Unit of Work While You Are Already On Another Branch

Normally this would require:

Stash/Push current work
Checkout master
Fetch latest and pull with rebase
Resolve any conflicts from rebase
Create the new branch from main
Switch to the new branch

With Git Town

git hack feat/new-feature

Example 2: Sync Main

The following steps would be performed by: git sync

[master] git fetch --prune --tags
[master] git add -A
[master] git stash
[master] git rebase origin/master
[master] git push --tags
[master] git stash pop

Example 3: New Branch From Main

Easy to quickly ensure you are up to date with remote and generate a new branch with your current uncommitted changes.

[master] git fetch --prune --tags
[master] git add -A
[master] git stash
[master] git rebase origin/master
[master] git branch feat/demo-feature master
[master] git checkout feat/demo-feature
[feat/demo-feature] git stash pop

Example 4: Quickly Create a PR While On A Branch for Seperate Set of Changes

This workflow is far too tedious to do without tooling like this.

Let’s say I’m on a branch doing some work, and then I recognize that another bug, doc improvements, or other change unrelated to my current work would be good to submit.

With git town, it’s as simple as: git town hack feat/improve-docs.

This pulls the latest, flips you over to the new branch, and sets you up to commit just the lines or files you need from all the pending changes that still show up. Commit those specific changes, sync, and you’ve now got an upstream branch with minimal effort that is also updated from main.

[feat/demo-feature] git fetch --prune --tags
[feat/demo-feature] git add -A
[feat/demo-feature] git stash
[feat/demo-feature] git checkout master
[master] git rebase origin/master
[master] git branch feat/demo-feature-2 master
[master] git checkout feat/demo-feature-2
[feat/demo-feature-2] git stash pop

Example 5: Ship It

When not using a PR-driven workflow, such as solo projects, then you can still branch and get your work over to main to keep a cleaner history with git town ship.

This command ensures all the sync features are run, while then initiating a squash of your branch, allow you to edit the squash message, rebase merge this onto main, and finally clean-up the stale branch.

More Examples

Check out the documentation from the creators: Git Town Tutorials

Other Cool Features

Automatically prune stale branches after PR merge when syncing
Handles perennial branches if you are using Git Flow methodology.
Extensible for other git providers.
Rename a local branch + remote branch in a single command
Handles a lot of edge cases and failures

Wrap-Up

When using git, leveraging some tooling like this can accelerate your workflow. I don’t think you need to be an expert in git to use this, as it helps simplify many workflows that are just too tedious to be diligent on when running manually.

You can also do much of this with git aliases, but Git Town has a pretty robust feature-set with a testing framework in place, edge condition handling, and it’s fast. Consider using it you’d like to improve your git workflow while simplifying all the effort to do it right.

Incremental and Consistent

Sheldon — Sat, 13 Feb 2021 09:53:31 +0000

Allen Holub

@allenholub

Saying that we do knowledge work but have no time for learning is like saying we do farm work but have no time for planting.

00:46 AM - 13 Feb 2021

It's really hard to prioritize when life gets busy, but it's important that continued improvement is a priority.

Great at Work: How Top Performers Do Less, Work Better, and Achieve More was a really interesting book.
The fact that small incremental improvement done daily can make such a difference is pretty interesting.

It's similar to Agile tenets in how to approach software design.
Smaller iterations with rapid feedback is better than large isolated batches work delivered without regular feedback.

If you find yourself saying, "But I don't have time" or "When I have some time" it might be indicative of a failure to grasp this.
When I catch myself saying this I try to reword it and say "Whenever I make time for this" instead.

You'll always have pressure on you.
The further along in your career and life you go, the more pressure is likely to be on you.

You have to "make" time for improvement and learning if it's a priority.

Working With Powershell Objects to Create Yaml

Sheldon — Tue, 09 Feb 2021 05:30:39 +0000

Who This Might Be For

PowerShellers wanting to know how to create json and yaml dynamically via pscustomobject.
Anyone wanting to create configs like Datadog or other tools dynamically without the benefit of a configuration management tool.
Anyone else wanting to fall asleep more quickly. (I can think of better material such as the Go spec docs, but hey, I can't argue with your good taste 😄)

YAML

It's readable.

It's probably cost all of us hours when debugging yaml that's nested several layers and an errant whitespace got in.

It's here to stay.

I prefer it over JSON for readability, but I prefer JSON for programmability.

Sometimes though, tooling uses yaml, and we need to be able to flip between both.

Historically I've used cfn-flip which is pretty great.

Enter yq

The problem I have with using cfn-flip is dependencies.
It's a bit crazy to setup a docker image and then need to install a bunch of python setup tools to just get this one tool when it's all I need.

I thought about building a quick Go app to do this and give me the benefit of a single binary, as there is a pretty useful yaml package already.
Instead, I found a robust package that is cross-platform called yq and it's my new go to. 🎉

Just plain works

The docs are great

Reading STDIN is a bit clunky, but not too bad, though I wish it would take more of a pipeline input approach natively.
Instead of passing in {"string":"value"} | yq it requires you to specify stringinput | yq eval - --prettyPrint .
Note the single hyphen after eval. This is what signifies that the input is STDIN.

Dynamically Generate Some Configs

I was working on some Datadog config generation for SQL Server, and found this tooling useful, especially on older Windows instances that didn't have the capability to run the nice module powershell-yaml.

Here's how to use PowerShell objects to help generate a yaml configuration file on demand.

Install

See install directions for linux/mac, as it's pretty straightforward.

For windows, the chocolatey package was outdated as of the time of the article using the version 3.x.

I used a PowerShell 4.0 compatible syntax here that should work on any instances with access to the web.

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
if (-not (Test-Path 'C:\tools\yq.exe' -PathType Leaf))
    {
        $ProgressPreference = 'SilentlyContinue'
        New-Item 'C:\tools' -ItemType Directory -Force
        Invoke-WebRequest 'https://github.com/mikefarah/yq/releases/download/v4.4.1/yq_windows_amd64.exe' -OutFile 'C:\tools\yq.exe' -UseBasicParsing
        Unblock-File 'C:\tools\yq.exe' -Confirm:$false
    }

Once this was downloaded, you could either make sure C:\tools was in PATH or just use the fully qualified path for our simple use case.

Get AWS Metadata

In AWS, I parsed the metadata for the AccountID and InstanceID to generate a query to pull the Name tag dynamically.

TIP: 💡 You must have the required permissions for the instance profile for this to work.
This is not an instance level permission, so you'll want to add the required DescribeTags and ListInstances permissions for using a command such as Get-EC2Tag

Import-Module AWSPowershell -Verbose:$false *> $null
# AWSPowerShell is the legacy module, but is provided already on most AWS instances
$response = Invoke-RestMethod -Uri 'http://169.254.169.254/latest/dynamic/instance-identity/document' -TimeoutSec 5
$AccountId = $response.AccountId

Pull Back EC2 Tags

Now we can pull back the tag using an EC2 instance filter object.

$filters = @(
      [Amazon.EC2.Model.Filter]::new('resource-id', $response.InstanceId)
  )
  $tags = Get-EC2Tag -Filters $filters
  $tagcollection = $tags.ForEach{
      $t = $_
      [pscustomobject]@{
          Name  = $t.name
          Value = $t.value
      }
  }
  Write-Host "Tags For Instance: $($tagcollection | Format-Table -AutoSize -Wrap | Out-String)"
  $HostName = $Tags.GetEnumerator().Where{ $_.Key -eq 'Name' }.Value.ToLower().Trim()
  $SqlInstance = $HostName

Switch Things Up With A Switch

The next step was to alias the instance.

The better way to do this would be to use a tag that it reads, but for my quick ad-hoc use, this just let me specific an explicit alias to generate as a tag in the yaml. Again, try to use the Datadog tagging feature to do this automatically if possible.

TIP: 💡 If you aren't familiar with PowerShell's switch statement, it's a nice little feature for making this evaluation easy to read.
For the breadth of what this cool language feature can do, check this article out:
Everything you ever wanted to know about the switch statement

switch ($AccountId)
{
    '12345' { $AWSAccountAlias  = 'mydevenv' ; $stage = 'qa' }
    '12345' { $AWSAccountAlias  = 'myprodenv' ; $stage = 'prod' }
    default
    {
        throw "Couldn't match a valid account number to give this an alias"
    }
}

Now, preview the results of this Frankenstein.

Write-Host -ForegroundColor Green ("
`$HostName        = $HostName
`$SqlInstance     = $SqlInstance
`$AWSAccountAlias = $AWSAccountAlias
`$stage           = $stage
 ")

Ready To Generate Some Yaml Magic

$TargetConfig = (Join-Path $ENV:ProgramData 'Datadog/conf.d/windows_service.d/conf.yaml')
$Services = [pscustomobject]@{
    'instances' = @(
        [ordered]@{
            'services'                   =  @(
                'SQLSERVERAGENT'
                'MSSQLSERVER'
                'SQLSERVERAGENT'
            )
            'disable_legacy_service_tag' = $true
            'tags'                       = @(
                "aws_account_alias:$AWSAccountAlias"
                "sql_instance:$SqlInstance"
                "stage:$stage"
            )
        }
    )
}

$Services | ConvertTo-Json -Depth 100 | &'C:\tools\yq.exe' eval - --prettyPrint | Out-File $TargetConfig -Encoding UTF8

This would produce a nice json output like this

One More Complex Example

Start with creating an empty array and some variables to work with.

$UserName = 'TacoBear'
$Password = 'YouReallyThinkI''dPostThis?Funny'
$TargetConfig = (Join-Path $ENV:ProgramData 'Datadog/conf.d/sqlserver.d/conf.yaml')
$Queries = @()

Next include the generic Datadog collector definition.

This is straight outta their Github repo with the benefit of some tagging.

$Queries += [ordered]@{
    'host'      ='tcp:localhost,1433'
    'username'  =$UserName
    'password'  = $Password
    'connector' ='adodbapi'
    'driver'    = 'SQL Server'
    'database'  = 'master'
    'tags'      = @(
        "aws_account_alias:$AWSAccountAlias"
        "sql_instance:$SqlInstance"
        "stage:$stage"
    )
}

TIP: 💡 Using += is a bit of an anti-pattern for high performance PowerShell, but it works great for something like this that's ad-hoc and needs to be simple.
For high performance needs, try using something like $list = [Systems.Collections.Generic.List[pscustomobject]]:new() for example.
This can then allow you to use the $list.Add([pscustomobject]@{} to add items.

A bit more complex, but very powerful and performance, with the benefit of stronger data typing.

This one is a good example of the custom query format that Datadog supports, but honestly I found pretty confusing in their docs until I bumbled my way through a few iterations.

$Queries +=    [ordered]@{
    # description: Not Used by Datadog, but helpful to reading the yaml, be kind to those folks!
    'description'             = 'Get Count of Databases on Server'
    'host'                    ='tcp:localhost,1433'
    'username'                = $UserName
    'database'                = 'master'
    'password'                = $Password
    'connector'               ='adodbapi'
    'driver'                  = 'SQL Server'
    'min_collection_interval' = [timespan]::FromHours(1).TotalSeconds
    'command_timeout'         = 120

    'custom_queries'          = @(
        [ordered]@{
            'query'   = "select count(name) from sys.databases as d where d.Name not in ('master', 'msdb', 'model', 'tempdb')"
            'columns' = @(
                [ordered]@{
                    'name' = 'instance.database_count'
                    'type' = 'gauge'
                    'tags' = @(
                        "aws_account_alias:$AWSAccountAlias"
                        "sql_instance:$SqlInstance"
                        "stage:$stage"
                    )
                }
            )
        }
    )
}

Let me do a quick breakdown, in case you aren't as familiar with this type of syntax in PowerShell.

$Queries += takes whatever existing object we have and replaces it with the current object + the new object. This is why it's not performant for large scale work as it's basically creating a whole new copy of the collection with your new addition.
Next, I'm using [ordered] instead of [pscustomobject] which in effect does the same thing, but ensures I'm not having all my properties randomly sorted each time. Makes things a little easier to review. This is a shorthand syntax for what would be a much longer tedious process using New-Object and Add-Member.
Custom queries is a list, so I cast it with @() format, which tells PowerShell to expect a list. This helps json/yaml conversion be correct even if you have just a single entry. You can be more explicit if you want, like [pscustomobject[]]@() but since PowerShell ignores you mostly on trying to be type specific, it's not worth it. Don't try to make PowerShell be Go or C#. 😁

Flip To Yaml

Ok, we have an object list, now we need to flip this to yaml.

It's not as easy as $Queries | yq because of the difference in paradigm with .NET.

We are working with a structured object.

Just look at $Queries | Get-Member and you'll probably get: TypeName: System.Collections.Specialized.OrderedDictionary. The difference is that Go/Linux paradigm is focused on text, not objects. With powershell-yaml module you can run ConvertTo-Yaml $Queries and it will work as it will handle the object transformation.

However, we can actually get there with PowerShell, just need to think of a text focused paradigm instead. This is actually pretty easy using Converto-Json.

$SqlConfig = [ordered]@{'instances' = $Queries }
$SqlConfig | ConvertTo-Json -Depth 100 | &'C:\tools\yq.exe' eval - --prettyPrint | Out-File $TargetConfig -Encoding UTF8

This takes the object, converts to json uses the provided cmdlet from PowerShell that knows how to properly take the object and all the nested properties and magically split to JSON. Pass this into the yq executable, and behold, the magic is done.

You should have a nicely formatted yaml configuration file for Datadog.

If not, the dog will yip and complain with a bunch of red text in the log.

Debug Helper

Use this on the remote instance to simplify some debugging, or even connect via SSM directly.

& "$env:ProgramFiles\Datadog\Datadog Agent\bin\agent.exe" stopservice
& "$env:ProgramFiles\Datadog\Datadog Agent\bin\agent.exe" start-service

#Stream Logs without gui if remote session using:
Get-Content 'C:\ProgramData\Datadog\logs\agent.log' -Tail 5 -Wait

# interactive debugging and viewing of console
# & "$env:ProgramFiles\Datadog\Datadog Agent\bin\agent.exe" launch-gui

Wrap Up

Ideally, use Chef, Ansible, Saltstack, DSC, or another tool to do this. However, sometimes you just need some flexible options for generating this type of content dynamically. Hopefully, you'll find this useful in your PowerShell magician journey and save some time.

I've already found it useful in flipping json content for various tools back and forth. 🎉

A few scenarios that tooling like yq might prove useful could be:

convert simple query results from json to yaml and store in git as config
Flip an SSM Json doc to yaml
Review a complex json doc by flipping to yaml for more readable syntax
Confusing co-workers by flipping all their cloudformation from yaml to json or yaml from json. (If you take random advice like this and apply, you probably deserve the aftermath this would bring 🤣.)

Nativefier

Sheldon — Mon, 25 Jan 2021 22:19:41 +0000

Ran across this app, and thought was kinda cool. I've had some issues with Chrome apps showing up correctly in certain macOS windows managers to switch context quickly.

Using this tool, you can generate a standalone electron app bundle to run a webpage in as it's own dedicated window.

It's cross-platform.

For a site like Azure DevOps, you can run:

$MYORG = 'foo'
$MYPROJECT = 'bar'
$BOARDNAME = 'bored'
nativefier https://dev.azure.com/$MYORG/$MYPROJECT/_boards/board/t/$BOARDNAME/Backlog%20items/?fullScreen=true ~/$BOARDNAME

If redirects for permissions occur due to external links opening, you might have to open the application bundle and edit the url mapping. GitHub Issue #706

/Users/$(whoami)/$BOARDNAME/APP-darwin-x64/$BOARDNAME.app/Contents/Resources/app/nativefier.json

Ensure your external urls match the redirect paths that you need such as below. I included the standard oauth redirect locations that Google, Azure DevOps, and Microsoft uses. Add your own such as github to this to have those links open inside the app and not in a new window that fails to recieve the postback.

"internalUrls": "(._?contacts\.google\.com._?|._?dev.azure.com_?|._?microsoft.com_?|._?login.microsoftonline.com_?|._?azure.com_?|._?vssps.visualstudio.com._?)",

Go R1 Day 27

Sheldon — Fri, 22 Jan 2021 21:54:21 +0000

progress

Iterated through AWS SDK v1 S3 buckets to process IAM policy permissions.
Unmarshaled policy doc into struct using Json-To-Struct.

Github Pages Now Supports Private Pages

Sheldon — Fri, 22 Jan 2021 02:07:36 +0000

I'm a huge static site fan (lookup jamstack).

What I've historically had a problem with was hosting. For public pages, it's great.

For private internal docs, it's been problematic. It's more servers and access control to manage if you want something for a specific group inside a company to access.

This new update is a big deal for those that want to provide an internal hugo, jekyll, mkdocs, or other static generate based documentation site for their team.

Access control for GitHub Pages - GitHub Changelog

Ensuring Profile Environment Variables Available to Intellij

Sheldon — Wed, 20 Jan 2021 23:05:12 +0000

Open IntelliJ via terminal: open "/Users/$(whoami)/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app"

This will ensure your .profile, .bashrc, and other profile settings that might be loading some default environment variables are available to your IDE. For macOS, you'd have to set in the environment.plist otherwise to ensure they are available to a normal application.

ref: OSX shell environment variables - IDEs Support (IntelliJ Platform) | JetBrains

Create an S3 Lifecycle Policy with PowerShell

Sheldon — Tue, 19 Jan 2021 00:00:00 +0000

First, I’m a big believer in doing infrastructure as code.

Using the AWS SDK with any library is great, but for things like S3 I’d highly recommend you use a Terraform module such as Cloudposse terraform-aws-s3-bucket module. Everything Cloudposse produces has great quality, flexibility with naming conventions, and more.

Now that this disclaimer is out of the way, I’ve run into scenarios where you can have a bucket with a large amount of data such as databases which would be good to do some cleanup on before you migrate to newly managed backups.

In my case, I’ve run into 50TB of old backups due to tooling issues that prevented cleanup from being successful. The backup tooling stored a sqlite database in one subdirectory and in another directory the actual backups.

I preferred at this point to only perform the lifecycle cleanup on the backup files, while leaving the sqlite file alone. (side note: i always feel strange typing sqlite, like I’m skipping an l 😁).

Here’s an example of how to do this from the AWS PowerShell docs.

I modified this example to support providing multiple key prefixes. What wasn’t quite clear when I did this the need to create the entire lifecycle policy collection as a single object and pass this to the command.

If you try to run a loop and create one lifecycle policy for each Write-S3LifecycleConfiguration command, it only kept what last ran. Instead, ensure you create the entire object as shown in the example, and then you’ll be able to have multiple lifecycle policies get attached to your bucket.

Good luck!

#tech #development #aws #powershell #devops

Leverage Renovate for Easy Dependency Updates

Sheldon — Thu, 14 Jan 2021 02:42:20 +0000

Renovate is a great tool to know about. For Go, you can keep modules updated automatically, but still leverage a pull request review process to allow automated checks to run before allowing the update.

This is particularly useful with Terraform dependencies, which I consider notoriously difficult to keep updated. Instead of needing to use ranges for modules, you can start specifying exact versions and this GitHub app will automatically check for updates periodically and submit version bumps.

Why? You can have a Terraform plan previewed and checked for any errors on a new version update with no work. This means your blast radius on updates would be reduced as you are staying up to date and previewing each update as it's available.

No more 5 months of updates and figuring out what went wrong 😁

Here's an example json config that shows how to allow automerging, while respecting minor/major version updates not enabling automerge.

Note that you'd want to install the auto-approver app they document in the marketplace if you have pull request reviews required.

In addition, if you use CODEOWNERS file, this will still block automerge. Consider removing that if you aren't really leveraging it.

Go R1 Day 26

Sheldon — Wed, 13 Jan 2021 03:25:48 +0000

Progress

Evaluated gorm usage best practices with Slack Gopher community.
Obtained a great example to get me started on go routine and channels usage with multi-database queries.