DEV Community: DealerOn Dev

Why Agile Frameworks Fail - A Psychological Analysis

Jonathan Eccker — Thu, 26 Mar 2026 16:30:10 +0000

I would like to paint a picture. If it's familiar, keep reading. If it sounds outlandish and not relatable, this is not the article for you.

You have a small company.

You are winging it from day to day, adapting and iterating with no, or minimal, process.

Everyone talks out loud of process, of predictability, of structure they will eventually implement. But your small team pushes forward with late hours, big over-the-weekend features that come with incremental wins, and while everyone is exhausted, the sentimentality is "at least we don't have big company culture". The entire team is vastly intelligent, capable to find patterns and angles on problem spaces that many others can't see.

Innovation is built in at every step. It's a core value, as is similar mantras like "have fun" and "employees first" (even though not a single person in the team knows how to maintain "work-life" balance).

You become successful.

You either get a big client or you reach a tipping point on income that allows you to grow ambitious and plan further than one month ahead.

So you grow.

You adopt a framework, the structure you always dreamed of. The word "Agile" got used somewhere in the description, so you call the framework "Agile". "We are doing Agile now."

And you slow.

Some of the former powerhouses shift into leadership. Some of the powerhouses remain attached to fast-moving projects that "must happen". And one or two others move on, attached to "small company culture". The most vocal quit, either attached to "small company culture" or feeling generally not enabled to provide the value they used to. It's a huge loss as they were specialists on technology or a specific vertical.

But that's part of the cost of growth, right?

You keep growing.

You go through a V2 of your "Agile" process that really pins down your process to make sure "quality is baked in at every step".

To some extent, the "Agile" framework helps. You are able to sustain multiple teams, and as you start measuring velocity, you get an understanding of effort-to-value ratios.

But most of the engineers and some of the management feel like they are focusing on metrics rather than deliverables.

Why are they so caught up with the numbers? They are meant to just be validation, not the product.

Everyone in leadership says that, but when it comes to getting things done, it's always about the metrics.

Do the numbers themselves look right? You aren't sure.

You most certainly don't feel like you're getting as much done as you had before.

What the Hell Happened?

You switched the motivational framework.

In an unstructured environment, especially in Engineering, motivations come from unsolved, complex problems, novelty, urgency, and personal interest. Responsibility is a byproduct or guiding factor, not a driver.

Throw those motivation factors into a Google search.

I'll wait.

Additionally, in small companies, there is a lot of agency and freedom to tackle problems that require unrefined requirements to be reduced to logical, repeatable patterns that are represented in a vastly complex system.

Throw "individuals with advanced pattern recognition and rules extrapolation and a pathological need for agency" into that search.

Wait, my Team Isn't ADHD or Autistic

It's entirely possible.

But the line for the two diagnoses revolves around "disability", which, by the social model, is scoped to the relationship between the individual and the environment, not the individual itself. In the right environment, this means symptoms of the disability can (depending on the severity or type of disability) become completely transparent or seem like "just part of the job".

It's basically accidental masking by being successful, and it is fairly common.

I have, personally, become fairly convinced that there exists an alarmingly high amount of undiagnosed ADHD and Autistic Leadership in Engineering that are unintentionally masking simply by having found the right environment to fit their motivational framework. This is partly informed by many tech giants speculating that they would likely have been diagnosed as Autistic when younger (see: Bill Gates).

On that point, it's worth taking a moment to note that this article exists in the space of "Lived Experience" (community discussions/personal experience) and "Emerging Theory" (professional speculation). "Institutional Knowledge" (DSM-5) is far behind the active discussion space (by design).

To give an idea of just how far behind institutional knowledge is, it was just a bit more than a decade ago that the DSM recognized that Autism and ADHD could both be diagnosable in the same individual, and there's now an estimated ~50% co-diagnosis rate.

The point being that discussions on neurodivergence are moving fast.

What I can confidently say is:
1) Most everyone already knew the Engineering world had a high ratio of neurodivergent representation. There exist studies on this already.
2) Society is going through an unprecedented wave of demasking and diagnosis. (If you're successful in Engineering, I suggest taking this test https://raads-r.net/ if you have some time to kill)
3) Even for those not severe enough to be diagnosed, Autistic/ADHD traits are present across "neurotypical" society. Those traits are going to be very common in engineering, which requires non-linear and highly cognitive processing. (search: Autistic traits continuous in general population).

How Does this Relate to Agile?

Great guiding question, me!

I would like to call attention to the Agile Manifesto. Not an Agile framework. The core philosophy.

https://agilemanifesto.org/principles.html

In particular:

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

This is, coincidentally, in line with the current directive for most neurodivergent therapy. Integration, not assimilation. In HR terms, we call it equity.

When you had low process and high agency, with constant stimulation driven by urgency, novelty, and personal passion, you had an environment that appealed to a neurotype that is very powerful when motivated.

This is a neurotype that will often put in 60 hours a week and like it (assuming they are actually motivated). This is a neurotype that will solve complex problems over a weekend that might take others months to even understand.

The misguided lessons society has taught itself about "work-life balance" have led us to adopt the 9-5 workweek, based on the assertion that it would work for everyone and that it's the box you must fit into to be successful.

Many companies then built a bunch of fancy metrics around this philosophy in a misguided attempt to show just how productive this 9-5 workweek was at driving motivation.

And neurodivergents, in response, invented the mouse wiggler.

In some jobs, time worked is the measurable deliverable.

Engineering is not one of those jobs. You deliver software, not time.

You're Suggesting 60-hour Weeks are GOOD?

Sustainably? Every week? Absolutely not.

I've personally had 60-hour weeks and 10-hour weeks.

I've solved months of value in 2 hours, and gone down the wrong rabbit hole for weeks.

AuDHD (Autism + ADHD, that's me) in particular often comes with a drive to solve problems. I'm either solving a problem or looking for a new problem to solve. Responsibility is a guiding factor, not the engine. Burnout stems from either working on long-term projects out of obligation (and not motivation) or from too little going on ("boreout").

There's a bit of nature vs. nurture in the discussion of where that "eternal antsyness" originates. I'd highly recommend doing some research on that if the question "should AuDHD be that way" popped into your head. It's a fascinating rabbit hole that will show you the bigger picture of just how much society shoots itself in the foot with misleading messaging and expectations.

You Just Implied Velocity Isn't Constant

I'm not implying.

I'm stating.

The illusion of constant velocity is one of the most harmful lies society tells itself.

Across a team and over a sufficiently long timespan, velocity can approach consistency. But not at a granular, individual level.

There's an important entry in the manifesto.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

And a second one.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Preference is for shorter time periods, but the extreme of "day to day velocity" becomes micromanaging (and risks encroaching on the neurodivergent need for agency, which has a high risk for causing burnout and turnover. Search: Pathological Demand Avoidance).

Two-week sprints are generally a good time frame for most teams to average out to a "constant pace", so those tend to be what get used. I do believe it's important for a team to have an established contract on their iteration team, especially since time-boxing is such an important concept for ADHD (which often comes with perfectionism).

But ultimately, at an individual level, the smaller the time frame you consider, the harder it is to say "spontaneous bursts of innovation and complex problem solving" and "predictable delivery speed" in the same breath.

There's a balance.

As a side note, search for "ADHD time blindness"; it will likely explain why you struggle with overcommitting or undercommitting in almost every sprint. Many neurodivergent employees try to plan around their highest velocity rather than their expected velocity. Combine that with a bunch of biology around dopamine dysregulation that I'm not going to get into here, and you get powerhouses that are incapable of predicting just how powerful they will be.

Wouldn't that Show up in the Metrics?

Enter Autism.

Pattern and rules extraction are our entire thing. (That's a reductionist statement for the sake of making a point, we have a lot of things, don't @ me)

Intuition is not intuitive for us. Big-picture thinking (holistic intuition) is often low, while expertise and wisdom (pattern-based intuition) are often higher, but take time to build.

This is by design in Engineering.

You want employees who are cognitive-forward; employees who are missing the caching layer of intuition. It means the problems get properly analyzed (cognition) with very few assumptions made (intuition), and a sustainable solution is implemented. Engineers don't just write code; they pattern-match and extract rules from requirements.

Intuition would tell you, "Don't re-invent the wheel."
Cognition would tell you, "But the wheels we have aren't built for this terrain. In a few weeks, we can invent the plane."
(And Intuition might then question, "Why are we even talking about wheels? We build houses.")

But when we haven't had time to iterate on the same problem space for long periods (building expertise), we default to repeatable processes and numbers to reduce the problem space to bite-sized, actionable information.

More often than not, this leads to gamifying the numbers. Mouse wigglers, not pulling in extra work when work is done faster, over-pointing efforts, under-pointing efforts, skipping process completely for certain efforts, etc.

In the absence of more concrete directives on deliverables, making sure those numbers look good, min/maxing them, is inevitable. It's not generally an active decision; it's just a piece of the puzzle we incorporate into the planning process when everything else is open-ended and doesn't match a pattern we are familiar with.

This is Still about Agile, Right?

Yes!

The core moment you went wrong, the single "what should we have done differently?" decision point, was picking a singular Agile Framework to prescribe to a full department.

These frameworks often prescribe, or are interpreted as prescribing:

Low, or no, rollover
No mid-sprint changes
Predictable velocity at a granular scale
All work to be completely scoped before committing
Time commits on work to be predictable and exact
Turning Engineers into code monkeys, rather than problem solvers, moving decision-making entirely into other teams (Architects, Product, UX, QA, etc.)

If you look at the core 12 principles of the Agile manifesto, every single one of those bullet points comes back to a misinterpretation of the Agile philosophy.

And most importantly, every single one of those bullet points is incompatible with the neurotypes you have built your team out of.

You have not created an environment for motivated individuals; you have created one for the "neurotypical" mentality, which so few people thrive in. (If you want another fun rabbit hole, search "does the 'neurotypical' person exist at the biological level". Spoilers: It's a sociopolitical term, not medical. Depending on how you draw a few lines in definitions in 'normal', it's not even a majority.)

Assuming You're Right, What Do We Do About it?

Be Agile, don't do Agile.

Provide the right environment.

The manifesto is the important part, not the frameworks.

You need different processes for different types of teams, different types of projects, and different budgets.

I want to call attention to an important entry in the manifesto:

Simplicity--the art of maximizing the amount
of work not done--is essential.

And another one:

The best architectures, requirements, and designs
emerge from self-organizing teams.

One-size-fits-all process is the problem, not the existence of process. Process is important, process enables, process protects. But doing less, quite literally, translates to doing more.

I can personally tell you that when I am mid flow-state on a big project, even transitioning a Jira ticket that carries no perceived value can be enough to break momentum. It's unbearable. I can physically feel the frustration from it in my veins.

Those without ADHD are likely thinking this is melodramatic. If you know someone with ADHD, ask them. This is a major reason why "context switching" is viewed as a cardinal sin among engineers. Got into the flow in the morning, but was forced into a meeting on a completely unrelated topic? It's a literal crap shoot on whether I get into that flow again.

Reduce process to only the process that enables development or protects a system.

Highly critical services in maintenance mode require high process. Lesser critical systems require less.

"Must do", non open-ended work (if you have buckets of work catered to specific important clients, for instance) is good in high process to self-regulate on commits.

If something is greenfield and doesn't affect production, does it need a full regression test suite and a Jira step for Product approval in a lower environment? Can you just demo on your local machine?

For the other Software Architects out there: Design systems to enable greenfield development to occur out of band from those critical services. This is one of the subtle but most valuable benefits of microservices. It lets you right-size the process.

Another quote:

Working software is the primary measure of progress.

Reduce metrics to those that drive SMART goals. SMART goals are the most empowering weapon for neurodivergent employees, who often crave explicitly defined deliverables (just don't overprescribe how they get TO those deliverables, see: Pathalogical Demand Avoidance again).

A Couple Other Principles Viewed through the Neurodivergent Lens

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.

Pattern-based and cognitive-forward decision-making takes time. Couple this with the low Implicit Social Cognition you get with Autism (difficulty to intuitively keep up with an ongoing discussion), you get employees who need longer than your 1-hour meeting to think or process.

Sure, you did sprint planning. That was important. You did due diligence on attempting to plan.

Plans are nothing; planning is everything.

An Engineer realizing a flaw in a plan a few days into a sprint is part of the process. Encourage text-based communication for asynchronous planning, which is very neurodivergent-friendly (giving time to right-size communication and process).

Business people and developers must work
together daily throughout the project.

I mentioned Pathalogical Demand Avoidance a few times.

This is the best way to avoid the problem. Make engineers feel involved in decisions. They are specialists on the system you are building, use that, don't silo it. Studies exist to prove performance gain from engineer engagement, if you need proof. See: https://link.springer.com/article/10.1007/s10664-023-10293-z

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

This might sound counterintuitive, because many people think of "anti-social" when they hear Autism. If you tried to initiate a discussion about the weather, I might give an awkward smile and do my best to extract myself from the physical pain that is "small talk".

But when it comes to an area of interest, you can't shut us up.

The literalness of "face to face" leaves much open to interpretation, but at the end of the day, vocal communication, at the very least (even if just Zoom), is significantly more efficient and effective time-wise. Just define your deliverables going into the meeting and organize the conversations with live note-taking to avoid endless tangents and rabbit-holing.

And finally:

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

This is, coincidentally, how many people with AuDHD simply live their lives. Iterate -> Analyze -> Consult (ideally) -> Iterate. An endless series of "focus hard, then reflect".

That, at its core, is what you're aiming for with being Agile.

Well, I'm Neurodivergent, and I figured it out, they Can Learn to Push Through it Just like I did

I want to call this out since I've actually heard it said, specifically in the context of prescriptive Agile frameworks.

This is the equivalent of telling someone with severe depression, "just smile like the rest of us". It's a form of generational trauma, has ableist undertones, and risks violations of ADA directives on handling ADHD and Autism as a disability.

The world's take on "undue hardship" and equity in ADHD/Autism is shifting. There are already successful lawsuits on underplaying the severity of mental disabilities in the workplace (including the usage of derogatory words like "lazy" or "disorganized").

Do some searching. Educate yourself. Become powered by neurodiversity, not disabled by it.

Why should you care about website performance?

Deborah — Thu, 21 Nov 2024 18:24:17 +0000

Picture this: You're on your phone, waiting with bated breath for midnight, when the latest greatest game console finally gets released. You Google the console to track down where to order it - the site is a few results down on the page, but you find it eventually. You click the link. The loading bar creeps tantalizingly across the screen...and then grinds to a halt! Not to be deterred, you toss your phone aside, open your laptop, and try the ordering site again. It takes a minute, but eventually loads. Huzzah! You hit the button to add the console to your cart...but your browser freezes! By the time you restart it and try the site again, the console is SOLD OUT. Furious, you vow then and there never to buy that brand of console again.

Ok, so perhaps it's a bit dramatic and contrived, but the purpose of this story is to illustrate the perils of poor site performance. Delayed page loads, slow interactions, and many other performance problems can lead to a frustrating user experience, and make it more likely that potential customers will bounce from your site.

Performance issues can be even more pronounced on mobile devices due to less processing power and slower network connections. This is especially significant considering that, in 2023, people used their mobile devices for browsing over 3 times more than desktop. Let that sink in. 75% of the time, your users will be using mobile devices that, by their very nature, increase the likelihood of encountering performance problems if you haven't put in the work to optimize. 75% of the time, there is an increased chance of a potential customer giving up on your site.

Google also factors your pages' performance into their search results ranking algorithm. That means it's theoretically possible that your page could get bumped down below a similarly relevant page if it doesn't perform as well, all other factors being equal. The farther your site is from the top of the search results, the less likely it is to be clicked on. The following chart illustrates the exponential drop-off in click-through rates as you go farther down the list of search results:

Source: https://www.searchenginejournal.com/google-first-page-clicks/374516/

Note how the click-through rate at position 2 in the list is almost HALF that of the top position. And if your site gets bumped to the dreaded second page of results? Well, less than 0.5% of users will even bother to look.

Going back to our would-be gamer at the start of this post, let's step through the story again with a newfound understanding of what happened. Our gamer started out using a mobile device, which automatically made everything run slower (certainly not the user's fault, but a factor nonetheless). When they Googled the console site, it took a tad longer to find since it showed up farther down the page than expected. This lower Google ranking likely resulted from the site itself having performance issues - as our gamer soon discovered for themself, first on their phone, then on their laptop. In the precious minutes they wasted in trying to find and load the website, the product they were seeking was sold out, and the console company lost a loyal customer.

Let this be a lesson: when setting up your website, you might be immediately concerned with making sure the core functionality works properly, the interface flows well, and the homepage design looks awesome - all important considerations, to be sure. But you should also be factoring site performance into every stage of your site's development, ESPECIALLY when thinking about the mobile user's experience. I'll show you how in future posts!

A Taxonomy of Software Failures

Tom Warner — Tue, 27 Jun 2023 21:12:09 +0000

(Somewhat inspired by this classic article about categorizing exceptions, which anyone who writes software should read.)

There are an infinite number of specific ways that a production software system can fail, and in my career I've encountered a vast array of them (of course, some percentage of them were my fault in the first place). But I've noticed that production failures tend to fall into a few broad categories, so my hope is that by sorting the categories out, it can be easier to anticipate and fix issues (and maybe even prevent a few. That would be nice!)

New Defect Released

This is by far the most straightforward failure. Someone wrote some new code, it got deployed, and it turned out to have an immediately obvious bug in it. This sort of issue can be mitigated by having a process of thorough code review, QA testing, integration testing, etc and it can also often (but not always!) be solved by having a robust and simple process for rolling back any newly released code. However, if for example the released code includes a dependency for something else in prod, or if it bundles some other features which can't be reverted out, your best fallback is having some sort of accelerated process for deploying hotfixes. Note that every obstacle in between writing and deploying code (review, permissions gate, deployment window, ...) which helps reduce the frequency of errors making it out to prod also slows the deployment of any fix for the errors that do make it to prod.

Existing Defect Noticed

When buggy code gets released, it's actually best for someone to notice immediately. If that doesn't happen, you may end up in this situation, where a bug has been out in production for plenty of time but it has only just become an emergency. There are many reasons for something like this to happen -- for example, a new customer may start using an existing feature in a way you didn't anticipate, or a new release may depend on the defective code in a high-impact way.

The best way to avoid this sort of thing is to avoid deploying buggy code in the first place, of course. However, since we live in an imperfect world and you will deploy buggy code from time to time, there are a few mitigations available. One is to make a priority of finding and fixing existing bugs -- decrease the friction for your users to report them, motivate developers to fix them, and generally invest in code quality. However, all that will fail from time to time, so the last fallback is again to shorten the turnaround time between "defect noticed" and "hotfix deployed".

External Failure

A large-scale software ecosystem collects external dependencies the way the floor underneath your couch collects dust bunnies and long-lost board game pieces. However, unlike the problem under your couch, which can be solved by a vacuum cleaner, you cannot go through and clean out the external dependencies because they provide solutions for your users. This is unfortunate, because sometimes these dependencies will fail, for circumstances entirely beyond your control. Sometimes third party software breaks entirely on its own, or sometimes they just SILENTLY CHANGE THEIR API FORMAT ONE DAY BECAUSE THEY ARE MONSTERS. The best you can do is proactively identify and catalog all the external dependencies that you have, make sure the systems which connect to them are as resilient as possible in case they fail, and (if possible) identify and implement ways to backup or cache external data so that you have temporary fallbacks available if you need.

Load/Infrastructure Failure

Though many would prefer not to think about this, your software ecosystem does ultimately rely on physical machines, existing in the real world, running your software on them. Sometimes these machines have physical problems. More commonly, especially if you are using the sort of cloud providers that let you mostly abstract that away, the work you're asking some piece of infrastructure to do exceeds the work that it is capable of doing, and you start running into all manner of complex and diabolical failure states. Your server runs out of memory, your database runs out of rows, your monthly subscription runs out of compute, or perhaps you have some kind of exponentially inefficient algorithm buried deep in your codebase that is suddenly running against too much data to handle in a timely manner.

This sort of thing is why devops people get paid so well. But obviously there are ways to be proactive, such as monitoring the resources you're using and the performance of your infrastructure. Having a well-architected environment in general is a huge help here because it makes it easier for you to fix or replace infrastructure on the fly.

I find it helps to be aware of these different failure cases because they cover just about anything that can go wrong with production software, so when something starts to break, an easy starting point is to try and figure out which of these (it may be more than one!) is happening. This sort of knowledge can also inform your attempts to be proactive about heading off problems before they emerge, or to be reactive and categorize the failures you've been experiencing in hopes of future improvement.

(Reposted from my personal blog here)

Git Learnt

Tom Warner — Thu, 11 May 2023 13:33:56 +0000

(ya know, if you want)

Git is a powerful tool, and by far the most effective way to unlock its potential is to use it on the command line. There's just one issue with that, which is that the syntax for Git commands is unintuitive, often overly verbose, and written at a very low level, so doing many operations you care about (like, say, "put me on the latest version of this specific branch from the remote") tend to involve several commands. And since the commands are often context-dependent, common operations may require different commands depending on the state of your local git.

So it's understandable that there are so many Git UIs, however, what I've found over the years of working with Git on the command line is that by setting aliases and local functions for common workflows, I can accomplish just about everything I need in short, simple commands. Here are some of those aliases, along with explanations of how they work.

All of these commands live in a GitHub repo where I've been collecting them, so you should be able to either clone it or copypaste them out to get them to work locally. Also, all the shell aliases and functions assume you're using a Unix-like shell (I use Git Bash even on Windows) but I'm sure if you really wanted to, you could port them over to something else like Powershell.

Put me on the latest version of this branch

This is one that somehow takes up to 3 separate commands in native Git, which is ridiculous because it's a core part of anyone's workflow. The alias I use is gfl (git-fetch-latest) and I have it configured to either take a branch name as an argument, and put you on the latest revision of that target branch, or without a branch name it puts you on the latest version of your current branch. Super useful for code reviews!

gfl() {
    if [ $# -eq 0 ]; then
        git pull
    else
        git fetch && git checkout "$1" && git pull
    fi
}

Usage:

$ gfl branch-name #pulls the latest version of branch-name from origin
$ gfl #with no arguments, pulls the latest version of the branch you're already on

Start a new branch from the commit I'm on

This is a common operation but of course the branch commands are janky and unintuitive (IN WHAT UNIVERSE DOES IT MAKE SENSE FOR git checkout -b TO CREATE A BRANCH AND git branch -D TO DELETE ONE). So you can either memorize the long form of this syntax, or make a short little alias (gcb for git-create-branch) and not have to think about it. Note that any existing local changes will not be affected by this command.

gcb() {
        git checkout -b "$1"
}

Usage:

$ gcb new-branch-name

Start a new branch from a specific tag

The workflow at my job involves using tags to mark specific builds, and mostly when we create new feature branches, it's those tags that we branch off. This is another one that's a fairly straightforward command in Git but it has a weird and verbose syntax, so I alias it to gbt for git-branch-tag.

gbt() {
        git fetch && git checkout tags/$1 -b $2
}

Usage:

$ gbt target-tag new-branch-name

Merge a specific tag into your branch

Sometimes at my job, the build we're targeting our feature branches against changes. When that happens, we should merge that newer commit (represented by a tag) into the feature branch we're working on. I use gmt for git-merge-tag to do this.

gmt() {
        git fetch && git merge $1
}

Usage:

$ gmt target-tag

Blow away all local work

Sometimes you're just having one of those days where you modify 18 files and then you realize it was all completely wrong. Right? It's not just me who has those days? Well, anyway, if you need to just wipe out every local change, you can use grhh (git-reset-hard-head). This is another one where I memorized the syntax years ago and then realized I didn't like typing the whole thing out every time I need it, but as usual the syntax is janky and awkward and there's no reason you should have to remember such a specific series of flags for such a basic operation.

alias grhh='git reset --hard HEAD'

Usage:

$ grhh #just make extra sure you don't have any local work you want to keep!

Make your submodules work

If you're dealing with submodules, it can be kind of a pain to keep them in a correct state, especially since operations that change your local branch don't touch the submodules. This is (again!) a verbose one-liner, but running it will get your submodules into exactly the state that your current branch expects them to be in--even if they're not checked out at all, or if they're nested. If you're dealing with submodules that change a lot, you may want to work this alias into the commands you use to switch branches. It's gsm for git-sub-module.

alias gsm='git submodule update --init --recursive'

Usage

$ gsm

Commit and push all your local work

A lot of the time when I'm working on a feature branch, I get to a good stopping point and I just want to push all my local work remotely. To do this in Git natively takes 3 commands, of course, but as you may have gathered, I hate typing any more letters than I absolutely must. So I set up gacp for git-add-commit-push to let me just enter a commit message and send it all upstream in one operation. The one caveat with this is that sometimes you do have changes on your local that you don't actually want to go up to the remote; for that situation I use the next command.

gacp() {
        git add . && git commit -m "$1" && git push
}

Usage

$ gacp "this is my sweet commit message"

Commit and push all your staged local work

If you have a bunch of local changes that you want to push, and some you don't, you can use this command to commit and push just the changes you've already staged -- use the git add command to stage, and you can check the output of git status (aliased below!) to see what is and isn't staged already. I call it gcp for git-commit-push.

gcp() {
        git commit -m "$1" && git push
}

Usage

$ gcp "this is my sweet commit message"

Pull from the remote, but without any weird error messages

If you have created a local branch and pushed it remotely, and then someone else has put newer commits on it, and you try to pull those commits down, Git is liable to spit out an annoying error message at you and fail. This is because technically the local<->remote linkage in Git works in two different directions, and by pushing your branch up from local, you do not default to setting it as the upstream target for when you do a pull. I'm sure there are complicated technical reasons that it works this way, but also I don't care because it's a dumb way for Git to work.

Anyway, I set up this alias (gpd for git-pull-down) that lets you just pull from the remote branch matching the name of your local branch without worrying about any of this, and also save some typing of course.

gpd() {
    head="$(git rev-parse --abbrev-ref HEAD)"

    if [[ $(git config "branch.$head.merge") ]]; then #there's already a merge target configured, just pull as normal from there
        git pull
    else
        if [[ $(git ls-remote --heads origin $head) ]]; then #there is an upstream branch existing with the same name as our branch
            git branch --set-upstream-to origin/$head #set merge target to upstream branch with same name
            git pull
        else #fail with explanation
            echo "Branch $head has no upstream or merge target! You will likely have to push first, or manually configure it"
            return 1
        fi
    fi
}

Usage

$ gpd

Print a diff of all locally changed files

This is actually one that's really easy to write and remember but I hate typing and I run it all the time, so I've aliased it down to gd for git-diff. Also I use diff-so-fancy to make the output of my diffs look frickin sweet and I suggest you do the same.

alias gd='git diff'

Usage

$ gd

Print the current status of your repo

Once again this is an easy one that I'm just too lazy to type out each time. I have it aliased to gs for git-status.

alias gs='git status'

Usage

$ gd

Hopefully these commands save you some typing, and also having to memorize some arcane and obnoxious syntax. My brain and my fingers have very little tolerance for either, so having a bunch of 2- or 3- character commands for the most common operations I run in Git directly improves my workflow, my happiness, and my sanity.

(Reposted from my personal blog here)

Modern Software Architecture is a Misnomer (sort of)

Jonathan Eccker — Wed, 06 Apr 2022 14:30:53 +0000

When I first was introduced to the term "software architecture", my mind immediately filled in a picture of someone bent over a massive flow chart, sculpting it into a shape that made sense. Finding the ideal shape that would solve everyone's problems in the most efficient and effective way.

I likened it to an actual Architect, outlining the shape and details of a sky scraper that would sustain for millennia.

And that's sort of the only actual parallel: building a solution that will sustain for the foreseeable future.

Let's define the industries

The building industry:

Has been around for 12000 or so years (https://en.wikipedia.org/wiki/List_of_oldest_known_surviving_buildings)
Has a fairly (pardon the pun) concrete and well developed list of fundamental infrastructure materials
Has stakeholders (landlords, tenants, etc.) that have a very finite list of requirements (working, living, retailing, tenant count, etc.)
Has generally well defined resource limitations (space, height restrictions, budget)
Has employees that have a pretty common foundation of knowledge and experience with materials and tooling

The software industry by comparison:

Has been around for less than 80 years (https://en.wikipedia.org/wiki/History_of_software)
Has a growing list of relatively volatile infrastructure materials (see: JavaScript frameworks)
In many cases has stakeholders that are still figuring out what they even are trying to solve and who they are trying to solve it for
Is still in the process of defining its resource limitations (look up any pricing for any cloud hosting service and you'll generally come out with more acronyms to google than answers on pricing)
Has employees that will have vastly different levels and types of experience with various tooling (partly due to the large list of relatively volatile infrastructure)

Very different industries, yes, the point?

The point is that the building industry can make a lot of assumptions and have concrete answers on what it's trying to solve and how it expects to solve it. The end result is large, monolithic, buildings expected to last hundreds, if not thousands, of years.

The software industry you can make very few assumptions and there often are not concrete answers, just educated (and sometimes uneducated) guesses. This is why modern Software Architects are inclined to focus on two concepts: standardization (increase assume-ability), and modularity (ease to switch in/out educated guesses).

If we built Software in the Building Industry

You've likely seen Software designed as if there were very concrete answers built on thousands of years of experience. It's a legacy monolith codebase. I like imagining every company has one, some large piece of PHP or classic ASP sitting in the corner gathering dust.

When you have confidence that you're solving the right problems with the right technology, you can mold all of your solutions into an all-in-one self-documenting codebase, and expect it to chug away for eternity.

Because everyone in the industry has some level of standardized experience with the tooling and infrastructure you used, you can hire pretty much anyone to maintain it! It'll last forever!

If we built Buildings in the Software Industry

You're handed the requirements for creating living space for people:

We don't know how many, if any, people will use your solution
We're pretty sure that a Bed, Microwave Oven, and Toilet for every person is MVP, but we may end up needing a gas oven.
There's definitely more appliances we'll need to add in, we can speculate at what those are but won't really know until we have people living there
We think it needs to interact with this other company's building, they support mail and radio (but only AM).
We hired one person who knows how to work with brick, and 3 people that know drywall but are very eager to learn brick.
We also don't have a crazy budget for this until people start using it, so make it as close to free as possible.

It's easy to think "those requirements should really be cleaned up", but the fact is that there isn't many years of experience that could tell us what will/won't work in the industry. We're all literally building the airplane while it's flying (https://www.youtube.com/watch?v=S_dgWl83cTM).

There's a few options here:

Build a Skyscraper

You proactively build one giant building that could fit any possible ask that comes up.

You added the support for gas in anticipation for the gas oven but the request never came. Now you have to spend resources keeping gas pipes up to standard and good repair.

Several appliances you guessed would come up did in fact get requested! Value! Some appliances you had not anticipated, like a sauna room, did not really fit into your design though so they had to be hacked in as part of the bathrooms.

With the delivery industry being so successful, the entire Kitchen was dropped as a supported product. Actually removing all of those kitchens would be a ton of overhead so you remove the appliances that aren't bolted to the wall or floor and leave the rest hoping that someone will be able to use them.

Halfway through the project you realized that Brick 2.0 came out, which has breaking changes with how you implemented the bathrooms. Brick 3.0 is already being talked about in the industry and hiring skilled employees to maintain your skyscraper built on Brick 1.0 is becoming difficult.

You implemented a mail box, radio, cable, and a Batman signal for external communication, but since you only utilize the mail box all of the other systems stop working over time as renovations happen around them. There's speculation that someone may be stealing your cable but there's not enough interest to look into it.

As the skyscraper becomes more popular, there's plenty of room for the inhabitants, but they never quite fully utilize even a decent chunk of the allocated space.

Build a House and plan for Renovations

You build a mid sized house that achieves the MVPs of the request, and make sure you leave room for features you think are coming up.

The request for gas ovens never comes, just means you have extra space in the walls.

You were able to do some Renovations to the house to account for the sauna, as more requests come in you're noticing that the house is starting to bloat in size and maintaining the electric wiring and Brick upgrades is starting to take more time and interfere with other work in the house.

Dropping the kitchen as a product was difficult, but doable with some planned renovations. It did require stopping work on other portions of the house for a bit though.

Eventually a request comes in for a [working] fireplace. But because of the positioning of the solar panels from a few requests earlier you don't have a good place to put a chimney.

As the house becomes more and more popular, room begins to be a problem. You now have to figure out how to build an exact copy of your house multiple times to accommodate more people.

Build a series of Huts

You break down the requirements to identify what particular solutions are being solved:

Providing a sleeping space
Providing a way to cook food
Providing a private area for personal needs

You build three huts, one for each ask. You actually found that Drywall was better suited for the sleeping space hut, Brick for the kitchen hut, and a newer technology Tile for the restroom hut.

Several other smaller applications got added to existing huts, but nothing that really significantly increased their size.

You considered adding the Sauna as an expansion of the restroom hut, but realized it solved problems for a different set of people so gave it its own hut. This way renovations to your sauna won't break your toilet.

Dropping support for the kitchen was super easy (barely an inconvenience). You just took a wrecking ball to it.

The Brick upgrade only really needed to be applied to a few huts. They did interfere with work in those huts but not any in other huts, so was scheduled for a time when the Brick huts would not be worked on.

As more people are drawn to your solution, you can create many more bedroom huts for virtually no effort, while not needing to create as many of the less used Saunas.

Back to software

This last example, the Huts, is what micro services are. They are the centerpiece of modern Software Architecture, and in fact are going the direction of becoming even more micro (serverless, think if a Hut was designed just to have a sink cuz you may need more sinks than toilets).

The big point to be made, is that the end goal of modern Software Architecture is not about creating codebases that survive until the end of time the way Building Architecture does.

Modern Software Architecture is about creating small, modular solutions that can be spun up and thrown out on a whim as a reflection of their parent industries that are still practically in their infancies.

Quality is a Measure of the Problems You Don't Solve

Jonathan Eccker — Thu, 17 Mar 2022 14:42:35 +0000

We live in an era of infinite problems. Unfortunately our resources do not scale at the same rate as our problems. The big takeaway being that we can't solve everything.

An important assertion to have in a world with finite resources is that our usage of those resources should be efficient. This is what quality is, maintaining a high confidence in resource efficiency (whatever the resources may be).

We spend an insane amount of resources to maintain that high confidence in resource efficiency. The entirety of the Software Architect role exists because it's accepted that problem spaces evolve faster than our solutions.

There's an extremely resource efficient solution that's always been right in front of us though, and has no additional layers of resource consumption to maximize quality: Solve fewer problems.

Not Solving Problems

It almost feels silly giving an example of how to not solve problems. There's billions of examples, I actually find it very fun taking careful note of successful products and identifying the problems they intentionally don't solve.

But for the sake of this, let's talk Uber.

It has come a very long way, and solves infinitely more problems than anyone would have ever expected it to solve. But lets go back to their roots.

Uber was a black car service. They solved the problem space of on demand black cars with trained drivers giving you a ride from point A to point B.

A list of SOME of what they did not solve:

Tipping
Multiple stops
Transporting anything but people
Non trained drivers
Curbside taxi style service (think traditional yellow cab)

I am very confident that the technology was there to support those from the start. It would have been "trivial" to add a radio button for "Pickup target: Takeout, Person" to slap in an entire new User Story and make a few extra bucks. But Uber took its time adding features.

The end result is a very high quality product that solves exactly the problems it set out to solve, no more, no less.

Why Not?

The question "Why Not" gets used a lot more often than it should. It's by nature dismissive of the nuance of a problem space and is an attempt to bypass decision making or discussion.

When it comes to adding functionality to a product, it is extremely easy to think "Why Not?" and slip in extra low hanging fruit as functionalities.

What you will find is that "features" added this way have a high risk of being buggy or misused. They weren't added as part of a planned feature that solves a very specific problem, and will not get the support that other intentionally planned solutions have. This is a very active detriment to quality.

What problems should you solve?

There are a thousand decision making frameworks for prioritizing problems. I think the most relevant one to the software industry that also carries technical direction is Domain Driven Design. Microsoft does a much better job giving an overview of this than I could: https://docs.microsoft.com/en-us/azure/architecture/microservices/model/domain-analysis

For those less technically oriented or those who aren't looking to pick up an entire architecture analysis framework right now, there are two primary questions to answer:

Why do consumers spend resources (time, money, etc.) using your product instead of alternative solutions?
What domain expertise do you have that sets you apart?

Answering those questions is critical. Products that aren't in line with providing value to a consumer or aren't being built with domain experience backing them are very likely to be low quality. These are excellent candidates to start opting out of solving (sunsetting their functionality, outsourcing to a team with more relevant experience, or switching to a third party system to provide required functionality).

It's not Lazy, it's Energy Efficient

To wrap this up, I want to make a cultural observation. We've grown up in a society that has an absolute phobia of laziness. It's understandable, the world has achieved an inconceivable amount in the last hundred, thousand, million years. No one wants to be the cog not turning in the machine.

What many have failed to notice though, is that this phobia has made us feel obligated to tackle every single problem that we come across. To quote Bo Burnham, "apathy is tragedy and boredom is a crime".

Going back to my first assertion, resources are limited. On some level then, we need to make sure those are being used efficiently. YOU are an extremely limited resource. Go ahead and be energy efficient.

Upgrade your Value Objects in 10 Steps

Eric Damtoft — Tue, 17 Aug 2021 20:40:06 +0000

In object-oriented programming, it's tempting to overuse built-in types. For example, if you think of an object representing a vehicle, that vehicle's model year may be represented as a 32-bit integer, and it's make as a string. For many purposes, this is OK, but defining strong types for those values within your domain can make your code more expressive and allow the compiler become a powerful ally in verifying that your code is being used the way you intended it to.

However, if you look at built-in types, there are a lot of hidden features they implement which can add value. A simple implementation of a domain-specific value-object will likely miss some of these features and eventually lead to frustration for the users of that code. We'll look at some ways to make a true, fully featured value object with all the bells and whistles you've come to expect from .NET types.

As an example, let's use a simple Coordinate with an X and Y value.

class Coordinate
{
    public int X { get; set; }
    public int Y { get; set; }
}

Upgrade 1: Immutability

Not all types should be immutable, but any type which fundamentally represents a value can benefit from immutability. Immutability will generally simplify logic and clarify how your type is supposed to be used.

public class Coordinate
{
    public Coordinate(int x, int y) => (X,Y) = (x,y);

    public int X { get; }
    public int Y { get; }
}

Notice the constructor? This is just for style. Feel free to use standard assignments, but tuple assignments make for a simple one-liner. If you have more than 2 or 3 values to assign, it's probably better to use classic assignments.

Upgrade 2: Make it a struct

Structs have a number of advantages over classes for types which fundamentally represent a single value. They're passed by value and usually only allocated on the stack instead of a reference to the heap so in many cases they'll avoid allocations and reduce garbage collection, thus improving performance. Structs are best avoided for mutable types because of how they're passed, but for a small immutable type like a coordinate point, a struct is the way to go. See more about when to use a struct vs a class here. You can also use the readonly keyword so the compiler will enforce that the type is immutable.

public readonly struct Coordinate

Upgrade 3: Equatability

A good value type should be equatable by it's values. I.E. a coordinate (1,2) should equal another coordinate (1,2). Along with the Equals and GetHashCode methods, we should make sure that you can use the == and != operators and implements IEquatable<Coordinate>.

public override bool Equals(object obj) => obj is Coordinate coordinate && Equals(coordinate);

public bool Equals(Coordinate other) => (X, Y) == (other.X, other.Y);

public override int GetHashCode() => HashCode.Combine(X, Y);

public static bool operator ==(Coordinate left, Coordinate right) => left.Equals(right);
public static bool operator !=(Coordinate left, Coordinate right) => !(left == right);

Notice some interesting syntax in the Equals method. Comparing 2 tuples is an easy way to compare a number of properties. This is again a style choice and feel free to stick with a classic equality check for each property.

Upgrade 4: Formattability

We'It's common to need to specify a way of formatting a value to a string. It's always good practice to add a ToString() method, but there are often a variety of ways to represent any value as a string. In our case, we'll consider the numbers which make up the coordinate and allow them to be formatted like any other number value. To make this work, we'll add the IFormattable interface to our class. You could probably imagine a more sophisticated implementation that allowed for custom delimiter, etc. but for simplicity, we'll just forward the format onto each value.

public string ToString(string format, IFormatProvider formatProvider)
{
    return X.ToString(format, formatProvider) + "," + Y.ToString(format, formatProvider);
}

This way, you can use it in interpolated strings such as $"coordinates: {c:00}" to result in "01,02";

Upgrade 5: Parsability

If you notice, most built-in types have a static Parse and TryParse method. These are useful, and implementing them following the established pattern will make it easy for users.

We'll skip the details of actually parsing a value for brevity

public static Coordinate Parse(string s) => CoordinateParser.Parse(s);

public static bool TryParse(string s, out Coordinate c) => CoordinateParser.TryParse(s, out c);

Upgrade 6: Convertibility

Being able to easily convert between types is crucial to making a user-friendly type.

We can add implicit and explicit conversions to and from other types. This will allow the c# compiler to automatically translate these types either with or without an explicit cast.

For our coordinate class, let's imagine we also had an Vector type with a magnitude and a direction.

public static explicit operator Vector(Coordinate c) => Vector.FromCoordinate(c);

Upgrade 7: Operators

We've already added equality operators, and conversions are also considered by c# to be operators, but depending on the structure, consider adding additional operators specific to your case. In our instance, we'll just add some basic math operations, but you can also consider use cases like DateTime - DateTime = TimeSpan where the types are representative of the actual meaning of the object.

public static Coordinate operator +(Coordinate c) => c; // doesn't do anything, but complements minus
public static Coordinate operator -(Coordinate c) => new Coordinate(-c.X, -c.Y);
public static Coordinate operator ++(Coordinate c) => new Coordinate(c.X+1, c.Y+1);
public static Coordinate operator --(Coordinate c) => new Coordinate(c.X-1, c.Y-1);
public static Coordinate operator +(Coordinate left, Coordinate right) => new Coordinate(left.X + right.Y, left.X + right.Y);
public static Coordinate operator -(Coordinate left, Coordinate right) => new Coordinate(left.X - right.X, left.Y - right.Y);
public static Coordinate operator *(Coordinate left, Coordinate right) => new Coordinate(left.X * right.Y, left.X * right.Y);
public static Coordinate operator /(Coordinate left, Coordinate right) => new Coordinate(left.X / right.Y, left.X / right.Y);
public static Coordinate operator %(Coordinate left, Coordinate right) => new Coordinate(left.X % right.Y, left.X % right.Y);

Upgrade 8: Deconstruction

If we want to easily deconstruct values out of our type, we can add a Deconstruct method.

public void Deconstruct(out int x, out int y) => (x, y) = (X, Y);

This allows us to pull values out using the same syntax as tuple deconstruction.

var (x, y) = new Coordinate(1,2);

Upgrade 9: Debuggability

This one is a bit niche, but can be quite helpful for providing a view of a value geared specifically towards developers. If you inspect a value in the debugger, it'll show you the value from ToString(), but you can customize this by adding a DebuggerDisplayAttribute. In this case, we'll keep it mostly the same as the ToString(), but you could add other useful debugger information or context here as well.

[DebuggerDisplay("({X},{Y})")]
public readonly struct Coordinate : IEquatable<Coordinate>, IFormattable

Upgrade 10: Documentation

This one may go without saying, but once you're done adding features, make sure to take the time to put good XML comments on everything public. Styles and coding guidelines vary, but in general, I tend to go with the principal of only commenting when necessary and to add clarity, but if you're writing a library to be used by others, it's best to take the time to comment everything.

Also, take advantage of more than just the <summary> tag. There are a number of other useful tags for XML comments which are especially valuable if you use tools like DocFX to automatically generate a documentation website with details and examples.

/// <summary>
/// Converts a string into a coordinate object
/// </summary>
/// <param name="s">A string containing a coordinate to convert</param>
/// <returns>A coordinate equivelant to the string contained in <paramref name="s"/></returns>
/// <exception cref="FormatException">A format exception will be thrown if the structure of the string is unexpected</exception>
/// <example>
/// This demonstrates basic usage of the Parse method:
/// <code>
/// var c = Coordinate.Parse("1,2");
/// </code>
/// </example>
public Coordinate Parse(string s)

Putting it all together

We started with a simple class:

public class Coordinate
{
    public int X { get; set; }
    public int Y { get; set; }
}

This might be fine for a simple serialization bucket, but once our upgrades are all done, we've got a class worthy of inclusion in a world-class .NET library. Here's the end result (with comments omitted for brevity):

[DebuggerDisplay("({X},{Y})")]
public readonly struct Coordinate : IEquatable<Coordinate>, IFormattable
{
  public Coordinate(int x, int y) => (X, Y) = (x, y);

  public int X { get; }
  public int Y { get; }

  public override bool Equals(object obj) => obj is Coordinate coordinate && Equals(coordinate);

  public bool Equals(Coordinate other) => (X, Y) == (other.X, other.Y);

  public override int GetHashCode() => HashCode.Combine(X, Y);

  public override string ToString() => ToString(null, null);

  public string ToString(string format, IFormatProvider formatProvider)
  {
    return X.ToString(format, formatProvider) + "," + Y.ToString(format, formatProvider);
  }

  public void Deconstruct(out int x, out int y) => (x, y) = (X, Y);

  public static Coordinate Parse(string s) => CoordinateParser.Parse(s);

  public static bool TryParse(string s, out Coordinate c) => CoordinateParser.TryParse(s, out c);

  public static explicit operator Vector(Coordinate c) => Vector.FromCoordinate(c);

  public static bool operator ==(Coordinate left, Coordinate right) => left.Equals(right);
  public static bool operator !=(Coordinate left, Coordinate right) => !(left == right);

  public static Coordinate operator +(Coordinate c) => c; // doesn't do anything, but complements minus
  public static Coordinate operator -(Coordinate c) => new Coordinate(-c.X, -c.Y);
  public static Coordinate operator ++(Coordinate c) => new Coordinate(c.X+1, c.Y+1);
  public static Coordinate operator --(Coordinate c) => new Coordinate(c.X-1, c.Y-1);
  public static Coordinate operator +(Coordinate left, Coordinate right) => new Coordinate(left.X + right.Y, left.X + right.Y);
  public static Coordinate operator -(Coordinate left, Coordinate right) => new Coordinate(left.X - right.X, left.Y - right.Y);
  public static Coordinate operator *(Coordinate left, Coordinate right) => new Coordinate(left.X * right.Y, left.X * right.Y);
  public static Coordinate operator /(Coordinate left, Coordinate right) => new Coordinate(left.X / right.Y, left.X / right.Y);
  public static Coordinate operator %(Coordinate left, Coordinate right) => new Coordinate(left.X % right.Y, left.X % right.Y);

}

Ultimately, how far down the road you want to go is a judgement call. For some cases, the first version of the type is all that's required and exactly what's needed. But, if you're writing a library or core domain code, it's probably worth breaking out all the stops to make the experience smoother and more predictable for your users.

Supercharge Your Coding in Visual Studio

Scott Hintze — Thu, 12 Aug 2021 20:04:36 +0000

Keyboard Shortcuts

Hold Down Alt when Selecting Text

Let’s say you have a list of variables and you decide to change the scope from public to protected. You could use find and replace on the selected text, or you could just copy/paste over each modifier. Another option is block selecting. Holding down the Alt key while selecting allows you to include multiple lines to the position of the cursor rather than the entire line. The result is that when you type, cut or paste, it applies to every line selected. You could select all the public modifiers and simply type or paste protected to edit them all at once.

Ctrl + X and Ctrl + V with nothing selected

Cut and paste? What is so special about these? In Visual Studio. If you have nothing selected, the cut command will cut the entire line and paste will insert the entire line above the cursor. When moving lines around, this method is very fast.

Ctrl + K + C (comment) and Ctrl + K + U (uncomment) with nothing selected

These commands will comment and uncomment the entire line when no text is selected. When text is selected, they will comment and uncomment the selection. The uncomment shortcut will uncomment even if no text is selected as long as the cursor is within commented code.

Better Commenting

As nice as the default comment shortcuts are, you still have to use both hands. That can get tedious if you are doing a lot. Here is an easier combo to make it even faster.
Go to Tools>Options, then find the Environment>Keyboard node.

In the “Show commands containing box” enter “comment” (no quotes).

Select the Edit.ToggleLineComment shortcut and click in the “Press shortcut key” input field.

Press Left Ctrl + Left Windows, then click “Assign”

Repeat the process for Edit.UncommentSelection using Left Alt + Left Windows

Setting these shortcuts allows you to keep one hand on the mouse or arrow keys to navigate the code and the other to comment and uncomment quickly.

Visual Studio has a Back Button?!

Ever feel you are a bit like Alice and falling down the rabbit hole when navigating code? Visual Studio has a back button that tracks a history of where your cursor has been. Just click to go back through and retrace your steps. You can also click the down arrow to see the history list.

Tedious Code

Repetitive code, such as mapping database tables to object models, can be a real downer to a good programming groove. Here’s how to use simple Excel formulas to write hundreds of lines of code in seconds.
Get a list of the variable names or database fields without any extra punctuation. Then, paste the list into column A of your blank Excel sheet.
Excel string formulas are fairly straightforward. The basic syntax is =”string literal” & A1 & “string literal”. The quotes indicate literal text. Use ”” to escape a double quote in the string. Let’s say your variable name is Id and is located in the A1 cell of your sheet and you want to create a public string accessor property for it. The formula would be =”public string ” & TRIM(A1) & “{ get; set; }” Now, you just use the Excel fill handle and pull it down your list to "write" that line of code for each variable.

Server To Server Google Api Credentials Without a Json File in .Net

Jonathan Eccker — Thu, 17 Jun 2021 14:44:11 +0000

All Google documentation indicates that you need a special JSON file for configuration of google calls. This breaks the standard configuration pattern that .Net uses (IConfiguration abstraction/layers, IOptions injection, etc.)

After poking around at the classes available in the Nuget Package, I was able to hook this all up for server side calls in a very simple way.

To cut to the chase, here's the code, note that this is code for interacting with the Calendar API but it'll be similar for other API services.

public sealed class CalendarEventRepository : ICalendarEventRepository
{
  private readonly GoogleConfiguration _configuration;
  public CalendarEventRepository(IOptions<GoogleConfiguration> options)
  {
    _configuration = options.Value;
  }
  public async Task<IEnumerable<CalendarEvent>> GetUpcomingCalendarEvents(CancellationToken cancellationToken)
  {
    var credentialJson = new
    {
      type = "service_account",
      project_id = _configuration.ProjectId,
      private_key_id = _configuration.PrivateKeyId,
      private_key = _configuration.PrivateKey.Replace("\\n","\n"),
      client_email = _configuration.ClientEmail,
      client_id = _configuration.ClientId,
      auth_uri = "https://accounts.google.com/o/oauth2/auth",
      token_uri = "https://oauth2.googleapis.com/token",
      auth_provider_x509_cert_url = "https://www.googleapis.com/oauth2/v1/certs",
      client_x509_cert_url = "https://www.googleapis.com/robot/v1/metadata/x509/" + Uri.EscapeUriString(_configuration.ClientEmail)
    };
    var credentials = GoogleCredential.FromJson(JsonConvert.SerializeObject(credentialJson)).CreateScoped(new[] { CalendarService.Scope.Calendar });

    var service = new CalendarService(new BaseClientService.Initializer()
    {
      ApplicationName = "<Your Application Name>",
      HttpClientInitializer = credentials
    });
    var eventRequest = service.Events.List(_configuration.CalendarId);
    eventRequest.MaxResults = 10;
    var result = await eventRequest.ExecuteAsync();
    return result.Items.Select(i => new CalendarEvent(HashToGuid(i.Id), i.Summary, DateTimeOffset.Parse(i.Start.Date), DateTimeOffset.Parse(i.End.Date))).ToArray();
  }

  private Guid HashToGuid(string input)
  {
    using (MD5 md5 = MD5.Create())
    {
      byte[] hash = md5.ComputeHash(Encoding.Default.GetBytes(input));
      return new Guid(hash);
    }
  }
}

The important points to note are:

You can use GoogleCredentials.FromJson and assign that to HttpClientInitializer
I did a .Replace("\\n","\n") on the private key because many configuration providers don't support new lines
The variables you'll need to add into configuration are: ProjectId, PrivateKeyId, PrivateKey, ClientId, ClientEmail (and CalendarId if you're using calendars)
The Google Nuget Package I'm using here is Google.Apis.Calendar.v3, version 1.52.0.2312

Rolling Back SSDT (And Why You Can't)

Jonathan Eccker — Thu, 17 Jun 2021 13:55:06 +0000

SSDT is a tool for source controlling SQL Schema.

How SSDT works

It works very similar to application builds. SSDT compiles "code" (CREATE scripts) into a "build" (a Dacpac), and you can deploy it (generally using SqlPackage) to a database.

You can even publish it from visual studio the same way you might an application. There's a ton of parallels in their functionality.

The Catch

In most applications, when you deploy a bad build and a fix is not available in a timely manner, you can typically just deploy the previous build to "roll it back".

This does not work with dacpac deploys. Dacpacs are designed to only deploy in a forward, progressive manner.

There's a number of reasons for this:

SqlPackage and the database will not know how to revert refactors that have been ran (this is the big one)
Columns that were added to populated tables cannot be deleted without disabling BlockOnPossibleDataLoss (very high risk)
Newly created objects will not be dropped unless you enable DropObjectsNotInSource

There's some other smaller issues you can run into in edge cases as well.

The Solution

It's generally pretty simple, to fix a bad dacpac/build, you simply create a new one with whatever refactorlog changes or table changes are needed to make the fix.

For higher risk changes (changing column names for example) it can be suggested to have a "just in case" dacpac ready that would undo the one you just deployed, in the event you need to urgently "roll it back".

The philosophy, much like many CI/CD philosophies, is to keep builds moving forward, never taking steps back.

It might also be worth noting there are mechanisms like synonyms and computed columns for minimizing risk of changing names.

Example

I have a very simple SSDT example project with a single Person table:

CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [Name] NVARCHAR(64)
)

I've created Dacpac 1 for this, and deployed it.

Let's roll out a change to the name column, using the Visual Studio refactor tool to refactor the column Name to FullName:

CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [FullName] NVARCHAR(64)
)

This also creates a refactor log entry:

<Operation Name="Rename Refactor" Key="813ae462-b83f-4dc8-ab48-12c56b09137b" ChangeDateTime="06/10/2021 17:37:09">
  <Property Name="ElementName" Value="[dbo].[Person].[Name]" />
  <Property Name="ElementType" Value="SqlSimpleColumn" />
  <Property Name="ParentElementName" Value="[dbo].[Person]" />
  <Property Name="ParentElementType" Value="SqlTable" />
  <Property Name="NewName" Value="[FullName]" />
</Operation>

I created Dacpac 2 for this, and deployed it. The script looks like:

PRINT N'The following operation was generated from a refactoring log file 813ae462-b83f-4dc8-ab48-12c56b09137b';
PRINT N'Rename [dbo].[Person].[Name] to FullName';
GO
EXECUTE sp_rename @objname = N'[dbo].[Person].[Name]', @newname = N'FullName', @objtype = N'COLUMN';
GO
-- Refactoring step to update target server with deployed transaction logs
IF OBJECT_ID(N'dbo.__RefactorLog') IS NULL
BEGIN
    CREATE TABLE [dbo].[__RefactorLog] (OperationKey UNIQUEIDENTIFIER NOT NULL PRIMARY KEY)
    EXEC sp_addextendedproperty N'microsoft_database_tools_support', N'refactoring log', N'schema', N'dbo', N'table', N'__RefactorLog'
END
GO
IF NOT EXISTS (SELECT OperationKey FROM [dbo].[__RefactorLog] WHERE OperationKey = '813ae462-b83f-4dc8-ab48-12c56b09137b')
INSERT INTO [dbo].[__RefactorLog] (OperationKey) values ('813ae462-b83f-4dc8-ab48-12c56b09137b')
GO

But oh no! Everything broke cuz we forgot to update the application first.

Trying to redeploy Dacpac 1, the script ends up looking like:

/*
The column [dbo].[Person].[FullName] is being dropped, data loss could occur.
*/
IF EXISTS (select top 1 1 from [dbo].[Person])
    RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
PRINT N'Altering [dbo].[Person]...';
GO
ALTER TABLE [dbo].[Person] DROP COLUMN [FullName];
GO
ALTER TABLE [dbo].[Person]
    ADD [Name] NVARCHAR (64) NULL;
GO

That doesn't actually undo the rename. It drops FullName and creates Name. Or, more accurately, it fails because there is data in the table.

So the resolution is to create a new build. I use the Visual Studio refactor tool to rename the column FullName to Name:

CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [Name] NVARCHAR(64)
)

Now the refactor log has 2 entries:

<Operation Name="Rename Refactor" Key="813ae462-b83f-4dc8-ab48-12c56b09137b" ChangeDateTime="06/10/2021 17:37:09">
  <Property Name="ElementName" Value="[dbo].[Person].[Name]" />
  <Property Name="ElementType" Value="SqlSimpleColumn" />
  <Property Name="ParentElementName" Value="[dbo].[Person]" />
  <Property Name="ParentElementType" Value="SqlTable" />
  <Property Name="NewName" Value="[FullName]" />
</Operation>
<Operation Name="Rename Refactor" Key="7c29f405-8585-41ed-8ae0-47ffb0ec8128" ChangeDateTime="06/10/2021 17:43:44">
  <Property Name="ElementName" Value="[dbo].[Person].[FullName]" />
  <Property Name="ElementType" Value="SqlSimpleColumn" />
  <Property Name="ParentElementName" Value="[dbo].[Person]" />
  <Property Name="ParentElementType" Value="SqlTable" />
  <Property Name="NewName" Value="[Name]" />
</Operation>

Generating this as the build Dacpac 3, the script comes out looking like:

PRINT N'The following operation was generated from a refactoring log file 7c29f405-8585-41ed-8ae0-47ffb0ec8128';
PRINT N'Rename [dbo].[Person].[FullName] to Name';
GO
EXECUTE sp_rename @objname = N'[dbo].[Person].[FullName]', @newname = N'Name', @objtype = N'COLUMN';
GO
-- Refactoring step to update target server with deployed transaction logs
IF NOT EXISTS (SELECT OperationKey FROM [dbo].[__RefactorLog] WHERE OperationKey = '7c29f405-8585-41ed-8ae0-47ffb0ec8128')
INSERT INTO [dbo].[__RefactorLog] (OperationKey) values ('7c29f405-8585-41ed-8ae0-47ffb0ec8128')
GO

Which successfully gets us back to a working application.

For science, lets look at what happens when this is deployed to a database that has not had either Dacpac 2 or Dacpac 3 deployed to it:

-- Refactoring step to update target server with deployed transaction logs
IF NOT EXISTS (SELECT OperationKey FROM [dbo].[__RefactorLog] WHERE OperationKey = '813ae462-b83f-4dc8-ab48-12c56b09137b')
INSERT INTO [dbo].[__RefactorLog] (OperationKey) values ('813ae462-b83f-4dc8-ab48-12c56b09137b')
IF NOT EXISTS (SELECT OperationKey FROM [dbo].[__RefactorLog] WHERE OperationKey = '7c29f405-8585-41ed-8ae0-47ffb0ec8128')
INSERT INTO [dbo].[__RefactorLog] (OperationKey) values ('7c29f405-8585-41ed-8ae0-47ffb0ec8128')
GO

It's smart enough to know it does not need to change the column Name to FullName and then back to Name. It just inserts the refactor logs, and does nothing else.

Manual Rollbacks

I'm going to call this out as a capability, but would really like to emphasize that it can introduce a lot of risk and is not suggested except for the most dire and urgent of situations.

If you were to manually undo a refactor, as in via non dacpac generated SQL execute a "ALTER TABLE..." script and delete the entry from "__RefactorLog", you can effectively put the database into a state as if the refactor had never happened.

This introduces a lot of complex, risky, variables. Especially if you are using database versioning, you could have a database that thinks it's on one version but is actually missing a piece of it. You could potentially have one environment that has undone one refactor but other environments haven't, so you get inconsistent behavior of subsequent deploys. If you botch the manual rollback, you're going to end up in a weird state where further dacpac deploys do weirder things that can potentially make your database state even worse.

So to summarize, except in very very rare edge cases (that we should design process to avoid), rollbacks for dacpacs don't really exist. Keep things progressing forward.

Dropping Objects with SSDT

Jonathan Eccker — Tue, 15 Jun 2021 13:55:57 +0000

SSDT is a tool for source controlling SQL Schema.

How SSDT works

It works ALMOST exactly like you would expect it to work: you add CREATE scripts to define objects. You publish this script utilizing a Dacpac which is essentially a build. SqlPackage uses this dacpac to diff what's in your build versus what's in the target database, and creates scripts to resolve differences.

The Catch

Accounting for human error means accepting the possibility that a typo can lead to an entire table to be dropped. Simply renaming a column or table and not checking in a refactor log, without checks could lose irrecoverable data.

The Solution

SqlPackage has two major mechanisms to enable avoiding this:

Dacpac deploys will NOT by default drop objects it finds in the target database that do not exist in the source. This means just deleting a table in SSDT will not try to drop it on a deploy. This does not apply to columns.
Before a dacpac deploy tries to delete any object that can hold data (I.E: a column, table) it does a select to check if there's data that would be deleted. If it finds data, it aborts the operation.

There are options you can enable in SSDT to disable these layers of protection. It is HIGHLY recommended that you do not enable these options except when you have time to pay extra attention to a deployment.

Drop Objects In Target But Not In Source

The SqlPackage parameter is DropObjectsNotInSource, in visual studio you can find this under Advanced -> Drop -> Drop objects in target but not in source.

This does NOT disable the second layer of protection (preventing data loss), but it will cause the deploy to ATTEMPT to drop tables/stored procedures/etc. that do not exist in the dacpac.

It is important to note that you can specify what types of objects to NOT drop when using this. For instance, if you don't use dacpac deploys to manage Users/Permissions/etc. you can set DoNotDropObjectTypes=Users,Permissions to NOT drop those.

Block Incremental Deployment If Data Loss Might Occur

The SqlPackage parameter is BlockOnPossibleDataLoss, in visual studio you can find this under Advanced -> Block incremental deployment if data loss might occur.

This is the big scary one you should be extremely careful with. Disabling it causes any drops that would occur to happen without checking for data first.

Example

I have a quick example SSDT project, it has a single Person table:

CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [Name] NVARCHAR(64)
)

If I delete Person.sql and generate a publish script:

USE [$(DatabaseName)];
GO
PRINT N'Update complete.';
GO

Nothing happens.

Now, if we enable DropObjectsNotInSource, and generate the script again:

USE [$(DatabaseName)];
GO
/*
Table [dbo].[Person] is being dropped.  Deployment will halt if the table contains data.
*/
IF EXISTS (select top 1 1 from [dbo].[Person])
    RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
PRINT N'Dropping [dbo].[Person]...';
GO
DROP TABLE [dbo].[Person];
GO
PRINT N'Update complete.';
GO

And once more, with both DropObjectsNotInSource enabled AND BlockOnPossibleDataLoss DISABLED:

USE [$(DatabaseName)];
GO
PRINT N'Dropping [dbo].[Person]...';
GO
DROP TABLE [dbo].[Person];
GO
PRINT N'Update complete.';
GO

Recommended Processes For Deleting Objects

I highly recommend avoiding disabling BlockOnPossibleDataLoss as much as possible. From experience, it is VERY easy for a small oversight or misunderstanding of a dacpac/build to lead to losing some critical data.

The Simple Process

The simplest course of action for dropping tables is to manually delete all rows in the table when you're 100% confident they are not used, then they'll be dropped with no complaints with a normal DropObjectsNotInSource.

A More Controllable Process

A process that takes a bit more overhead but provides more confidence before dropping objects is to deprecate tables/columns/etc. before dropping them.

You can utilize a deprecated schema (you can use schema specific permissions to enforce applications not accidentally using things in this schema) that you move tables/etc. into to mark them for deletion. Or append _deprecated to column names to mark them as dead. This will also give you time to roll back the deprecation if you do find something still using it.

Then, on some regular basis, you can go through and delete all deprecated objects at once. When you do this you will want to pay extra attention to the generated script BEFORE you deploy it. Make sure it's only going to drop what you expect it to drop.

You can do this in visual studio by selecting Generate Publish Script instead of Publish in the Publish window (I wish those weren't right next to each other). This is actually a good habit to get into every now and then to get insight into what deploys actually do.

The SSDT Refactor Log

Jonathan Eccker — Fri, 11 Jun 2021 18:02:17 +0000

SSDT is a tool for source controlling SQL Schema.

How SSDT works

The Catch

There's a somewhat big catch. The above process relies on being able to match objects in the dacpac to objects in the target database. If you rename a table, how would it know that [RenamedTableName] is the same table as [TableName]? From SqlPackage's perspective, it would think that [TableName] got dropped and [RenamedTableName] was created. (It won't actually drop it unless you tell it to, but that's a blog for another time)

The Solution

The solution is to be more explicit with when certain refactors happen. The dacpac carries instructions on refactors that happened that SqlPackage would not be able to intuitively know.

There's two mechanisms that enable this:

The <projectName>.refactorlog file that gets checked in that lets you record refactors
The __RefactorLog table that keeps track of which refactors have already been ran on a database

.RefactorLog File

The .refactorlog file consists of XML, mainly just a list of <Operation> elements. Each of those represent an individual refactor.



<Operation Name="Rename Refactor" Key="3a7b06a1-5e83-47f9-95d7-2a2fbaf2a20e" ChangeDateTime="06/10/2021 15:16:10">
  <Property Name="ElementName" Value="[dbo].[Country]" />
  <Property Name="ElementType" Value="SqlTable" />
  <Property Name="ParentElementName" Value="[dbo]" />
  <Property Name="ParentElementType" Value="SqlSchema" />
  <Property Name="NewName" Value="[CountryInfo]" />
</Operation>

The above is an example of a refactor that changes a table Country's name to CountryInfo. I wouldn't worry too much about learning the syntax for these entries, if you are using SSDT in Visual Studio it creates them for you.

__RefactorLog Table

This is a VERY simple table that just keeps track of which refactors have already been ran. Note that it automatically gets created when you deploy a dacpac to your database.

It only has one column, OperationKey, which just holds a list of GUIDs. Those are the GUIDs you can see in the Key field in the Operation entry shown above.

When you deploy a dacpac, it will fill this table with every refactor that got ran, to avoid it trying to run the same refactor every single time (which would obviously fail in most cases).

Example

Here's a brief example of changing a column name. I've started with a very simple SSDT project with just one table in it:



CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [Name] NVARCHAR(64)
)

Now to test it WITHOUT the refactor log, let's just change the column name Name.



CREATE TABLE [dbo].[Person]
(
  [Id] INT NOT NULL PRIMARY KEY IDENTITY(1,1),
  [FullName] NVARCHAR(64)
)

Generating a script for what this would do (doable from the publish window in Visual Studio), we get:



/*
The column [dbo].[Person].[Name] is being dropped, data loss could occur.
*/
IF EXISTS (select top 1 1 from [dbo].[Person])
    RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
PRINT N'Altering [dbo].[Person]...';
GO
ALTER TABLE [dbo].[Person] DROP COLUMN [Name];
GO
ALTER TABLE [dbo].[Person]
    ADD [FullName] NVARCHAR (64) NULL;
GO

Obviously this is not what we were going for.

Undoing the manual change, and utilizing the Visual Studio Refactor option (I HIGHLY suggest using this as opposed to manually making refactor logs. It also will automatically update stored procedures, functions, views, etc. that were referencing the renamed table/column for you):

It has now created a ExampleDatabase.refactorlog (because this was the first refactor), with a single Operation entry:



<Operation Name="Rename Refactor" Key="0517c5b8-8ae1-4642-ba65-9465fa2daf3c" ChangeDateTime="06/10/2021 15:36:05">
  <Property Name="ElementName" Value="[dbo].[Person].[Name]" />
  <Property Name="ElementType" Value="SqlSimpleColumn" />
  <Property Name="ParentElementName" Value="[dbo].[Person]" />
  <Property Name="ParentElementType" Value="SqlTable" />
  <Property Name="NewName" Value="[FullName]" />
</Operation>

Generating the script again, we get:



GO
PRINT N'The following operation was generated from a refactoring log file 0517c5b8-8ae1-4642-ba65-9465fa2daf3c';
PRINT N'Rename [dbo].[Person].[Name] to FullName';
GO
EXECUTE sp_rename @objname = N'[dbo].[Person].[Name]', @newname = N'FullName', @objtype = N'COLUMN';
GO
-- Refactoring step to update target server with deployed transaction logs
IF OBJECT_ID(N'dbo.__RefactorLog') IS NULL
BEGIN
    CREATE TABLE [dbo].[__RefactorLog] (OperationKey UNIQUEIDENTIFIER NOT NULL PRIMARY KEY)
    EXEC sp_addextendedproperty N'microsoft_database_tools_support', N'refactoring log', N'schema', N'dbo', N'table', N'__RefactorLog'
END
GO
IF NOT EXISTS (SELECT OperationKey FROM [dbo].[__RefactorLog] WHERE OperationKey = '0517c5b8-8ae1-4642-ba65-9465fa2daf3c')
INSERT INTO [dbo].[__RefactorLog] (OperationKey) values ('0517c5b8-8ae1-4642-ba65-9465fa2daf3c')
GO

Which is the intended result.

Rolling Back Refactors

At some point I'll have an entire article on rolling back Dacpac deployments, or more accurately how you CAN'T roll back Dacpac deployments.

I think it's worth mentioning here, however, that there isn't really a mechanism for undoing refactor log changes. You can't simply deploy an older dacpac and have it undo refactors - it wouldn't even know what to undo.

The best (safest, most structured, controllable) mechanism to undo a refactor log is to create a new refactor that undoes it, and deploy that via a newer dacpac. SSDT is about always having builds move forward, even if "moving forward" is changes that undo the previous "move forward".

As someone who's manually deleted from the __RefactorLog table and undone schema changes by hand in the midst of urgent, super hot, situations, I can definitely say that the potential complications manual management of the __RefactorLog can introduce will almost never be a worthwhile risk versus moving forward with an "undo" refactor log committed.