On professional standards

Building software occupies an unusual space, it's certainly more of a vocation along side plumbers, electricians and technicians, however there remains an enormous scope for creativity and problem solving because unlike those aforementioned vocations computers more-or-less lack any physical or chemical laws governing our application of techniques or which can constrain our fantastically complicated creations.

I'd go as far to say that the debate about whether or not people need to have computer science degrees is wholly misguided. Scientists ... let's say people with an analytical mind, a fundamental training in the "laws of nature" of computers, these people may make excellent engineers, but science is not engineering, and a computer sciences degree is absolutely not a guarantee of being a capable engineer, just as a physics degree doesn't qualify someone to engineer buildings, bridges or skyscrapers, even if it might equip them with the right analytical skills.

I am generally opposed to certifications and standards, as I believe that easy access to technology-focused knowledge work is such an enormous enabler for individuals, that the floodgates for new boot-camp graduates and interns should be wide, wide open to encourage as many people as possible into the field, as the skills base widens, and as accessible low- and no-code tools proliferate the volume of software in the world continues to grow, and more and more opportunities continue to exist where profitable businesses need extremely highly skilled engineers to assist them, and those engineers can command a premium salary or rate.

Technology enjoyed this way can line the pockets of everyone concerned, the fresh-faced rookies, the 200 year old jaded witches and wizards (architect years are like dog years.), and the company who is profiting all the time (maybe with a smaller gross margin than they wished..) from the software we build.

That all said, in the absence of professional standards such as those which exist in the form of building, HVAC, electrical and plumbing codes, it continues to be exhausting and draining to constantly be held back by sub-standard work, and for that definition of sub-standard to be entirely subjective.

For all the progress a software architect (or someone in a Principal or Distinguished role) can make in fostering a culture of collaboration, design, deliberate decision making, and modelling in collaboration with stakeholders from diverse disciplines (towards business or IT operations), the environmental pressure to make changes at any cost are everpresent, and in the absense of objective measures (including standards & certifications) things can rapidly devolve into the famed "ball of mud" non-architecture which has more in common with the Shanty towns than anything you might recognise as deliberate architecture.

Just as Shanty towns are often shaped by the environmental constraints, lack of resources and rapid growth, software projects can suffer the same. Shanty towns can gentrify info middle-class neighbourhoods, but more often than not, at least in software the growth on the fringes out-paces the gentrification and investment in the core.

The architect(s) must focus their energy, and choose their battles wisely.

Ultimately the architect(s) require the support of the tradespeople working hands-on on the code every day. Often programmers lament that their business and project counterparts don't grant them time for re-factoring, writing tests, or conducting load tests, however their reasoning is likely flawed, nobody negotiates (at least not over the table) those things with their HVAC, plumbing, or electrical technicians.

Those tradespeople have stacks of standards, international and regional to adhere to. An electrician will not add a new circuit to a pre-1950s house (which in many parts of the world lack the 3rd wire in the wall sockets which carries a protective earth to protect against electrocution) without performing the remedial work to re-fit the house with modern wiring, no matter the cost. The liability, insurance, loss of license and with it, their livelihood is a cost so high that with sub-standard wiring in your outdated home, professionals cannot, and will not take shortcuts.

The rise of the open office may indeed correlate with the cost of installing adequate HVAC systems in closed-office environments, and tenants would likely violate their lease and put their employees' health in danger by taking shortcuts, thus HVAC professionals will also not take the risk of assuming the liability of contributing to sub-standard work.

Technology however doesn't directly have these kinds of standards in place, sure there's security standards such as ISO 27001, the GDPR (article 15) mandates that access to data is logged to assist in preserving the data subject's rights, financial and telecommunications software is required to keep data for a long time for auditing, and certain fields such as the payment card industry specifically mandate against storing data.

With all those "non-functional" requirements, however enforcement and policing is extremely lax. Article 15 of the GDPR should render it illegal for any software to show any data without leaving a trace, however most Django, Rails, Express.JS or similar applications will happily serve a GET request and log practically nothing in the logs, which may or may not even be persisted. I make no judgement on whether or not the global invalidation of practically every piece of software on the planet is a bug or feature of the GDPR, but our industry is steadfast in it's refusal to even address the topic.

Digressing somewhat to professional standards, most seasoned architects and engineers would begin to raise questions about whether using some or all of a dynamically typed language, without CI/CD, with a privileged database user account, with access to an SSH console would begin to constitute malpractice, if we drew parallels to tradespeople in the physical world if would be akin to working on a live electrical panel, or sourcing fresh air for HVAC From next to a busy highway, or recycling water from a bathroom into a kitchen, etc, etc.

Less clear, again is perhaps the use of certain paradigms such as stateless- or functional-programming, event sourcing, certain IPC strategies, and various combinations of the above, one may not forget that just like a building is built once, and lived in forever, code too is built once, and lived in for a long time, perhaps it is worth investing.

A passing slight at Agile, before I wrap up, which is that often Agile is used as a tool to keep things moving, no matter the total cost, as long as the immediate costs are bearable, in a strongly opinionated ecosystem where technology choices are made for you, this may be appropriate. I might be tempted here to draw parallels between "renting" space someone else designed, living out of a small studio apartment, to how it is to "rent" space in an opinionated inflexible framework, there may come a time when really investing into "where you live" as an engineer in a business context makes more sense, but having rented for the first few years of your life, you may lack the maturity to do sufficient, just-in-time planning; a delicate skill which takes years and years of vocational training to master.

Of course, just as hand-building a shed in your garden under a certain size is not subject to building regulation, nor should all software be subject to regulation, we must, at all costs preserve the accessibility of our industry at large, and preserve it as a creative practice in which people can innovate, but we also urgently need to develop a philosophy of our own, and begin to identify objective measures of good and bad software, I might start with something like:

• Some application of schema definition tools Any polymorphic type system, json schema, AVRO schema, etc, etc. Polymorphism (i.e interface types, traits, generics, etc) is important to avoid hard static typing, but dynamic typing leaves too much surface are for testing. With a proper type schema, parameterized testing becomes feasible, and the total set of inputs and outputs becomes a much smaller, but likely still infinite set. Type systems also provide a great platform for documentation, IDEs, language servers, generating documentation, and evolving a schema over time. Avro is outstanding here, as is GraphQL as schema definition languages, the rest (all of them.) fall down on one or more vital area.
• Proper application of an "aggregate root" pattern For example, avoiding the "active record" pattern, and grouping locally related objects such that data access is controlled by the "root" (i.e in a "invoice has line items" example, the invoice would be the entry-point for changes to lineitem quantity to apply business rules, calculate taxes, check customer standing, and stock control. Restricting a "model" to the core few objects of a system. ActiveRecord isn't an objectively bad pattern, but when all objects are addressable without constraint, it can be nearly impossible to maintain a strong aggregate-root based model, to leverage the often more powerful Repository Pattern, or Event Sourcing, and it's difficult to leverage the extensive capabilities of the underlying data stores.
• Proper application of modelling tools and design strategies Framework-over-database is precisely the absence of software design, in today's environment the developers who operate in that level will soon find themselves in real trouble from low-and-no-code backed by a Firebase database or a Google Sheets sheet. Just as a highly professional building crew will not be enthused about trying to turn a Shanty Town shack into a built-to-code residence, software architects and highly professionalized teams will be reluctant to
• Proper managing of side-effects and transactional guarantees An operation such as booking a cinema ticket, or securing a package holiday should have very clear guarantees about the transactional properties. It should not be possible to book 1 of n tickets in a mult-seat booking, or book a flight without a hotel if the customer tried to secure both. In the absence of support from your data store, some kind of "two-phase" transactions can be emulated for which established patterns exist. Controlling for side-effects (such as API calls, IO, reading random numbers, reading from external data sources) are harder to control for, but again practices are well established in some communities to have entirely deterministic execution of some or all codepaths, and simulation tools which will will rigorously "prove" a "pure" (no side effects) model, these tools are effectively unused entirely by business software, to the detriment of practically everyone.
• Proper Auditability Probably through some kind of event-sourcing mechanism (e.g log everything that happens as the source of truth, and derive projections from it, PDFs for your stakeholders, and various read-only databases which can be hyper optimized for your use-cases). Event sourcing is the oldest record keeping strategy on the planet, and with the blistering power of modern computer systems almost any amount of data can be managed in an event-sourced fully audited manner, with proper caching strategies, careful treatment of read-cursors, and proper aggregate partitioning schemes this architecture can be potentially infinitely scalable. Regrettably it only works right if you apply it along side the other proper things enumerated in this list.
• Proper user- and authentication-strategies Conflating a single human being with a single account is a sin committed by practically every piece of software on the planet, ever. That being said decoupling the human who is authorized from the "account", and permitting many-to-many-to-many relationships enables enormous agility in software. Such strategies would permit seamless "customer authorizes admin to log into their account" for support without requiring super-admins (violates the GDPR), would assist elders who are not intimately familiar with technology to delegate access to their accounts to trusted relatives temporarily or permanently, and allows developers under careful control to impersonate, or share accounts on testing environments. Here are mature and established standards such as OAuth 2.0 which have stood for a decade or more, applied correctly they are enormously liberating. Role-based-access-control (RBAC), super user "admin" accounts, and similar anti-patterns can be successfully avoided if OAuth is applied properly with a sympathetic understanding of the nuance in in the spec, however again this requires taste, and expertise, and must overcome the ever-present draw to grab the simplest library in the language's package registry and copy 10 lines from a README and move on to the next ticket.

Application of all of those patterns above, at precisely the right time, not too soon and not too late; which requires intuition and experience, something which is regrettably entirely subjective, it also inflates enormously the cost in some places, but radically reduces cost over time as a framework-over-relational-database model stagnates, and becomes intractable, a properly designed system can be an everlasting joy to maintain and offer an enormous competitive advantage.

I think Agile and XP practices, the easy accessibility of framework-first development, and the overwhelming ratio of inexperienced, to experienced programmers means that the ideas above will never reach mainstream adoption.

For every senior, experienced programmer who takes time to understand the needs of the stakeholder, the regulatory and environmental (i.e existing code, existing practices), and attempts to produce high quality, well-thought-out work, which is sympathetic to the entire nuanced multi-dimensional spectrum of needs.... there will be ten less experienced people who lack the experience necessary to build for maintenance and will through malice or inexperience (incompetence?) offer a "competing", but sub standard solution.

Sometimes you don't need top-of-the-line high skill plumbing, and you literally need the software equivalent of a bucket and hose, but business stakeholders often can't tell the difference, and if a more experienced engineer commits to a certain time frame, scope and cost, there will often be that less experienced engineer promising the business what they need, right this second, for half the price, with no up-front discussion. To both the inexperienced engineer and the business stakeholder, they are apparently offering the same thing as the more experienced engineer, but the more experienced engineer is offering a different, significantly higher quality product, so the "market" cannot behave rationally in the absence of regulation or objective standards.