DEV Community: Andrew Tetzeli

DevOps Fail: "Windows Update Zero-Day Being Exploited to Undo Security Fixes"

Andrew Tetzeli — Wed, 11 Sep 2024 20:21:31 +0000

In not-good news for DevOps, Microsoft released vulnerable software as part of its Updates subsystem. The flaw allowed the rolling back of patches to -- you guessed it -- other security flaws. Security Week

We're waiting for it to reach the update-to-patch-flawed-update-to-fix-flawed-update-in-the-prior-update stage.

Stay tuned. We'll keep you posted.

From the Microsoft bulletin:

“Microsoft is aware of a vulnerability in Servicing Stack that has rolled back the fixes for some vulnerabilities affecting Optional Components on Windows 10, version 1507 (initial version released July 2015)."

Big implications for DevOps: German BSI developing measures to prevent another CrowdStrike-style fiasco

Andrew Tetzeli — Thu, 15 Aug 2024 19:08:56 +0000

A very large and top-rated government cybersecurity center, the German Federal Office for Information Security (BSI), has taken the lead in taking steps to fend off another CrowdStrike-style fiasco.

The BSI's actions should have big implications for DevOps, and not just at CrowdStrike and Microsoft.

The BSI has been in talks with both CrowdStrike and Microsoft about their DevOps following the July 19 global IT meltdown. As a result, BSI will be working with both companies to ensure that systems can be started in a minimum safe mode, even if serious errors occur.

BSI's long-term goal is to have new and resilient components designed and implemented offering the same functionality and level of protection as before, but which require less invasive permissions to operating systems. This aims to minimize the impact of software errors.

BSI has been in direct contact with Crowdstrike in Germany and in the USA since the incident on 19 July 2024. Following the immediate measures taken by the software vendor to prevent further incidents and the provision of an initial workaround for the affected systems, preliminary analysis reports on this incident were continually discussed between Crowdstrike and BSI and subsequently published. Based on the discussions, the evaluation of the available analyses and continued feedback from the vendor, BSI has initially developed the following measures:

Short-term measures until 15 August 2024

Impact analysis of the security incident in Germany
Continuous tracking of the recovery rate of affected systems (as of 25 July 2024 21:54 CEST and according to Crowdstrike, 97 percent of all systems with Windows sensors are already back online)
Merging already issued short-term warnings with expected incident-related CVEs based on the established CVD process

Medium-term measures until 30 September 2024
Evaluation of the upcoming detailed and final analysis report (root cause analysis)

Review of the current and the improved test concept of Crowdstrike by BSI in coordination with other international partner agencies and discussion of necessary adjustments with Crowdstrike
Clarification of future measures to ensure a rapid rollout of business logic/signatures while strictly guaranteeing the operational stability of customer systems
Testing the effectiveness of the progressive and closely monitored update rollout process to customers as already announced by Crowdstrike with extended telemetry analyses by Crowdstrike for immediate detection of faults after installation of the updates
Raising the awareness of organizations using Crowdstrike products about fundamental operational risks (cf. https://www.crowdstrike.com/terms-and-conditions-de/) and creating sufficient operational redundancies for critical deployment scenarios

Long-term measures until 31 December 2024

Discussion of concrete possibilities for evaluating the vendor's software development processes by independent third parties based on announcements already made by Crowdstrike
Establishing a cooperation between BSI, Crowdstrike, and Microsoft with the objective to ensure booting of the system at least in a restricted mode, even in the event of serious malfunction of the EDR tool
Initial discussions with all relevant stakeholders on the architecture of EDR tools to increase their resilience

Further measures in 2025

Design and implementation of new, more resilient architectures for running EDR tools with the minimum required privileges while maintaining the same functionality and same level of protection
Involving all other software vendors in this product category, all relevant operating system platforms and, in general, providers of products with (currently still) high privileges
BSI is in continued contact with the vendor Crowdstrike and with Microsoft regarding the operational and strategic processing of the security incident expecting concrete results and solutions. In the meantime, Crowdstrike has published a large amount of additional information that already describes the initial implementation of the above measures.

https://www.bsi.bund.de/EN/Service-Navi/Presse/Pressemitteilungen/Presse2024/240729_Folgemassnahmen_Crowdstrike.html

These measures by the BSI are important and necessary. They will enhance DevOps, and should result in improved operability of systems and continuity, for businesses and consumers. They also work toward needed reform and improvement of not just DevOps but governmental guidance. They represent a growing trend toward inter-sector cooperation between state entities and private software companies.

More AI, please. It can aid coding and is not a threat -- survey.

Andrew Tetzeli — Sun, 04 Aug 2024 15:33:45 +0000

The latest Stack Overflow Developer Survey -- hey, Dev.to, let's do some more surveys, too! -- shows developers welcome AI as an aid to coding and do not feel threatened by it at all. Survey.

I would agree that "AI is largely non-threatening because it has not replaced the human in the loop." AI can be a complement and is nowhere near being a replacement to human coding.

AI extensions to commonly used development platforms like Visual Code or Visual Studio code can be useful, as borne out by the survey. It's worth seeing how much and how far the standalone AI coding aids progress for software development.

Perhaps unsurprising is that developers believe AI is not yet quite ready for or good at solving complex tasks. As the AI models and their backends become more sophisticated, however, and with even more computing resources dedicated to them, that situation should improve. It's most likely just a matter of time.

AWS vs. DevOps

Andrew Tetzeli — Thu, 01 Aug 2024 01:09:38 +0000

Without explanation, Amazon Web Services has cut back on its DevOps services. DevClass article.

Back in June 2024 AWS CodeCommit quietly stopped onboarding new customers. That's not good. Not good at all, for AWS, its customers, or DevOps.

An entity the size of AWS exerts influence and pull on other large companies and their approaches to DevOps. AWS's maneuver could leave openings for other companies, however, to step up their DevOps offerings.

It will certainly for the short-term drive more people to GitHub or GitLab. They have more robust DevOps environments and features than most.

Other ripple effects may follow. Stay tuned.

CrowdStrike vs. DevOps

Andrew Tetzeli — Sun, 21 Jul 2024 00:07:20 +0000

Let's start by getting the "WTF" out of the way. WTF, by the way.

Now, the CrowdStrike debacle reads like a dystopian novel, some sort of amalgam of Dostoevsky meets Kafka meets Heinlein.

CrowdStrike's buggy patch, itself absurd, unleashed absurdly damaging and widespread upheavals and disruptions to normal life around the world.

CrowdStrike's actions have presented us with _The Curious Case of CrowdStrike vs. DevOps. _ We see that CrowdStrike has something against DevOps.

In one corner, we have CrowdStrike -- a Goliath, publicly traded and now publicly berated -- and in the other, DevOps, a logic-based methodology for producing secure, functioning software.

It would seem that CrowdStrike got a Technical Knock Out (TKO) against DevOps. CrowdStrike definitely succeeded in stomping all over good DevOps.

Yet it was a success of abject failure.

Good DevOps ultimately prevails. It keeps itself and its adherents alive and kicking, developing and releasing software that runs and certainly doesn't cause an easily avoidable global meltdown by preventing hundreds of thousands, millions, of systems from booting up and operating.

Time will tell how CrowdStrike fares after this colossal snafu that fubar'ed so many computers and services.

We'll be dissecting this case study of _The Curious Case of CrowdStrike vs. DevOps. _ for a long time, and into the future, if not the figurative corporate corpse. We'll have to see how the latter part plays out.

DevOps will keep on keeping on. We'll get more and more into refining, explaining, and hopefully applying it.

I look forward to being part of that process. And I hope you'll join me.