24 Jul CrowdStrike Bugged Update Causes Serious IT Problems

CrowdStrike update was massive without these checks, resulting in tumultuous systemic failures. Adequately, it’s one of the worst cyber and IT failures in history.

In the late 90s, before the turnover to the year 2000 (or Y2K), a deep fear resonated around the world. Referred to as the “Y2K” Bug, tech experts shared deep concern that the shift to the new year would cause a serious systemic bug across numerous hardware systems. Not just personal computers, but federal level and business systems. Due largely in part to the hard work and diligence of said experts, however, the rollover to the new millennium year went off without any major hitches.

The Y2K Bug was a cautionary tale on the potentially serious consequences of unchecked changes to technological systems. Unfortunately, it’s the precise opposite that occurred with the CrowdStrike disaster.

What Happened

CrowdStrike is a widely utilized security platform. Nearly every large-scale business using Windows-based software and operating systems takes advantage of CrowdStrike. While it’s not a bad thing to rely on experienced, veteran service, the CrowdStrike rollout reveals a wide variety of catastrophic flaws. We’ll dig into them in this article.

The wide-scale shutdown of well over 8 million IT systems utilizing Windows software started with a faulty CrowdStrike update. This update, while rudimentary on its own, resulted in a logic error, causing millions of systems to experience the dreaded “blue screen of death.” Again, these were not isolated to personal computers. The wide-scale failure impacted organizations of all sizes. Airports, for instance, had to ground flights when the error occurred.

But why did a simple update cause a serious failure of various global IT systems? For one, CrowdStrike is a widely utilized security software. Embedded in tens of thousands of IT infrastructure, that means any failure of CrowdStrike directly affects those who use it. Because there were no backups or alternative systems, those reliant on CrowdStrike suffered a major blow in downtime.

Furthermore, CrowdStrike’s update rollout was handled in a dangerous manner. Ideally, when any major software enterprise releases patches, it’s done on a small scale. Therefore, if a patch or update causes “software breakages” and system failures, the damage is minimal and additional changes can be made. But the CrowdStrike update was massive without these checks, resulting in tumultuous systemic failures. Adequately, it’s one of the worst cyber and IT failures in history.

Why It Happened

While the exact details are still forthcoming, we know a few basic details. First, while the rollout failure was far-reaching, it was not caused by a cybersecurity breach or attack. Likelier, a lack of proper checks and balances prevented the detection of the error. Again, small patches are the trusted method in case of severe error.

A lack of oversight, quality controls, testing, and proper Q and A are also likely culprits. CrowdStrike works with Microsoft, and Microsoft is one of the world’s leading providers of office software. Therefore, Microsoft not quality checking the update before it’s pushed released creates numerous control questions while revealing the dangers of unchecked patches. It’s also possible there was a demand for speed and “efficiency,” forcing the update before it was ready.

How Is It Fixed

Now comes the bigger question: how is the CrowdStrike issue resolved on impacted devices/systems? CrowdStrike and Microsoft have since released recovery tools for affected industries. But, even the recovery tools/process have specific criteria adding to the issue’s complexity.

For instance, those affected by the downtime need to download a recovery tool and run it via bootable media, such as a USB flash drive. The recovery also requires at least 8GB of RAM and running Windows 10 or later. However, those reliant on outsourced IT infrastructure may not have access to the systems. And, these recovery operations must be performed on each device. Given the number is estimated in the millions, this is not a simple task.

More information is available about recovery directly from CrowdStrike.

The fallout from the CrowdStrike failure will require analysis for months to come. From hospitals, and airports, to even banks, numerous systems were directly compromised. The failure demonstrates the problems with mass reliance on single services, while also showcasing even simple updates can have catastrophic repercussions when handled incorrectly.

If you’re concerned about downtime, need IT support, or want additional info about the CrowdStrike fallout, reach out for help. Contact Bytagig today for additional information.