This past July, the most significant outage in IT history disrupted the nation, affecting industries from public transit to healthcare. CrowdStrike, a leading cybersecurity firm, issued a defective update, causing widespread outages. Instead of strengthening defenses against malware and other cyber threats (as initially intended), the update led to a crash of over 8.5 million systems globally, resulting in an estimated $10 billion in damages. Amidst this chaos, the Vontas IT team sprung into action with incredible efficiency, trust, and teamwork, transforming the crisis into an opportunity to strengthen customer relationships.
About the CrowdStrike Outage
CrowdStrike designed the “Falcon Sensor” security software to further protect their customers against rising cyber threats. However, on July 18th at 11:09 PM, CrowdStrike pushed a faulty update to the software. Upon loading the file, Microsoft Windows systems had memory issues and could not boot afterward. This event triggered over eight million devices to crash worldwide, affecting 60% of Fortune 500 companies. The impact was felt across the globe, with banks, hospitals, public transit systems, and airlines experiencing significant disruptions.
Instantaneous Impact
The consequences were immediate and severe. The first report of a Vontas outage was at 12:34 AM, with the Azure Virtual Desktops running Vontas OnRoute crashing. Eventually, all Vontas Cloud customers experienced downtime, with an average outage lasting six hours. Around 4 AM, our team determined steps to resolve the problem, and shortly after, they identified the issue as part of the global outage. Vontas internal operations were also impacted, with 60% of user laptops and over 90% of servers and infrastructure crashing.
Springing Into Action
As the CrowdStrike update wreaked havoc, our IT team demonstrated efficiency and composure. A mere three and a half hours after discovering the initial resolution, the IT team restored our internal and user systems. And this was no easy task—we use BitLocker to encrypt data, meaning each individual laptop needed a recovery key. Marc Pilarczyk, Vontas Internal IT Manager, reflected on the effort: “It’s about 200 employees. So we had the 200 laptops and our mission-critical and production servers up, fixed, repaired, and running by 9:30 in the morning.” This quick response highlighted the team’s preparedness and ability to operate under pressure.
The IT team’s calm and collected approach was crucial in minimizing the impact of the outage. They quickly identified the problem, coordinated their efforts, and implemented a solution in record time. While some major airlines like Delta took nearly a week to recover, Vontas was back up and running in just a few hours. This efficiency restored operations and also minimized downtime for customers, allowing them to resume activities with minimal disruption.
To reinforce our commitment to reliability and customer satisfaction, our team prioritized keeping customers informed throughout the incident. The swift recovery and clear updates encouraged customers to trust our ability to manage even the most unexpected challenges.
Resolving the outage was a collective effort involving our internal IT team, Vontas D&S, and Lunavi. Together, we responded quickly and comprehensively. Even under pressure, we were able to effectively work as a team.
Customers Come First
In the face of this unprecedented IT disruption, our team prioritized what matters most—our customers. We understand they rely on us to deliver consistent, reliable solutions, especially during challenging times. When the CrowdStrike update triggered widespread outages, our internal IT team restored services quickly, minimizing the downtime that could have severely affected them.
Throughout the incident, we kept our customers informed every step of the way. Their trust is invaluable, so we ensured effective communication and quick responses. Even under pressure, our focus remained on delivering the high level of service that our customers deserve.
Takeaways From the Outage
The CrowdStrike update posed a significant challenge, but our response turned a potential disaster into an opportunity to tap into our strengths. Through quick problem-solving, transparent communication, and strong teamwork, we’re proud to have rapidly restored operations and reinforced our customers’ trust and confidence.