Transcript
Host: Have you ever clicked on a mysterious email link, and suddenly had your computer shut down? Or heard of a friend of a friend whose identity was stolen online?
These are examples of cybercrimes – a form of crime that costs about a trillion dollars a year, globally.
And one of the biggest forms of cybercrimes that businesses face today?
Ransomware.
Alban: Once we realize that there is somebody who's nefarious on the other end, trying to cause this harm, we're kind of in the dark.
Allan Liska: Walking in and seeing thousands of workstations encrypted can be, can be devastating. It's almost always the worst day of a cybersecurity leader's career.
Jason: A ransomware type attack is kind of like a hurricane hitting a house because you're not quite sure of the effect.
Marcus Brakewood: When they delete the backups, it prevents the primary response a company would do – would be restoring their data from backup. And then all their data is encrypted and they can't get access to it.
Host: This is Speed to Modern Tech, an original podcast from KPMG. I'm Tori Weldon. Each episode, we'll bring you a problem many businesses are facing, and the story of how technology was used to tackle it. Today, the technological challenge of managing fallout from ransomware attacks, and how businesses can steal themselves.
Host: It was Sunday Feb 21, 2021. Alban Brooke was visiting an art gallery in downtown Jacksonville with some friends. They enjoyed the show, met the artist, then parted ways.
Alban: And then I'm driving home and the head of support starts texting me. And she says like, “Hey, are you seeing this? Like, we're under attack. And I'm like, oh my gosh, what? This is just dramatic. There's nothing happening.”
Host: Alban is the head of marketing at Buzzsprout. It’s one of the largest podcast hosting companies in the world, publishing more than 100,000 active podcasts.
Alban: And then I just see a flood of people reaching out. Why is my podcast not working?
Host: The Buzzsprout team rarely works on weekends - but this was an exception. Something was seriously wrong.
Alban: I get home, sat down in a cushion chair and my office, grabbed my laptop and that was 1:00 PM. And I didn't stand back up until 1:00 AM. Like I just sat there and typed and typed and typed and typed.
Host: Buzzsprout was experiencing a ransomware attack -- it meant that podcasters couldn't access their website, or app, to publish new episodes. Podcasts were not being distributed properly. So listeners couldnt find the shows they wanted to hear. In other words, everything was down.
The attack was what’s called a Distributed Denial of Service attack -- or DDos attack for short. Ddos attacks work by sending excessive traffic to a website, so that it becomes impossible for anyone else to access it.
Alban: Think of, you know, you have a small bodega, or you have a small little store. And normally you're getting five or six people into your small store at a time. And all of a sudden 400 people cram into your store. Well, now your normal customers have no ability to come in and buy whatever they wanted to buy. And that's pretty much what had happened to us. That all of a sudden, we just have so many people requesting our podcasts, episodes, the Buzzsprout site, the Buzzsprout app, everything was just completely flooded.
Host: For a hosting provider, whose service is making content available at all times, a Ddos attack can be an existential threat to business.
Alban: And we have a lot of podcasters that on Sunday, they're uploading their content for the week and on Monday, that's the biggest day Buzzsprout has in the week. And those two days that's when Buzzsprout was under attack.
Host: During those two days, it was all hands on deck. Buzzsprout's small team worked around the clock to deal with different aspects of the attack -- for example, understanding if sensitive data was compromised. Finding the source of the traffic. Getting servers back up so people could access the site. Responding to clients' questions.
Thankfully, the attacker had not accessed, or stolen, any sensitive data. Things like client credit card numbers were safe.
It was a relief to share that piece of good news with clients -- but Buzzsprout still didnt know when their podcast services would be restored.
Alban: I think it was really the inability to give people a timeframe that was the most frustrating. In normal situations, people are writing in saying, when do you think you'll have this resolved? We'd say worst case scenario, it's going to be 30 minutes. You know, we're going to have to restart a ton of stuff, but it will be fine. Once we realize that there is somebody who's nefarious on the other end, trying to cause this harm, we're kind of in the dark.
Host: So instead of concrete answers, they shared live-time updates through social media, emails and blog posts.
Alban was on Twitter, responding to messages and sharing updates, when one particular DM caught his eye.
Alban: I see messages in our Twitter DMs, and it's just somebody kind of like, snarky, going, “Hey, hope you're enjoying this, you know, attack” or something. And I realized it was the attacker reaching out to us. And then he starts making requests for money and sending a Bitcoin address and says, “Hey. All up to you. If you want to turn this all off, pay me $2,500 and it all ends.”
Host: Alban realized this wasn't just a Ddos attack -- it was a ransom Ddos attack. This person was asking for money, and promising that if they were paid, the attack would end.
So the Buzzsprout team had a decision to make: pay the ransom, or refuse, and keep fighting back.
Alban: We pretty quickly went, “okay, there's no way we're going to pay this,” because they're not telling you the truth. People will often attack you and then you say, “okay,” and you give them $2,000. But what you've actually done is now funded the second attack on you, which will be much more intense because they're going to spend that $2,000 to take you down for days and days, where they make a much larger demand.
Host: The Buzzsprout team chose not to pay the ransom. They shared the update on Twitter, and received a lot of positive reception from Buzzsprout users, who encouraged them to keep fighting back against the attack.
Competitors, who'd been attacked by the same person, also reached out and offered support.
Lastly, they shored up their defenses, by upgrading their infrastructure during a brief period when the attacker backed off.
Alban: When we were able to get the site back on, we immediately did a massive infrastructure upgrade, pretty much the biggest that we could've done. We migrated servers. We did everything possible to basically upgrade our tiny little store into a Superstore so that the denial of service would have to be so much larger to overwhelm us.
Host: It was a grueling 27 hours for the team - with customer relationships and trust on the line. Luckily, they weathered the storm -- thanks in part to deploying system upgrades.
Like many companies, these upgrades had been in the planning stage for a while. Unfortunately, it took an attack to make it happen.
Alban: Brian who led our infrastructure team, he had drawn up lots of these plans for migrations. You know, we're going to go through it very methodically. We're going to let everyone know about scheduled downtime. We will find the one hour window where Buzzsprout is used the least.
And then all of a sudden, when we're getting attacked, you know, we're looking around going, it would've been really great if we'd made some of these changes sooner.
Host: The ransom DDos attack Buzzsprout experienced is just one type of ransomware attack. Attacks can take a number of different forms -- and for businesses, this makes it even harder to prepare, especially as they continue to change and evolve.
Allan Liska: Ransomware is generally defined as anything that prevents you from accessing your systems or tools or data.
Host: That's Allan Liska, a US-based cybersecurity expert. He's spent more than two decades in the business, and has seen a real evolution in what constitutes ransomware. Typically, people think of ransomware as something that encrypts files on a network.
Allan Liska: We also now see on the other end, what we still consider ransomware groups, they're not encrypting any files – they're just stealing them and then they're holding those files hostage. “Hey, if you don't pay us, we're going to release these sensitive files for everybody to see.” So there's really a broad definition of the term ransomware.
Host: Another evolution? Lone-wolf attacks, like the one Buzzsprout experienced, they’re less and less common. Increasingly, attacks are coming from organized groups that function similarly to legitimate businesses – think hundreds of employees, with different people playing different roles. There's also a growing network of ransomware services that support these groups.
Allan Liska: A ransomware attack in 2000 14, 15, 16 was kind of like knocking over a liquor store.
Right? You ran in, you smashed and grabbed, you got your cash and then you left. Right? There wasn't a lot of planning that went into it. It was, you know, sending out literally millions of phishing emails, compromising websites or ad distribution capabilities to exploit vulnerabilities in browsers.
You land on a single machine. You encrypt everything on that machine. And then you automate the process of going and getting the ransom. A more modern ransomware attack is kind of like Ocean's 11. There's more planning that's involved in one of these attacks.
Host: Today, many attackers spend weeks - or even months - in a company's computer system to learn its weaknesses, before even launching the ransomware.
Allan Liska: You have to map the network, understand what it is –– so what we call the sort of the reconnaissance phase. Then you have to figure out where all the sensitive files are, so you can steal them. You have to get access to the domain controller. And then from the domain controller, you have to push the ransomware out and encrypt all the files on your system, and disrupt the activity. And then they're sitting back waiting for you to come in and negotiate with them.
Host: Negotiations have high-stakes and can be risky -- there's no guarantee that paying the ransom will stop the attack. And, like Alban from Buzzsprout mentioned, it's possible that the ransom money will fund more attacks.
Another aspect of modern attacks? Ransom demands have shot through the roof.
Allan Liska: You've gone from, a couple hundred to a couple thousand sometimes even, you know, maybe, maybe $10,000 to now, ransom demands regularly in the millions of dollars.
Host: Despite the recent changes in ransomware attacks - one thing remains the same for the folks experiencing it:
Allan Liska: Seeing thousands of workstations encrypted can be devastating. It's almost always the worst day of a cybersecurity leader's career. There's a lot of despair, there's certainly some panic, and then kind of the resignation that, “All right, we've got a long slog ahead of us. We've got to get through it.”
Host: Globally, cybercrime costs about a trillion dollars a year. One trillion dollars. That's 50% more than what was predicted back in 2018.
And on an individual level for businesses, ransomware attacks can be devastating -- sometimes the damage takes months to repair, and worst case, they can even force businesses to close.
But dealing with a ransomware attack isn’t a one-and-done solution. There’s a series of strategic decisions, processes, and technology upgrades.
So how do companies respond to attacks in the moment, and recover their business? And how can they protect themselves against future ransomware attempts, especially as attacks become more and more sophisticated?
Jason Haward-Grau: A ransomware type attack is kind of like a hurricane hitting a house because you're not quite sure of the effect, and it depends to a significant degree on how much of a blow you get from the hurricane as it comes through. There's a big difference between a tropical storm and a category five.
Host: That's Jason Haward-Grau. He’s a managing director at KPMG in the cybersecurity practice. With over a decade of experience in the industry, Jason has seen how hard it can be for business leaders to think clearly when a ransomware attack – or a hurricane – rips through.
Jason Haward-Grau: So often it's a bit of shock. And then it's confusion. And then it's a little bit of panic. And then you follow up with an urgent, immediate need for, “I need to know what's going on. I need to understand how this has happened. I need to know what are we doing about it. How quickly can we recover?” And all those things actually tend to jam up the poor single IT team who are desperately trying to figure out what's going on.
Host: That's where Jason and his team come in -- to support the decision making processes, and manage the IT operations.
Jason Haward-Grau: Where we come in as the masters of disaster, or the chaos coordinators, is we help our client build out the plan, build the resources to respond and then actually manage expectations.
Host: According to Jason, ransomware attacks are always a surprise, and always cause panic.
So when his team was called to help a global manufacturing company respond to a severe ransomware attack, they were prepared for the worst.
Marcus Brakewood: An easy way to describe how messy it was is it was Christmas time in Chicago and the systems to turn on the heat were down. That's how messy it was.
Host: That's Marcus Brakewood, Jason's colleague and fellow cybersecurity expert at KPMG.
He stepped in as IT director during the recovery process. One of the first challenges he had to overcome — the sheer number of different IT systems involved.
The company facing the ransomware attack had recently acquired several smaller companies.
This created a patchwork of IT systems on different standards, with different different levels of security. The patchwork presented vulnerabilities for the attackers -- vulnerabilities the client wasn't even aware of.
Marcus Brakewood: They had probably grown through 60 acquisitions of small and medium businesses and had never truly invested in an IT to bring them together on standard and secure platforms. And those attackers found the vulnerabilities, escalated their path to being able to access the entire company.
Host: These vulnerabilities allowed the attackers to carry out a long reconnaissance period, undetected. They scanned the system for at least 90 days. Then, they gained administrator access to the entire company, and waited until a long weekend to encrypt all the data.
This meant it took a long time for the company to even realize what was happening.
To boot, the attackers also deleted most of the company backups.
Recovering data would mean stringing together backups from hundreds of different sources across the patchwork of systems.
Marcus Brakewood: When they delete the backups, it prevents the primary response a company would do – would be restoring their data from backup. So they've got no backups to restore from. And then all their data is encrypted and they can't get access to it. So at this point, they're unable to process orders, ship, product, report, their financials.
Host: Based on policy and principle, the company decided they would not pay the ransom. So where to start? And how to unlock the business?
Host: Marcus and Jason built a plan of recovery that had three phases: reach 'business as acceptable' stage; return to ‘business as usual’; and finally, help the business systems be as secure as possible.
Marcus Brakewood: The first thing you want to do is communicate. So what we did is we set up a directory with usernames and passwords and email. And it's brand new users, brand new passwords, brand new email, because what they had before was completely encrypted and down.
And this allowed the client to both email within the company, email their client's, email their suppliers, and just have basic communication.
Host: Even though their other systems weren’t running, they could use paper record keeping. This meant the company could continue communicating and doing business with their clients.
Marcus’s team also ensured that the attacker was booted out of the system, flagged possible points of entry, and put controls in place to stop re-entry. This is like making sure doors and windows are closed when the hurricane is still happening.
Jason Haward-Grau: I don't get any water in, or the wind. It's not open to the elements. People can't just walk into my house.
Jason Haward-Grau: And then you have to start thinking about, “Well, okay, what's my priority? Do I care about my kitchen? Do I care about my bathroom? Am I worried about bedrooms first? What's the most important things for me in my house?”
Host: This is like deciding which systems are most essential to restore first.
Marcus Brakewood: We had to get agreement with the C-suite on what systems they needed to run their business for, I'd say, the next two to three months.
Host: Company leaders agreed the most essential system was the financial ERP - their enterprise resource planning system. It's used to manage day to day business activities like accounting and supply chain operations. The ERP was especially critical, because without it, the client wouldn't be able to file its 10-K annual report with the Security and Exchange Commission (SEC) - essentially their annual report to investors. The ERP needed to be running again within just a matter of weeks.
At the same time, Marcus needed to restore as much data as possible. This was a challenging process, since the attackers had deleted most of the softbackups. But luckily, one server was taken offline for maintenance just before the attack. It housed some key data that Marcus' team was able to recover.
With those important tasks underway, Marcus and Jason moved on to business as usual, and business transformed.
Marcus Brakewood: After we brought this client to a business as acceptable state, they needed to transform and run their IT in a safe way that would work for all their acquired companies.
And we put together a bet on where they wanted to invest to build a more secure IT. And that bet was in Microsoft Azure.
Host: They leveraged all the security products Microsoft offers within Azure and created a hardened Microsoft Azure cloud. Then, they extended all those security controls to the disparate data centers at the acquired companies.
This brought security and visibility across the organization. Finally, the systems had the same security standards.
After several months of hard work, it was business as usual, and transformed to be more robust against future attacks.
Jason Haward-Grau: The best thing organizations can do is plan for a attack like this and ensure that they are building the right capabilities, and have some playbooks defined and tested so that they understand what can they accept from a risk perspective for something like this to occur. Because unfortunately it isn't a question of if – it's more likely a question of when.
Host: You've been listening to Speed to Modern Tech, an original podcast from KPMG. I'm Tori Weldon.
Todd Lohr: And I'm Todd Lohr, the head of technology enablement at KPMG. If you want to know more about the technologies and the people you heard about in this story, click on the link in the show notes.
Host: And don’t forget to subscribe and leave a review in your favorite podcasting app. We'll be back in two weeks with more stories.
