Fortunately, and this is where Backblaze comes into the picture, we had a tertiary backup location that was offsite in the cloud, not tied to our domain or our normal user accounts, where we were sending another round of backups. Fortunately, we had credentials to log in there, saved in a password manager in the browser, a cloud password manager, which allowed us to get in. We were able to pull down Active Directory and restore. What’s funny about that is we actually had to restore it to an eSports gaming PC, because we just didn’t have the hardware to put it on anything.
We ended up negotiating back and forth with LockBit. They decided to lower the ransom payment to about $750,000 to avoid posting the data they’d exfiltrated from us on the dark web.
We decided to go with our gut instinct that they don’t have sensitive data or that at least the data they have isn’t overly sensitive, and opted not to pay the ransom, at which point…they posted all that data on the dark web, and we found out it’s only a little over 2 gigs. The file listings that we got from them ended up being the only files that they had, and they were trying to bluff us into paying for that. We ended up not paying. We found maybe four Social Security numbers.
We did not have to pay the threat actors anything. I estimate we were probably out around $300,000, based on recovery fees, insurance deductibles, hours employees had to be pulled off from other projects to work on remediation, lawyer fees, etc.
Could you walk me through which systems went down and which weren’t affected?
Zach Lewis: Some of our primary systems were cloud-based and SaaS-based. When the environment went down, our student population didn’t know anything. They were able to continue going to class and submitting homework assignments.
Our Active Directory environment, however, went down, and our on-premises site and services did as well. Building automation was down, and we no longer had control over heating and cooling. Security systems were down. Cameras weren’t up, and door badges weren’t working. We couldn’t change door schedules, but fortunately, they were stored. The schedules were stored locally in each badge reader, so doors were still working. Lighting systems around the buildings were down.
With Active Directory unavailable, we couldn’t create user accounts. We couldn’t change passwords. We couldn’t delete users.
Our password manager also wasn’t available for that primary backup. That was definitely a surprise we hadn’t anticipated, and we were fortunate that someone had kept their Backblaze login credentials in their personal password manager. A little against policy, but in that instance, it worked out for us…because without it, we wouldn’t have recovered.
We also learned that you need to use out-of-band (OOB) communication during an attack.
Learning how the threat actor got in, we discovered that they came in through what we think is a personal laptop belonging to one of our employees. We think the account was compromised on a personal end-user device that wasn’t managed by us, likely via a phishing email where they submitted credentials. Then the threat actor used those credentials to come in over the VPN on that personal device into our environment. They found some cached credentials on a server that allowed them to escalate. They got higher-end privileges that eventually led them to the hypervisor.
In our negotiations with them, they gave us a doc showing 10 usernames and passwords they’d been able to crack or compromise during the breach, so we knew they had elevated permissions. They had created some backdoor accounts as well in our system.
How long did that recovery period take?
Zach Lewis: We learned that they entered our environment on April 1st. They were in there for about 13 days before they triggered any sort of attack. Data dropped around the 12th of June. We were fully remediated and operating again by around the first week of July.
It sounds like the cloud backup was the key.
Zach Lewis: I’ll call it the chef’s kiss. It kept us alive and functioning. Without it, we would not have been able to restore.
The best recommendation is to have three tiers, which is what we did. But also having one of those tiers be fully separate from your environment. We want a backup solution that’s not tied to our network or our user accounts.
What takeaways do you have?
Zach Lewis: Looking at how the threat actors came in, it was definitely a compromised user account. Some controls around that user account and how they accessed our environment failed. They failed primarily because we didn’t take enough time to validate configuration changes. We bake in a little more time now in our environment when we do a project to validate that configurations are true and that our assumptions are accurate. That is one.
Two, password availability. That’s definitely something we’re doing and reflecting on. I don’t want to be in a situation again where I can’t access my backups.
Then just keep a robust backup strategy and test it. We test from on-premises, we test from the cloud, we test from Backblaze. We test every quarter from any place we have backups going, validate they work, and document that whole process. If my system administrator isn’t here, if my network guy isn’t here, whoever is available can use that documentation to restore a server from any of these environments and get us operational again.
If an organization finds itself a victim of an attack, what advice would you give them?
Zach Lewis: If they have cyber insurance, definitely activate it. But if you do, that should be your first call. You should know your policy really well. They’re going to be able to bring in a lot more resources than probably your team can handle on their own.
I don’t employ anyone on my team who’s ever negotiated with a threat actor. So having someone who’s familiar with that is really good. Outside counsel who understands this landscape; our general counsel, while great in the legal sphere, had no experience with ransomware and cyberattacks.
They should definitely be filling out an IC3 report, which is the Internet Cybercrime Complaint Form that goes directly to the FBI. If you end up having to pay a ransom, they are really good at getting that money back, especially if it’s paid in crypto at the very early stages of that payment.
CISA (the Cybersecurity and Infrastructure Security Agency) is there to help you, guide you through remediation steps, and share best practices. If you have nothing at all, there are some free resources out there that can at least help you get going again.
Keep in mind, out-of-band communication, as we mentioned, is huge.
Try to stay calm; there’s going to be a lot that comes at you. It’s very stressful; it’s going to be very hard.
We see attacks increasing year over year. We’ve gotten no better as an industry at lowering attacks or stopping attacks. We’ve gotten better at recovering. I think that speaks to robust backup strategies, as that’s the default factor for people in restoration.
Kari Wilson: One additional thing is something called Object Lock, which is immutability, and it’s a feature in the platform. It’s literally just a toggle of a button to turn it on. What it does is lock the data so it can’t be encrypted, modified, or deleted. You always have a copy there that cannot be touched. That’s a best practice that we recommend as well. A lot of people have backups, but they either don’t enable object lock, or a surprising number aren’t using it yet, and even more aren’t testing those backups.
Zach Lewis: Immutability does not mean non-deletable. Having a strategy that keeps data from being changed, read, and deleted is huge.
Healthcare and hospitals specifically, their primary go-to in the event of a ransomware attack is to pay the ransom. It is just because they don’t want to interrupt patient care. It’s the fastest way for them to get operational. They need to really look at that backup strategy, that resilience factor. Where’s their data, how’s it secured? Getting back to foundational security practices around identity and data governance to keep those attacks from happening, so we can stop paying the threat actors and maybe alleviate the pressure on healthcare a little bit, because they’re getting slammed.
