… for Visual Effects
Tip #1765: Surviving a QNAP Ransomware Attack
Robbie writes: “Please learn from our misfortune.”
This article, written by Robbie Coblentz, first appeared in ProVideoCoalition.com. This is a summary.
I’m heading to a shoot and my phone rings. It’s Jake, my senior producer.
“Boss, I think we’ve been hacked.”
And with that starts a loooong week of recovery, troubleshooting, and formatting. Our QNAP actually had been hacked.
Quick background. I have a small video production company that produces commercials, brand films, and TV programming.
We are a PC-based shop, with all machines connected to 48TB NAS via a closed 10 gig ethernet network. The NAS, a QNAP TS1685, is stocked with 4TB drives and striped into a RAID 6 configuration. That gives us 40 TB of usable space with the safety net of being able to survive 2 drive failures. The QNAP services four edit suites and a few other computers for browsing and offloading
The QNAP has four 1-gig ethernet ports and a single 10-gig Ethernet port. The 10-gig port services the edit suites. One of the single gig ports connected to our traditional network and was outward-facing to the internet. That was part of the problem.
Up until now, my backup strategy was based around the idea that a hardware failure was the most likely — and dangerous — problem we would face.
Typically, we have at least four copies of all footage shot.
We burn footage cards on an iMac via ShotPut Pro to a bare hard drive (copy 1) along with a copy to a locally attached RAID 5 (copy 2). Then, the footage is loaded into an active project folder on the NAS (copy 3). Once the bare drive (copy 1) reaches capacity, we make an LTO copy (copy 4). When the project is complete, we archive to another bare drive (copy 5) for mastered projects. When that drive is full, it gets an LTO copy (copy 6). The RAID 5 and NAS copies get deleted once everything is mastered off.
We make a Chronsync backup of the NAS every night using an older RAID system to give a near-line-identical copy. Technically, that would be the seventh temporary copy. In this case, 7 wasn’t our lucky number.
The Chrosync backup was made after the hack had occurred, so the ransomed files copied over the last known good copy. And we didn’t have archiving on.
So if you are keeping score at home, that’s a bunch of copies of the footage, but only one copy of projects, image, animation, and music files — all typically smaller than 20 MB. That was our Achilles heel.
Read the full article as Robbie describes how they recovered, how his backup strategy changed and how they are moving forward.
You don’t need to be a big company to get hacked. You just need to connect your servers to the Internet.
Here’s the link.