… for Codecs & Media

Tip #1286: Optimize RAIDs for SSDs

Larry Jordan – LarryJordan.com

Optimizing RAIDs for SSD drives will improve performance.

RAIDs (Redundant Array of Independent Disks/Drives/Devices) consist of a number of hard drives or SSDs grouped together into a single unit so that they appear to the computer as a single device. Because there is more than one hard drive in a RAID, they offer greater performance and storage.

As we continue shifting storage from spinning media, which holds a LOT, to SSDs, which are FAR faster and more flexible, we need to rethink how RAIDs are configured. This configuration is done using “levels.” There are different levels of RAIDs, identified by numbers:

NOTE: Another benefit to SSD-based RAIDs, is that there is no latency. Because there is no mechanical movement, data can be retrieved much more quickly.

• RAID 0 – Fast, inexpensive, no data redundancy. Requires a minimum of two hard drives inside the RAID enclosure. The more drives you add, the faster the performance, as performance and storage capacity are the sum of all drives in the RAID. However, if you lose one drive, you’ve lost ALL your data. Most often used when speed combined with low cost are paramount.
• RAID 1 – Complete data redundancy. Generally only uses two hard drives inside the RAID enclosure. Often called “mirroring,” each drive is a complete copy of the other. Most often used for backing up servers or when on-set for DIT media work. Has the speed and capacity of the slowest single drive in the system.
• RAID 3 – Medium-fast, data redundancy. Requires a minimum of three drives, as one drive is reserved solely for parity data. Should one drive die, your data is safe. This technology is no longer in common use, replaced by the faster performance of RAID 4 or 5 systems.
• RAID 4 – Very-fast, data redundancy. Similar to RAID 3, requires a minimum of three drives, as one drive is reserved solely for parity data. Should one drive die, your data is safe. This is the preferred RAID format for SSD drives because of how the data is stored on the drives. When compared to a RAID 5, RAID 4 with SSDs is about 25% faster on reads.
• RAID 5 – Very fast, data redundancy. Requires a minimum of three drives and shares parity data across all drives. Most often found with four or more drives inside. If one drive goes down, your data is safe. This is the preferred choice for RAIDs containing spinning media (traditional hard disks). Used for both locally-attached storage and servers.
• RAID 6 – Fast, extra data redundancy. Requires a minimum of four drives. This version protects your data in the event two hard drives die at the same time. More expensive than RAID 5, but, generally, the same physical size. Like the RAID 5 this is most often used connected to just one computer. Not as fast as a RAID 5.
• RAID 10 (or 1+0) – VERY fast, totally redundant. Requires a minimum of four drives, but is more often created by combining two matched RAID 0’s into a RAID 1. This provides the speed equivalent of a RAID 0, with the data redundancy of RAID 1. As RAIDs continue to drop in price, this can be a less-expensive way to create systems that rival the performance of a RAID 50.
• RAID 50 – VERY fast, data redundancy. Generally the domain of very large RAIDs, this format combines the speed of RAID 0 with the redundancy of RAID 5 by dividing the RAID into sections, where you can lose a drive in each section without losing data. These systems generally cost more than $10,000 and contain at least twelve drives. Generally used in network and server situations where multiple users need to access the same data.
• RAID 60 – VERY fast, extra data redundancy. Generally the domain of very large RAIDs, this format combines the speed of RAID 0 with the redundancy of RAID 6 by dividing the RAID into sections, where you can lose two drives in each section without losing data. These systems generally cost more than $10,000 and contain at least twelve drives. Generally used in network and server situations where multiple users need to access the same data.

EXTRA CREDIT

RAID 4 is the preferred option for SSD-based RAIDs.

… for Codecs & Media

Tip #1256: What’s the “Ideal” Computer?

Larry Jordan – LarryJordan.com

It doesn’t have to be perfect, it simply needs to get the job done efficiently.

My wife has a saying: “The perfect is the enemy of the good.” Nowhere is that more true than in technology; and it is driving us all nuts.

What this saying means is that we spend too much time looking for the perfect system, when a system that may be less than perfect is still more than adequate is enough. This is especially true when it comes to storage.

As an example, I’m in the process of upgrading my server for faster performance and greater capacity. However, last night, as I was exporting my weekly webinar, I measured how fast Final Cut creates a ProRes 4444 file: 85 MB/second. Even if I had storage that clocked in at NVMe speeds – 2500 MB/sec – my exports would not be any faster, because FCP X can only calculate these files so fast.

1080p media needs less than 40 MB/second to edit, while 4K media needs less than 70 MB/sec. Storage that goes 300 MB/second will edit at the same speed as storage that goes 2500 MB/second.

I’m not saying faster storage is a bad idea, clearly, multicam editing, HDR or larger frame rates require more horsepower than simple HD. However, what I am saying is that we need to ask ourselves a bigger question: Where will extra speed actually help? For example, if I only edit one project a week, spending a lot of money improving export speed is not meaningful compared to the time it takes to edit the project in the first place. Sadly, faster storage does not help me think any faster. I wish it did.

Another example was provided by Gloria. She owns a high-end 2019 Mac Pro. She’s worried that Thunderbolt 4, which hasn’t shipped yet, will make her system obsolete.

Well, ALL computers become obsolete at some point, but when it comes to performance, Thunderbolt 4 is the same as Thunderbolt 3. And, even when new gear is released, as it always is, all our current gear will still work exactly the same as it does now.

I get dozens of emails each week from editors happily editing on Mac Pro systems that are 10-12 years old. Clearly not state of the art, but fully capable of doing the work they need to get done – on time and on budget. I get even more emails from editors stressing over whether they need a 3.2 GHz or 3.3 GHz CPU.

My advice is stop trying for perfection – unless the search itself is something you enjoy. Instead find a system that meets your needs. Most of the time, good enough is also fast enough. And “future-proofing” is a fool’s errand.

Jan Frederickson, of WLS-TV, had a sign on her wall that I think about daily: “It’s better than perfect, it’s done.”

That is a reassuring statement.

… for Codecs & Media

Tip #1122: RPMs Really Do Make a Difference

Larry Jordan – LarryJordan.com

Faster hard drive RPM speeds equals faster data transfer rates.

I’ve spent the last two weeks optimizing my network and two of my locally-attached RAIDs. In this process, I found myself with two identical fully-optimized 4-drive RAIDs; except one was filled with 7200 RPM drives and the other with 5400 RPM drives.

So, I decided to test them to see how much RPM speed affects data transfer speeds.

What I learned is that 7200 RPM drives increase data transfer speeds by 30% on average. (See screen shot.)

For the complete details on my tests and the results, read this article.

… for Codecs & Media

Tip #1081: Improve the Responsiveness of a Server

Larry Jordan – LarryJordan.com

An NVMe SSD cache card can speed file directory operations and make a server more responsive.

I’ve spent a lot of time this week thinking about how to improve the speed and responsiveness of my storage. Here’s a relatively inexpensive way to improve the responsiveness of a server, especially for smaller workgroups. When it comes to storage there are three elements we can adjust:

• Storage capacity – measured in gigabytes – is how much the device holds
• Bandwidth – the speed it transfers data to and from the computer
• Responsiveness – how fast it responds to requests for data

Looking at these in more detail:

• Capacity – measured in terabytes – we are all familiar with. From experience, we know that we can’t have too much capacity. It seems that hard disks are either empty or full.
• Bandwidth – measured in MB/second – is how fast we can move data from one device to another.
• Responsiveness – measured in milliseconds – is how fast a storage device responds to a request for data.

Most older people celebrate a birthday with a nice dinner. Me? I upgraded my server. I have a Synology DS 1517+ that’s around 3 years old. I connect to it using 1 Gb Ethernet from a variety of different computers.

One of the problems with my storage is that, because my network only consists of 2-3 users, the server is not heavily used. Which means that a lot of the time the drives stop spinning, or spin slower, to save energy because there’s nothing happening across the network.

This means that when I access a server volume from my Mac, it takes several seconds for it to wake up and display a file directory or open a file.

Most of the time, while annoying, this isn’t a big problem. But, all my media is stored on the server, when I’m doing a live webinar this delay drives me nuts.

So, I finally decided to do something about it: I added an SSD card as a cache to the server. SSDs are marketed to database users as a way to improve the responsiveness of I/O operations. And I’m sure it does that. But there are also benefits to media creators in terms of making the server feel much more responsive.

NOTE: The specific hardware that I added were a Synology E10M20-1 Ethernet Adapter, which also holds a Synology SNV3500-400G 400 GB NVMe SSD to accelerate the storage cache. (The card uses an m.2 form factor.)

I was amazed at the difference. Folders pop open almost instantly, even though the disks are not yet up to speed. Navigating is almost as fast as the internal SSD on my computer. While I haven’t, yet, connected the 10 Gbps Ethernet port – though that’s coming – just adding SSD makes a big difference. I don’t feel like I’m waiting on my storage anymore.

NOTE: The SSD does not speed file transfers, but it does make moving around and finding things inside the server much faster.

If you feel that your server is a bit “laggy,” look into adding an SSD card as a cache. It will make your system feel much peppier, even if your server, like mine, still uses spinning disks for storage.