DATA MANAGEMENT & THE NETFLIX APPROACH

One of the biggest mistakes that I’ve seen other filmmakers do is mishandle data, and by data I mean the valuable footage (audio and video) they spend time recording. This is something that’s important and not just for professionals either but for filmmakers at any level.

This is because how you backup and archive your data will often determine if your film gets made at all. But luckily Netflix have been kind enough to tell you specifically how to do it.

THE NETFLIX APPROACH (aka 3/2/1)

In terms of backup and archival of data Netflix follow the standard 3/2/1 policy which is as follows:

• Hold at least three copies of all original camera files (OCF) and audio.
• Store the OCF and audio copies on at least two different types of media.
• Keep at least one of these backups in a different geographical location from the others.

A type of media in this case refers to the medium of storage. So SSDs, HDDs, and LTOs would be types of media.

The above guidelines are to ensure safety through redundancy by spreading backups across geographical locations and also eliminating the potential for technological failures arising from using one type of media

“When working on set I will offload large amounts of data simultaneously to master and backup drives as well as a “shuttle drive” which is taken to post-production by courier on a daily basis.“

The above is the bare minimum and an extremely simplified summary of how data is handled on-set. More details of data wrangling will be covered at a later date, but the important thing to note is the 3/2/1 approach is an industry standard.

For more details please refer to Netflix’s guidelines on archival of production assets: https://partnerhelp.netflixstudios.com/hc/en-us/articles/360000581207-Production-Assets-Data-Management

CAN I IGNORE IT?

Before production can begin (for professional productions) insurance companies require you to have a plan of action in place. This requires a document to be drawn up detailing the approach you intend to take. Only when they’re happy with it will they agree to insuring the production.

If they find you have broken the terms of the agreement then that coverage will be rescinded.

But beyond that if you don’t have adequate safety measures in place you could literally lose everything you’ve shot over the course of the production.

In that case ignore at your peril.

TYPES OF MEDIA

Some types of media are faster, some slower, and some are more prone to failure. If you have a big wallet I would go for some variant of the NVME SSD drives, preferably in an enclosure that can handle the speed.

But here’s a rough summary of the differences between the two general groups of common storage types:

• SSDs / Flash Storage (solid state drives) come in many different “flavours” with some of them being incredibly fast depending on the connectivity technology they’re using, but the capacities are limited when compared with HDDs. They also have a limited read/write capacity with the professional drives being more robust.
• HDDs (hard disk drives) are mechanical drives instead of purely digital drives like SSDs. They work by moving an arm over thin platters and reading the data. The fuller they get the slower they can become. But the mechanical parts will fail even if they are left sitting on the shelf unused, with HDDs being continually replaced in big data centres. However the capacities are enormous.

RAID arrays: Both of these types of media can be linked together in an enclosure as part of RAID array. Depending on the RAID settings you use this can combine speed and / or storage capacities so they act as one storage drive. But they can also be set for redundancy, so even if one of more drives fail then the data can be recovered.

Out of these I use large capacity HDDs built for continual use in a RAID array (look for drives like the WD Red series), and then SSDs (the faster the better) for secondary backups. But especially for fast shuttle drives, as keeping those in circulation to keep post-production on-track is one of the concerns of the data manager/wrangler/digital loader.

But remember: The big drives you buy from the store are nearly always HDDs, or contain them at least, and they will break from mechanical or electrical failure over time, or seize from lack of use if they’re not “spun up” every so often. Do NOT rely on them for permanent storage.

See below for more permanent storage options.

Recovery of HDDs and SSDs: SSDs with “TRIM” enabled cannot be recovered if data is deleted. But in other cases, especially with HDDs, data can sometimes be recovered after hardware failure or formatting. If you’re local to the Midlands and need to recover data I recommend Essential Data Recovery: http://www.essentialdatarecovery.co.uk/

BACKUPS vs ARCHIVES

There is a difference between the two terms. Backups are shorter-lived and temporary copies of data, while archives are longer-term and permanent solutions to storing data.

Professionally LTOs are used to store data on magnetic tape for up to 30 years. This is far longer than the 7 years for HDDs, and more cost effective than SDDs.

Often post-production companies will do this for an extra fee.

By the time of writing the latest LTO standard is LTO-9 which can fit up to 18TB on a single cartridge, but are only backwards compatible up to 2 generations (LTO-9 drives can read and write LTO-8 and 7). However LTO drives are incredibly expensive and can cost many thousands of £ with a single LTO-9 cartridge over £100.

But if you can pay someone to create them for you then they are still more cost-effective than SSDs, more stable than HDDs, and far more space efficient than large numbers of drives.