Having worked with DITs (the horrible acronym for Digital Intermediate Technician, the person responsible for the delivery of digital media from the set to post) over many years, and more recently, doing the job of a DIT on a couple of productions, I’ve come to one very simple conclusion: the process could be better.
That isn’t to say that most DITs do a terrible job; on the contrary, many are extremely passionate and conscientious. Rather it’s that they approach the role from a cinematography standpoint, and almost never from an IT background. Historically, most DITs were barely-trained people provided by the rental company, who knew how to operate a particular digital camera, and how to copy files on a Mac.
Problem 1: data transfer
GUI-based copying (whether through Windows or Mac), has several flaws:
- It batches copy operations together into one lump. If a single file fails, you have to restart the batch
- It doesn’t verify files
- It’s slow
- It’s prone to human error
- It doesn’t handle errors very well
- If it does encounter an error, it will stop the entire process
- Having multiple copy operations tends to cause problems
and on and on.
Solution 1: rsync
Rsync is a universal file copying application that runs from a command-line. It’s even built-in to the Mac OS. It’s extremely robust, and recommended in many situations. It’s startlingly easy to use, and there are even GUIs available for it.
Open terminal, and type:
rsync -avhP /Volumes/A001_C001/ /Volumes/RushesStorage/Day01/
And watch it go. You can cancel with control-c and run the same command to resume it. I defy you to find anything that will copy files faster. The only thing to pay attention to is the trailing slash at the end of the paths. More information on rsync.
Problem 2: rented storage media
Every production I’ve worked on has rented storage media (SD cards, CompactFlash Cards, LaCie drives and so on), and needless to say, every one of them has had at least one take that had to be aborted due to an error with said media. Disk drives are sensitive things at the best of times, and this is one of the reasons used disk drives lose value.
Flash storage, which is most popular with digital cameras for its high performance is especially problematic. The problem is, it can also be prohibitively expensive.
Solution 2: buy storage media when possible
The real solution here is proper planning. There is rarely a good argument for renting storage media. Cost is certainly a factor, but question what the cost is of having to reshoot due to drive failure? Also, consider planning to sell the media after a shoot, recouping much of the cost (and probably putting it on par with the cost of renting the media in the first place).
Problem 3: daily storage system
Low to mid-budget productions all use external (USB or Firewire) disk drives for immediate transfer of rushes by the DIT. On a given day, the camera(s) will produce anywhere from a few GB to 1 (or more) TB of data, which gets dumped onto these drives for an undetermined amount of time.
There’s nothing inherently wrong with this, but we can do better.
Solution 3: network attached storage
Rather than using a stack of G-RAIDs, consider using one or more network attached storage (NAS) systems. There are so many benefits to this, it’s incredible everyone isn’t doing it:
- Data is better protected, instantly
- All the data is available without having to swap through drives
- Data is accessible to multiple people, simultaneously
- You can run other services (like a media server)
- Data can be encrypted easily
- It’s lighter
- It’s almost the same cost*
I’ve used the following kit on the last two productions, giving me all the benefits above:
This will give you 9 TB of RAID 6 (meaning you can lose 2 disks). Why sacrifice 6 TB of storage space? From Wikipedia:
RAID 6 provides protection against data loss during an array rebuild, when a second drive is lost, a bad block read is encountered, or when a human operator accidentally removes and replaces the wrong disk drive when attempting to replace a failed drive
The QNAP is considered reasonably high-end (it even has HDMI out, imagine that!) by NAS standards, and so if price is a concern, there are many cheaper options available. I personally recommend the QNAP because it has decent Mac support, and is very user-friendly.
* Compare this to the equivalent in G-Technology Disks: 5 x 2 TB G-RAID ~$1450. If you opt not to use RAID storage, you then of course get more storage for your money with the NAS, and the difference in price becomes negligible.
And if you want to be really paranoid about backing up that data (as well you should), then you can just get an HDD dock, some extra internal hard drives, and copy data onto those as well.
Problem 4: inefficient workflow
Every DIT I’ve ever worked with does things the same way:
- Receive camera media
- Plug into laptop
- Copy to external media
- Return camera media
- Make second copy to other disk drive
- Wait for copying to complete
- Watch media directly from external drive
- Give one of the disk drives to director for rushes
Solution 4: more efficient workflow
With a NAS setup, here’s a better approach:
- Receive camera media
- Plug into NAS
- Tell NAS to start copying to itself and/or other connected drives (yes, you can still use rsync in most cases)
- Return camera media
- Everyone who wants to watch needs only connect via ethernet
This now gives rise to a new problem:
New problem: DIT has too little to do
Solution: Do something useful and beneficial, like logging all that data as it’s being shot.
Posted: November 4th, 2012
Tags: data management
, digital production
Comments: No comments