As well as having known each other for over half our lives, we also worked together on several projects, most notably “Films by Surreal Road”, an attempt to utilise digital production and distribution to create independent low-budget films. That didn’t work out well enough to be commercially viable, but it was a great experience, and the test films we created, which Scott either wrote, produced or starred in (putting his acting skills to great use), still live on.

Scott was always on a personal crusade to try and right injustice, and in 2008, around the time Surreal Road switched its focus to software development, he had decided that he wanted to turn his attention to far more noble causes, such as supporting human rights, and in particular, protecting privacy in an age where the subject is becoming increasingly important and complex. So it was that he enrolled into a college and started studying for a Law degree, which he would have completed by next summer.

It’s weird to have so much of his history recorded somewhere, what with both of us having been technophiles, and his love of writing. I still have thousands of emails from him (a great many of them being multi-page monologues) going back to 2003, when we came up with the company name, Surreal Road (looking back now, I can see that we also considered such exciting names as “Pupil of Dilation” and “Bloodshot Eye”), as well as the everyday ones he sent me that showcase his special brand of dark humour and sum him up better than anything I can come up with, such as:

30/03/2003: oOPS, I just woke up two days late.  It’s not entirely my fault though.  I have a fucking bug or summink again and spent the last day and half stuck in a fucking dream about hamsters in pink skirts only to wake up in solidyfied sweat.  Mum says its sunday but im not quite sure I believe her.

not to mention the countless voicemails he left me that I saved over the years because they were so damn entertaining. Each of these little treasures will no doubt provide me with a great source of nostalgia in the future and I’m glad to have kept them.

Scott Lust, c.2004

We will miss you.

*Wave and AIFF are container formats that are used in conjunction with various audio codecs, which may impose limits on each property.

So here goes:

1. The first line of the batch file must contain the headings of the relevant fields, separated by tabs. These must be named precisely (see below for possible field headings).
2. The fields “Name”, “Media Start”, “Media End” and “Reel” are required.
3. The information for clips must be provided on one line per clip.
4. Folders in a project are defined on a line by starting with an asterisk (*), a space, and the folder name (e.g. “* AAA_cut_aways”). All clips listed after that label will be placed are in that folder. Strangely though, FCP seems to ignore these labels when importing a batch list.
5. The file must be plain text.

As of Final Cut Pro 6.0.3, any of the following field headings can be used:

• Name
• Duration
• In
• Out
• Media Start
• Media End
• Tracks – must be in the form “1V,2A” (for 1 video track and 2 audio tracks in this case)
• Good – “Yes”/”No”
• Log Note
• Label
• Label 2
• Audio
• Frame Size – must be in the form “pixel_width x pixel_height” (e.g. “1920 x 1080″)
• Vid Rate – must be in the form “25 fps”
• Compressor
• Data Rate – must be in the form “2048/sec”
• Aud Rate – must be in the form “96.0 KHz”
• Aud Format
• Alpha
• Reverse Alpha – “Yes”/”No”
• Composite
• Pixel Aspect
• Anamorphic – “Yes”/”No”
• Field Dominance – “Upper (Odd)” or “Lower (Even)”
• Description
• Scene
• Shot/Take
• Angle
• Reel
• Master Comment 1
• Master Comment 2
• Comment 1
• Comment 2
• Master Clip – “Yes”/”No”
• Offline – “Yes”/”No”
• Last Modified – Must be in the form “Tue, Mar 4, 2008, 17:19″
• Film Safe – “Yes”/”No”

Other things to note:

• Although many people are under the impression that right-clicking on a sequence and choosing Export->Batch List will produce a list that only contains clips used in the selected sequence, this is actually not the case. Exported batch lists will always contain all clips in the project. UPDATE (25/August/09): See the comment by Josh below…
• Exporting a batch list from FCP will only export field data for columns that are visible in the project. So if you have the “Master Comment 1″ column hidden, that data won’t get exported.
• The frame rate gets determined by the project that the list is imported into, and is not specified in the list directly.

UPDATE (07/April/09): It was brought to my attention that there is an issue with manually creating text files for Batch Lists. It seems that yet another requirement of the Batch List is that every line must be terminated by ASCII character 13 (rather than ASCII character 10). Not doing so will produce the error that one or more headings are invalid.

I have created an AppleScript that will convert files that are producing this error.

Batch List Text Conversion AppleScript

If you change the brightness for any part of the image, all the selected pixels are affected in the same way.

In one of my previous rants, I proposed that resolution was more important than bit-depth… Since then, I’ve been wondering why, in practice, this is never the case. After some thought, I realised that the way digital colour correction works is different from changing the exposure of film.

Changing the exposure of film, on microscopic level, affects the image like this:

The individual grains are not affected uniformly. It’s that a percentage of the grains become exposed (or not), not that every grain becomes more exposed. To the viewer, the result from a distance is that the affected region is brighter. It’s also the reason that grain structures are visible.

Until now, it didn’t make sense for colour correction software to work like this: the resolution of images was too small to make sense. However, for 4k and beyond, non-uniform (or dithered) colour-correction may very well yield superior control over the colour-correction process.

So the question I find myself asking is, why don’t any high-end colour grading systems offer this kind of functionality?

After all, the vast majority of discussions of Red camera workflow are necessarily theoretical, after all very few people are using it for anything other than test footage. The editing stage is often the first stumbling block, as actually getting any of the footage into an editing suite to play around with can be challenging. There’s also a much more problematic issue that deserves attention: ensuring that the finished edit can actually be used for an online somewhere.

There is a popular workflow with Red footage which is to edit the R3D files directly in FCP. The benefit of this is that it is very simple: you just load the QuickTime proxies into a project and snip away as normal. You can even do your online in FCP if you wish, putting Final Cut’s colour correction and effects tools to good use. You can then export to a wide variety of formats, including some uncompressed* variants.

*Note: I’ve not done any tests on this workflow to see if working in this manner produces a “true” uncompressed result (compared to converting the RedCode data directly to an uncompressed format first, and editing with that). It is possible that somewhere along the path, probably at the Red proxy file, some amount of compression is introduced.

However, I’m not a big fan of workflows like this, because they make the process proprietary and reliant on specific software configurations. It’s entirely feasible that at some point RedCode will be treated in the same manner as formats such as DV, universally recognised, and with predictable results regardless of your editing system. Furthermore, we didn’t want to limit ourselves to having to use FCP for cutting (although ultimately that was exactly what we used). And finally, I was concerned that the cases of clips with duplicate  names would give us grief when un-picking the edit in preparation of the online.

The method we used in the end was to convert everything using the Redline command-line processor (from Red), by creating a batch script using our Redemption… database. We set everything to output 1920×1080 DVCPROHD movie files (which would retain the original timecodes) but prefixing our own reel numbers to each file. There were a couple of bugs in the Redline code (which are being addressed) that prevented the iso value of each file being read automatically so we had to plug those numbers in ourselves. It was also apparent that it was necessary to apply a gamma-curve during the conversion process to prevent everything looking flat: helpful for the online process perhaps, but not so much for the edit.

We then had a folder of movie files that could (in theory at least) be loaded into any editing system. We went with Final Cut, and an additional step was that we had to change the reel number FCP had assigned (based on the filename I assume) to our reel number.

That done, the editor took over and started to cut, as if working with any other footage. Meanwhile, we could start looking into getting the material prepped for the 4k online.

As a side note, one of the other things we were able to do easily due to the digital nature of all this was have our proprietary database automatically match the shoot notes to the Red data logs, so we were also able to produce a reference log for each clip.

The third and final part will look at using Autodesk’s Lustre to online the 4k files.

There are a lot of available options for processing the clips, for the purposes of this test, I turned off sharpening and noise-reduction, set the detail to high, the ISO to 320, and applied a rec.709 gamma curve.

It’s possibly a little difficult to tell from this image, but the Lanczos and Catmull-Rom filters produced the sharpest results compared to the others. Although a difference matte between those two methods suggested there was a difference, I just couldn’t see it. I suspect that the difference in processing time between the two would also be neglible, even over a lot of frames, so I’ll probably just go with Lanczos when the time comes to actually prep the images for the online. What’s interesting is that the default filter used by the Red software is
Mitchell, which is probably the most middle-of-the-road in terms of sharpness.

As tests go, this one’s not bullet-proof, for example I didn’t test to see how the gamma curve affects the process, so I have no idea whether using a different gamma curve affect the interpolation (this would probably depend upon the order the Red software performs its processing, if it interpolates first and then adjusts the gamma, then there would be no difference between curves). The other problem is that this really only provides half the story– until we get this into a grading suite and see how it looks at speed it will be difficult to say for sure that one filter is perceptually better than another. I also can’t compare to how it would look to something similar shot in-camera at 2k, because we didn’t shoot anything 2k (although the general buzz around the Red forums is that shooting 2k produces far inferior results, at least with the current firmware).

The original 4k frame, along with the converted DPX versions, as well as layered Photoshop versions can be downloaded from here… if anyone wants a closer look (please note all images are Copyright 2008 Entitled Productions).

There were a few oddities about the camera itself- there is no back focus plane indicator anywhere on the camera body, so our focus puller had to manually measure where it was and mark it with a piece of tape. The record and power buttons are also very easy to press accidentally when setting up.

The camera internally numbers the clips by reel number and take. The take number is incremented every time you press record, and the reel number is incremented every time you change the flash card. However, there are a couple of things with this method that make it unreliable: first of all, the reel numbering is not consistent; there were several occasions where the number either didn’t change automatically, and even when set manually it seemed to just use whichever number it liked. The other thing is that starting and stopping the camera doesn’t necessarily imply a complete take; for instance on a few occasions we recorded a slate just before a set up, so you really need to go through and properly log everything afterwards. What all of this means is that you can potentially end up with two or more clips that have the same name (I’m looking at a few like that right now), and changing the name just in finder or windows explorer feels like a very bad idea, not least of all because the metadata is also written into the RedCode file itself, and it doesn’t appear that you can change that at all. So what I did straight away was just ignore the reel and clip numbers from the camera and number them sequentially myself. It’s also another reason why it’s critical to slate footage in some way whenever possible, or at the very least, try to log shot material soon after the shoot while it’s still fresh in your mind. We slated some of the shots in the traditional way, but I also kept a log of everything as I went along, including the reel number I’d assigned and the corresponding RedCode filename.

The only serious problem we had with the camera was right towards the end of the shoot, when we were getting errors whenever the record button was pressed, at which point it would instantly stop recording. Initially we suspected it was to do with the flash card we were using, but it turned out the camera was just overheating- 5 minutes later it was up and running again. The flash cards themselves worked well, they’re very robust as you’d suspect- all the precautions that other people have mentioned, like anti-static bags and so on, seemed completely unnecessary; in fact we didn’t lose a single bit of data from the flash cards the whole shoot.

What we did as far as saving the data was basically having a laptop with a USB flash card reader (again, contrary to what other people were saying, any old reader works fine), I would create a folder per reel (the “right” reel number as opposed to the one that the red camera said it was, and move the contents of the flash card to the folder. I moved it rather than copying it, because that way I could guarantee everything had gone across. The way Macs move files is to copy first and delete anyway, so in the event of a transfer error (there was one during the whole shoot, due to someone accidentally cutting off the power to the disk drive we were saving to…) nothing was actually lost and we could pick up where left off.

Finally each of these folders was converted to a DMG mac disk image file for various reasons, the nice thing about doing that was that you end up with a very portable read-only file per reel, which checks itself whenever you open it– and these become the original camera reels. Then it’s a matter of backing them up to tape and copying them to whoever needs them (the other nice thing abour Red is that it’s all very compact, we had just over 250GB of data for the whole shoot).

I’ve created an Automator workflow that does all of the transfer side of things automatically, but only works on Mac OS 10.5, and requires quite a powerful machine.  Grab it here… and submit feedback here…

The next part will look at preparing the rushes for the edit.