How to deal with the data deluge involved in 4K work

Gary Adcock

I am going to be honest. The future in 4K is pretty bleak for everyone who chooses to cling onto their existing workflows and hardware. 

Be forewarned, those productions scratching and clawing their way through their first 4K project utilizing Firewire drives for data management and editing.

Its not just all your cameras and lenses that have been outdated in the deluge 4K. Just to maintain functionality, your computer, storage solutions and monitoring all have to be re-evaluated and upgraded.                                                   

 

Why is 4K necessitating that upgrade? Your existing HD monitor is only handling one-fourth of a 4K native frame. When working with compressed ProRes files, the data rate currently required for playback is barely within the rounding error as compared to the 1.6GB a second required when finishing in 16bit color for High Dynamic Range (HDR) content using the A.C.E.S. Workflow from  AMPAS

I am going to limit this discussion to 10bit RGB, where working in a 24p 4K project will still require processing as much as 800MB each second.  It will potentially destroy your home life with all but the fastest hardware configurations, while you wade through laboriously slow renders and data transfers. 

Overwhelming amount of data being generated    

Computers use was never intended to handle the overwhelming amounts of data we are generating on set and the geometrically greater amount of data created by post. It is not uncommon for 2TB of natively acquired, RAW camera data to magically blossom into more than 12TB of working files between the initial edit and finish.

This breaks down to the amount of data created vs. the throughput required to playback that same data in realtime.

Its just not feasible to add a bunch of loose drives in a pile on your desk to solve the volume and throughput. Working in 4K will force a re-evaluation in your conception about workflow.

The existing infrastructure for portability onset has clearly reached its limits. Cheap portable drives are no longer suitable for editing in a 4K world, yet the existing computer interface types are all lacking.

Firewire800 was replaced with eSata and USB3 drives are only now achieving the speeds necessary to be able to handle realtime playback of 4K. 

Yet the latest USB 3.1 specifications barely handle playback of 10bit 4K content at anything faster than 30fps, even Thunderbolt2 workflows delivering 4K 60p are limited to displaying just 8bits of color because of bandwidth limitations.

The technological association between devices is fairly well calculated. I have lived through the nightmares and assure you that cameras and computers built in the same general timeframe will have fairly matched capabilities for the end user. 

Moving forward in your future in post or production, I leave you with a small piece of advice:  The computer you edit your project on should never be older than the camera on which you created that project.

Next week: Handling 4K connectivity, the here and now.