RAW File Bit Depth Changes with ISO

Let’s begin with the fact. The usable bit depth of your RAW file depends on the ISO used to shoot the image.

I discovered this while working on the RAW histogram feature in Kuuvik Capture. To make the RAW histogram usable, we have to scale the data coming from the RAW file. This scaling ensures that the left side of the histogram represents pure black and the right side represents pure white. Technically scaling is done by first subtracting the black level from each pixel, then mapping pixel data from the [0, white saturation] interval into the [0, 1] interval.

Black level is the value your sensor emits when no photons reach a given pixel. This is calculated utilizing a black masked area along the edges of the sensor (see my former post on this).

White saturation is the value from the given pixel when it’s completely full – that is more photons reaching the pixel will not generate a higher value. This depends on physical attributes of the sensor. We do a series of measurements for each sensor to determine its value. The higher the white saturation the more tones your RAW file contains.

What surprised me during the initial white saturation measurements is that with most of Canon’s cameras this value changed as I changed the ISO. Some cameras even present different white saturation in different exposure modes (Av and M for example).

The following graph shows the result from these measurements converted into usable bit depth for four cameras up to ISO 6400.

bit-depth-vs-iso-2For the mathematically inclined, usable bit depth is calculated with the formula:

\(\log_2 (w – b)\)

Where \(w\) is the white saturation and \(b\) is the black level.

The roughly 0.3 bit difference between the lowest and highest values doesn’t seem that large at first sight, but this means that you lose 15% of the tones at ISO 640 compared to ISO 800. To put it another way it’s a 1/3 stop difference.

Implications

Avoid non-full-stop ISOs.

The truth is that both ISO 500 and ISO 320 are exposed at ISO 400, putting a 1/3 stop “digital exposure compensation” value into the RAW file. For the ISO 320 setting this produces an overexposed image, which should be pulled down 1/3 stop. The downside is that you lose 1/3 stop of both tonal and dynamic range. The upside is that there will be less perceived noise, which can be helpful in some situations (and which is the basis of lots of false myths)

Avoid ISOs < 200 on crop-sensor Canons.

As you can see on the graph above, bit depth on these machines are less below ISO 200 than on or above it.

What about the 1D X?

Some of the 1-series bodies are not prone to the 1/3 stop bit depth loss. For example the 1D X starts to show this behavior at ISO 12800. The 1Ds Mark III produces the exact same bit depth at each ISO. And the 1D Mark IV works like the 5D Mark III.

So my practice is to use just full-stop ISOs and forget about ISO 100 on crop-sensor bodies.

Canon WFT-E7 First Impressions

There are situations when you need remote camera control. No, I’m not speaking about the “let’s control the camera from an iPad two meters away” kind of pointless exercises, I’m talking about real needs. Think about cameras mounted on the roof of the stadium, situations where you would scare wildlife away with your presence, or downright dangerous places where you don’t want to spend more time than absolutely necessary.

I bought Canon’s Wireless File Transmitter WFT-E7(B) to assist in implementing some of my photographic ideas, and also to develop network (both wired and wireless) tethering support for Kuuvik Capture. While it’s clear that Canon’s transmitters are not standing in the forefront of today’s technology, if you could live with their quirks and limitations, then they could do the job well.

The Pricing Issue

Well, I paid 520 EUR (roughly $675, excluding taxes) for the WFT. Although this is some $175 less than the list price, it is pretty expensive. The bill of materials for this device is hardly more than $50, so this asking price is a bit irritating. As a comparison: an Apple Airport Extreme base station that packs comparable amount of software costs $179. Considering the functionality of the WFT, my opinion is that Canon could sell huge amounts of these at $199 or even at $299. But the $849 list price is simply unrealistic.

The bad news is if you need full, 100% USB-equivalent remote control functionality, then you have to swallow the price, and buy the device. There are some products, such as Camranger, that offer partial solutions, but you need a WFT for the whole thing.

Integrated Wi-Fi (and WFT software) on the 6D certainly points to a good direction, and I hope more cameras with integrated WFTs will follow. This is how it should be done in 2013.

The Hardware

When the first wireless transmitters had appeared for Canons, they were ugly bricks that connected to the camera with a cable, but in exchange they supported multiple cameras. Then Canon made their mind, and started to sell dedicated transmitters for each camera model (think 5D2 and 7D). With the WFT-E7, the ugly brick returns. In theory it will support future cameras, but I found no sign on the Net whether it works with anything except the 5D3. I plan to investigate this in the near future.

Its exterior finish acts like a dust magnet, and also files off small skin particles of your hand, so it’s a challenge to keep it clean.

Mounting the WFT is another challenge. I’m using Arca-Swiss compatible tripod heads/plates, so screwing the WFT under my camera is not an option. As a short term solution I simply attached a keyring to the supplied neoprene case, and hung the device on my tripod hook. In the long term the keyring will be replaced with a carabiner that will attach to that hole on the tripod base. Canon packages two cables with the device: one is too short and the other one is too long for me. This is why I reel up and velcro the longer cable to the case.

The good news is that the WFT is powered by the same LP-E6 battery that powers the 5D3.

Operating Modes

You can choose from the following operating modes (which are mutually exclusive):

  • Remote camera control (EOS Utility).
  • WFT Server, which is a web-based remote control facility.
  • You can upload your shots to an FTP server.
  • Show your images on a DLNA compatible TV set.
  • Can sync time between cameras.

The only mode that gives you full remote control is the EOS Utility mode. This is what I’m using, so will skip the others.

Network Configuration

This is the first quirk. For some unknown reason, the device can’t operate as an access point. So you either connect it to an existing network; bring your own access point (another box, batteries, etc); or create and use an ad-hoc Wi-Fi network (which can work only in “g” speed, and its WEP encryption isn’t something considered secure). This is a major oversight on a $675 device.

I won’t talk about basic network configuration, because you can find pretty good and detailed guides at Canon’s Digital Learning Center. I will, however, talk about another quirk: pairing.

Pairing

To control a WFT-equipped camera, you have to pair it to the computer and application you plan to use it from (I’ll call this computer/application pair an endpoint).

You must configure your networks settings on the 5D3 using the Connection wizard. No matter how experienced you are in IP network configuration, this is the only way you can get your WFT to do the pairing at the end.

Below is a screenshot of the network camera manager in the private beta version of Kuuvik Capture 1.1 showing my paired (and connected) 5D Mark III.

Network camera manger in Kuuvik Capture 1.1 beta

Network camera manager in Kuuvik Capture 1.1 beta

Working with the WFT

This is the best part: once paired and connected, you can forget about the WFT. Everything looks and works as if your camera was connected via USB. Yes, Wi-Fi can be slow (especially ad-hoc “g” speed networks), so large RAW file downloads can take a while. But otherwise the functionality is completely identical.

You only have to pair your WFT to Kuuvik Capture once, but you must watch for the connection sequence later on. WFTs advertise themselves on the network using Bonjour. When you turn on a paired WFT, it will advertise itself for a minute. During this time Kuuvik Capture (or another software) must connect to it. If the camera does not receive a connection request within a minute, it will shut down the WFT! From this point on things become pretty stochastic. Sometimes after a while the camera turn the WFT on again, sometimes you have to disable and re-enable EOS Utility mode on the camera to retry the connection.

The recommended sequence is:

  • Start Kuuvik Capture.
  • Turn on the paired camera. It might need half a minute or so to start advertising itself. When it appears in Bonjour, Kuuvik Capture will automatically connect to it.

Conclusion

During the last few weeks of testing the WFT-E7 worked as expected – offering stable, completely USB-equivalent connection. I don’t have a reason no to be satisfied with it. I just try not to think about its price.

Oh, I’m sure 6D owners will break into a smile seeing the ugly brick hanging under my tripod…

How to Make Focusing a Tilt/Shift Lens Easier

The tilt movement is used in technical cameras as well as DSLR tilt/shift lenses to precisely adjust where the plane of focus is on the image. Focusing with tilt is a tedious process (described here and here), but the results always worth the time!

There was a big pain point in using DSLR T/S lenses: checking what you have done. The viewfinder isn’t enough for that with today’s high resolution bodies, so you have to zoom in and check different points on the image using magnified live view. The adjust either tilt or focus. Then check the points. Then refocus… I had some images where I spent more than half an hour on fine tuning focus!

I said “was” – as it was the case before Kuuvik Capture’s Split View feature came along. I’m using this since I was halfway into developing the first prototype, and man, it can save lots of time! No, it won’t think instead of you, but the ability to quickly and visually asses what you have accomplished is priceless. It is also a great tool for learning how to focus a tilt/shift lens.

So watch the video below, and if you are using a Canon EOS-1D X, 5D Mark III or 6D with any of Canon’s great tilt/shift lenses, then grab Kuuvik Capture’s beta now! It’s that good (OK, don’t believe me, try it for yourself ;)).

The Adventure of the Mendacious Histogram

Digital exposure optimization is a controversial topic. Although the notion of “exposing to the right” is widespread, widely accepted with a large group of advocates, camera manufacturers “doesn’t seem to get it”. But there’s much more technical stuff behind this than simple ignorance. In this post I’ll shed some light on how complicated digital exposure optimization could be.

Let’s start with ETTR. Two of my masters, Michael Reichmann and Jeff Schewe had extensively written about the topic, so instead of replicating it I would encourage you to read the following articles and Jeff’s new book. It is imperative to grasp the idea so that you can understand the rest of the post.

  • Expose (to the) Right – The original article from 2003.
  • Optimizing Exposure – A rather utopistic view of the problem. Reminds me of Adams’ (Douglas, not Ansel) Total Perspective Vortex – as it extrapolates a whole universe not from a fairy cake, but from the fact that increasing exposure reduces shadow noise. Anyway, a good read on what would be really needed from a photographers point of view, even if its not possible with the current technology.
  • The Digital Negative – Jeff’s book collects the majority of information about digital exposure right in its first chapter.

To summarize: increasing exposure has advantages to the shadows and the amount of information retained in the RAW file. That’s great. But here comes the million dollar question: how much one should increase exposure being hundred percent confident that highlights aren’t blown or destroyed (and thus, keeping all the possible information)? You should read Ctein’s article on the dangers of ETTR regarding lost highlights.

I hope you are confused enough about whether to ETTR or not and how can you really assess over-exposure. Don’t be afraid, this is the where our adventure begins.

Before we embark on it, let me rephrase the question: as overexposure is terminal to the data (details) in the overexposed area, how can one avoid it with confidence? Regardless of whether you ETTR or not, this is important. Imagine a bright yellow flower for example. Overexposing one or more channels will destroy fine color variance – which is a bad thing (except if you will run the image through some ugly lo-fi filter, of course, in which case the things I’m writing about is totally unimportant to you). Also, what I’m writing about is for RAW shooters only. JPEG guys only get what they see, so these topics does not apply to them.

Let’s begin!

One Image – Three (Different) Histograms

The histogram is the primary tool for assessing exposure on a digital camera. But what your camera shows has only a little resemblance to the RAW data recorded. This is because all JPEG settings, such as white balance, color space, sharpening, contrast, etc. influence the histogram display. As a result, the histogram on the LCD comes from the JPEG preview of your RAW file. In an ideal world, one could set the histogram into “RAW mode” which would instruct the camera to calculate the histogram from the RAW data instead of the JPEG preview.

Canon 5D III, Auto WB

The majority of the parameters mentioned above can be zeroed (like sharpening, contrast) on the camera. The problem-child is white balance, where we have little influence by default.

Take the image on the left, for example. The RGB histogram shows gross overexposure in the red channel, you can even see overexposure warning “blinkies” in Elmo’s eyes.

Based on the camera’s histogram one would lower the exposure to avoid overexposing Elmo’s… wait! Blinkies warn for overexposure in the eyes, not the red fur, even if the red channel was blown. Something isn’t kosher with this…

Histogram from RAW Data

On the right is a histogram generated from the same image with Kuuvik Capture utilizing gamma-corrected RAW data. As you can see reds are far from being overexposed. You can also see a different distribution in the RAW histogram, with the peaks are being at distinctly different places. This is because the RAW histogram in KCapture is not white balanced.

White balance is set by multiplying data coming from a channel with a number (the white balance coefficient) to “scale” it to reach the desired white point. It is represented by a four element vector (one number for each channel, in RGGB order). You can use exiftool to examine these coefficients, they are displayed as WB RGGB Levels As Shot. The actual values for the above image are: (2185, 1024, 1024, 1526). That is, you have to multiply the red channel by 2185/1024 = 2.13 to get the white balanced image. You can easily see from the RAW histogram that multiplying reds by 2.13 on this image will blow the channel out – in that white balance setting.

Sidebar: white balance is always represented as RGGB coefficients internally, not with color temperature/tint as RAW converters and cameras present this data to you. Color temperature is an “artificial” construct to handle these numbers in a more user-friendly way. And the way these coefficients are converted into color temperature is a proprietary process for each converter. This is why you get completely different white by using the same Kelvin value in different converters.

Now let’s take a look on what RAW converters, Adobe Camera RAW 7.3 to be exact, think about the same image (Capture One displays a similar one).

Histogram in ACR 7

Part of the black magic of RAW conversion is graceful handling of the roll-off into overexposure. The fact that a non-overexposed channel can be blown during white balancing is responsible for the converter’s ability to do highlight recovery. They are mostly taming the data that is blown only by the currently set white balance. As the common myth goes: RAW files have more headroom in the highlights. And as with most of the myths, there is truth lurking behind it: because the white balance is not fixed in RAW captures, converters have the ability to extract more information from it than you would be able to get from a JPEG, where clipped highlights (even if clipped by the current white balance setting) are lost forever.

Above I shown you the case when the JPEG histogram shows overexposure, while the RAW histogram doesn’t. It could happen the other way around, which is even more dangerous. I recommend you to read Alex Tutbalin’s article on white balancing problems for an example. Alex is the author of Libraw, the library Kuuvik Capture is also using for extracting RAW data from proprietary file formats, such as Canon’s CR2.

On Gamma Correction

If you read Alex’s article, you saw the Rawnalyze tool, and if you try both that and Kuuvik Capture, you’ll get different histograms. Why? Because what Rawnalyze displays is the rawest raw data possible. That is, it doesn’t map the camera’s black level to the left side and the maximum saturation level on a given ISO to the right of the histogram (in other words, it does not scale the data). In KCapture I wanted to make the RAW histogram to look familiar to photographers (including myself), so instead of blindly displaying the raw data the app does a little processing. The processing consists of scaling (so black is on the left and white is on the right instead of somewhere in the middle of the histogram), and gamma correction.

By default RAW data is linear, that is the highest exposure stop occupies the entire right half of the histogram, the next 1/4 of it, and so on. The result is that a linear RAW histogram pushes all the data to the left, and makes it hard to judge shadow exposure and see whether we have a clipping there. Instead KCapture corrects this data the same way it happens during RAW conversion: by applying a gamma curve to make exposure stops equal sized on the display.

Mapping Theory to Practice

Let’s draw some conclusions. First, 1) white balanced in-camera histograms are not suitable for checking overexposure. RAW histograms are markedly better in this, but 2) RAW histograms can only show physical overexposure of channel(s), and are blind to white balancing induced highlight clipping. Most of that clipping is curable in the converter utilizing some form of highlight recovery, however. The 3) final word on highlight clipping is said by the RAW converter after white balance has been set.

(2) is why I called Michael’s second article an utopia. When we wrote the specification for Kuuvik Capture back in December 2011, our goal was to implement ETTR optimization described in that article. It turned out rather quickly that you can’t do this unless you have the final white balance set – which will not happen until later in the process. And even if you can do this, you shouldn’t always ETTR. Sometimes artificially overexposed images will push the noise from the shadows into the sky (scroll down to the ETTR section in the linked article for an example). And even ETTR could be behind skies turning purple.

(3) is the real reason why camera manufacturers are unable to show the same histogram as your converter (unless they are using the same algorithms, of course – which is highly unlikely).

So how do I use this information in practice?

When I’m shooting tethered (which is the majority of cases when I do landscape work), I rely on Kuuvik Capture to check the physical exposure from RAW histograms. If your physical exposure is bad it won’t get any better during RAW conversion. What I look for here is potentially uncorrectable overexposure (non-specular highlights) and as a Canon shooter who’s cursed with muddy shadows, I check for underexposure. I usually push the exposure to the right when the shadows are in danger or when there’s plenty of room for the highlights.

Then I pass the image to Capture One for the final decision. Capture One 6 had some issues with highlight roll-off handling with the 5D3, so I had to back-off a bit from extreme highlights, but v7 fixes this problem (be sure to use the v7 process).

Free (that is, non-tethered) shooting is a different beast. One needs a trick to cancel the side effects of white balancing.

Unitary White Balance

Guillermo Luijk came up with the idea of UniWB in 2008. UniWB is basically a custom white balance that sets the WB coefficients to 1 (hence the name unitary). It could be useful in the field if you can live with the ugly green images (to avoid that I usually just use UniWB for exposure tuning and switch back to AWB for the real shot).

The real downside of UniWB was the tedious process of obtaining the magenta target image. Having control over the WB coefficients, you can obtain the UniWB setting in Kuuvik Capture just with two clicks: on the Set Unitary White Balance item on the Camera menu.