Imaging Resource a reliable reference?

I have been using the Compare Images screens at Imaging Resource to look at how different cameras handle High ISO in comparison to each other. In particular I look at the Still Life.
But, recently, as I looked more closely I have begun to believe there are significant differences in exposure and focus between shots, which would make it hard to do a real comparison.

Anyone else use (or have used) this site, and opinions?

What sorts of variations are you seeing? We really try to control all the variables as best we can, given the limitations of a 3-D target, some necessary variation in lenses, and differences between cameras.

To make sure that the results are directly comparable, we adjust exposure so the shots turn out with exactly the same brightness value (123 +/- 1-2 units) for the middle grey on the mini-MacBeth chart at the bottom. Depending on the camera's tone curve, there may be minor variations in shutter speed needed to achieve that. We also make small adjustments (a couple of inches one way or the other) in the distance between the lights and the target, to make exposure adjustments of less than the 1/3 EV that most cameras can manage. For Canon, most Nikon, Pentax, and Sony, we use the same model lens, a Sigma 70mm f/2.8 macro, the sharpest lens we've ever tested for SLRgear.com. For other platforms, we use the sharpest lens we've been able to find, but they are different lenses, which means different T-stop values, which would also lead to different shutter speeds. (The actual amount of light transmitted by the lens at a given aperture, here's the Wikipedia description of this: F-number - Wikipedia, the free encyclopedia.) Different lenses also mean different shooting distances, as is also the case with cameras with different sensor sizes. We control the shooting angle, but adjust the distance so the subject always fills the frame to the same extent.

We've tried to standardize on an aperture that's as small as we can manage (for depth of field) without diffraction-limiting. Somewhat arbitrarily, we've settled on f/5.6 for cameras with larger sensors, and f/4 for ones with smaller. Recently, we've seen some cases with intermediate-sized sensors where f/5.6 is diffraction limiting, so we've gone to f/4 for some of those as well.

As to focus, we *try* to always focus the same, pointing the camera at the neck of the vinegar bottle, as it has good contrast and is roughly in the middle of the depth of the target itself. We also shoot at a slight downward angle, so the plane of best focus aligns better with the arrangement of objects in the scene.

Of course, though, we're dealing with a 3-dimensional subject, so variations in depth of field with different sensors will come into play. Full-frame cameras have shallower depth of field, so some parts of the subject in front of or behind the plane of focus may be rendered less sharply than by some smaller-sensor cameras.

There have also been some minor changes over time, as we've gradually tightened our control over the variables. In particular more recent shots have much better-controlled exposure levels (less variation in the shade of the gray swatch), and at some point, we switched from shooting at f/8 for better depth of field to f/4 or f/5.6, to avoid diffraction limiting in the lens. We've also significantly improved our control over framing and shooting angle in recent years, as compared to some shots you might have seen back when we first began using the current target.

Let me know what you're seeing, if it doesn't fall into one of the categories I've just discussed. We try to control for everything we can, and there are unavoidable differences between cameras, but if we're missing something, I surely want to know about it.

- Dave Etchells
Founder & Publisher
Imaging-Resource.com

Thanks for the response.

In particular I am seeing some color variation which shows up best on the circular scale in the lower right hand corner of the Still Life as a brownish cast, rather than white. I also see a fair amount of difference in sharpness which I am attributing to focus and potentially depth of field (most visible in the same area).

What started me thinking about ths was doing side-by-sides between the Olympus OM-D E-M5. I have a hard time crediting it with being significantly sharper and better color balance at ISO 3200 than the Nikon D7000 or D600.

If it is, I may need to switch camera systems.

EDIT: And, I had assumed that depth of field was a factor in the Full-frame models. But, the DX shots (D7000, D5100 and D3200) are the ones that make me wonder if focus is an issue. Especially, since I don't see anything close to the same sharpness and color clarity in the Panasonic GH2 shots at ISO 3200.

Again, thanks for the response. I really appreciate it.

Thanks for the further info, I can maybe shed at least some light...

On the color of the scale, I need to check in the lab, but I think the plastic might be aging and yellowing in the process. The lighting is pretty consistent, and the color balance of the OMD is on the blue side if anything, so it's got to be the target itself. Thanks for pointing this out, I'll look into it and try to get a duplicate of the scale if at all possible. (If it turns out it isn't possible, we'll put a note on the scale itself that its color is changing and we can't do anything about it. - In fact, even if we do change it, we should attach a note to it, so people will understand not to rely on it for judging color casts.)

On the issue of sharpness, there's a couple of factors, but the net of it is that the E-M5 does appear/I] sharper than the D7000. To get a better sense of it, and what might be going on, I checked our resolution target shot, which is the best place to see details of what might be going on, sharpness-wise.

I see two key differences between the E-M5 and D-7000. First, the E-M5 has a very weak anti-aliasing (aka "low-pass") filter, as witness the color artifacts that you can see in the horizontal resolution wedge there, at the center of the frame. This in itself will make images appear crisper than those from a camera with a stronger low pass filter, which appears to be the case with the D7000. The downside is that you might sometimes get color artifacts like this if you have a subject with very fine, high-contrast detail in it. (Generally, though, I think that Olympus struck a pretty good balance with the E-M5.)

The other difference is in the amount of sharpening the two cameras apply to their JPEG images by default. I believe you can adjust this via the menu systems of both, but by default, the E-M5 is sharpening its JPEGs a lot more than the D7000 is. Again, you can see this pretty clearly in the res-target shots: You'll notice that there are distinct white "halos" around the edges of the numbers and solid elements in the E-M5's image. You don't notice them so much in the resolution wedges (at least I don't), but the effect on finely-spaced detail is actually even more dramatic, in the way it makes the details "pop".

The problem with lots of sharpening is that it can actually destroy really fine detail, in ways that prevent you from ever getting it back. I personally prefer the approach that Nikon's taken with the D7000, as it leaves more of the detail that is there undisturbed, so you can sharpen things to your liking post-capture in Photoshop, etc.

I do think that the E-M5's less aggressive low pass filter has a significant impact, though.

Another difference that favors the E-M5 in this comparison is its 4:3 format. To be consistent between formats, we always frame for top-to-bottom size, so images from 3:2 cameras with otherwise identical pixel pitch will simply be wider than those from a 4:3 camera. The D7000 and E-M5 have almost identical pixel counts, but the E-M5 isn't using any of those to make its field of view wider horizontally. As a result, its pixel pitch relative to the subject is actually noticeably finer, which results in the subject appearing larger on your computer screen. So, while they're both ~16 megapixel cameras, the effective resolution of the E-M5 relative to the target area is actually higher.

Finally, on the question of the sharpness of the GH2 at ISO 3,200, there's a whole *world* of things that companies do to suppress noise at high ISOs, all of which impact image quality. Noise reduction is a fine art, with many tradeoffs to be made in different directions. Low ISO shots will tell you a lot about a camera's ultimate resolution, but if you're strongly interested in high-ISO shooting, you need to look at the high-ISO test images. There's often very little correlation between how two cameras images compare at low ISO and how they compare at high.

In general, high-ISO noise reduction tends to suppress a lot of fine detail, especially in areas of more subtle contrast. You'll also frequently see manufacturers backing off on color saturation at high ISOs, because that helps the noise issue a lot.

Phew.

I hope all that was helpful, and thanks for bringing the color of the circular scale to my attention.

Now I'd better get back to the work that's burning my toes here at IRHQ.

Thanks for the lengthy (and highly informative) responses. And, BTW, welcome to Mac-Forums. Hope you will find time to visit more often.

Imaging Resource has been one of the main sites I use for camera comparisons, and I am looking forward to your upcoming reviews on the Nikon D5200 and the V2.

EDIT: After rereading your posts. A few things in case there is any misunderstanding.
1) All my observations are related to High ISO images (3200 and above). FWIW I almost never look at the low ISO shts, because they ALL look pretty good
2) I am pretty sure the brownish cast I mention isn't due to darkening of the scale. Will try to do a screen shot when I get a chance to better show what I am seeing

Screenshot showing brown overtones

As promised, here is a side-by-side of the Oly and Nikon D600 at ISO 3200. The D600 is on the right.


FWIW I didn't see the Brownish tones in most of the other Nikon ISO 3200 shots.
Hope this helps in finding the cause.

Here is a second comparison at ISO 3200. This time the Nikon D7000 is on the left and the D600 is still on the right.