Lens tests – and why you should (usually) ignore them

This is part one of a series of articles concerning lens tests and performance…

Lens tests! The Internet is littered with them. I’ve even done some. Photography magazines regularly feature them. Photographers ponder their contents and base purchasing decisions on them.

But here’s a truth – in general lens tests are of limited value and buying a ‘better’ lens won’t make you take better pictures. The only time a lens test is really useful is when it identifies a real turkey – a lens that is no better than a glass jar.

OK, I’m exaggerating a little. Lens tests have their place, but you have to know how to read them otherwise they may cost you some serious money for no discernible benefit.

The key phrase is discernible benefit. There is no doubt that some lenses are sharper, faster, have less distortion, less chromatic aberration and better bokeh than others. But will you actually see the difference in your pictures? Let’s have a look at these, and other, areas and see if they will benefit you.

Image Stabilisation (IS)

If you hand hold then whether or not a lens is stabilised is relevant. Unless, of course, you have a camera that provides in camera stabilisation. But, in general, unless you only ever shoot using a tripod, image stabilisation is a plus with few, if any, drawbacks.

Whether it is worth the extra £££ for such a lens depends on many things. Wide angles don’t really need IS unless you are shooting at really slow shutter speeds. Macro lenses don’t benefit a lot from IS when used to take extreme close ups. Canon do a hyper expensive 100MM macro lens with IS designed for macro use but most tests I’ve read don’t think it provides significant benefits.

From 40mm onwards, IS provides real handholding advantages in non-macro scenarios.

Conclusion: IS is an advantage, but a landscape photographer can probably live without it.

Distortion

Most cheap lenses, and many not so cheap lenses, distort. Wide angled lenses in particular distort – trying to cram so much of the scene onto a tiny imaging chip (even a full frame one) and keep straight lines straight is a huge technical challenge and many lenses just cannot do it.

But does it matter? For many scenes, the answer is No. But if the distortion is obvious (curved seascape horizons or buildings that are leaning over, for example) then it almost matters. But software usually can correct it. So, it matters on a technical level but not in practice.

So, if a lens test complains that a lens has distortion, in most circumstances, that complaint can be ignored. I wouldn’t spend a ton of money getting a lens with less distortion unless that lens gave me many other benefits.

The only exception would be if the test shows that the lens has weird distortions that cannot be corrected in software. Then, I’d avoid that lens.

Also, be aware that correcting distortion in software comes at a price – after correcting the distortion the image has a to be cropped a little.

So, your ultra-wide lens will be slightly less ultra-wide. In practice, this really doesn’t matter.

Professional Architectural Photographers would probably disagree and get lenses with little or no distortion and I wouldn’t argue with that – but specialist professional users will always want to get the best equipment.

Conclusion: Ignore any criticisms that a lens distorts unless the distortion cannot be corrected in software or you are a professional and require images to always be distortion free.

Chromatic Aberration

This can usually be corrected in software. It is irrelevant unless it is incredibly bad.

Vignetting

Most lenses, especially wide angles, vignette when used at wide open apertures. This is usually irrelevant. Most wide angled lenses are never used wide open but are stopped down to gain depth of field. Longer lenses, such as those used for portraits, may well be used wide open, but in those circumstances vignetting is irrelevant as the subject will not be placed in the corner or the frame.

In any case, vignetting, unless it is truly terrible, can be corrected in software.

Conclusion: vignetting can be ignored.

Flare

All but the most expensive lenses produce ‘flare’ if the sun is included in the image. Some expensive lenses also produce flare.

If a review stated that the lens flares even if the sun isn’t in the frame or is nowhere near the edge of the picture then I’d pay attention to it. But even then, a good lens hood may prevent the problem.

Conclusion: Almost all lenses can produce flare. What matters is how easily the lens flares. I’d avoid lenses that flare easily unless I know I’m not going to use it in such conditions.

Bokeh

If a lens is going to be used to capture an in focus subject with an out of focus background, then bokeh (the appearance of the out of focus background) is important. Portrait and wildlife photographers in particular will want the bokeh to be excellent.

Wide angled lenses rarely produce decent blurred backgrounds due to the depth of field they produce, even when shot wide open. For this reason, landscape photographers usually couldn’t care less about it.

Cheaper medium telephoto to long lenses, particularly zooms, generally produce lower quality bokeh than their expensive counterparts.

Conclusion: Portrait, wildlife, wedding, macro and street photographers should pay attention to bokeh quality. Landscape photographers can usually ignore it.

Build quality/weather sealing

A professional photographer may use his equipment every day, in poor to terrible weather and in harsh, difficult conditions. For them, the build quality of both camera and lens is truly important. For more casual photographers, lower build quality and a lack of weather sealing can be compensated for by being careful and knowing the limitations of their system.

Sometimes compromise is needed. High quality comes at a cost – this equipment weighs far more and costs far more. A hiker, for example, would benefit from high quality equipment as it is less likely to break if it gets knocked or the weather turns bad. But a hiker won’t enjoy the extra bulk of the equipment.

Conclusion: Build quality matters if you use your gear in bad weather or regularly in challenging situations. The rest of us can usually do well with lower build quality and no weather sealing by exercising reasonable care and knowing when to put the camera in a protective bag.

Lens speed

The maximum aperture of a lens determines:

• The amount of light reaching the autofocus system of the camera
• The brightness of the viewfinder in a DSLR (but not in an EVF)
• The smallest depth of field the lens can produce
• The fastest shutter speed the lens can provide at any given ISO setting

In some circumstances these factors are important. Wildlife/sports/action photographers need their camera’s autofocus to be fast and accurate and the more light reaching the AF system the better. A faster lens will still autofocus well with a teleconverter fitted.

Viewfinder brightness matters – it does help to be able to see what you are taking, especially in low light. EVFs and Live View systems brighten the image automatically, so this may not be a factor for you.

Portrait photographers, in particular, need to be able to isolate their subject and blur the background. A 135MM F4 lens probably wont be suitable, whereas an F2 optic will do very nicely. Landscapers and general purpose photographers probably won’t need such shallow depth of field – Landscape photographers in particular want maximum depth of field, so a fast lens really isn’t important.

Action photographers need fast shutter speeds, as do street photographers looking to ‘freeze’ the action. In low light they would probably prefer to sacrifice some depth of field to gain a fast shutter speed rather than shoot at a high ISO level.

But fast lenses have drawbacks:

• They are heavy. Often really heavy. The Canon 70-200MM F2.8 IS II lens weighs 1490gm whereas the 55-250MM f4.0-5.6 IS STM weighs a paltry 375gm. I know which one I’d prefer to carry up a mountain.
• They are expensive. The aforementioned 70-200mm costs almost £2,000 whereas the cheaper 55-250MM costs less than £300. Is the quality of the 70-200mm better? Probably. But it certainly isn’t 7 times better.

Conclusion: Lens speed matters to specialists. Landscapers, macro photographers and general purpose users can generally ignore it.

Autofocus Speed

Professional sports and wildlife photographers need fast and consistently accurate autofocus. The rest of us don’t need blazing speeds.

Lens sharpness

I’ve saved the least important to last. What?? Sorry, but lens sharpness is the most talked about and usually irrelevant factor in making a lens purchase. The simple truth is, almost without exception, all lenses are sharp. True, not all lenses are sharp all the time and some are sharper than others. But, if you work within a lens’s limitations, you’ll get good results.

Here’s a truth you need to keep in mind when evaluation sharpness. Lens sharpness is evaluated by viewing an image at 100% magnification on a computer screen. On Windows, a typical screen has 72 dots per inch. So, viewing a 6,000 pixel wide image at 100% is about the same as viewing the same image, at close range, printed a on a seven foot wide sheet of paper. When did you last do that?

Of course, a typical print is about 300DPI – so my 6,000 pixel image would usually be printed at a maximum of 20 inches wide… And there’s the thing to remember: when you view an image ‘normally’ – on screen or by printing it – you simply will not see it at the extreme magnification that the lens has been tested at.

If you are intending to produce huge prints, then lens sharpness may be a factor but it’s still less important than you may think.

Here are some common sharpness ‘facts’ you should usually ignore:

• The lens is soft at the edges wide open
• The lens is soft wide open
• The lens sample was decentred
• The lens has field curvature

Soft edges wide open usually don’t matter. The main reason for shooting wide open is to get a blurred background… In low light, street photographers may sacrifice depth of field to get a faster shutter speed at low ISO settings, but are they really going to put important parts of the scene in the corners?

Image softness wide open usually doesn’t matter. At F1.2 the depth of field is so shallow that a real world subject is only going to be in focus in a small (maybe very small) area of the picture. Stopping down to F1.8 or F2 usually resolves softness issues and, really, does it matter if the picture was shot at F1.2 or F1.8? The reasons for buying such are fast lens are listed above. Using it wide open is not one of them…

Lens decentering is noticeable when shooting a flat object wide open. If shooting brick walls or lens test charts is your thing then this is a key factor. Otherwise, you can ignore it.

Field curvature – this is as irrelevant in real world conditions as lens decentering.

And that’s the key point, actually. Lens tests no more measure the quality of a lens than an IQ tests measure a human being’s overall worth. They tell you something about the subject, but not how they’ll perform in the real world.

Typically, to assess a lens the tester takes a picture of a lens test chart or a flat subject such as a brick wall. Many of the lens’ weaknesses will be magnified in this scenario and its strengths will be negated.

In the real world, many of a lens’ weaknesses just evaporate – landscapers stop their lenses down to gain depth of field. This eliminates the effects of field curvature and decentering and uses the lens at its maximum sharpness.

Even wildlife and sports photographers usually stop down a little to gain depth of field. The great bird photographer Arthur Morris usually shoots his 600mm f4 lens at f5.6 for these reasons.

Portrait photographers (and their subjects!) may actually appreciate a little bit of softness when shooting wide open and they certainly won’t care about corner and edge sharpness.

Macro photographers will often want edge to edge sharpness but they will stop down to gain depth of field. So will landscapers. And most general purpose photographers will get good results from their lenses in real world situations by shooting at medium apertures.

If you are in the habit of cropping images, maybe as an alternative to buying an expensive super-telephoto lens, then centre sharpness will matter. At medium apertures, most consumer zoom lenses are decent enough to tolerate some cropping, but there is always going to be a limit to this technique.

Lens sharpness is not the end of the story. Once the image has been captured it still can be processed on a computer. Serious photographers shoot RAW images and will sharpen the image in stages – straight after RAW processing, after adjustments and finally for printing/displaying on screen. Users of DXO’s RAW converter benefit from tailor made lens sharpening, provided their camera and lens is supported (which is usually the case). DXO ‘knows’ which areas of an image are soft and sharpens these areas more strongly. The result is usually a very evenly sharp image. Modern sharpening tools, such as Focalblade and Topaz Detail are really good at enhancing images, and result in very acceptable photos even if the lens isn’t the sharpest.

One final thought regarding lens sharpness. A test may show that a lens is 10% sharper than the one you currently use. That may sound good, but will you really see the difference on screen or when printed? A huge print may show the difference. Viewing the picture at 100% may show the difference. But in usual, real world conditions you probably won’t be able to tell the difference. So, is it really worth spending hard earned £££ to get that improvement? You’d probably be better off getting a new tripod or using the money for a photo trip…

So what are lens tests good for?

By all means read lens tests. In very rare cases they may tell you that a lens is hopeless and should be avoided. Other than that, use a lens test to ascertain a lens’ limitations and then avoid exceeding those limitations.

Here’s a list of limitations and how to compensate for them:

• Lens speed: Only meaningful if you must have a sharp subject and a very blurred background. Using a higher ISO, image stabilisation or a tripod can usually compensate when using a slow lens in low light. If you shoot landscapes, then fast lenses provide little benefit.
• Chromatic aberration and distortion: ignore this unless it is very bad. Correct this in software. If you shoot JPEGs then the camera will usually correct these automatically.
• Image stabilisation: useful to have. Not that important for wide angled lenses.
• Autofocus speed: Important for sports/action/wildlife photographers. Irrelevant for landscapers. Mostly irrelevant for general purpose photographers.
• Bokeh: Only relevant if you need blurred backgrounds.
• Sharpness: Once you know the optimal apertures for a lens’ sharpness, ask yourself if you can live with it. Most lenses are sharp from f4 to f11 and these are the most generally used and useful apertures. If you really need optimum sharpness wide open then shop around for such a lens. Most of us wont need this. Don’t spend huge amounts because a lens test say the more expensive one is 10% sharper. You won’t notice the difference.

See also Ken Rockwell’s article on lens sharpness…