|
Computational Photography
Now here’s a chapter you won’t find in many photography books. This chapter is really a look to the future of photography. I’m sure most of you will agree that the future lies completely in the digital domain (barring niche/specialist areas) – therefore to see into the future, you could do a lot worse than looking at what is going on in the fields of computer science and electronics – specifically those relating to imaging software and hardware. Research like this is usually found in the realms of academia – university computer science or electronics departments, although many major corporations such as Canon and Sony also have extensive research facilities.
The term ‘computational photography’ refers to the creation of photographs using means that rely heavily on computer software algorithms. This often requires dedicated hardware to provide the algorithms with the necessary inputs to create the desired effect.
Computational photography is a very broard term and can be used to describe lots of topics of research including (far from an exhaustive list):
- image scaling
- dynamic range compression (tone mapping)
- colour management
- image completion (e.g. auto filling of gaps in a picture)
- image compression
- digital watermarking
- flash / no-flash imaging
- high dynamic range (techniques and sensor designs)
- refocusing after the shot
- photography using multi-camera arrays
One of the leading centres for research into computational photography is the Department of Computer Science at Stanford University. Here is their definition of computational photography ...
|
Computational photography refers broadly to sensing strategies and algorithmic techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. Representative techniques include high dynamic range imaging, flash-noflash imaging, coded aperture and coded exposure imaging, photography under structured illumination, multi-perspective and panoramic stitching, digital photomontage, all-focus imaging, and light field imaging. |
Even big names like Microsoft are carrying out work in computational photography so it is an area for all photographers to keep their eye on, as over the next 5 or 10 years, the current research topics may well make their way onto mainstream cameras and as photoshop plug-ins.
Let's take a look at some of the more interesting of the topics mentioned above.
High Dynamic Range photography - the principle is fairly well understood and we will cover this in a dedicated section. However there is research at the moment looking at tone mapping algorithms. These are required to convert your high dynamic range image back down to a range that can be displayed on a normal display.
Flash / No-flash photography - some of the well known problem with flash photography is that you can get a very harsh light, faces appearing bright and glossy, light levels that quickly fall off with distance, reflections off nearby objects etc.
Techniques are being developed to allow the automatic combining of two images - one taken with the flash and the other without - to create a result that takes the best bits of both images.
Image Deblurring with Blurred Noisy Image Pairs - a long exposure hand held shot will inevitably contain camera shake. If you use a fast shutter speed you will get a stade image that might well be very dark and therefore if boosted, very noisy. Research at Microsoft takes this pair of images (the blurred one and the noisy one), and attempts to remove the blur by using the noisy image to identify the location of the edges within the image. We think this is unlikely to be of a great deal of use in mainstream photography, but it would be useful is specialist scientific / medical / forensic photography.
Automatic Scene Completion - imagine you have a photograph which is perfect except for some foreground detail (we are not talking about a small object which can just be cloned out). Research is going on whereby you mask out the region of the image you don't like, feed the original image and mask into some software which then uses a database of millions of images and tries to create one or more plausible resultant images by matching the 'keep' part of your image with one or more images in the database - and then blends them seamlessly.
Refocus your photograph after you've taken it
This must surely be one of the most amazing tricks of computational photography. And the first time you see it demonstrated - we would challenge you not to be blown away! It is so convincing that you may wonder why its not already a feature on every camera (or at least the top of the range models).
One day we may get round to putting up a dedicated page on this, however for now a very brief explanation...
The camera that can do this is called a "plenoptic camera" or "4D light field camera". It is based on research carried out at MIT in the early 90s by Edward Adelson and John Wang however a working prototype based around a Contax 645 16MP medium format digital camera was developed at Stanford by Ren Ng for his PhD.
It makes use of a standard CMOS or CCD type sensor, however inserted in front of the sensor is an array of microlenses (different from the microlenses in front of every pixel to improve light gathering on a DSLR). These additional microlenses effectively split the sensor up into lots of discs - for example there might be 200 sensor pixels in each disc.
|
In a traditional camera, all the rays of light coming from a point that the camera is focused on converge at a point on the sensor. What the extra microlenses do is split this up so that the rays that all converge in a traditional camera are now recorded by different sensor pixels within the disc. (i.e. to record one pixel of information you are actually using a lot more pixels on the sensor since you need to capture all the different light rays). The resultant image appears like a series of small discs (example shown right). Adding all the pixels in the disc back together gives you a resultant pixel as though the microlenses were not even there. |
 |
Now if you want to refocus on another object, you can't do it the normal way by refocusing the lens as the picture has already been taken. However another way of changing the focus point is if you could move the position of the focal plane (the sensor). And because the microlenses you inserted have recorded all the rays entering the disc at different angles - you can effectively do this.
above: image is in focus on the sensor - all rays from the tip of the arrow converge to a point.
In the image above - the subject (the arrow) is further away than the focus point and the rays will not form a sharp point on the sensor but what is known as a circle of confusion (you can see the three rays are spread out on the sensor). However you'll also notice that the rays do converge at a point in front of the sensor. If you could add these rays up, you would be able to get the image in focus even though the lens was not focusing them on the sensor.
With the plenoptic camera, you have in fact recorded all the individual rays - however as they are spread out - they are actually recorded by different actual sensor pixels in different discs. So instead of summing all the pixels in the same disc, you sum certain pixels from different discs to give you the resulting pixel. By doing this you are effectively moving the sensor position, and hence moving the point that is in focus either nearer or further away from the camera than the original focus point (the one that the main lens was actually focused on).
AMAZING!!!
Go to the Refocus Imaging Inc. website to see it in action. This is a new startup company to commercialise the research.
The downside is that a pixel in the resultant image corresponds to the disc so if you have 200 pixels in the disc, your image sensors native resolution will drop by the same amount - i.e. a 16MP sensor will become less than a 0.1MP sensor. Of course you can have less pixels in each disc and not take such a hit on resolution, but your ability to refocus is reduced.
|
