kixor.net

Project: Depth from Defocus

Overview

My goal in this project was to implement a method to obtain the depth of a scene given several images of varying focus. This is accomplished by the realization that the blurriness of out of focus objects gives hints to their depth. This problem is known to be ill-posed [1]. Most solutions to the problem involve tradeoffs between efficiency, noise, precision, or any other number of variables.

I have chosen to implement the method described in [2]. The paper describes the process in detail and others [3] have used the technique to achieve reasonable estimations of depth. The process involves a spatial domain convolution/deconvolution and makes the assumption that the scene is planer with respect to the focal plane. This method requires only two images in order to estimate a solution and the camera parameters can be varied by focus plane depth of aperture size.

All project files can be downloaded.

Derivation

This is a brief overview of the derivation of the depth from defocus solution. A more detailed derivation is found in the paper.

When an image is focused in front or behind the image plane, a blur circle is formed on the image plane [2][4][5]. This blur circle is scaled by a factor according to the depth of the point causing the circle. Given the lens equation, the radius of of the blur circle can be calculated as follows:
$R = s \frac{D}{2}[\frac{1}{f} - \frac{1}{u} - \frac{1}{s}]$
Given that blur circle appears as a circular gradient of decreasing brightness on the image plane, a gaussian point spread function can be estimated:
$h(x,y) = \frac{1}{2\pi\sigma^2}e^{-\frac{x^2+y^2}{2\sigma^2}}$
Using this PSF and the focused real world image, a defocused image can be obtained by convolving the focused image with the PSF:
$g(x,y) = h(x,y) * f(x,y)$
Then, using a spatial domain transform and manipulations in polar space, a deconvolution filter using the laplacian of the image and the standard deviation of the PSF is obtained:
$f(x,y) = g(x,y) - \frac{\sigma^2_h}{4} \nabla^2g(x,y)$
Finally, given two images of different focus, we can begin solving for the standard deviation of the PSF (essentially the blur radius):
$f=g_1-\frac{1}{4}\sigma^2_1\nabla^2g_1$
$f=g_2-\frac{1}{4}\sigma^2_2\nabla^2g_2$
Then, in order to solve for the radius in the presence of noise, the two equations are equated and the laplacian replaced with the mean of both laplacians:
$g_1-g_2 = \frac{1}{4}(\sigma_1^2-\sigma_2^2) \nabla^2g$
Squaring and summing in a small region gives rise to the following integral:
$G^2 = 16 \frac{ \int\int(g_1-g_2)^2\,dxdy} {\int\int(\nabla^2g)^2\,dxdy}$
The root is:
$G'=(\sigma_1^2-\sigma_2^2)$
The sign is ambiguous due to squaring of previous terms. This is resolved by estimating the sign using the gray-scale variance of the two images. The variance corresponds to the 'blurriness' of the images.

Finally, using a relation between the two standard deviations involving the two camera parameters, a solution for the second image's blur is found based on the previous integral and camera parameters:
$\sigma_2 = \frac{G'-\beta^2}{2\beta}$

Implementation

My implementation is very similar to the one described in the [2]. A minor difference lies in the final estimation of the blur radius. Since the original focus of the method was with accurate estimation of the depth of a planer object for auto-focusing, the radius was taken to be the mode of a filtered histogram over the entire image. I am interested in obtaining a depth map over the entire image, so I did not perform the histogram filtering. Instead, in order to be robust in noise, a median filter is applied to the resulting blur estimation. This removes outliers caused by noise.

A note on the filters used in the implementation. The entire premise of [2] is the treating of the image as a cubic polynomial. The recommended filters in this case are based on Chebyshev polynomials and are described in [6].

In general, my implementation cannot be used to extract reasonable depth maps for most objects. This may be in part to the equifocal assumption present in this method. Nonetheless, while the depth maps are too flat to match the actual depth, the depth is generally close to the measured focus planes from the images. The measured depth is often several centimeters shorter than the actual depth from the camera. It should be noted the the authors of the paper described a similar occurrence, which they attributed to improper design in the camera. For the purposes of consistency, I shall do likewise.

Results

All images were processed at 300x300 with a filter size of 16, which results in a kernel size of 33 pixels. For most of my tests, I found the bluriness estimate to be less than convincing. Only in the synthetic test did it result in an estimation that clearly corresponded to the image blur. It is possible that image noise or the lack of magnification normalization causes this issue. Clicking on an image will show a full-size version

Synthetic test
This constructed image with two planer surfaces. The right half of the image is blurred to give the impression that it is closer to the camera. Note that the camera parameters are not valid, and thus the depth is not either (only the shape).

While the far plane suffers from a large amount of noise in the depth map, the shape is preserved. It should be noted that the blurriness estimate is very accurate, far more so than any of the other images. The holes in the 3d image are caused by points being removed by the noise filter.

Close image	Far image
Close laplacian	Far laplacian
Bluriness Black = in image1 White = in image2
Integral solution	Root of integral
Blur radius	Depth map
3d top	3d side

Heart sheet

This is a planer sheet textured with small hearts. The sheet lies angled to the focal plane, such that the portion at the bottom of the image is closer. In this image, only the heart outlines present depth information. The large white areas and the insides of the hearts have very little texture and thus cannnot be used for depth estimation. The results still are shaped close to the scene's, likely due to the large filter window.
Actual scene
Close image (focused at 0.99m)	Far image (focused at 1.17m)
Close laplacian	Far laplacian
Bluriness Black = in image1 White = in image2
Integral solution	Root of integral
Blur radius	Depth map
3d top	3d side

Small vase

A small vase, with a slightly convex top. The vase is fairly flat. There is an opening in the middle of the pot. The resulting depth map for these images is slightly flat compared to the actual object. This may be caused by the swirly texture on the vase. The bluriness estimation is does not seem to match the image data (bottom of the image in focus in close image, top in focus in the second image)The hole in the vase has no texture data, and the resulting depth map for it is invalid.
Actual scene
Close image (focused at 1.27m)	Far image (focused at 1.36m)
Close laplacian	Far laplacian
Bluriness Black = in image1 White = in image2
Integral solution	Root of integral
Blur radius	Depth map
3d top	3d side

References

[1] H. Jin and P. Favaro, A variational approach to shape from defocus, Proc. IEEE European Conference on Computer Vision, 2002
[2] Subbarao, M. and Surya, G., Depth From Defocus: A Spatial Domain Approach,IJCV, 1994
[3] Kyuman J., et al., Digital shallow depth-of-field adapter for photographs, The Visual Computer: International Journal of Computer Graphics, 2008
[4] A. Pentland, A new sense for depth of field, IEEE Trans. Pattern Anal. Mach. Intell., 9:523-531, 1987
[5] J. Ens and P. Lawrence, An investigation of methods for determining depth from focus, IEEE Trans. Pattern Anal. Mach. Intell., 15:97-108, 1993
[5] P. Favaro and S. Soatto, Learning depth from defocus, Proc. IEEE European Conference on Computer Vision, 2002
[6] Meer, P. and Weiss, I., Smoothed differentiation filters for images, Proceedings., 10th International Conference on Pattern Recognition, 1990

Project: Depth from Defocus

contact info on the about me page