We offer a position as an

Early Stage Researcher (M/F/D)

in the Marie Skłodowska-Curie Innovative Training Network Project MENELAOSNT in the field

Deep Depth from Defocus (Deep DFD) for near range and in-situ 3D exploration (ESR5)

 

Acquiring 3D geometry of the scene is essential for a series of applications like scene understanding, navigation, robotics etc. Among the existing approaches, those using passive 3D sensing are more cost-effective and allow the use of compact, standard systems. Two hints for depth estimation by passive sensing are the blur and geometrical deformations due to perspective representation. The blur is due to camera lens, that represents any point, which is not in the focal plane, by a disk (circle of confusion) in the image plane. The following figure explains blur formation:

Geometrical optics calculations provide the following equation for Δ:

This equation shows that the blur D depends on depth  and, conversely, that depth can be extracted from blur. The approach is known as Depth from Defocus (DFD). Grossmann pioneered the DFD technique, by proposing a method for circular apertures [Gro]. Since then considerable progress has been made. A cornerstone is the works of Levin et al. [Lev], who replaced the circular aperture by a coded aperture, that turns the circle of confusion into an arbitrary shaped Point Spread Function (PSF). The coded aperture was designed so that the PSF patterns corresponding to various depths to be easily discriminated. Refined methods based on single or multiple apertures evolved in the next years [Lin, Zho]. Despite the improvements, the DFD used solely remains severely limited by the ambiguity in depth estimation with respect to the focal plane and dead zone, and by the heavy calculation needed by the algorithms.

In parallel to DFD, machine learning methods relying on geometric deformation have been developed. The success of Deep Learning (DL) in a series of applications has leveraged this approach and methods for inferring depth from multi-focused images were proposed [Eig, Liu]. Recently, DL was used to combine geometry and defocus for extracting more accurate depth maps [Car]. The results certify that including blur in 3D equation can provide real benefit for depth extraction and that the joint use of structural and blur information overcomes current limitations of single-image DFD.

The main objective of the thesis is to propose, develop and test DL based methods that exploit the naturally arising blur in the images to its full potential in order to obtain more accurate 3D maps. The use of hints like coded apertures or pairs of stereoscopic images should be considered. To this end, the ESR will identify and study the cues that carry depth information in the images, will propose an architecture for the neural network and will implement it by using free DL libraries (e.g. Tensor Flow, Keras, PyTorch). To train and test the network, the ESR will use pairs of true depth maps and images from existing databases [e.g. NYU]. If the images are all-in-focus, they will be synthetically defocused by using a realistic blur model and real camera parameters. To quantify the test results, appropriate error metrics will be selected from the related literature. The network validation will be carried on indoor images taken with a DSLR camera. An additional sensor will be used in order to obtain the ground truth. A final validation will be done for in the wild images.

The successful candidate will be employed for a maximum period of three years full-time equivalent and receives a generous financial package plus an additional mobility and family allowance according to the rules for Early Stage Researchers (ESRs) in an EU Marie Sklodowska-Curie Actions Innovative Training Networks (ITN). A career development plan will be prepared for each fellow in accordance with his/her supervisor and will include training, planned secondments and outreach activities in partner institutions of the network. The ESR fellows are supposed to complete their PhD thesis by the end of the 3rd year of their employment. For more information please visit the Marie Sklodowska-Curie Actions Innovative Training Networks website.

 

YOUR TASKS

  • Achieve more accurate depth estimation
  • Speed-up of depth map generation
  • Fusion of blur and geometric and/or binocular information in order to overcome the limitations of basic DFD method

PROFILE

  • A Master of Science in Computer Science Science is required. It could comprise to the full range of mathematical, physical, engineering and technology disciplines related to sensor data acquisition and programming
  • Proficient English level
  • Image analysis, neural networks, programming languages, basic knowledge on optics could be an advantage.

PLANNED SECONDMENTS

  • CiTIUS, Santiago de Compostela, Spain, Prof .Dr. P. López Martínez, 3 months, study the possibilities to integrate the coded aperture into the sensor and to understand the impact of sensor technology on the camera performance.
  • INSITU, Santiago de Compostela, Spain, Prof. Dr. P. Arias Sánchez, 5 months, refine the camera specification by studying the applications.

ADDITIONAL INFORMATION

References

  1. [Gro] Grossmann, Paul. “Depth from focus”. Pattern Recognition Letters 5, no. 1 (1987): 63-69.
  2. [Lev] Levin, Anat, Rob Fergus, Frédo Durand, and William T. Freeman. “Image and depth from a conventional camera with a coded aperture”. In ACM Transactions on Graphics (TOG), vol. 26, no. 3, p. 70. ACM, 2007.
  3. [Lin] Lin, Jingyu, et al. “Separable coded aperture for depth from a single image.” IEEE Signal Processing Letters 21.12 (2014): 1471-1475.
  4. [Zho] Zhou, Changyin, Stephen Lin, and Shree K. Nayar. “Coded aperture pairs for depth from defocus and defocus deblurring.” International journal of computer vision 93.1 (2011): 53-72.
  5. [Eig] Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. NIPS (2014)
  6. [Liu] Liu, Fayao, Chunhua Shen, and Guosheng Lin. “Deep convolutional neural fields for depth estimation from a single image.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.
  7. [Car] Carvalho, Marcela, et al. “Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks?” Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  8. [NYU] https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

University POLITEHNICA of Bucharest is the oldest and most prestigious engineer school in Romania. Its traditions are related to the establishment, in 1818, by Gheorghe Lazăr, of the first higher technical school with teaching in Romanian, at the Saint Sava Abbey in Bucharest. In 1832, it was reorganized into the St. Sava College. On 1 October 1864, The School of Bridges and Roads, Mines and Architecture was established, which becomes, on 30 October 1867, The School of Bridges, Roads and Mines, with a duration of 5 years. Under the leadership of Gheorghe Duca, on 1 April 1881, the institution acquires a new structure, under the name of The National School of Bridges and Roads; on 10 June 1920, the Politehnica School of Bucharest was founded, with four departments: Electromechanics, Civil Engineering, Mines and Metallurgy, Industrial Section. From November 1920 the name changes to POLITEHNICA of Bucharest. On 3 August 1948, the Polytechnic Institute of Bucharest was founded, which initially included 4 faculties and in which, since 1950, have appeared most of the current faculties. Based on the resolution of the Senate of November 1992, the Polytechnic Institute of Bucharest became University POLITEHNICA of Bucharest.

The position will be located at

University Politehnica of Bucharest
Splaiul Independenței nr 313, Sector 6
060042 București, Romania

Supervisor: Prof. Dr. Daniela Coltuc

Planned Recruitment date: 1st September 2020.
Eligibility Criteria and Mobility Rule

ESRs must, at the date of recruitment, be in the first four years (full-time equivalent research experience) of their research careers and have not been awarded a doctoral degree. Full-Time Equivalent Research Experience is measured from the date when the researcher obtained the first degree entitling him/her to embark on a doctorate (either in the country in which the degree was obtained or in the country in which the researcher is recruited), even if a doctorate was never started or envisaged. Researchers can be of any nationality.

ESRs must not have resided or carried out their main activity (work, studies, etc.) in the country of the recruiting beneficiary for more than 12 months in the 3 years immediately before the planned recruitment date. Compulsory national service, short stays such as holidays, and time spent as part of a procedure for obtaining refugee status under the Geneva Convention are not taken into account.