Researchers from Princeton University and the University of Washington have developed a high-resolution color camera roughly the size of a coarse grain of salt.
This new sensor technology combines meta surface optics and machine learning models to reconstruct images via the nano-optic imager. Specifically, the research paper detailing the technology says ‘Nano-optic imagers that modulate light at sub-wavelength scales could enable new applications in diverse domains ranging from robotics to medicine. Although metasurface optics offer a path to such ultra-small imagers, existing methods have achieved image quality far worse than bulky refractive alternatives, fundamentally limited by aberrations at large apertures and low f-numbers. In this work, we close this performance gap by introducing a neural nano-optics imager,’
Previous micro-sized cameras (left) captured images with low detail, false color and distortion. The new system, neural nano-optics (right), produces sharper, full-color images. Image courtesy of the researchers.
The camera relies upon a technology called a metasurface, which includes 1.6 million cylindrical posts. Each post is roughly the size of the human immunodeficiency virus (HIV). Each post features unique geometry and functions like an optical antenna. Per Princeton, ‘Varying the design of each post is necessary to correctly shape the entire optical wavefront.’
Machine learning-based algorithms turn the light information from each post into an actual image. Further, the image quality surpasses anything other previous ultracompact cameras have been able to achieve. ‘A key innovation in the camera’s creation was the integrated design of the optical surface and the signal processing algorithms that produce the image. This boosted the camera’s performance in natural light conditions, in contrast to previous metasurface cameras that required the pure laser light of a laboratory or other ideal conditions to produce high-quality images,’ said Felix Heide, the study’s senior author and an assistant professor of computer science at Princeton.
‘Our learned, ultrathin meta-optic as shown in (a) is 500 μm in thickness and diameter, allowing for the design of a miniature camera. The manufactured optic is shown in (b). A zoom-in is shown in (c) and nanopost dimensions are shown in (d). Our end-to-end imaging pipeline shown in e is composed of the proposed efficient metasurface image formation model and the feature-based deconvolution algorithm. From the optimizable phase profile, our differentiable model produces spatially varying PSFs, which are then patch-wise convolved with the input image to form the sensor measurement. The sensor reading is then deconvolved using our algorithm to produce the final image. The illustrations above “Meta-Optic” and “Sensor” in (e) were created by the authors using Adobe Illustrator.’ Image and caption credit: Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar & Felix Heide / Princeton University and the University of Washington
Previous micro-sized cameras captured fuzzy, distorted images. The new nano-optics technology produces much crisper, better images with more accurate color and expanded field of’It’s. ‘It’s been a challenge to design and configure these little nano-structures to do what yo’ want,’ said Ethan Tseng, a computer science Ph.D. student at Princeton who co-led the ‘tudy. ‘For this specific task of capturing large field of view RGB images, it was previously unclear how to co-design the millions of nano-structures together with post-processing algo’ithms.’
Co-lead author Shane Colburn, Ph.D. student at the University of Washington Department of Electrical and Computer Engineering, dealt with this problem by creating a computational simulation to automate testing of different nano-antenna configurations. Colburn is now an affiliate assistant professor at the University of Washington.
‘Compared to existing state-of-the-art designs, the proposed neural nano-optic produces high-quality wide FOV reconstructions corrected for aberrations. Example reconstructions are shown for a still life with fruits in (a), a green lizard in (b), and a blue flower in (c). Insets are shown below each row. We compare our reconstructions to ground truth acquisitions using a high-quality, six-element compound refractive optic, and we demonstrate accurate reconstructions even though the volume of our meta-optic is 550,000× lower than that of the compound optic.’
Image and caption credit: Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar & Felix Heide / Princeton University and the University of Washington
Fellow Ph.D. student and coauthor James Whitehead, fabricated the metasurfaces based on silicon nitride. The metasurface design can be mass produced at a lower cost than lenses in a traditional camera, per the study.
The team’s approach itself is not novel. However, combining surface optical technology with neural-based processing is. The micro camera may have significant use in medical settings to enable minimally invasive endoscopy. It can also improve imaging for robots with size and weight constraints. Possibly thousands of the tiny cameras could be placed in an array, turning a surface into a camera.
The study can be read in full here. Its authors include Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar and Felix Heide.