Fiddler Crab Eye View Inspires Gwangju Institute of Science and Technology Researchers to Develop Novel Artificial Vision

The researchers develop an amphibious artificial vision system with a panoramic field-of-
view based on the Fiddler crab’s eye structure
Artificial vision systems are implemented in motion sensing, object detection, and self-
driving vehicles. However, they are not suitable for changing external environments and
are limited to a hemispherical field-of-view (FOV). Addressing this issue, researchers from
GIST have now developed a novel artificial vision with 360° FOV that can image both
terrestrial and aquatic environments. The system, modeled after the eye structure of the
fiddler crab, could help realize the all-weather vision and panoramic object detection.
Artificial vision systems find a wide range of applications, including self-driving cars, object
detection, crop monitoring, and smart cameras. Such vision is often inspired by the vision of
biological organisms. For instance, human and insect vision have inspired terrestrial artificial
vision, while fish eyes have led to aquatic artificial vision. While the progress is remarkable,
current artificial visions suffer from some limitations: they are not suitable for imaging both
land and underwater environments, and are limited to a hemispherical (180°) field-of-view
(FOV).
To overcome these issues, a group of researchers from Korea and USA, including Professor
Young Min Song from Gwangju Institute of Science and Technology in Korea, have now
designed a novel artificial vision system with an omnidirectional imaging ability, which can
work in both aquatic and terrestrial environments. Their study was made available online on
12 July 2022 and published in Nature Electronics on 11 July 2022.
“Research in bio-inspired vision often results in a novel development that did not exist
before. This, in turn, enables a deeper understanding of nature and ensure that the
developed imaging device is both structurally and functionally effective,” says Prof. Song,
explaining his motivation behind the study.
The inspiration for the system came from the fiddler crab (Uca arcuata), a semiterrestrial
crab species with amphibious imaging ability and a 360° FOV. These remarkable features
result from the ellipsoidal eye stalk of the fiddler crab’s compound eyes, enabling panoramic
imaging, and flat corneas with a graded refractive index profile, allowing for amphibious
imaging.
Accordingly, the researchers developed a vision system consisting of an array of flat micro-
lenses with a graded refractive index profile that was integrated into a flexible comb-shaped
silicon photodiode array and then mounted onto a spherical structure. The graded refractive
index and the flat surface of the micro-lens were optimized to offset the defocusing effects
due to changes in the external environment. Put simply, light rays traveling in different
mediums (corresponding to different refractive indices) were made to focus at the same
spot.

To test the capabilities of their system, the team performed optical simulations and imaging
demonstrations in air and water. Amphibious imaging was performed by immersing the
device halfway in water. To their delight, the images produced by the system were clear and
free of distortions. The team further showed that the system had a panoramic visual field,
300 o horizontally and 160 o vertically, in both air and water. Additionally, the spherical mount
was only 2 cm in diameter, making the system compact and portable.
“Our vision system could pave the way for 360° omnidirectional cameras with applications in
virtual or augmented reality or an all-weather vision for autonomous vehicles,” speculates
Prof. Song excitedly.
And it might be soon!

***

Reference
Authors: Mincheol Lee 1,2 , Gil Ju Lee 3,4 , Hyuk Jae Jang 3 , Eehyung Joh 1,2 , Hyojin Cho 1,2 , Min Seok Kim 3 ,
Hyun Myung Kim 3 , Kyeong Muk Kang 3 , Joong Hoon Lee 3 , Myungbin Kim 1,2 , Hongwoo Jang 5 , Ji-Eun
Yeo 3 , Frédo Durand 6 , Nanshu Lu 5,7,9,10,11 , Dae-Hyeong Kim 1,2,8 , and Young Min Song 3
Title of original paper: An amphibious artificial vision system with a panoramic visual field
Journal: Nature Electronics
DOI: https://doi.org/10.1038/s41928-022-00789-9
Affiliations:
1 Center for Nanoparticle Research, Institute for Basic Science (IBS)
2 School of Chemical and Biological Engineering, Institute of Chemical Processes, Seoul National
University
3 School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology
4 Department of Electronics Engineering, Pusan National University
5 Texas Materials Institute, University of Texas at Austin
6 Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
7 Department of Aerospace Engineering and Engineering Mechanics, The University of Texas at Austin
8 Department of Materials Science and Engineering, Seoul National University
9 Department of Electrical and Computer Engineering, The University of Texas at Austin
10 Department of Biomedical Engineering, The University of Texas at Austin, Austin
11 Department of Mechanical Engineering, The University of Texas at Austin
About the Gwangju Institute of Science and Technology (GIST)
The Gwangju Institute of Science and Technology (GIST) is a research-oriented university
situated in Gwangju, South Korea. Founded in 1993, GIST has become one of the most
prestigious schools in South Korea. The university aims to create a strong research
environment to spur advancements in science and technology and to promote collaboration
between international and domestic research programs. With its motto of “A Proud Creator
of Future Science and Technology,” GIST has consistently received one of the highest
university rankings in Korea.

Website: http://www.gist.ac.kr/

About the authors
Young Min Song is currently a Professor in the School of Electrical Engineering and
Computer Science at Gwangju Institute of Science and Technology (GIST), Korea. He
received his Ph.D. in Information and Communications from GIST in 2011. From 2011 to
2013, he was a postdoctoral research associate in the Department of Materials Science and
Engineering at the University of Illinois at Urbana-Champaign (UIUC), USA. Over the
decades, he has predominantly focused his efforts on developing bio-inspired optics and
photonics. His group is developing advanced optoelectronic sensors and systems,
multifunctional nanophotonics, and optical healthcare systems.
Dae-Hyeong Kim obtained Ph. D. degree in Materials Science and Engineering from
University of Illinois at Urbana Champaign (UIUC) in 2009. He was a post-doctoral research
associate at UIUC from 2009 to 2011. He joined Seoul National University in 2011 and is
currently a Professor in the School of Chemical and Biological Engineering at Seoul National
University. Since 2017, he has also served as an associate director of Center for Nanoparticle
Research at Institute for Basic Science (IBS). He has been focusing on the research of
nanomaterials and deformable devices and their application to bio-integrated and bio-
inspired electronics.
Mincheol Lee is a post-doctoral research associate at Seoul National University. He received
his Ph. D. degree in the School of Chemical & Biological Engineering from Seoul National
University in 2021. He has been focusing on the research of stretchable electronics based on
ultrathin semiconductor films and their application to wearable, implantable, and bio-
inspired electronics.
Gil Ju Lee is currently serving as an Assistant Professor in the Department of Electronics
Engineering at Pusan National University since 2021. He has intensely focused on the
research of advanced photonics and optics for next-generation optoelectronics and optical
systems. Before coming to Pusan National University, he completed his Postdoctoral
training and doctoral course at Gwangju Institute of Science and Technology (GIST).
Hyuk Jae Jang received his B.S. from Department of Physical and Semiconductor Science at
Dongguk University and is a Ph. D. candidate under the guidance of Prof. Young Min Song in
GIST. His current research interests focus on bio-inspired nano-optics and imaging device
systems.