For many, the first time they realized that the Metaverse would transform the world was when social media giant Facebook changed its name to Meta in 2021. Although initially perceived as just virtual reality glasses that visually simulate the real world, no one expected that the ancient oriental craftsmanship of origami would be the first to achieve a key breakthrough from virtual to real.

At 16:00 London time on May 29, Nature Machine Intelligence published online the latest research results of Prof. Hanqing Jiang's team at Westlake University, "Active Mechanical Haptics with High-Fidelity Perceptions for Immersive Virtual Reality." This was the first time that a high-fidelity active mechanical haptic interaction system was proposed and developed, bringing a new dimension of haptic perception to the Metaverse.

This meana that by drawing on the art of origami, they made the virtual world in VR glasses not only visible and audible, but also “touchable”.

To access the paper, please visit: https://www.nature.com/articles/s42256-023-00671-z

Metaverse: A realm beyond touch

American writer Neal Stephenson coined the word “metaverse” in 1992 in his science fiction novel Snow Crash. It refers to the state where the real world mixes with the virtual world created by augmented reality (AR) technology and virtual reality (VR) technology. The book was named one of the All-Time 100 Novels by Time magazine.

Three decades later, the fantasy world Stephenson created in his book is almost a reality. Controversial or not, the Metaverse has provided our social life with new channels and possibilities. Michael Abrash, chief scientist of Meta Reality Labs, said that VR and AR are setting off the second wave of transformation in the history of personal computers, and that the two technologies would spark the next generation of consumer digital products after PCs and smartphones.

Current VR interaction is limited to visual and audio experiences, where one could see but not touch. However, touch plays such a key role in how people sense the surroundings. The human body receives sensory information through the five senses: sight, sound, smell, taste, and touch. Sight and sound are the main contributors to information reception, with taste and smell playing auxiliary roles, but it is touch that holds a fundamental position (through which we gain a sense of ownership, which is key to distinguishing between subjects and objects).

Researchers have tried many ways to simulate touch, mostly through vibration or pressure compensation to create "passive touch", such as in the case of handles with vibration. John Rogers, an expert in soft electronics at Northwest University, invented a vibratory system that could be attached to human skin. In 2021, Meta also released haptic gloves that were lined with around 15 ridged and inflatable plastic pads that provided a haptic sensory experience when the user grabbed objects.

Yet the haptic senses recreated by such methods are more like the vibration of our phones or the vibration of the seats in 4D cinemas. The vibration first happens on the devices. It is then transferred to the human body, which is far from how human-centered physical perceptions actually feel.

This is in line with the scientific research into haptics. In 2021, American physiologist David Julius and Armenian-American neuroscientist Ardem Patapoutian were awarded the Nobel Prize in Physiology or Medicine for their discoveries of receptors for temperature and touch. Their peers researching sound and sight already won the laurels back in the 1960s.

The origami muse: Adding magic to VR

Jiang wrapped up his 15 years of teaching at Arizona State University in May 2021, and officially joined Westlake University on June 18^th, 2021. One of the first projects he initiated at Westlake was on flexible electronics and soft/hard heterogeneous materials. He was also the one that introduced the concept of “origami and kirigami-based mechanical metamaterials,” artificial structures with mechanical properties defined by their form rather than composition.

Jiang combined his research on materials with the booming prospects of the Metaverse. “Origami material might be soft, but we could make it stiff by folding it. We can also adjust the level of stiffness through robots,” he said.

Jiang also formulated the concept of active mechanical haptics. Unlike the shoulders, chest, waist and back where we usually receive sensory perceptions, the human hands and feet are usually active in touching and sensing the physical environment. His team chose to start with mechanical haptics (stiffness or hardness) and stimulate the feeling when our hands and feet come in contact with objects.

They developed a high-fidelity active mechanical interaction system that incorporated two sets of interactive devices using origami of different textures and sizes. One part of the interactive system was a handheld device that triggered partial senses, while the other was a pedal device that initiated full-body sensing. When using the handheld device, users could experience the level of stiffness or softness of the object they were grabbing; when using the pedal device, users could have a full-body experience of the features of the ground in a virtual environment through active movement.

The passive deformation of the curved origami structure inside the hardware device was actively triggered by the user during the interaction process, creating the active mechanic haptics. With the help of motors, the curved origami can bend at different angles and generate different reaction forces, providing users with varying "elastic" feedback on their hands and feet.

The following image offers a visual representation of the system. The middle part of the device is made up of two thin, folded plastic films interspersed together in an X shape. When you press down on the films, depending on the angle and force, your hand will receive a different bounce in return. The changes are sent to your brain, and your brain would, based on the level of stiffness, decide whether what you are touching is, for example, cotton, wood, or steel. The same logic applies to the feet. Our brain can also deduce if we are walking on the road, grass, or ice, depending on the hardness sensed.

This is how we could stimulate haptic senses in the Metaverse.

A partial structure of the high-fidelity active mechanical interaction system

03 The beauty of mechanics

Jiang has named his lab “Beyond mechanics with societal impact” and plans to dive deep into this intersection. The main team members include postdoctoral fellow Zhuang Zhang, who graduated from the Department of Mechanical Engineering at Shanghai Jiao Tong University; Zhenghao Xu, a visiting scholar from the Department of Mechanics, Zhejiang University; Sentao Chen, a Ph.D. student who did his undergraduate study in the Department of Engineering Mechanics at Tsinghua University; postdoctoral fellow Pingdong Wei, who graduated from the Department of Chemical Engineering at Wuhan University; and Emu Luoqian, a research assistant who won the first prize in ROBOTAC China.

If we say the inspiration of the high-fidelity active mechanical interaction system is origami, then the challenge is how to fold it. The folding needs to transfer active haptics in a limited space. Anyone can fold a paper plane, but only the truly gifted can make it into an art.

Jiang invited us to try the handheld device. It was round, with five points for each of the five fingers. Under each point were the X-shaped interspersed plastic films. The 10 motor-powered interspersed films could be turned into different angles, forming different levels of stiffness. The complementary pedal device was similar – there were X-shaped interspersed steel films beneath it and the curved origami was controlled by strings.

The handheld device

To our surprise, this groundbreaking system was not created with expensive components. All parts, including the plastic films in the handheld device or the steel ones in the pedal, were bought online and could be substituted by other elastic materials. The core component was the curved origami that changed according to the virtual scene. The hardware devices could be made bigger and smaller, as long as the core structure remained consistent. This meant that once there was a suitable application scenario, this system could easily be commercialized.

In the last part of their research, Jiang and his team verified the human sensory representations, including subjective user surveys and objective electromyography and heart rate measurements. The curve in the figure below represents the heart rate changes in the volunteers when they were on the pedal device. The team stimulated the haptic senses of walking on icy ground, with a twist that the ice suddenly collapsed. When that happened, the pedal instantly dropped, leading volunteers to believe that they were falling. This was how they recreated a real haptic sense in the virtual world.

Heart rate and EMG measurements

“Our research needs to be innovative and should make a societal impact.” This is how Jiang sees his work. His research has opened a new realm for VR interaction and provided guidance for the further integration of metamaterials in the metaverse. His team hopes to expand the use of VR in areas such as entertainment, teleoperation, medical diagnosis and treatment, and rehabilitation.

Jiang’s team would continue improving the haptic experience. Currently, they are working to integrate flexible electronics into the system to realize the combination of active and passive haptics. They hope to apply the origami theory to larger-scale experiences and realize interaction in larger scenarios.

“Say that we are playing a motor racing game. Can we ‘grow’ a motorbike that you can touch and feel from the ground? Can we ‘grow’ chairs and boats?” This is how Jiang envisions his future work.

It may be a fantasy today. But as history has proven many times and again that the boundless imagination of scientists may ultimately shape our future.