Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

Chinese scientists have developed “magic stickers” that allow you to share your sense of touch with robots.

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

Is it reliable to let a robot do nucleic acid sampling? 

In the past two years, a number of R&D teams have developed systems that allow medical staff to remotely control robotic arms to complete nucleic acid sampling. However, compared with “starting directly”, there is a difficulty in these operations – it is difficult to grasp the importance of starting. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲A new intelligent throat swab sampling robot system jointly launched by Zhong Nanshan, an academician of the Chinese Academy of Engineering and the Shenyang Institute of Automation, Chinese Academy of Sciences

After all, it is not the hands of the medical staff. The medical staff can only use their rich experience to imagine how much force they should control to control the machine to poke the nose and throat of the inspected person. 

But if the robot can synchronously feedback the sense of touch to the medical staff, will the problem be solved? 

On January 14, a research result co-authored by scholars from City University of Hong Kong, Dalian University of Technology, Tsinghua University and University of Electronic Science and Technology of China was published in Science Advances, the top international academic journal. The robot VR system project in the paper solves the above problems. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

The research team has developed a flexible electronic skin patch that can be used as a wireless human-computer interaction system for robotic VR. Just put on this “magic sticker” and humans can share their sense of touch with machines. 

The patch can be as small as non-sensing, or as large as a top, and feedback tactile sensation through vibration strength and frequency. When the user wears the VR device, he can remotely control the robot to complete the fine movements as if he were in person. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲ Demonstration of nucleic acid detection application of electronic skin in intelligent robot VR

Most of the VR immersive games on the market in the past often required users to install bulky and bulky haptic sensing hardware devices such as exoskeleton controls to increase the haptic experience. The new electronic skin is expected to make it easier for users to have real tactile sensations in the virtual world. 

How to realize haptic feedback in robot VR? What difficulties did the research solve? What new applications will there be in the future? 

Recently, Zhixi exclusively interviewed Yu Xinge, the corresponding author of this Science paper and a doctoral supervisor in biomedical engineering at City University of Hong Kong, to understand the behind-the-scenes story of this research, and have a deeper understanding of its technical principles and application prospects. . 

Welfare of this article: With the outbreak of the new crown epidemic and the normalization of epidemic prevention work, robots have shown their talents in epidemic prevention work. The high- quality report ” Research Report on Epidemic Prevention Robot ” is recommended , which can be obtained by replying to the keyword [ Smart Things 263 ] in the chat column of the official account. 

01. Remote control by medical staff ,  robots help you test nucleic acids

The start of the robot VR project will start from the new crown pneumonia epidemic that swept the world in early 2020. 

At that time, the number of confirmed cases of the new coronary pneumonia epidemic continued to rise, the medical resources of medical staff were in short supply, the efficiency of nucleic acid sampling was low, and the infection risk of medical staff was very high. This made Yu Xinger think, can a remote non-contact control sampling robot be developed? 

Nucleic acid sampling robots have many benefits, including reducing medical costs and alleviating infection risks. Zhong Nanshan’s team, Tsinghua University’s Department of Computer Science Professor Sun Fuchun’s team, and researchers from the University of Southern Denmark have all developed an intelligent nucleic acid sampling robot system. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲The fully automatic nasopharyngeal swab sampling robot exhibited at the 2021 Service Trade Fair (the source of the picture is the Global Times)

However, these robots controlled by remote control handle have high requirements on the operation accuracy of medical staff. 

Yu Xinge’s team observed that during the operation, doctors need to judge the operation process through the screen and imagine how strong the feedback is, which often requires doctors with clinical experience to control. 

So, can haptics be shared between doctors and robots, so that robots can synchronize on-site haptic feedback to remote medical staff? 

A research project to apply electronic skin as a wireless human-robot system in robotic VR was thus launched. 

The researchers put forward an idea that as long as the smart electronic skin is “affixed”, when the doctor remotely controls the robot, he can perceive the operation process through vision, hearing and touch in all directions, which is similar to the zero-distance feeling of the surgical process. Lower the bar for certifying doctors for surgical robots. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲Application demonstration of electronic skin in intelligent robot virtual reality

However, to turn the vision into reality, there are still many hurdles to overcome. 

In the progress of the robot VR project, the biggest problem encountered by Yu Xinger’s team is the two-way feedback link between the robot and the human body. This is also the part of the project that takes the longest time. 

There are two core difficulties: First, after the electronic skin captures the movement of the human body, how to accurately transmit the instructions of this movement to the robot? The second is how to accurately feedback the sense of touch obtained by the robot to humans? 

In addition, the robotic VR research process not only involves biomedical engineering, which Yu Singer’s team is good at, but also involves materials, circuits, mechanical optimization, wired feedback paths, software development, and interface optimization. 

Therefore, they cooperated with many outstanding talents from Dalian University of Technology, Tsinghua University and University of Electronic Science and Technology of China, and finally made progress in such a huge application engineering project. 

02. The skin can also be changed into a top  , with vibration mapping haptic feedback

Going back to the research itself, the overall dimensions of the control panel for the new e-skin are 57mm x 39mm x 0.8mm. 

The researchers used a multi-layer stacking layout in the electronic skin, in which a layer of skin-colored elastic silicon (polydimethylsiloxane, PDMS) acts as a soft adhesive for the skin, interconnected by a polyimide (PI) layer. A series of chips and sensors, including resistors, capacitors, Bluetooth modules, microcontroller units (MCU), bridges, and soft sensors and actuators developed by the researchers. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲The composition of CL-HMI electronic skin

The designs in the e-skin are based on well-established mechanical design rules into a filamentous serpentine structure, making the entire system stretchable. Another layer of skin tone elastic silicon (PDMS) is used to encapsulate all functional components. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲Snake-shaped structure design in electronic skin

“The final form of the electronic skin depends on the function we want to achieve, it can be a small patch that is almost insensitive, or it can be a pair of gloves, a top or a onesie,” said Yuchinger. 

Extending to robotic VR, the paper says, the researchers wore four electronic skin-integrated patches to control a 13-degree-of-freedom humanoid robot capable of performing most human movements. 

Equipped with corresponding sensors, the robot can simultaneously experience the pressure from the forearm, upper arm, thigh, side of the thigh, and abdomen. 

During the experiment, the number of bending sensors and tactile actuators can be freely selected according to the specific application, and the VR equipment can be worn at the same time to synchronize the visual information obtained by the robot electronic eye. 

In this way, the robot VR system can be used as an operation platform to remotely control the robot in a wireless operation mode. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲ Simulation experiment of glove-shaped electronic skin manipulating remote control car

Regarding the implementation of haptic feedback, Yuchinger said: “We replace the sense of touch with different vibration intensities and frequencies.” The range of vibrations may range from low-frequency tapping vibrations to states similar to mobile phone vibrations. 

During the study, the researchers established mapping standards for users, who were trained and familiarized with the vibration feedback gradually, but so far, the mapping standards have varied by body part, user, and many other factors. Yu Xinger told Zhixing that they are also exploring techniques to achieve more realistic mapping. 

03. Transmitted over the Internet  , easy response in less than 50 milliseconds

In addition to tactile perception, the wireless transmission function of the robot VR system is equally magical. 

Seven bending sensors and five actuators are installed on this electronic skin. These sensors and actuators can be directly connected to the analog-to-digital converter (ADC) and general purpose input/output (GPI/O) interface of the MCU in the e-skin for multi-channel sensing and actuation. 

The e-skin includes sensors to read information, wireless transmitters to send information, and small vibrating magnets that facilitate tactile feedback. 

The sensors consist of wires placed in a zig-zag pattern that are pulled straighter when the patch is bent, which transmits data about body movements to the robot when the arm is bent and straightened. 

According to the researchers, these sensors can convert human motion into electrical signals that are further processed by the MCU and wirelessly transmitted to the target robot. At the same time, the pressure sensor on the robot detects the contact with the external environment, and controls the vibration intensity of the haptic actuator through the Bluetooth module to provide the user with haptic feedback. 

To further illustrate the performance of the electronic skin’s real-time wireless operation, the researchers measured the cyclic response speed of the electronic skin at a distance of 1-5 meters, including signal sensing, data transmission, signal reception, and braking response. 

The research results show that the factors affecting the wireless transmission distance of the electronic skin are: the thickness of the elastic silicon encapsulation layer on the top skin color, and the complex biological tissues in the human body may absorb electromagnetic radiation. 

In terms of improving the tactile feedback of robot VR, Yuchinger said: “We can now solve the above problems by adjusting the position and structure of the electronic skin, and at the same time, we will cooperate with more professional experts in the field of communication to integrate related technologies and continue to explore robot VR. potential in the field of wireless communication transmission.” 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲Experimental data of wireless transmission distance of electronic skin

In addition to the Bluetooth operation mode, the electronic skin also supports two wireless communication modes, Wi-Fi and Internet. 

For Wi-Fi mode, as long as the operating area is covered by the same Wi-Fi network, the wireless transmission range can be extended to hundreds of meters. In the Internet operation mode, the wireless transmission distance can be extended to any place covered by the Internet, and its response time ranges from 30.2 to 47.8 milliseconds. 

04. Create an offline version of the Metaverse , in addition to listening and watching, you can also touch

In addition to the medical field, haptic feedback can also be applied to social, entertainment, prosthetic sensing, etc., to further enhance the user experience. 

If VR equipment, tactile feedback and existing online video software are combined to extend from hearing and vision to touch, across the screen, users can feel the real feeling of shaking hands and hugs with distant relatives and friends. 

In the “Metaverse” concept that exploded last year, people need to build a virtual avatar of themselves in the virtual world.To make this virtual avatar empathize with humans, it must be inseparable from the evolution of haptic technology. 

As early as 2018, Yu Xinger has been focusing on haptic feedback research for a long time, exploring the combination of VR, robot and electronic skin. Previously, the skin VR progress developed by his team was published in the international academic top journal Nature at the end of 2019. The skin VR system is equipped with near field communication technology (NFC), and people thousands of miles away can wirelessly control the tactile drive of skin VR by touching and sliding the screen. 

In his opinion: “Robot VR can be understood as an offline version of the Metaverse.” 

In the real world, many tasks with high risk factors and extreme environments require robot assistance, which requires a closed-loop human-computer interaction system that integrates tactile feedback and sensing. 

Yu Xinger said that to control a robot in the virtual world, the user only needs to send instructions to the software, while in the real world, there are more interaction links between instructions and hardware, making it more difficult to control the robot. 

Robotic VR systems are mainly aimed at human-computer interaction. Yu Xinge also added that from the perspective of precise perception of robots, their research team has already achieved relevant results. Now the size of the vibration unit in the electronic skin has been successfully reduced to nearly 1/4 of the previous one, and the unit integration per unit area has also been reduced. Greatly improved. 

In terms of entertainment applications, Yu Xinger gave us an example. In the movie “Iron Armored Steel Fist”, robot fighting replaced violent and bloody fighting sports. In the future, using robot VR and skin VR to control robot duels is also a potential application scenario in this field. 

Create an offline version of the Metaverse, robot VR helps to make nucleic acids remotely, and appear on the Science sub-issue

▲ “Iron Armored Steel Fist” man and robot synchronized action clips

Robotic VR can also be used for prosthetic sensing in the health field. 

Special populations who have lost upper or lower limbs are often unable to grasp the strength of their prosthetic limbs through observation alone, and the installation of electronic skins on the ends of their amputations is expected to improve this problem. 

Yu Xinger explained that while the electronic skin controls the motion of the prosthetic limb, it can also feed back information such as the hardness and weight of the object to the electronic skin at the end of the amputation, so that the user can feel the real strength and hardness feedback, so as to control the strength of the limb. size. 

According to Yu Xinge, they are also preparing for start-up companies to explore the implementation of robot VR. Currently, they are more inclined to industrial partners such as medical applications. 

05. Conclusion: a new way of human-computer interaction , or a new outlet for robot applications

The emergence of the closed-loop human-machine interface provides an updated interaction method between the human body and the robot, which further increases the application scenarios of the remote control robot. In addition, the two-way feedback also provides the possibility for the robot to perform more complex tasks. Examples of teleoperated high-degree-of-freedom robots in teleoperation, prosthetic control, and medical applications represent a wide range of potential applications in different fields. 

It is very important to develop remote control technology with both precise tactile sensing and feedback functions. Yuchinger said that they are still exploring the diversity of electronic skin morphology, the accuracy of tactile feedback, and the efficiency of wireless transmission… 

In the future, the Metaverse may be an important area for the application of robotic VR systems. However, Yu Xinger believes that the research of this project is far more than that. The skin, as the largest tactile sense of the human body, has more development potential. However, the basic exploration time in this field is still very short, and basic research is still necessary at this stage. Essential. 

We have become accustomed to hearing and visual perception in our daily life. If tactile perception can be further popularized, will it make the virtual world more and more realistic? We are also looking forward to a future where we can not only hear and see remotely, but also actually feel virtual spaces. 

Posted by:CoinYuppie,Reprinted with attribution to:
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2022-01-28 08:59
Next 2022-01-28 09:01

Related articles