Implementing a Virtual-Physical Environment Manipulation System based on ROS using Python and Three.js

Let's explore the architecture of the Virtual-Physical Environment system. On the frontend, we utilize the Vue framework and Three.js for visualizing our virtual world, encompassing a robotic arm and a 3D environment.

Moving to the middle layer, the backend employs Python to establish a connection between the frontend through WebSockets and the physical world via Rosbridge.

In the third section, which is the local side, we have the robotic arm and IP camera. The robotic arm, based on ROS, receives messages from Rosbridge, while the IP camera covers and monitors the physical environment. The captured data is then sent to the frontend for display on pages using WebRTC.

The demonstrations are presented as follows:

The JavaScript code for receiving ROS messages from both the physical and virtual robotic arms is as follows:

The messages are displayed on the console of the physical robotic arm.

The messages are printed on the console of the client webpages.

The entire interaction flow between the virtual (Unity and Three.js) and physical environments of the robotic arm based on ROS, facilitated by ROS TCP Connector to Unity and ROSBridge to Python, is as follows:

The topic will be presented at the upcoming Conf42.