Search and rescue operations are challenging due to the limited time in which to locate the subject, the hazards imposed on the rescuer, and the difficulties of the non-local distribution of the full rescue team. Team ARM IT is developing a stereoscopic immersive head mounted display to control an in-house engineered twin GoPro camera gimbal system mounted on an unmanned aerial vehicle. This will allow rescuers to manipulate an unmanned aerial vehicle to assist search and rescue missions safely and effectively through telepresence and enhanced situational awareness.
We plan to test our hypotheses by prototyping, testing, and refining individual components of the system through the use of flight simulation software coupled with a game engine with a physics modeled UAV model using ROS and onsite physical testing. Visual object recognition tasks, viz. finding important objects such as people, delivering small packages, such as survival kits, and additional spectral imaging, such as thermal imaging as extensions to the projects that are also being made to enhance the rescuers’ situational awareness and aid in the overall rescue of the subject.
By providing a realistic sense of the UAV environment enhanced with relevant information, our project will reduce the danger to the rescuers and provide cognitively natural situational awareness.
The multidisciplinary team includes a fully licensed remote pilot.
Mission
The purpose of Team ARM IT is to improve search and rescue functions for UAVs through the implementation of mixed reality technologies. By providing a more realistic sense of the UAV environment through stereoscopic video, augmented with data gleaned from the surroundings, we seek to improve the efficiency of the operator for search and rescue functions.
Technical Material Box
As a way to render aid more quickly, a deliverable payload was devised. Such payload was to be mission specific and to provide materials desperately needed to those in dire circumstances. To this end four basic scenarios were envisioned: mountain, flood, forest fire, and earthquake. Each payload was tailored to the respective environment while taking into consideration weight, volume, and cost of each component of the payload.
Gimbal
The key custom device in the project is a 3-axis brushless gimbal designed to carry two cameras in a stereoscopic configuration. Because the gimbal needs to be positioned at exact angles, motors with built in encoders (rotational position feedback) are used. Our gimbal was custom designed, printed, and built by our team, in-house, to specifically fit the needs of our project.
We plan to test our hypotheses by prototyping, testing, and refining individual components of the system through the use of flight simulation software coupled with a game engine with a physics modeled UAV model using ROS and onsite physical testing. Visual object recognition tasks, viz. finding important objects such as people, delivering small packages, such as survival kits, and additional spectral imaging, such as thermal imaging as extensions to the projects that are also being made to enhance the rescuers’ situational awareness and aid in the overall rescue of the subject.
By providing a realistic sense of the UAV environment enhanced with relevant information, our project will reduce the danger to the rescuers and provide cognitively natural situational awareness.
The multidisciplinary team includes a fully licensed remote pilot.
Mission
The purpose of Team ARM IT is to improve search and rescue functions for UAVs through the implementation of mixed reality technologies. By providing a more realistic sense of the UAV environment through stereoscopic video, augmented with data gleaned from the surroundings, we seek to improve the efficiency of the operator for search and rescue functions.
Technical Material Box
As a way to render aid more quickly, a deliverable payload was devised. Such payload was to be mission specific and to provide materials desperately needed to those in dire circumstances. To this end four basic scenarios were envisioned: mountain, flood, forest fire, and earthquake. Each payload was tailored to the respective environment while taking into consideration weight, volume, and cost of each component of the payload.
Gimbal
The key custom device in the project is a 3-axis brushless gimbal designed to carry two cameras in a stereoscopic configuration. Because the gimbal needs to be positioned at exact angles, motors with built in encoders (rotational position feedback) are used. Our gimbal was custom designed, printed, and built by our team, in-house, to specifically fit the needs of our project.
Head Tracking
The movements of the user's head while wearing the Oculus Rift are mapped directly to the movement of the gimbal, allowing for natural head movement and vision in almost all directions. This allows for more intuitive searching by the user as opposed to the limited viewpoint of a stationary camera.
The movements of the user's head while wearing the Oculus Rift are mapped directly to the movement of the gimbal, allowing for natural head movement and vision in almost all directions. This allows for more intuitive searching by the user as opposed to the limited viewpoint of a stationary camera.
Testing
We will be testing our project in several ways, using both native tests of our system and a custom built flight sim, which will be utilized in a volunteer study to determine the effectiveness and ease of use of our system. This study will allow us to make tweaks to our system in terms of intuitiveness and preferences of the user. The flight sim is built by our team specifically to mimic our system and provide the user the optimal and most accurate experience without having to learn to how to pilot or operate the system.
We will be testing our project in several ways, using both native tests of our system and a custom built flight sim, which will be utilized in a volunteer study to determine the effectiveness and ease of use of our system. This study will allow us to make tweaks to our system in terms of intuitiveness and preferences of the user. The flight sim is built by our team specifically to mimic our system and provide the user the optimal and most accurate experience without having to learn to how to pilot or operate the system.
Overlays and Image Recognition
Our system will come with a pre-built overlay on the VR environment designed to aid the user in their search and provide valuable, time sensitive information about the environment. For example, the overlay can inform the user of the weather conditions, the wind speed, and the altitude of the vehicle at the current moment. The overlay will also come with image recognition, with the hope that it will assist the user in picking out hard to find people in the disaster environment.
An Interconnected System
With all of these separate components, we will be able to create a closely interconnected and communicative system with the ability to actively aid and support the user in their search and rescue efforts. Each component plays a part in maintaining the overarching system and its many functions. The ground control station, UAV, and computer processing system all work together to ensure the consistent operation of our system and assist rescuers in their search efforts.
Our system will come with a pre-built overlay on the VR environment designed to aid the user in their search and provide valuable, time sensitive information about the environment. For example, the overlay can inform the user of the weather conditions, the wind speed, and the altitude of the vehicle at the current moment. The overlay will also come with image recognition, with the hope that it will assist the user in picking out hard to find people in the disaster environment.
An Interconnected System
With all of these separate components, we will be able to create a closely interconnected and communicative system with the ability to actively aid and support the user in their search and rescue efforts. Each component plays a part in maintaining the overarching system and its many functions. The ground control station, UAV, and computer processing system all work together to ensure the consistent operation of our system and assist rescuers in their search efforts.