Gestr Hazard Notification System
REAL TIME AI to REVOLUTIONIZE BICYCLE SAFETY
The start to changing how we think about recreational safety
RISING BICYCLE ACCIDENTS
BICYCLE MOUNTED SAFETY
AUDIBLE WARNING ALERTS
Our Client, Augmented Sense Technologies, came to us with a problem; a surging increase of bicycle-related deaths across the country, as can be seen in the graph below. With the rising trend of bicycle deaths, our client saw a need and had the idea of the Gestr Hazard Notification System to solve it. In the long term life of the project, the device will protect a variety of recreational users who have also experienced an increase in fatalities, with the focus on bicycle safety for now.
The Gestr Hazard Notification System (HNS) is a state-of-the-art bicycle safety device unlike any of its kind. The product developed will be a first-to-market device. The next revolution of embedded and IoT devices will involve AI deployed locally without the need for the cloud. The revolution is just beginning and the Gestr HNS is a part of it. A consumer product utilizing AI to protect cyclists has never been created before and will change the way we think about bicycle and recreational safety by demonstrating effective safety results that can greatly impact a rider’s ability to stay safe.
With the Gestr Hazard Notification System attached to the back of a bike, the rider is alerted in ample time to be able to avert and avoid a hazard before it causes any harm to the biker. With this capability, riders will have an augmented ability to take safety into their own hands and not trust the drivers, other bikers, or pedestrians on the road behind them.
The Gestr HNS utilizes Computer Vision, Artificial Intelligence, and Radar to provide a high degree of safety to its users. The device developed provides advanced object tracking, accurate collision prediction, and user alerting capabilities that not only allow the user to avoid a hazard but also convey where the hazard is coming from. The device will drive the future technologies of recreational safety devices.
Live Zoom Chat
Use the link below to join us live from 8:00 – 10:30 a.m. on April 29.
Join from PC, Mac, Linux, iOS or Android: https://mines.zoom.us/j/9852303936
Dial: +1 253 215 8782 (US Toll) or +1 346 248 7799 (US Toll)
Meeting ID: 985 230 3936
- Jonathon McNabb
- Nayef Haidar
- Daniel Parr
- Maxwell Dumler
- Lucas King
- Darrian Schade
- Tony Dobaj
- Augmented Sense Technologies
Client: Tony Dobaj
Project Advisor: Robin Steele
Technical Advisor: Dr. Christopher Coulston
Luxonis Support: Brandon Gilles
Imagine biking on a road and a silent electric vehicle is approaching from behind. You can’t see it or hear it, but a device on your bicycle detects the vehicle and notifies you, without needing to take your eyes off the road in front of you. This capability is realized in the Gestr Hazard Notification System, a cutting-edge device in road safety. This product attaches securely to the back of a bicycle to detect and warn of any approaching hazard that is on a collision course with the rider. One day, we believe this system could be an integral and important part of riding safety, much like a helmet.
The Gestr HNS uses advanced algorithms utilizing machine learning and AI to enhance rider safety. The prototype detects vehicles, people, cyclists, and other potential hazards coming towards the rider from behind, all in real-time. The system uses artificial intelligence to classify a hazard and determine if it is currently (or likely to become) a threat to the rider, based on the object’s identity, velocity, and trajectory. This data can be sent to the user’s smartphone, which will alert the user to the vicinity of the hazard using 3D sound. This prototype has proven to be a functional proof of concept and is easily extended to a product that offers 360-degree protection. The ability to do this in real-time on a localized device that is not connected to the internet is revolutionary, and wouldn’t be possible without cutting-edge technologies like computer vision, machine learning, and smart antennas.
The design approach was unique because no individuals or company have ever accomplished what we were trying to accomplish. The closest existing product implemented radar to show the user the location of objects, but did not have any sort of hazard detection or alerting features. This project aimed to have a much more sophisticated design, with set goals and client expectations as follows:
To help realize the technological capabilities of the project, the team did a deep dive into researching several embedded hardware platforms that could support augmented awareness. These included the Google Coral (AI), Kendryte (AI), SiSpeed (AI), Luxonis (AI+CV), and Texas Instruments (RADAR) hardware options. The team went through several design paradigms such as trade studies, decision matrices, and performance evaluations to determine which device’s capabilities would best fit our design requirements.
After testing all hardware options, the team settled on using the Luxonis (CV+AI) and Texas Instruments (RADAR) hardware. After the research phase was complete the team began brainstorming design solutions and creating a development plan for the rest of the project.
In order to successfully meet all of our clients’ requirements, it was essential to have a good plan in place that outlined development steps, project timeline, broke up the work between team members, and provided a clear and unambiguous testing plan. The team a document called the “gestr Hazard Detection Module Development Test Plan” to help support the development phase. This document includes project phases, GANTT charts, a project work breakdown structure, best design practices, a thorough testing plan, and documentation requirements. The team created this plan and followed it in order to successfully complete the MVP by the end of the spring 2021 semester.
GESTR HNS MINIMUM VIABLE PRODUCT (MVP)
The above image shows the final MVP for the project. The modules that perform Radar, Computer Visions, and AI operations are identified. The MVP is about 3.9 inches in height.
HNS HARDWARE AND SUBSYSTEMS
Radar, Computer Vision, Depth AI, Software
The Texas Instruments AWR1843 radar development board makes use of a smart antenna – a 2D microstrip antenna array that uses digital signal processing (DSP) to optimally steer the antenna beam and process the received data. This particular radar antenna operates in the frequency range of 76 to 81 GHz and has a detection range of about 80 meters. The HNS used the radar data in its range detection, object tracking, trajectory estimation, and speed algorithms.
The computer vision capabilities are supported by the Luxonis development kit. The development kit includes two monochrome cameras and one RGB camera. These cameras allow for stereo vision and frame sampling capabilities.
The development kit allows for quick sampling of frames. With this design, the Gestr HNS can provide 30 FPS video without any delay. With this frame data, the Gestr HNS software is able to run edge detection, depth estimation, and more to help with hazard prevention algorithms.
The embedded artificial intelligence processing is a key capability of our project. In order to achieve real-time detection, the code cannot run in the cloud or depend on an internet connection. Furthermore, to meet the response time requirements of the project, the AI subsystem needs to process data fast and provide neural inference in real time. The team decided to go with the Depth AI software stack to meet these requirements.
Depth AI was created by Luxonis and is an embedded spatial AI platform built around Myriad X – a complete ecosystem of custom hardware, firmware, software, and AI training. It combines neural inference and depth vision all into a single platform.
Currently, Depth AI is the only platform that can interface with the Myriad X. Luckily for the team, the embedded AI platform does an excellent job of this, while minimizing the data processing time.
The team used Depth AI to train neural networks, apply object classification, process depth data, and minimize the false positives.
The Gestr HNS incorporates a custom Bluetooth service to interact with smartphones. When a hazard is detected, a Bluetooth packet is sent from the Gestr HNS device to a mobile device and alerts the user of the hazard. After the phone receives information about the hazard, it will play an audio message out saying “Hazard – Car” or “Hazard – Bike”. This warns the user of a potential collision before it happens and allows them to avoid the hazard.
The software architecture is highly advanced, reliable, and minimizes response time between the first stage of the system (data sampling) to the final stage (hazard reporting). The software planning, design, development, and testing took up a majority of the team’s time over the course of the project.
Getting less than 150 ms for an embedded system with massive amounts of incoming raw data, is a feat the team is extremely proud of. Here is a high-level overview of the software architecture that allowed us to complete this.
To see some of the software subsystems in action, check out the software demos section.
Live Hazard Architecture
HNS SYSTEM ARCHITECTURE
A big part of the Gestr HNS project was systems engineering and bringing all the subsystems together. The following diagram shows the connectivity/wiring diagram for the developed prototype.
The diagram shows the Luxonis raspberry pi demo board as the main hub of the system. On this board has a raspberry pi and Myriad-X integrated together. Connected to this hub is the radar data transfer line and Bluetooth dongle. The power bank distributes power to all devices that need it.
1: RADAR DATA SAMPLING
Problem: The default TI software for receiving the RADAR serial data was written in MATLAB. MATLAB is not feasible for consumer products and is not compatible with Gestrs’ embedded environment.
Solution: The team had to study the MATLAB code to determine how it reads and processes data from the radar. Once this was understood, the team developed Python code to process the radar data
Iterations: The team began RADAR development through testing frameworks and code written in MATLAB. This code read serial data from the RADAR module and graphically displayed the RADAR data to the user, all within a MATLAB GUI interface. After the team verified and evaluated the RADAR results from the MATLAB scripts, the team wrote a similar python script that would read in the serial data into the Gestr software. From there, the team added filtering capabilities into the sampling scripts to support a minimized system response time.
2: RADAR LOSING DATA CONNECTION
Problem: The TI RADAR board has had some intermittent trouble being recognized by a computer. There have been several instances where a computer cannot detect the presence of the TI board when connected via USB.
Solution: The team suspected that overheating may be a contributing factor. The radar draws a lot of current, but does not have a satisfactory heat sink. The current solution was to provide adequate ventilation to the TI board, but future iterations should incorporate a heat sink.
Iterations: The team first began realizing this issue midway through the spring semester. The first iteration involved de-powering the unit to allow it to cool down. As the team began to investigate the issue more, more ventilation was added to the casing to allow for more natural cooling. After adding this, the team no longer saw any issues with connectivity.
3: RASPBERRY PI VIDEO RECORDING
Problem: The Raspberry Pi’s processing power is limited. For testing purposes, the team needed to record massive amounts of data so that scenarios could be played back over and over again. The Pi has trouble handling the data buffering capabilities and the system would crash.
Solution: A long-term solution will be to use a better processor with a real-time operating system. The short-term solution was to develop and test the Python scripts on a laptop, and only run the scripts on the Raspberry Pi when the recording was not necessary.
Iterations: When the team realized there was a need to record data for playback, we first attempted to implement this on the raspberry pi. This quickly proved to be a challenge. In order to achieve this feature, the team implemented multiprocessing algorithms and developed the recording software on a laptop for testing/development.
4: INCORRECT STEREO DEPTH READINGS
Problem: Luxonis distance data was faulty. The Luxonis is advertised as being able to calculate distance to a detected object. When experimenting with this capability, the team discovered that the distance data ranged between 0.5 m and 2 m, regardless of the actual distance.
Solution: The team reached out to Luxonis for support. This was a tricky problem to troubleshoot, but fairly simple to fix. The team had to properly calibrate the stereo cameras and store the calibration files on the device’s EPROM, in order to get accurate distance data.
Iterations: The team first began to see this issue after creating the graphing/visualization capabilities. On the graph, it was obvious that the stereo depth distance was incorrect. After realizing the issue, the team began to perform isolated tests with people and cars at set distances. This gave the team a ground truth to evaluate results. The team reached out to Luxonis, who pointed the team in the right direction and gave guidance on calibration steps. After the device was calibrated, the team re-evaluated the results by comparing visualized results to ground truth measurements.
5: NEURAL NETWORK CLASSIFICATION RESOLUTION
Problem: There are few different types of neural networks that are supported by the Myriad-X chip. The initial model used by the team had terrible classification resolution for vehicles, people, and bikes. After an object was 10m away, the system could no longer identify it.
Solution: The team suspected that a neural network trained with images that had objects farther away may have a better classification range. The team found such a model that was pre-trained by Intel. This model classified objects at 20m-30m and did a much better job of detecting objects.
Iterations: The team noticed these challenges right from the start. The initial demo model included with the Luxonis development kit was trained for 15 different objects including: Airplanes, Cats, Dogs, Goats, Bikes, Cars, etc. Furthermore, the model included was trained on zoom in or close up images of these objects. In our testing, it was clear that as objects got further away from the module, the system would begin to fail to recognize them. To fix this, the team began searching for alternative neural networks that were compatible with the hardware. The team found a model trained by Intel for vehicles, pedestrians, and cyclist that had better distance resolution and was a perfect use case for our project. The team implemented the model into our software, and tested the results. After implementing the model, the detection distance increased by a good margin.
6: SYSTEM RESPONSE TIME
Problem: There is a lot of data going through the Gestr HNS system. This requires a fair amount of compute power. When done incorrectly the system can become out of sync and very lose its real-time processing capabilities.
Solution: The team implemented multiprocessing and parallel sampling in order to accomplish the system response requirement.
Iterations: The system architecture went through many iterations before being able to have a less than 150 ms response time. In the first revision of the software, all data and algorithms were executed consecutively. This caused many issues and slowed down the response time by a ton. In the next phase of software architecture development, the team began to research multiprocessing and data pipelining. After redesigning the software architecture around these concepts, the team implemented multiprocessing into the Gestr HNS software stack. The results were obvious with an immediate speedup in response time. Finally, the team continued to refined various algorithms across the Gestr HNS software stack to continue minimizing response time.
7: GRAPHING AND VISUALIZATION
Problem: Although the final product will use audible alerts rather than visual alerts, a GUI showing data points from the Luxonis and TI sensors was helpful for accurate development and testing. Most graphing libraries tested slowed the system down by 300% or more.
Solution: The team developed a parallel GUI using multiprocessing using a supported python library to show a real-time polar plot of raw and processed data. Being able to visualize data in real-time helped optimize troubleshooting.
Iterations: The first attempt at including a graph to visualize data slowed the system down by a very large amount. The software was unusable. To fix this, the team began researching alternative python supported graphing methods that could handle large amounts of graphing data without any slow down to the system. The team found a solution, and implemented it into the project.
Several steps need to be taken to turn this into a marketable product:
- The hazard-detection algorithms will need to be made more robust to account for all possible edge cases.
- A real-time operating system (RTOS) and more powerful processor will replace the Raspberry Pi that we used, decreasing the response time and allowing for more advanced CV algorithms.
- A helmet-mounted 3D auralization system will need to be designed, to alert the user to hazards without the use of a distracting screen on a mobile device.
- The HNS and helmet system will need to be controlled by a mobile app. The development of such an app has been started, but needs further refinement. This will allow for an interface between the Gestr HNS and the user.
- The different technologies should be integrated into a system with a single PCB, to allow for minimized module size.
- A universal housing unit should be developed that will attach to bicycles, motorcycles, and other outdoor recreational equipment. The team has started this design with an initial CAD drawing of an injection-molded housing, shown in the next section.
It is estimated that the Hazard Notification System will cost between $130-$150, for both the sensor system that attaches to the user’s equipment and the 3D auralization system that attaches to the user’s helmet. This price takes into consideration the estimated cost of production and the price range most stakeholders were willing to spend on such a product.
MOCKUP PRODUCT CASING DESIGN
recordings demonstrating Gestrs’ capabilities in real time
STEREO VISION DEPTH
For short range distance (< 20m), Gestr HNS incorporates stereo cameras to estimate object distance. The heat map generated is based on source code from Luxonis and different colors correspond to varying depths. As the person gets further away, the colors get lighter.
For long range distance (> 20m), Gestr HNS incorporates mmWave Radar to find and track objects behind the bike
CRITICAL PATH TRAJECTORY ESTIMATION
This module uses a potential hazards past movement to predict its future path
LICENSE PLATE DETECTION
Accountability was extremely important to our stakeholders! Automated license plate detection allows for automated license plate recording when the rider is at risk. Video courtesy of Luxonis.
Meet the Team
Hi, I’m Jonathon! I was born and raised in Houston, Tx, and lived there most of my life. After coming to Colorado for a few backpacking trips, I fell in love with the mountains and ended up at Mines. In my free time I enjoy playing video games, playing the Cello, and getting into a fun game of Ultimate Frisbee. I also really enjoy project management and systems engineering design! Currently, I am on track to graduate with a double degree in CS and EE, graduating this December. After that, I hope to start working full-time for an aerospace startup: Lunar Outpost.
Born in Lebanon but raised around the world, seeing all the diversity of culture and technology gave me a passion for mechanical engineering and a desire to work on technology that will affect people. My love for motorcycles has made this project incredibly interesting and fun to work on, as well as grounding it to fully understand its value.
I hail from the icebox of Gunnison, CO where I was known as Wolverine for my envious mutton chops. Since acquiring my first college degrees in math and history, I have shaved my bodacious mutton chops, gotten married, and moved to Golden, where I am finishing a bachelor’s in electrical engineering. After graduation, I will be starting a career as a control systems engineer for water systems.
Born and raised in the Midwest, engineering topics have been my passion since childhood. Whether I was building vending machines, taking apart the family car radio (thanks mom and dad), or dismantling my many bikes – mechanics and the outdoors have piqued my interest. Pursuing a BS in Mechanical Engineering and Minor in Robotics and Intelligent Systems.
I am from Seattle Washington, where as a young kid I rode my bike to school, friends houses and anywhere in between. This project fit right in with my passion for biking and applying my passion for engineering. In my free time I enjoy playing soccer, basketball and spending time with friends and family. I am currently a senior pursuing a BS in electrical engineering and plan to continue my education with a MS in computer science.
Being an avid mountain biker, this project became a passion of mine and I am very satisfied with my experience on this team. I am excited to wrap up this semester and move on to the next stage of my career with the solid base Mines has helped me build.