By the end of this year, Google’s Waymo self-driving program plans on expanding its public self-driving ride-sharing program in Arizona, and in the next decade there are about 46 companies working to do the same. Many of these systems don’t yet use thermal imaging, but after a demonstration of how effective it is at identifying objects, I’m not sure I’d get in a self-driving car that didn’t have it.
(Full Disclosure: FLIR wanted me to see how its thermal imaging technology could be easily incorporated into self-driving cars so their PR team showed up at our office with a Chevy Suburban with some cameras mounted to the roof. It was driven by a human but they still got the point across.)
FLIR, which claims to have about a 70 percent market share in thermal imaging technology but is not the only thermal imaging company trying to get in the self-driving game, has already offered automotive applications for the last few years in the form of visual driver assistance systems in high end cars from BMW, Mercedes, Audi and others. In this form, a display in the car uses the thermal imaging as a form of night vision to help spot animals and pedestrians in the road at night.
What FLIR is trying to do now is just apply that same concept to self-driving technologies, except instead of serving as a visual aid for a human driver, the thermal imaging acts as another visual layer for self-driving software to scan in real time and react to the environment around the car.
Here’s a video that FLIR paid The Drive contributor and former Jalopnik columnist Alex Roy to do to explain it all:
In an effort to advertise thermal imaging’s self-driving functionality, FLIR has developed its own visual awareness software and a new camera for demonstration purposes. The way it works is very straightforward.
The camera is no larger than a GoPro, ideally planted somewhere on the top of the car, just behind the windshield in the middle of the width of the roof, which is where a lot of new cars placing cameras anyway. The application I witnessed was very simple, with a FLIR camera and a standard camera mounted side-by-side, connected to a relatively powerful gaming laptop via USB cables. In this application, some of the capabilities of the setup is limited by the computer hardware and the fact that we were using USB connections.
On the demonstration screen inside the car, I saw a side-by-side view of what the standard camera was seeing, and a synced-up feed of what the thermal camera was seeing in black and white, in real time. Since it was dark outside, the thermal image was picking up a remarkable amount more data than the standard camera.
While in our demo, the thermal feed was in black and white—black being cooler areas of the image with white being the warmest areas of the image, and hundreds of shades of gray in between—it can be programed to have temperature ranges register in various colors, like the average range of human body temperature, for example.
Pedestrian figures stand out in the image as moving, glowing white human-shaped figures, with a clear outline that any person watching could instantly recognize, which is a good sign for any computer program trying to do the same. The way the computer analyzes the video feeds works with over 14,000 pre-programmed annotated visuals.
Basically, FLIR has taken recorded footage and gone through and annotated, or marked down, which shapes are cars, which shapes are bicycles, animals, people, etc. and fed that data back into the software so the computer can recognize and track similar-looking objects in real time.
I watched as we drove a couple of blocks around the Jalopnik office in Manhattan. Thanks to Benjamin Franklin or George Hudson or whichever brain-genius came up with daylight savings time, it was already nearly pitch black by 5:30 p.m., and the standard camera was picking up next to zero pedestrians and vehicles, despite street and ambient light.
But the thermal imaging camera was picking up a constant stream of cyclists and people on sidewalks, in crosswalks, jaywalkers, people in the back of box trucks, and even someone’s head poking around a tree.
I also watched as the FLIR camera and annotation software highlighted almost every parked car on the side of the road, every car in front of us and cars passing by in the intersection ahead. The only vehicles it didn’t seem to pick up was a parked mail truck, which was essentially just a giant grey box on screen, and a New York City bus. A FLIR representative told me the system didn’t see the bus because they didn’t have enough data from their test drives in California to have accurately annotated city buses yet. Presumably because California doesn’t believe in public transit, though I’ve been assured in the past that Silicon Valley is going to invent the bus any day now.
It’s important to note that this thermal imaging technology is not meant to replace traditional cameras, radar or LIDAR, the laser imaging system already found on most self-driving cars that does the three-dimensional mapping work. Instead, it’s meant to be a passive layer supplementing the recognition software in the car.
The biggest issue with thermal imaging is the prohibitively expensive cost of thermal imaging cameras, which can be over $1,000 per unit, which may be too much to ask of companies that may find the technology redundant. But the company is confident that it can get the cost of the cameras below $1,000 as more companies incorporate the technology into their self driving cars. The other main issue with the tech is its limited functionality as a backup, additional passive layer to imaging systems.
Thermal imaging doesn’t pick up street light colors, road sign markings, lane markings, and a lot of three-dimensional information is lost in the middle-toned greys of the surroundings. But what matters—moving cars, cyclists, animals and people—stands out remarkably better than a standard video camera. And to my eyes, objects can be more reliably tracked by the software, particularly at night.
FLIR even went through the trouble of re-enacting the fatal self-driving Uber incident, where a Volvo XC90 running Uber’s self-driving systems and hardware struck and killed a jaywalking pedestrian pushing a bicycle across the street at night, while Uber’s employee in the driver’s seat watched television on their phone. In FLIR’s recreation, the thermal imaging software recognized the pedestrian nearly 80 meters away, and around 30 seconds before they were clearly visible to me on the normal camera.
But beyond the self-driving aspects, there’s also something comforting about watching the system recognize obstacles and people on a screen in real time, something companies should keep in mind as they try to convince the masses to hop in the back seat of a driverless car.
When it comes to self-driving cars, however, the more safeguards and redundancy, the better.