Several million photos that might contain polar bears; a call to jury duty; and a Microsoft Hackathon.
This series of events led NOAA Fisheries scientists at the Alaska Fisheries Science Center to team up with tech industry talent to develop Artificial Intelligence (AI) for Arctic marine mammal research.
The team is developing AI models to detect ice seals, polar bears, and other marine mammals in images taken during aerial surveys of Arctic sea ice. The AI system will complete image analysis onboard the flight, greatly reducing image storage requirements. It will save scientists years of tedious work, and make Arctic marine mammal population assessments faster and more efficient.
Finding Marine Mammals on Ice
Ice seals—ribbon, ringed, spotted, and bearded are dependent on Arctic sea ice for pupping, resting, and completing their annual molt. Polar bears range over enormous expanses of the ice to hunt these seals.
These species play key roles in the Arctic marine ecosystem and are a vital resource for Alaska Native communities. To effectively manage and conserve these animals, it is crucial to know how many there are, and how their populations change over time.
Collecting this essential information presents scientists with a host of challenges. First, they have to get to some of the most remote places on the planet. Then they fly over the Bering, Chukchi, and Beaufort Seas, searching vast ice fields for widely dispersed, relatively small, often camouflaged animals. The data they collect need to be analyzed as quickly and accurately as possible to inform management decisions.
Scientists have met these challenges with hard work and innovative technology —with some help from a serendipitous meeting.
Erin Moreland, the Alaska Fisheries Science Center Marine Mammal Laboratory scientist who leads the AI project, explains how it evolved:
It started when we shifted from observer-based aerial surveys to cameras. Cameras let us cover a lot more area in a shorter time. We can fly higher and faster since high-resolution cameras can record animal sightings much more quickly than a human observer can find animals looking out an airplane window. These broadly distributed animals can be difficult to find in hundreds of thousands of photos taken from the air. We use thermal cameras to detect the warm bodies of animals on the cold sea ice. Seals look like a bright “hot spot” in the thermal image. By covering the same areas with both infrared and color cameras, we detect animals in thermal images and reference the high-resolution color images to confirm the hot spot is from an animal and identify the species. In one million images, there may be 7,000-9,000 animals. It took us a year to find the animals in the images after the survey season. If images could be processed on board, we would land with a smaller dataset, greatly reducing our post processing workload and timeline.
Marine Mammal Science Meets Tech Industry Expertise
In recent years, Seattle has become the nation’s fastest growing high-tech hotspot. So maybe it is not surprising that while on jury duty, Moreland connected with two experts in Artificial Intelligence. One of the jurors was Microsoft software engineer Peter Baer.
Meanwhile, Moreland’s colleague Josh London was attending a scientific conference when he saw a presentation by Dr. Lucas Joppa, Chief Environmental Officer for Microsoft.
“After some initial discussions, we decided to submit our project for the 2018 Microsoft Hackathon,” Moreland said. “The Hackathon is an opportunity for employees to showcase skills and try new things. All Microsoft employees and interns are free to participate in any of thousands of projects. Our project was in the “AI for Good” category and 14 Microsoft engineers signed up to try to solve our problem.”
“It’s a natural partnership,” London said. “We have all these images we need to analyze, and Microsoft is right here, just across Lake Washington from our NOAA offices. Our large data set gave them an opportunity to work on an important and tangible problem that fit nicely within their AI for Earth project goals.”
AI for Earth is Microsoft’s five-year, $50 million commitment to apply AI to the issue of sustainability through grants, tools, education and training.
Microsoft is helping NOAA develop an algorithm to tackle the first challenge of determining if seals are present in any given image.
“Though Microsoft is a large tech company, we recognize that most of the environmental science expertise lies outside of our four walls,” Joppa said. “We’re committed to putting data and AI to work for the planet by empowering others, such as NOAA’s Alaska Fisheries Science Center, with our latest technology to better monitor, model and ultimately manage our planet’s natural resources,” Joppa said.
University of Washington student, Yuval Boss, is working on technology that is able to identify different seal species. He is a student of software engineer Joe Redmon, renowned for his work developing a faster AI approach to image analysis. He is also an intern with Xnoir.ai, a startup that specializes in energy efficient AI.
“Detecting objects that are indistinguishable to the human eye from an airplane, like seals that can be as small as two pixels, is a huge challenge,” said Ali Farhadi, co-founder of Xnor.ai “The advanced AI technology from xnor.ai enables accurate and complex AI models to run efficiently in real-time on resource constrained compute platforms, at high altitude without any connectivity. In this case, we’re able to help NOAA process critical wildlife data faster, and with greater accuracy so we can make an immediate and lasting impact on these communities.”
Testing AI in the Sky
This spring the team flew with the new AI system in the field. The goal was to test the system on ice seals in Kotzebue Sound and expand the model to include polar bears. The team is now finalizing development of the system.
"Our engineers have been working with NOAA since 2015 to develop automated image analytics to reduce the burden of having to manually browse through hours of video," said Anthony Hoogs, senior director of computer vision at Kitware. "We were able to help Erin and her team by developing an in-flight image acquisition system designed so that the first stage of image analysis could happen during the survey flight. We hope to demonstrate that capability in the upcoming months."
Microsoft and Xnor are continuing to refine the AI algorithms with plans for testing them in the field next year. Our hope is that other animals like walrus and fox are also picked up. “We want to collect as much biological data as we can,” said Moreland. "Surveys of frozen ocean basins are expensive and dangerous, so the more data we can extract from the effort to use and share with other agencies, the better."
The ultimate goal for the future is to miniaturize the AI system to the point where it could fly on uncrewed aircraft systems.
However, Arctic marine mammal research will benefit from the new technology much sooner. “With this system we can do our work much more efficiently,” said Moreland. “We really can do more with less.”
*Thermal camera detection techniques and on-board imaging system were developed with support of U.S. Fish and Wildlife, NOAA’s Uncrewed Aircraft Systems Program, and NOAA’s Office of Science and Technology.
