After high-profile crimes, authorities sometimes have to sort through mountains of video footage taken by security cameras, TV crews, and even cell phone users near the scene. Today human analysts do most of this painstaking work. But developments in artificial intelligence and computer vision—including facial recognition and the ability to track individuals across multiple cameras—will allow computers to pre-process much of the footage. This will free up investigators to focus on the more complex tasks that only a human can perform.
These new technologies could offer great benefits for fisheries, where the labor-intensive process of collecting data and processing it might someday be aided by computer vision. For instance, many fishing vessels carry observers who collect information on the boat’s retained catch and bycatch—that is, the fish and other animals inadvertently caught and then discarded at sea. By tallying what and how much each boat takes out of the water, observers enable the fleet to ease up as it approaches its annual limit. This feedback loop from fishing boats to managers and researchers is crucial to sustainably managing a fishery.
A Vision for the Future
But human observers are expensive. So, these same people—fishermen, managers and researchers—are asking … could video cameras be brought onboard, with the catch accounting done by computer vision instead? The answer is … maybe. Right now, technologies being developed for military and homeland security applications, as well as by Google and Facebook, might someday make this possible. Several NOAA scientists, alongside industry and academia partners, are also working to transfer those technologies to the fishing industry.
The basic idea is that digital video cameras would record all fish brought onboard and all fish or other animals discarded as bycatch. Then, the computers would identify the species passing in front of the cameras and, in the case of fish, estimate their weight. These two pieces of data would allow computers to tally the catch for each species.
Estimating weight is a relatively straightforward computational problem. A computer can measure length if it knows how far a fish is from the camera, and it can then use standard conversions to get a weight. New technology to track a fish as it moves across the frame or between cameras would allow for multiple estimates that can then be averaged for a more accurate result.
Programming Computers to Recognize Fish
Recognizing different species of fish, on the other hand, is a challenging problem for a computer to solve. “It’s not like having a fish in hand,” said Kresimir Williams, a biologist at NOAA’s Alaska Fisheries Science Center. “A computer can’t just grab the fish and turn it over to get the cues it needs to figure out what species it is.”
But computers are good at capturing what Williams called “discriminating features.” Facial recognition algorithms developed by Google and Facebook rely on the ratio of several features—the distance between the eyes divided by the distance from eye-to-nose, for instance—to identify a human face. In fish recognition, the ratio of eye-to-mouth over eye-to-tail might serve the same purpose.
Williams is working with Jenq-Neng Hwang, a University of Washington computer engineer whose work is funded in part by NOAA, to develop fish recognition algorithms. They have found that six discriminating features are enough to reliably distinguish between several common species of fish.
Of course, it all depends on the species in question. In some fisheries, where there are relatively few and distinctive-looking species, video monitoring might have great potential in the near term. In the case of West Coast rockfish, on the other hand, there are over 60 different species, and telling some of them apart is a challenge even for experienced human observers. For those fisheries, the promise of video monitoring may be a long way out.
Next Steps
There’s also the question of operational flexibility. “Human observers collect a wide range of data,” said Farron Wallace, another NOAA scientist working on video monitoring research. “They collect data on endangered species and marine mammal interactions as well as catch and bycatch.” Computer systems will never be as flexible at problem-solving as humans are. Not for the foreseeable future, anyway.
“Still,” Wallace said, “that doesn’t mean that camera systems can’t be really important in the near future.” As with investigators pouring over footage from a crime scene, the idea is not to replace human analysts, but to give them tools so they can work more efficiently.. Programs that sift through hours of video and then queue up just the key moments might increase efficiency without sacrificing human ingenuity.
“Any software that makes the process more efficient,” Williams said. “That’s the direction we want to go.”
