Monitoring Today for a Sustainable Tomorrow: Podcast #3

Electronic Monitoring Podcast Ep 3

Electronic Monitoring Podcast - Episode 3

TRT 19:30

Announcer:
Welcome to On the Line a NOAA Fisheries podcast.

Interviewer:
A large part of sustainably managing our nation’s fisheries revolves around data collection. Traditionally, NOAA has relied on a combination of catch data from fishermen, independent observers, and shoreside dealers. But in some fisheries today, it’s feasible and even safer to use electronic monitoring or “EM” to collect data. The technologies range from electronic reporting of fishing trip data by fishermen, to using video cameras and gear sensors to capture information.
Today on On the Line, we’re back for the third and final episode in a series about electronic monitoring with Brett Alger. Brett is NOAA Fisheries National Observer Program’s Electronic Technologies Coordinator. Brett has a Master’s degree in Fisheries Management and Science from Michigan State University and he grew up fishing on the Great Lakes.
Brett tell us, how does a typical commercial fishing trip involved in the Electronic Monitoring Program begin?

Brett Alger:
A lot of fisheries around the country have pre-trip notification requirements whereby the vessel operator or the vessel owner has to notify NOAA Fisheries in advance of sailing and that’s historically been used to deploy observers on fishing boats. Well nowadays you can use those same pre-trip notifications systems to determine exactly which trips you’re going to conduct video review on. But once those are completed, you know, upon arriving at the actual boat to take a fishing trip, there’s typically requirements for the vessel to do some type of diagnostics test on the EM system to make sure everything is working properly and a program is usually going to have a series of business rules if something goes wrong with the system. System outages, camera outages are really rare and so there are some times where the vessel may be required to stay in port until an issue is fixed, but the vast majority of issues that do come up you can resolve those on your own or maybe over the phone with a technician so that vessels can depart and go fishing as they had intended. So, the take-home here is yeah, typically there’s some requirements maybe a day or two before you take your trip and then upon arrival at the boat, vessels are able to go sailing.

Interviewer:
And how does everything work once a crew is out fishing?

Brett Alger:
Every vessel is going to have a unique and tailored vessel monitoring plan that spells out the various requirements. Usually, cameras are required to be turned on once fishing gear is deployed for the first time and typically, most of the time the cameras stay on for the duration of the fishing trip until its conclusion. The vessel monitoring plans also have requirements for maintaining the system during operations, such as keeping the cameras clean, not standing in the way of camera views. But what is required in each of those vessel monitoring plans varies across fisheries, across gear types, across vessels. So it really depends on what the EM is being used for. If the EM program is used more for compliance, the fishing crew may have not as many on deck requirements other than following the catch retention and discard protocols, such as specific positions on the boat where they’re allowed to discard their catch. If the program is also used for collecting finer scale catch data such as species identification information or weight estimates of the catch, then there may be additional requirements for the crew to sort catch, move and handle the catch in camera view to facilitate collecting those finer scale data. And that may mean they move catch across a measuring board or they move the catch into totes in view of certain cameras. So, under what I would say is a good to a really good scenario, the EM system requires minimal maintenance and minimizes the impact to the crew in terms of their on deck protocols.

Interviewer:
And how do things like the environment and weather affect an EM system?

Brett Alger:
Fisherman go out in all kinds of conditions that you would experience on land, but then you add in the factor of high seas and wind and wave action on the ocean and obviously it gets pretty dangerous at times, to be blunt. But even when the seas are calm you can still get sun glare on metal objects or the water. Or during nighttime you can still get a reflection of the deck lighting on different objects. So, there’s no question that external factors beyond the vessel and crew impact the data quality and so it’s important to try and uncover those particular issues and develop mitigation strategies. Sometimes it’s as simple as changing a camera angle or having the captain and crew monitor certain issues with more diligence.

Interviewer:
So it sounds like the crew are an integral part of operating the EM system.

Brett Alger:
Definitely. There’s no question that the crew and the EM system need to operate together in tandem. When those aren’t working well together it can affect the data quality. You might have a blocked camera just because somebody is standing two feet over in one direction rather than the other. So EM programs try to examine the imagery and the data quality relatively quickly so that they can provide feedback to the crew in order to make improvements over time.

Interviewer:
And once a trip ends, how’s that imagery retrieved?

Brett Alger:
Most programs still rely upon transmitting hard drives from the EM system either by staff collecting the hard drives from the vessel or the captain may be required to send them through the mail. This is pretty straightforward and for the most part we don’t see loss of data or loss of hard drives. There are some programs and EM service providers that are starting to explore wireless data transmission via cellular or satellite or Wi-Fi. But there’s still some issues with that currently. File sizes from really long fishing trips certainly they’re really large data files and so wireless options aren’t always available to all ports in all regions at least in a cost-effective and timely way. So, we need to continue working on expanding data transmission options and reducing file sizes and I think artificial intelligence and machine learning will play a large role in shaping that future.

Interviewer:
Setting aside artificial intelligence and machine learning for a moment, how’s the imagery currently reviewed?

Brett Alger:
All of the various data are collected from the hard drive and processed through uniquely developed software that can integrate the time, the date, and location with the imagery and the other sensor data from the EM system. This allows the video reviewer to collect information from start to finish of a trip based on the reviewer protocols and whatever the summary data calls for in that particular program. The reviewer typically gathers species ID information and weight estimates from discards. They may flag certain events or certain issues, and then they’re gonna send some type of compliance report or summary data to managers and scientists. In some programs it may be that the reviewers collect data from all the trips or all the hauls, and in some cases it may be that they just collect data from a portion of the trips or a portion of the hauls.

Interviewer:
In one of our previous episodes you mentioned that video review is one of the more expensive parts of an EM program, what’s being done to reduce those costs?

Brett Alger:
In terms of human video review, programs are exploring ways to review a portion of the hauls or a portion of the trips, and still get the necessary information to meet the data needs of the program. For example, if EM systems are running on all trips and the video is reviewed on all trips, whatever the cost may be, one immediate opportunity to reduce costs is to review video from half the trips rather than all of them, or even far less than that. That may work in some programs if the EM data is validating the vessel reported discards you may get away with reducing video review. It may not work if the EM data is necessary to be gathered on all trips. So long-term, artificial intelligence and other technologies will reduce costs even more.

Interviewer:
Can you tell us more about how artificial intelligence and machine learning plays a part?

Brett Alger:
Almost any EM program or pilot project nowadays is exploring some flavor of AI and machine learning to aid in greatly decreasing the costs and the timeliness of the data while improving the accuracy of the data. In very simple terms, if you can develop algorithms that assist or even replace some of what a human review must do, it can greatly reduce the data transmission, the data review, and the storage costs of your program.

Interviewer:
Can you give some examples of what’s being developed?

Brett Alger:
Sure, some programs are developing AI to determine when fishing gear is going into the water or coming out of the water, or determining if there are crew on deck or not. Both of those applications are a strong indicator of fishery catch being on the deck of the vessel, right? If you identify the gear has come out of the water and there’s fishing crew on deck that’s a pretty obvious sign that you’re gonna be looking for catch to come aboard. And so the lack of catch being on deck or the lack of crew being on deck may allow the human reviewer to process through the video footage much faster and only focus on when there are actual catch events. Building upon those, algorithms are being developed to use AI for species identification, or weight estimation, counting fishing hooks, which starts to semi or almost fully automate the video review process. And then the human reviewer then is used more as a validator helping get data on species maybe that the algorithms are a little more uncertain about.

Interviewer:
So how are those algorithms developed?

Brett Alger:
Well, this is where I start to get outside of my skill set a bit. But this is how I would explain it… If I showed you a hundred pictures of a dog and a hundred pictures of a cat, you would learn how to distinguish between the two, so that when I showed you a brand-new picture, but I didn’t tell you what it was, you’ve learned how to distinguish dogs from cats. Artificial intelligence for electronic monitoring isn’t really that different. We are trying to build image-based datasets to train a computer how to perform certain tasks and eventually the computer or the algorithm is smart enough to make educated guesses about the imagery you are providing it. So whether or not there’s fishing crew on deck or gear in the water or not, those are much simpler, versus if you’re trying to use artificial intelligence to identify a whole range of potential fish species, that’s a lot more complicated. So, using my example from before, distinguishing all species of cats and dogs takes a lot more data to train your algorithms and fish identification is no different. We are trying to build out large image-based datasets across a whole range or variety of different species so that we can train the algorithms and implement them in our Electronic Monitoring Programs.

Interviewer:
And how do you actually train the computer or the algorithm?

Brett Alger:
One fundamental step is annotating or labeling the imagery, and by that I mean, either a human or buy some automated means, you need to label what you see on the screen. If it’s a fishing vessel with catch on board you need to label or draw boxes around the fishing gear, the crew, the catch, the measuring boards, the totes, etc., etc. And if you’re wanting to do species identification, you need to label the individual fish seen on the screen to distinguish each species from one another. A great example of this is actually Facebook. So for years we’ve been tagging pictures of ourselves, our family, our friends, and telling Facebook who is in every picture. And then Facebook has been able to take all those labels and annotations and send their guests back to us. “Is this you Brett? Is this your dad? Is this your sister?” And so over time they’ve gotten better and better and better at automatically identifying everyone because we’ve been annotating all of this imagery for them. Well, with electronic monitoring we of course have a much steeper hill to climb. We have to annotate the data ourselves and build out the image library to teach the artificial intelligence before we can even effectively integrate the AI into the EM data flow process. So we’re starting to see some early wins, but we certainly have a ways to go.

Interviewer:
Can you give us some examples of these annotated image libraries that are being used right now?

Brett Alger:
Within NOAA Fisheries each of our regions has developed annotated image libraries tailored to their specific programs. I think the Alaska Science Center has more than a million labeled images across a bunch of different fish species. I think the Northeast Fisheries Science Center has built a library that’s may be a few hundred thousand images that are labeled. But long-term we’re trying to build a single national image library so that any EM program can push data into the system or pull data out for algorithm development. So for example, an algorithm developed for a long line fishery in the Atlantic to count fishing hooks could be used in the same type of fishery in the Pacific. So we’re hopeful that by bringing all of those different datasets together into a single library we can develop and refine our artificial intelligence tools more rapidly.

Interviewer:
Are there any public image libraries?

Brett Alger:
The one that I’m familiar with the most in Fisheries is called fishnet.ai which was designed specifically for supporting AI development. Its past focus was on tuna fisheries in the Pacific. But I think long term they’re hopeful to gather additional datasets from other fisheries and expand. Of course any datasets that we are able to make publicly available, such as data collected from a research survey, they’d be a natural fit to partner and work with. But outside of that I know that they are efforts in some discrete projects run outside of NOAA to gather data from electronic reporting applications, such as having private recreational anglers take pictures of their catch and send them in so that it can be turned around in the application to help automatically identify fish caught in the fishery. These types of applications are used in other sectors. I use one for identifying flowers and plants. There’s a bunch that exist for birds and wildlife and other different types of applications.

Interviewer:
Yeah I’ve seen those. Well, getting back to EM, you mentioned that it’ll actually help “decrease costs and the timeliness of data, while improving the accuracy…” Can you talk us through how that works?

Brett Alger:
Starting on the vessel itself you might have an algorithm running on the EM system that is looking for crew on deck, so if it sees crew all cameras run and if it doesn’t then maybe it has the ability to temporarily turn off certain cameras. Or maybe if the EM program is looking for discard events only, then the AI running on the vessel can flag those events. Those are examples of AI operating at the edge, meaning while the data is being collected out on the fishing boat. Those are small applications that may reduce the amount of data that is being collected on an individual trip which 1) brings you closer to wireless data transmission, 2) you’ve already potentially reduced how much data needs to be stored, and 3) you’ve already started to pre-process the data before it’s even put in front of a human reviewer.

Interviewer:
Gotcha, and then once the data’s been retrieved and is being reviewed by a person, there’s also savings there, is that right?

Brett Alger:
Exactly. You are using the reviewers time more effectively and the AI may be identifying most species and events and the reviewer is double checking certain things and able to process through a trip much faster. And over time the AI may be able to start improving the accuracy of species identification and the weight estimates beyond, or at least commensurate with, the human reviewer so that the data quality is improved as well.

Interviewer:
So where do you see electronic monitoring in let’s say five or ten years from now?

Brett Alger:
I am a little biased because it’s my role within NOAA Fisheries to develop and expand technologies in fisheries. But I do see a future where EM is a lot more prevalent than it is now. Right now there’s probably just under 600 vessels in the U.S. across maybe fifteen different programs and pilot projects, but as were able to develop AI tools and data transmission costs plummet with the expansion of 5G data and satellite deployments, that will help boost expansion I think. Additionally, I think we’re getting closer in trying to standardize some of our program components. That will allow technology service providers to scale their services and their products across the U.S. and even the world, rather than right now it’s a lot more fragmented what we see today. So I’d say maybe 7 to 800 commercial fishing vessels in the next few years, maybe over a thousand vessels in five years, but then who knows after that?

Interviewer:
And what leads you to estimate those figures?

Brett Alger:
Just seeing and hearing the interest grow. I get contacted by fisherman or different groups that are interested in EM for their fishery, but aren’t necessarily on the current map, if you will. Off the top of my head I think there are four to six fairly prominent fisheries in the U.S. that have expressed interest, want to start a pilot project, etc. In my experience with technology in fisheries is that rarely something gets started and then just die on the vine. I think even setting fisheries aside, anyone can see the role that technology and AI will play in the world’s future. So it’s a matter of time before we see that play out in fisheries whether that’s in three years, or five years, or 10 years, or whatever beyond that. So my estimates are just that, they’re estimates. They’re guesses. There are a lot of other factors that can accelerate or impede growth and that sometimes has nothing to do with the technology itself. But EM will continue to grow no doubt I think, long term.

Interviewer:
So how can listeners get more information?

Brett Alger:
Well people can find more on our programs from our national electronic monitoring website and from there they can read about a range of topics we’ve discussed here but also start to dive into some of the regional programs and learn more about what’s in their backyard. But even if someone doesn’t live on the coast, I encourage them to find out more about fisheries monitoring and ask questions about the seafood they buy at their local grocery store. It may have been caught in a fishery that uses EM. Actually, I would double-down on that. I would say anyone should ask more about where their seafood comes from. Who caught it? Is it from the U.S.? Was it imported? Where was it imported from? It’s important to have an engaged seafood consumer sector and I highly recommend anybody to buy U.S. seafood. They’re the best managed fisheries in the world.

Interviewer:
Well Brett, it’s been great to have you here, thanks for sharing all this information about NOAA’s Electronic Monitoring programs. Any final thoughts?

Brett Alger:
Well I appreciate you taking the time and shining a spotlight on our Electronic Monitoring programs. Hopefully I’ve been able to capture the complexities and the challenges with electronic monitoring, but also the opportunities and the potential growth and the impact that we can hope to see in the future. I certainly want to close by mentioning and thanking the countless people within and outside of NOAA Fisheries that are pushing electronic monitoring forward. It is most certainly a team effort. I encourage anyone listening to reach out to me with any more questions and I’d be happy to discuss or point them in the right direction. So thanks for the conversation Kent.

Interviewer:
You've been listening to the NOAA Fisheries’ podcast On the Line. Join us again next time to hear more stories about ocean life and ocean science. For more information visit NOAA Fisheries at: fisheries.noaa.gov. On the Line is a production of the NOAA Fisheries Office of Communications. Thanks for listening.