Effort Survey Improvements
In its 2006 review of NOAA Fisheries’ methods for gathering, estimating, and reporting recreational fishing activity, the independent National Research Council recommended fundamental changes to how we conduct our surveys, and how we engage with our partners and stakeholders. Over the past several years, we have been working on ways to improve our fishing effort estimates on the Atlantic and Gulf coasts. This is the information about how many fishing trips people make.Back to top
Why We Made the Change
NOAA Fisheries surveys households in coastal states to estimate recreational fishing effort. This information is coupled with our complementary on-site surveys of angler catch rates to come up with an overall picture of recreational fishing activity.
Since 1979, NOAA had estimated effort on the Atlantic and Gulf coasts through the Coastal Household Telephone Survey. The CHTS used a method called random-digit dialing (RDD) to target households in coastal counties. For years, RDD had been widely accepted as an effective survey method, and focusing on the coastline had been the best way to find saltwater anglers. However, there are also several well-known shortcomings with this approach:
RDD was inefficient at identifying anglers. Many calls went to households where no anglers lived, and we did not contact anglers who lived inland.
With more people abandoning landlines for cell phones, which were not included in the telephone survey, a growing number of potential anglers had become unreachable. Currently, only 45% U.S. households have a landline phone, down from nearly 100 percent as recently as 1998.¹
Response rates, or the number of people who actually pick up the phone and answer the questions, have been declining for all telephone surveys. This is true whether for fishing effort, public opinion polls, or attitudes about a commercial product or service.
In addition, our research suggests that people may not do as well remembering all their fishing activity when asked over the phone.
All of these issues can skew survey results and result in bias. Possible sources of bias include undercoverage (not reaching all anglers), nonresponse (people not answering the survey), and measurement error (inaccurate answers by respondents). MRIP conducted a series of pilot studies to determine the best way to redesign our effort survey to reduce these potential biases.
¹Blumberg, SJ and Luke, JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2017.. National Center for Health Statistics. December 2017Back to top
From 2008 to 2015, MRIP conducted six pilot studies to determine the most accurate and efficient survey to estimate marine recreational fishing effort on the Atlantic and Gulf coasts. A 2012 study conducted in Massachusetts, New York, North Carolina, and Florida compared a mail survey design for estimating recreational shore and private boat fishing effort with the Coastal Household Telephone Survey design that had been in use on the Atlantic Coast and Gulf of Mexico since 1979. MRIP subjected the final report from the pilot project to external peer review in 2014, and certified the new survey design in February 2015 as a suitable replacement for the CHTS. The pilot studies showed that the FES is much less susceptible to potential sources of bias than the CHTS. The mail survey design reaches more anglers, achieves higher response rates, and is less prone to possible recall errors. Survey estimates from the pilot study were 2.6 times higher than CHTS estimates for private boat fishing and 6.1 times higher for shore fishing.
Goals of the Pilot Studies
- Identify a better method than random digit dialing (RDD) to survey anglers.
- Determine the best way to use license and registration information in our surveys.
- Establish the most effective methods for maximizing angler response.
What the Pilot Studies Tested
The following methods were tested (for shore and private boat fishing only):
- Telephone survey using license information only, in place of RDD.
- Telephone survey using RDD and license info together.
- Mixed-mode telephone and mail survey.
- Mail-only survey using license info and U.S. Postal Service database.
Key Results of the Pilot Studies
- In the four states included in the pilot study, mail survey estimates of total effort were 2-6 times higher than Coastal Household Telephone Survey (CHTS) estimates.
- More people responded to the mail surveys than the telephone surveys.
- Estimates using mail surveys can be produced in a timely fashion.
- Licenses and registrations cannot be our only source of contact information because there are too many anglers who are unlicensed, introducing potential bias.
- However, using license and registration lists to identify potential anglers from the samples drawn from the postal address list increases the mail survey efficiency and lowers costs.
The FES allows us to contact more anglers, including anglers who were not covered in the telephone survey, which has had an impact on the estimates. From the results of our extensive series of pilot studies, we knew the mail survey would produce higher estimates of fishing effort. Because of the complex relationship between effort estimates and other data that go into determining fishery health, higher effort estimates alone do not necessarily mean that overfishing has occurred or is occurring.
To develop a transparent, inclusive means of converting to the new methods, MRIP created a cross-disciplinary Transition Team consisting of managers, stock assessors, scientists, and state partners. The charge of the team was to produce a comprehensive strategy to transition from the Coastal Household Telephone Survey (CHTS) to the Fishing Effort Survey (FES). After in-depth discussions, the team developed a three-year plan (PDF, 34 pages) to ensure that potentially significant impacts to the science and management processes were well thought out and accounted for, to the best of our ability, before transitioning to this new methodology for estimating effort.Back to top
One might assume that higher effort numbers would automatically mean that there are fewer fish out there to catch, but this is not necessarily the case. Our studies indicate that the increase in effort estimates is due to the fact that the FES does a better job of estimating fishing activity, not a sudden rise in fishing. In other words, the CHTS under-estimated fishing activity. After three years of side-by-side comparison, we were able to evaluate differences between FES and CHTS estimates and develop a mathematical model to re-estimate past fishing activity. Now we can look back at historical estimates, calibrate the variance, and predict how much higher past fishing activity was than originally estimated. Because the number of fish being caught is an indicator of fishery health, if effort rates were actually higher in the past than we estimated, then it is possible we were underestimating the number of fish in the population to begin with.Back to top