Effort Survey Improvements
In its 2006 review of NOAA Fisheries’ methods for gathering, estimating, and reporting recreational fishing activity, the independent National Research Council recommended fundamental changes to how we conduct our surveys, and how we engage with our partners and stakeholders. Over the past several years, we have been working on ways to improve our fishing effort estimates on the Atlantic and Gulf coasts. This is the information about how many fishing trips people make.
Why We Made the Change
NOAA Fisheries surveys households in coastal states to estimate recreational fishing effort. This information is coupled with our complementary on-site surveys of angler catch rates to come up with an overall picture of recreational fishing activity.
Since 1979, NOAA had estimated effort on the Atlantic and Gulf coasts through the Coastal Household Telephone Survey. The CHTS used a method called random-digit dialing (RDD) to target households in coastal counties. For years, RDD had been widely accepted as an effective survey method, and focusing on the coastline had been the best way to find saltwater anglers. However, there are also several well-known shortcomings with this approach:
RDD was inefficient at identifying anglers. Many calls went to households where no anglers lived, and we did not contact anglers who lived inland.
With more people abandoning landlines for cell phones, which were not included in the telephone survey, a growing number of potential anglers had become unreachable. Currently, only 45% U.S. households have a landline phone, down from nearly 100 percent as recently as 1998.¹
Response rates, or the number of people who actually pick up the phone and answer the questions, have been declining for all telephone surveys. This is true whether for fishing effort, public opinion polls, or attitudes about a commercial product or service.
In addition, our research suggests that people may not do as well remembering all their fishing activity when asked over the phone.
All of these issues can skew survey results and result in bias. Possible sources of bias include undercoverage (not reaching all anglers), nonresponse (people not answering the survey), and measurement error (inaccurate answers by respondents). MRIP conducted a series of pilot studies to determine the best way to redesign our effort survey to reduce these potential biases.
¹Blumberg, SJ and Luke, JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2017.. National Center for Health Statistics. December 2017
From 2008 to 2015, MRIP conducted six pilot studies to determine the most accurate and efficient survey to estimate marine recreational fishing effort on the Atlantic and Gulf coasts. A 2012 study conducted in Massachusetts, New York, North Carolina, and Florida compared a mail survey design for estimating recreational shore and private boat fishing effort with the Coastal Household Telephone Survey design that had been in use on the Atlantic Coast and Gulf of Mexico since 1979. MRIP subjected the final report from the pilot project to external peer review in 2014, and certified the new survey design in February 2015 as a suitable replacement for the CHTS. The pilot studies showed that the FES is much less susceptible to potential sources of bias than the CHTS. The mail survey design reaches more anglers, achieves higher response rates, and is less prone to possible recall errors. Survey estimates from the pilot study were 2.6 times higher than CHTS estimates for private boat fishing and 6.1 times higher for shore fishing.
Goals of the Pilot Studies
- Identify a better method than random digit dialing (RDD) to survey anglers.
- Determine the best way to use license and registration information in our surveys.
- Establish the most effective methods for maximizing angler response.
What the Pilot Studies Tested
The following methods were tested (for shore and private boat fishing only):
- Telephone survey using license information only, in place of RDD.
- Telephone survey using RDD and license info together.
- Mixed-mode telephone and mail survey.
- Mail-only survey using license info and U.S. Postal Service database.
Key Results of the Pilot Studies
- In the four states included in the pilot study, mail survey estimates of total effort were 2-6 times higher than Coastal Household Telephone Survey (CHTS) estimates.
- More people responded to the mail surveys than the telephone surveys.
- Estimates using mail surveys can be produced in a timely fashion.
- Licenses and registrations cannot be our only source of contact information because there are too many anglers who are unlicensed, introducing potential bias.
- However, using license and registration lists to identify potential anglers from the samples drawn from the postal address list increases the mail survey efficiency and lowers costs.
The FES allows us to contact more anglers, including anglers who were not covered in the telephone survey, which has had an impact on the estimates. From the results of our extensive series of pilot studies, we knew the mail survey would produce higher estimates of fishing effort. Because of the complex relationship between effort estimates and other data that go into determining fishery health, higher effort estimates alone do not necessarily mean that overfishing has occurred or is occurring.
To develop a transparent, inclusive means of converting to the new methods, MRIP created a cross-disciplinary Transition Team consisting of managers, stock assessors, scientists, and state partners. The charge of the team was to produce a comprehensive strategy to transition from the Coastal Household Telephone Survey (CHTS) to the Fishing Effort Survey (FES). After in-depth discussions, the team developed a three-year plan (PDF, 34 pages) to ensure that potentially significant impacts to the science and management processes were well thought out and accounted for, to the best of our ability, before transitioning to this new methodology for estimating effort.
Transition and Implementation Timeline
- Planning (2014): MRIP established a Transition Team consisting of members from states, councils, commissions, and NOAA Fisheries. The Atlantic and Gulf Subgroup of the Transition Team developed a plan for the transition from the telephone survey design, or Coastal Household Telephone Survey (CHTS), to the mail survey design, or Fishing Effort Survey (FES).
- Benchmarking (2015-2017): The mail survey design was conducted side-by-side with the telephone survey design for three years. This allowed for measurement and evaluation of consistent differences in the statistical estimates produced from each.
- Development of a calibration model (2016-2017): MRIP staff and expert consultants evaluated consistent differences between estimates produced by the new mail design and the legacy telephone design to determine possible sources of bias in the legacy design that could explain those differences. Experts from NOAA Fisheries used this information to develop a calibration model for re-estimating recreational catch statistics.
- Discontinuation of the CHTS and full implementation of the FES (2018): After the three-year benchmarking period, NOAA Fisheries discontinued the use of the telephone survey. All future estimates will be based on the new mail-based FES.
- Re-estimation of historical catches (2018): After the final 2017 estimates were available and a calibration model was peer reviewed and approved, revised catch and effort statistics for the entire time series (1981-2017) were produced. The calibration was based on the combined FES/CHTS benchmarking comparison and the revised APAIS adjustment outputs.
- Incorporation of revised statistics into stock assessments (2018-2020): The revised catch statistics will be incorporated into key stock assessments as soon as possible. It will take several years to update all managed stocks with new data. Managed stocks were ranked and assessments prioritized with input and guidance from the Atlantic and Gulf Subgroup.
- Incorporation of revised statistics and new ACLs into management actions (2018-2021): Once both revised catch statistics and new assessment results become available, management actions will begin to use both for decision-making as soon as possible.
Calibration Model Development and Review
Calibration is a critical step in the transition to a new survey design. When survey designs or sampling methods are changed, the new methods may produce results that consistently differ from those derived from the replaced methods. Because stock assessments and fisheries management depend on comparable time series of recreational catch statistics, historical catch statistics produced from a previous design need to be converted to what they would have been had the new survey method been used all along (i.e., into the same “currency” as those produced from the new method). Calibrating the data is the technical process that converts legacy estimates and allows scientists and fishery managers to make apples-to-apples comparisons. It also provides a framework that decision-makers can use for integrating new catch statistics into science and management activities.
A team of NOAA Fisheries staff and consultants created a calibration model to re-estimate the fishing effort statistics back to 1981 from the CHTS “currency” to FES “currency.” The model accounts for the change in survey methods and the shift from landline telephone use to cell phone-only households. The model was peer reviewed and accepted by a panel of independent experts.
We completed a similar process to adjust historical catch rate estimates produced by the Access Point Angler Intercept Survey, the shoreside survey conducted by the states that collects information on angler catch from Maine to Mississippi. This adjustment accounted for any effects of the 2013 change to an improved sampling design for the intercept survey. The approach was peer reviewed and accepted by a panel of independent experts.
One might assume that higher effort numbers would automatically mean that there are fewer fish out there to catch, but this is not necessarily the case. Our studies indicate that the increase in effort estimates is due to the fact that the FES does a better job of estimating fishing activity, not a sudden rise in fishing. In other words, the CHTS under-estimated fishing activity. After three years of side-by-side comparison, we were able to evaluate differences between FES and CHTS estimates and develop a mathematical model to re-estimate past fishing activity. Now we can look back at historical estimates, calibrate the variance, and predict how much higher past fishing activity was than originally estimated. Because the number of fish being caught is an indicator of fishery health, if effort rates were actually higher in the past than we estimated, then it is possible we were underestimating the number of fish in the population to begin with.
Infographic: FES Stakeholder Guide (PDF, 4 pages)
Webinar Slides: Transition to Improved Survey Designs (PDF, 34 pages)