Fishing Effort Survey Research and Improvements

Pilot Study Evaluating Measurement Error in the FES

"Evaluating Measurement Error in the MRIP Fishing Effort Survey" is one of several studies NOAA Fisheries conducted to evaluate potential sources of bias in the FES as part of our continuous improvement process. This report was published in August 2023. 

Key Pilot Study Findings

• While the sequence of questions in the FES is based on a well-researched and standard survey practice to ask easier questions prior to more challenging questions, findings from our pilot study suggest this may not always be optimal.
• The FES asks respondents to report their fishing activity over a 2-month period and a 12-month period. In the pilot study, we changed the question order and first asked about fishing trips in the previous 12 months.
• Switching the sequence of questions resulted in fewer reporting errors and illogical responses, and effort estimates that were generally 30 to 40 percent lower for shore and private boat modes than estimates produced from the current design. However, results varied by state and fishing mode. 
• Study limitations: the pilot study was conducted over 6 months with a smaller sample size than the full FES sample. 

Next Steps

• We are planning a larger-scale follow-up study to gain a clearer understanding of the differences in effort estimates between the current design and a revised design. It’s important to note that we don’t know if we will get the same results in this follow-up study. 
• The revised design includes changing the order of questions and also increasing the administration of the survey from every two months to monthly. 
• The follow-up study will be conducted over the full course of 2024 with the revised design administered alongside the current design.
• Monthly sampling will produce more frequent estimates, which is a priority of our regional partners, and a shorter respondent recall period may also minimize reporting errors. 
• It’s important to determine through this study the combined impacts on reporting errors of more frequent sampling with changing the order of questions.

Why do you think a revised questionnaire may result in more accurate reporting?

The pilot study’s revised question order resulted in fewer observed illogical responses compared to the current questionnaire. Specifically, respondents were less likely to report more trips for the 2-month period than for the 12-month period when asked to report fishing activity over the 12-month period first. 

In addition, based on anecdotal information from angler interviews, as well as the results of previous pilot studies, we suspect that anglers are eager to report fishing activity, so they may do so at the earliest possible opportunity when questioned, which may lead to reporting trips outside of a given timeframe.

How was the FES question order designed in the current survey?

The current FES was developed over several years through a series of pilot studies and was peer reviewed. During the FES development, we also conducted several rounds of cognitive interviews with anglers to identify possible sources of confusion in the questionnaire. Question order bias occurs when responses to a prior question on a questionnaire affect responses to a subsequent one. In some survey designs, questions may need to be asked in a particular order or grouped in a logical way to minimize confusion for respondents.

Peer review of the FES included three separate reviews: a panel of independent survey statistical experts recruited by the Survey Methods Section of the American Statistical Association; five members of the MRIP independent consultant team who had no involvement with the FES development; and the National Academies of Sciences, Engineering, and Medicine’s 2017 MRIP review committee. All reviews highlighted that the FES is a significant improvement over the former telephone survey for estimating recreational effort.

The sequence of questions in the current FES is based on a well-researched, standard practice to ask easier questions prior to more challenging questions. This strategy generally improves response rates and response accuracy. In the case of the FES, recalling fishing activity that occurred during the most recent two months should be easier in theory than recalling fishing activity that occurred during the previous 12 months.

As with all of the surveys we administer, we have extensively tested the FES for various other common survey biases that can impact data accuracy, such as non-response and adequate coverage of the recreational fishing population.

Should scientists and managers continue to use FES estimates in assessments and decision making while the study is being conducted?

This is a decision to be made by the stock assessment scientists and fisheries managers, as they determine the best use of all available data to inform their decision making. However, we will work closely with our partners to make informed decisions on how to proceed in light of the pilot study findings. Until we have the full-scale study results in hand, data from the FES remains the best and sometimes the only available science for tracking relative year-to-year and long-term effort trends.

This follow-up study is necessary to properly determine combined potential impacts to effort estimates when the survey is administered more frequently and the order of the questions changed.

How will these findings impact stock assessments and fisheries management decisions?

We won’t have all the answers regarding potential impacts to catch and effort estimates or stock assessments until we’ve completed the follow-up study, but we are committed to working openly and transparently with our partners.

We do know that transitioning from administering the FES monthly as opposed to every two months would allow for more timely production of preliminary catch and effort estimates to help inform management decisions.

If the agency does shift to a revised design based on the findings of the follow-up study, while the magnitude of the effort estimates may change, fishing activity patterns over time (increases and decreases in activity) are expected to remain similar. Stock status is relatively consistent when this trend information has not changed.

What are the next steps if the follow-up study aligns with the findings of the initial study?

If this new study supports the question order and one-month wave design change, NOAA Fisheries would initiate a transition to the revised design in collaboration with our state, council and commission partners.

As part of the transition, we would update calibration methods (methods to re-scale historical catch and effort estimates using the former survey design to updated estimates using the revised design) to account for how these changes would impact catch and effort estimates. Following a peer review of the calibration methods, we would re-estimate the historical time series of effort estimates to align with the revised FES design, which would be used to collect recreational fishing effort data moving forward. The revised time series of estimates would be incorporated into stock assessments and fisheries management decisions once impacts of the survey change are evaluated and understood.

The timeline for the calibrated time series of catch and effort estimates becoming available is anticipated no later than the normal release of final 2025 estimates in mid-April 2026. This is contingent upon successful development, implementation, and review of the updated FES calibration model. The earliest a new survey design, if warranted, could be fully implemented would be in 2026.

Why We Made the Change to a Mail Survey From a Telephone Survey

In 1979, the Coastal Household Telephone Survey (CHTS) began to collect data about fishing effort by dialing a random sample of residential households in Hawaii and along the Atlantic and Gulf coasts. While random digit dialing (RDD) was a standard sampling methodology for conducting household surveys, there were known limitations to this approach. As these limitations grew more pronounced over time—due in large part to a decline in the use of landline telephones—the accuracy of the survey’s estimates began to suffer.

In its 2006 review of the methods we used to collect recreational fishing data and report recreational catch, the National Research Council recommended fundamental changes to our data collection techniques and acknowledged the limitations of collecting information through random-digit dialing. In response, the Marine Recreational Information Program (MRIP) began to explore ways to improve how we understand and estimate the number of fishing trips taken by shore and private boat anglers in Hawaii and on the Atlantic and Gulf coasts. In 2018, we adopted a new Fishing Effort Survey (FES).

Coverage Error

Since the early 2000s, the percentage of adults living in homes with landline telephones has steadily declined. According to estimates from the 2018 National Health Interview Survey, more than half of all American homes don’t have landline phones. Among homes with landline and wireless phones, 42% receive all or almost all of their calls on a mobile device. In other words, 15% of American homes can be considered wireless-mostly, and 70% have effectively excluded themselves from being sampled by landline-based telephone surveys.

From 2000 on, the exclusion of wireless-only households from the pool of households we sampled meant our samples were growing less and less representative of the general population. Landline-only households, for example, report older residents and fewer children, and are more likely to be composed of single women and to report health and mobility problems. These demographic differences are correlated with differences in fishing activity: the landline-only households—the only households that could be reached by the CHTS—are much less likely to report fishing than the general population.

Measurement Error

The CHTS contacted households without prior notice; asked initial respondents a series of screener questions to determine whether anyone in the household fished during the two-month reference wave; and expected initial survey respondents to immediately describe household-level fishing activity without the benefit of memory cues. Most initial respondents were women, who were much less likely to report household-level fishing activity than men. Under this “gatekeeper effect,” the people who were answering our calls did not always accurately report—and in many cases, underreported—their households’ fishing effort.

Pilot Studies and Peer Review

Between 2007 and 2013, MRIP conducted a series of pilot studies to identify a more accurate and efficient way to estimate fishing effort. These pilot studies provided the basis for the design of the FES.

Methods Tested

• A telephone survey that used fishing license information rather than random-digit dialing as its sampling frame.
• A dual-frame telephone survey that used fishing license information and random-digit dialing as its sampling frame.
• A dual-frame mail survey and a dual-frame, mixed-mode telephone and mail survey that sampled anglers from state databases of licensed saltwater anglers and residential address frames maintained by the U.S. Postal Service.
• A mail survey whose design was revised in an effort to address weaknesses identified in prior studies, increase response rates, and eliminate biases resulting from inaccurate matches to the address frame.

Assessing Non-response Bias

In 2013, a non-response follow-up study demonstrated no significant differences in fishing activity between those who initially responded to the FES and those who responded to a follow-up questionnaire. This suggests non-response to the FES is not a significant source of bias. Routine comparisons between preliminary and final estimates of fishing prevalence also show no significant differences between early and late survey responders.

Results

• The FES provides a more representative sample of the population we survey.
• The FES is less susceptible to bias resulting from non-response and non-coverage.
• The FES gives more household members more time to provide complete answers, which is believed to produce more accurate responses to questions about fishing activity.
• The FES is a more efficient method of collecting fishing data and a superior approach for monitoring recreational fishing effort.
• Differences between CHTS and FES estimates can largely be attributed to differences in fishing activity between the households in each survey’s sample frame. These differences in fishing activity are correlated with differences in demographic characteristics, such as age, gender, and number of children at home.

Peer Review

In 2014, Development and Testing of Recreational Fishing Effort Surveys: Testing a Mail Survey Design (PDF, 56 pages) underwent rigorous peer review. Reviewers included staff from NOAA Fisheries’ Office of Science and Technology, three independent experts selected by the American Statistical Association’s Survey Research Methods Section, and five members of an external expert consultant team. Reviewers provided comments on the methods, results, and conclusions described in the report, and concurred with the overall findings and recommendations to implement a single-phase mail survey design.

In 2015, the design of the FES was certified (PDF, 6 pages) as a scientifically sound and suitable replacement for the CHTS.

Transition from the Telephone Survey to Mail Survey

To minimize the potential for disruptions to fisheries science and management, a cross-disciplinary Transition Team of state and federal partners, scientists, stock assessors, and managers was established to develop a process for transitioning from the CHTS to the FES. The team’s Atlantic and Gulf Subgroup prepared a transition plan and timeline (PDF, 34 pages).

Benchmarking

From 2015 through 2017, the FES and CHTS were conducted side-by-side. During this period, estimates produced by the CHTS were considered the best available for use in scientific assessments. The FES Transition Progress Report (PDF, 10 pages) describes the results from the first full year of this benchmarking period.

Developing a Calibration Model

Between 2016 and 2017, NOAA Fisheries staff and independent expert consultants worked to develop a calibration model to re-estimate statistics produced by the CHTS, which would soon be discontinued as a legacy survey design.

Calibration is a critical step in transitioning to a new recreational fishing survey. Because our work to foster healthy, productive, and sustainable marine fisheries requires a consistent, long-term time series of recreational catch statistics, calibration must be used to place historical estimates into the scale of the new survey design to allow for meaningful comparisons to be made.

Discontinuing the CHTS and Implementing the FES

On December 31, 2017, the CHTS was discontinued. As of January 1, 2018, the FES is used to produce all federal estimates of fishing effort.

In March 2018, Transition Team co-chairs led a NOAA Central Library Brown Bag seminar about the transition process. In July, they delivered a presentation (PDF, 34 pages) via webinar about the transition process and our work to calibrate historical effort estimates.

Re-estimating Historical Recreational Fisheries Statistics

Once our calibration model was peer reviewed and approved, we converted effort estimates dating back to 1981 to the scale of the FES. (A similar peer-reviewed process was used to adjust historical catch estimates following the 2013 transition to an improved Access Point Angler Intercept Survey (APAIS) sampling design.) In 2018, federal stock assessments began to incorporate these calibrated recreational fishing statistics.

Technical Workshops and Reports

In 2019, we participated in a multi-day workshop with the South Atlantic Fishery Management Council’s Scientific and Statistical Committee, held to explain the differences between CHTS and FES-produced estimates. Following the workshop, the committee agreed the FES marks an improvement in our work to measure fishing activity and endorsed the use of estimates calibrated to and produced by the FES in assessing stocks.

Similar presentations were delivered to the Atlantic States Marine Fisheries Commission's Management and Science Committee, and to the Mid-Atlantic Fishery Management Council's Scientific and Statistical Committee.

Implications from the FES Transition

For stocks assessed to date, this increase in effort from the FES in historical catch estimates has generally resulted in a retrospective increase in estimates of fish stock abundance, especially for those fisheries with large recreational components. Learn how recreational catch estimates inform stock assessments, and how stock assessments inform fisheries management.

More Information

• Recreational Fishing Surveys
• FES At-a-Glance
• FES Annual Reports
• FES Outreach and Educational Resources
• MRIP Survey Design and Statistical Methods Manual