Effort Survey Improvements

In its 2006 review of the methods we used to collect recreational fishing data and report recreational catch, the National Research Council recommended fundamental changes to our data collection techniques and acknowledged the limitations of collecting information through random-digit dialing. In response, the Marine Recreational Information Program (MRIP) began to explore ways to improve how we understand and estimate the number of fishing trips taken by shore and private boat anglers in Hawaii and on the Atlantic and Gulf coasts. In 2018, we adopted a new Fishing Effort Survey (FES).

Why We Made the Change

In 1979, the Coastal Household Telephone Survey (CHTS) began to collect data about fishing effort by dialing a random sample of residential households in Hawaii and along the Atlantic and Gulf coasts. While random digit dialing (RDD) was a standard sampling methodology for conducting household surveys, there were known limitations to this approach. As these limitations grew more pronounced over time—due in large part to a decline in the use of landline telephones—the accuracy of the survey’s estimates began to suffer.

Coverage Error

Since the early 2000s, the percentage of adults living in homes with landline telephones has steadily declined. According to estimates from the 2018 National Health Interview Survey, more than half of all American homes don’t have landline phones. Among homes with landline and wireless phones, 42% receive all or almost all of their calls on a mobile device. In other words, 15% of American homes can be considered wireless-mostly, and 70% have effectively excluded themselves from being sampled by landline-based telephone surveys.

From 2000 on, the exclusion of wireless-only households from the pool of households we sampled meant our samples were growing less and less representative of the general population. Landline-only households, for example, report older residents and fewer children, and are more likely to be composed of single women and to report health and mobility problems. These demographic differences are correlated with differences in fishing activity: the landline-only households—the only households that could be reached by the CHTS—are much less likely to report fishing than the general population.

Measurement Error

The CHTS contacted households without prior notice; asked initial respondents a series of screener questions to determine whether anyone in the household fished during the two-month reference wave; and expected initial survey respondents to immediately describe household-level fishing activity without the benefit of memory cues. Most initial respondents were women, who were much less likely to report household-level fishing activity than men. Under this “gatekeeper effect,” the people who were answering our calls did not always accurately report—and in many cases, underreported—their households’ fishing effort.

Pilot Studies and Peer Review

Between 2007 and 2013, MRIP conducted a series of pilot studies to identify a more accurate and efficient way to estimate fishing effort. These pilot studies provided the basis for the design of the FES.

Methods Tested

• A telephone survey that used fishing license information rather than random-digit dialing as its sampling frame.
• A dual-frame telephone survey that used fishing license information and random-digit dialing as its sampling frame.
• A dual-frame mail survey and a dual-frame, mixed-mode telephone and mail survey that sampled anglers from state databases of licensed saltwater anglers and residential address frames maintained by the U.S. Postal Service.
• A mail survey whose design was revised in an effort to address weaknesses identified in prior studies, increase response rates, and eliminate biases resulting from inaccurate matches to the address frame.

Assessing Non-response Bias

In 2013, a non-response follow-up study demonstrated no significant differences in fishing activity between those who initially responded to the FES and those who responded to a follow-up questionnaire. This suggests non-response to the FES is not a significant source of bias. Routine comparisons between preliminary and final estimates of fishing prevalence also show no significant differences between early and late survey responders.

Results

• The FES provides a more representative sample of the population we survey.
• The FES is less susceptible to bias resulting from non-response and non-coverage.
• The FES gives more household members more time to provide complete answers, which is believed to produce more accurate responses to questions about fishing activity.
• The FES is a more efficient method of collecting fishing data and a superior approach for monitoring recreational fishing effort.
• Differences between CHTS and FES estimates can largely be attributed to differences in fishing activity between the households in each survey’s sample frame. These differences in fishing activity are correlated with differences in demographic characteristics, such as age, gender, and number of children at home.

Peer Review

In 2014, Development and Testing of Recreational Fishing Effort Surveys: Testing a Mail Survey Design (PDF, 56 pages) underwent rigorous peer review. Reviewers included staff from NOAA Fisheries’ Office of Science and Technology, three independent experts selected by the American Statistical Association’s Survey Research Methods Section, and five members of an external expert consultant team. Reviewers provided comments on the methods, results, and conclusions described in the report, and concurred with the overall findings and recommendations to implement a single-phase mail survey design.

In 2015, the design of the FES was certified (PDF, 6 pages) as a scientifically sound and suitable replacement for the CHTS.

Transition

To minimize the potential for disruptions to fisheries science and management, a cross-disciplinary Transition Team of state and federal partners, scientists, stock assessors, and managers was established to develop a process for transitioning from the CHTS to the FES. The team’s Atlantic and Gulf Subgroup prepared a transition plan and timeline (PDF, 34 pages).

Benchmarking

From 2015 through 2017, the FES and CHTS were conducted side-by-side. During this period, estimates produced by the CHTS were considered the best available for use in scientific assessments. The FES Transition Progress Report (PDF, 10 pages) describes the results from the first full year of this benchmarking period.

Developing a Calibration Model

Between 2016 and 2017, NOAA Fisheries staff and independent expert consultants worked to develop a calibration model to re-estimate statistics produced by the CHTS, which would soon be discontinued as a legacy survey design.

Calibration is a critical step in transitioning to a new recreational fishing survey. Because our work to foster healthy, productive, and sustainable marine fisheries requires a consistent, long-term time series of recreational catch statistics, calibration must be used to place historical estimates into the “currency” of the new survey design, allowing for “apples to apples” comparisons between the two.

Discontinuing the CHTS and Implementing the FES

On December 31, 2017, the CHTS was discontinued. As of January 1, 2018, the FES is used to produce all federal estimates of fishing effort.

In March 2018, Transition Team co-chairs Dave Van Voorhees and Kelly Denit led a NOAA Central Library Brown Bag seminar about the transition process. In July, they delivered a presentation (PDF, 34 pages) via webinar about the transition process and our work to calibrate historical effort estimates into the FES “currency.”

Re-estimating Historical Recreational Fisheries Statistics

Once our calibration model was peer reviewed and approved, we converted effort estimates dating back to 1981 to the “currency” of the FES. (A similar peer-reviewed process was used to adjust historical catch estimates following the 2013 transition to an improved Access Point Angler Intercept Survey (APAIS) sampling design.) In 2018, federal stock assessments began to incorporate these calibrated recreational fishing statistics.

Technical Workshops and Reports

In 2019, we participated in a multi-day workshop with the South Atlantic Fishery Management Council’s Scientific and Statistical Committee, held to explain the differences between CHTS and FES-produced estimates. Following the workshop, the committee agreed the FES marks an improvement in our work to measure fishing activity and endorsed the use of estimates calibrated to and produced by the FES in assessing stocks.

Similar presentations were delivered to the Atlantic States Marine Fisheries Commission's Management and Science Committee, and to the Mid-Atlantic Fishery Management Council's Scientific and Statistical Committee.

Implications

While estimates derived from the FES are much higher than estimates derived from the CHTS, this does not necessarily mean there are more people fishing or fewer fish to catch. In fact, our research shows this change in effort estimates is not due to a sudden increase in fishing, but to the fact that the FES more accurately measures the amount of fishing taking place.

Because the CHTS underestimated fishing effort, using a calibration model to convert historical effort estimates to the “currency” of the new survey design raised historical estimates across the time series. For stocks assessed to date, this increase in historical catch estimates has generally resulted in a retrospective increase in estimates of fish stock abundance, especially for those fisheries with large recreational components. Learn how recreational catch estimates inform stock assessments, and how stock assessments inform fisheries management.

Continued Improvements

As part of our commitment to continual improvement, pilot studies are planned to explore the:

• Impact of question order on reports of fishing activity.
• Potential for non-response error, which may confirm prior findings that fishing activity does not significantly differ between responding and non-responding households.
• Role of boat registration data, demographic distribution, and other information in improving sampling efficiency.

More Information

• Recreational Fishing Surveys
• FES At-a-Glance
• FES Annual Reports
• FES Outreach and Educational Resources
• MRIP Survey Design and Statistical Methods Manual