Recreational Fishing Survey Design Certification

Certified survey designs and estimation methods meet a shared set of standards and undergo independent peer review.

National

Collecting recreational fisheries data and producing recreational catch estimates require continuous improvement. But if data derived from new or improved collection or estimation methods are to support science and management, the methods must be scientifically sound, and the data should be incorporated into the historical time series with minimal disruption. To that end, NOAA Fisheries has established a process for certifying survey and estimation methods as capable of producing the best scientific information available.

Certification means a survey design or estimation method is eligible to receive technical and/or financial support from NOAA Fisheries for implementation and ongoing improvement. For a new survey to be used in the assessment and management of sustainable fish stocks, survey sponsors must work with the Marine Recreational Information Program to determine:

How to incorporate estimates derived from new or improved survey designs into the historical time series.
How to account for differences between the estimates produced by the legacy survey design and those produced by the survey that will replace it.

These policies and procedures are described in the following documents:

Policy Directive 04-114: Implementing Recreational Fishery Catch and Effort Survey Design Changes (PDF, 4 pages)
Procedural Directive 04-114-01: Guidance and Procedures for the Transition Process for Modification of Recreational Fishing Catch and Effort Methods (PDF, 5 pages)
Procedural Directive 04-114-02: Guidance and Procedures for the MRIP Certification Process (PDF, 10 pages)

Since the certification process was established, NOAA Fisheries has certified seven survey designs and estimation methods. Survey sponsors may contact the MRIP Program Management Team for more information.

Certification

New and existing survey designs and estimation methods may be considered for certification. To earn certification, these designs and methods must be documented, adhere to applicable standards, undergo peer review, and receive a recommendation for approval from the MRIP Executive Steering Committee. Final approval by the NOAA Fisheries’ Director of Scientific Programs certifies the survey design or estimation method as documented. The steps of the certification process are described in more detail in Procedural Directive 04-114-02 (PDF, 10 pages).

Documentation

Survey designs and estimation methods must be documented and tested before the certification process begins. Documentation should include in-depth descriptions of: methods; survey manuals; survey instruments; data formats and structures; sample draw and estimation processes and program codes; and an evaluation of pilot test results.

MRIP Standards and Independent Review

Certified survey designs and estimation methods must meet applicable survey and data standards and undergo peer review against the following criteria:

Sample survey designs follow formal probability sampling protocols, with known inclusion probabilities at all sampling stages.
Estimation methods appropriately weight sample data to account for the sampling design. Both point and variance estimates are produced.
Methods are in place to measure and/or correct for potential biases due to under-coverage, non-response, and/or response errors.
The sensitivity of the accuracy of the survey to potential sampling and non-sampling errors is understood, and measures to evaluate, reduce and/or limit that sensitivity are described.
The sensitivity of the survey design to potential implementation errors is documented, and measures to evaluate, reduce, and/or limit that sensitivity are described.
New survey design and/or estimation methods are compared to the design and/or methods they will replace, as well as any other certified survey components currently used to estimate the same population parameters. The relative statistical validity and efficiency of each are described.
The survey design and/or estimation methods are collecting data and/or producing information that meet science and management needs.

Implementation

The decision to certify a survey design is not the same as a decision to implement a survey design or to use the estimates it has produced. The cost and practicality of changing methods must be evaluated, and the fit of the new estimates against science and management needs must be explored.

Transition

Because stock assessments and management actions must be based on a consistent, long-term time series of recreational catch, new surveys cannot be incorporated into the cycle of science and management until differences between old and new estimates are examined and accounted for. For this reason, NOAA Fisheries requires the sponsors of any certified survey design or estimation method to work with the MRIP Transition Team to describe how historical estimates will be placed into the “currency” of the new survey design.

The Transition Team includes representatives from NOAA Fisheries, regional fishery management councils, interstate marine fisheries commissions, and states, and will determine whether a formal plan must be developed and executed before estimates from a new survey design are incorporated into stock assessments, economic analyses, or management decisions. When appropriate, this plan may include time for benchmarking and calibration.

Benchmarking allows new and legacy survey designs and/or estimation methods to be implemented side-by-side. The resulting estimates are examined; consistent differences between them are measured and evaluated; and possible sources of bias that could explain these differences are identified.
Calibration accounts for the sources of variation and bias that may be contributing to consistent differences between old and new survey estimates. It can be used to convert historical estimates to the “currency” of the new survey design, allowing for “apples to apples” comparisons between the two.

The steps of the transition process are described in more detail in Procedural Directive 04-114-01 (PDF, 5 pages).

Accountability

The design and implementation of certified surveys and estimation methods are subject to periodic peer review. In addition, survey sponsors must submit annual reports that describe the certified survey’s data collection procedures, response rates, and resulting estimates. If an annual report indicates a certified survey design is not adhering to applicable standards or is not implemented as certified, a review will be launched to determine:

If the survey design should maintain its certified status.
If the survey should continue to receive funding.
If the statistics derived from the survey data should continue to be eligible for use in federal stock assessments and management actions.

Survey sponsors can request a review of any intended changes to a certified survey design to determine whether these changes are substantive enough to require a new certification request.