Survey Statistics Overview
The Marine Recreational Information Program (MRIP) strives to make its process for producing catch estimates as clear and transparent as possible. Below, we outline the fundamental mathematical concepts behind survey statistics, including sample sizes, weighting, percent standard error, and the two main sources of error that can occur during a sample survey (sampling error and non-sampling error). More information is available on materials and resources. We also answer questions from our constituents through our e-newsletter. If you would like to be added to the distribution list, please contact us at NMFS.MRIP@noaa.gov.
Overview
Survey design
You must consider many factors when designing a complex survey. Elements of the design can impact the efficacy, budget, and precision of the statistical output. To ensure that all factors are balanced, you can adjust the survey design’s elements stratification, clustering, and sample size.
Sample selection
Once the survey design is complete, you must select a sample that adheres to your design. Ultimately, the goal of sample selection is to obtain a sample representative of the entire population of interest. Having a representative sample will reduce the error, specifically the sampling error, inherent in all estimates derived from sample data.
Data collection
After selecting your sample, it's time to field your survey and begin collecting data.
Estimation
After fielding your survey and cleaning and analyzing the data, the next step is to create statistically valid estimates. The estimation process must consider the survey design to ensure that all units in the sample are represented properly in the final point estimate. To do this, weighting is introduced, and each point estimate has an associated measure of precision called the percent standard error to help gain a better understanding of what we don't know from the sample.
Sampling
A sample survey uses data from a subset of a population (the sample) to estimate characteristics of the whole population.
There are two broad categories of sampling: probability sampling and non-probability sampling. MRIP surveys use probability—or random—samples to estimate population values. In probability sampling, each member of the target population has a known, non-zero probability of being included in the sample. Generally, samples are randomly selected from a comprehensive list of population members, commonly referred to as the sample frame. Different probability sampling techniques, such as simple random sampling, stratification, and cluster sampling, may be used to improve the efficiency and precision of a sampling design, resulting in unbiased samples that are representative of the target population.
In non-probability sampling, the relationship between the sample and the target population is unknown. Consequently, it is not possible to know if a sample is unbiased. Examples of non-probability samples include convenience samples, quota samples, and volunteer or opt-in samples in which the sample members self-select into the survey. Generally, non-probability samples are not used to estimate population values.
Error
All surveys include some amount of error. Survey errors are classified into one of two types: sampling or non-sampling. Collectively, sampling and non-sampling errors determine the accuracy of a survey estimate. Properly designed surveys attempt to minimize both types of errors through careful planning, testing, and analysis. The evaluation of survey errors should be an ongoing process throughout the life cycle of any survey.
Sampling error
A sample does not include all members of a population. Consequently, an estimate based on a sample is likely to differ from the actual population value that would result from a complete census of the population. Sampling error is inherent in all sample statistics because of random variation among samples. The size of the sampling error depends upon the sample size, sample design, and natural variability within the population. Increasing the sample size generally reduces the sampling error.
The most commonly reported measure of sampling error is the “standard error,” which is a measure of the spread of independent sample estimates around a true population value. In MRIP, sampling error is reported as percent standard error, which expresses the standard error as a percentage of an estimate. The lower the PSE, the greater the confidence that the estimate is close to the true population value.
Non-sampling error
Non-sampling errors include any type of error other than the sampling error that can impact an estimate. A non-sampling error that results in a systematic difference between a survey estimate and the “true” population value is commonly referred to as bias. Non-sampling error can arise from insufficient coverage of the target population, inaccurate response or measurement, nonresponse, and data processing errors.
-
Coverage error: Coverage error occurs when members of the target population are omitted, duplicated, or wrongly included in the sample frame. Omissions from the sample frame, or undercoverage, will result in biased estimates if those who are excluded have different characteristics from those who are included. Overcoverage resulting from duplicating or including out-of-scope units can result in bias and sampling inefficiencies.
- Measurement or response error: Measurement error occurs when respondents provide incorrect responses to survey questions. Measurement error can result from poorly worded or ambiguous survey questions, faulty recollection of activities or events (recall error), inconsistent delivery of survey questions by interviewers (interviewer error), or intentional misreporting.
- Nonresponse error: Nonresponse error occurs when individual sample members are unwilling or unable to participate in the survey. This will result in bias if nonrespondents have different characteristics than respondents.
- Data processing error: Data processing errors can occur during preparation of the survey data. Examples include data entry errors, coding errors, and data editing errors.
Sample Sizes
The sample size is the number of units you measure in a sample survey. For example, if you have a bag of 100 black and white marbles and you pull out 10 at random to estimate the number of each color in the bag, your sample size is 10. When producing catch estimates, MRIP samples angler trips from the entire population of saltwater recreational anglers.
In survey statistics, there are two very important things to understand about sample sizes. The first is that the more samples you draw, the more precise your estimate will be. The second is that population size does not determine the precision of your sample. Although this often strikes many people as counterintuitive, your sample size of 10 marbles will give you the same level of precision regardless of whether the bag contains 100,000, 1 million, or 100 million marbles. As long as the population size is larger than the sample size (i.e., you’re using a survey instead of a census), precision is calculated by looking at the difference between the value (or measurement) result of each sample and the point estimate calculated from that sample. The actual formula for calculating precision is more complex (see PSE section <insert link>), but the major takeaway is that a public opinion pollster, for example, can predict the votes of millions of people from a sample size of just hundreds of voters.
Increasing sample sizes often comes with tradeoffs. The more you invest in sampling, the less you have for other science and management activities. As MRIP develops, tests, and certifies improvements to our surveys to make sure they are free of the potential for bias, we are working with our partners and stakeholders to determine the level of sampling necessary to provide the level of precision necessary to meet their science and management needs depending on the location, species, time of year, amount of fishing activity, etc.
Weighting
MRIP uses weighting—a statistical method to make sure each sample unit (fishing trip, measured fish, etc.) is properly represented—when calculating final catch estimates.
For example, if you had a bag of 100 assorted black and white marbles and drew a random sample of 10, you could say that each marble represents 10/100 of what’s in the bag. In statistical terms, each sample has an equal “weight” of 10. Each sampled marble represents 10 marbles, itself plus 9 others not sampled from the bag.
But let’s say you had two bags of 100 marbles. If you drew 10 from bag A and 20 from bag B, you could not simply add up the results of all 30 marbles to calculate an estimate. That’s because the marbles from bag A carry a weight of 10, but each marble from bag B represents 20/100 of the total, for a weight of 5. So it’s twice as likely that one of your samples comes from bag B versus bag A. If you treat them equally, you assume that the contents of the two bags are the same. And as discussed above, any time you make untested assumptions, you’re less likely to identify bias.
In sampling, each one of these bags is called a stratum (i.e., subgroup). To get an accurate estimate, you must calculate the weight of each stratum to account for potential differences among groups.
Along with making sure MRIP’s estimates are accurate, weighting serves another purpose. If our survey design is free of bias, and we know what weight to apply to each sample unit, we can choose to spend more time sampling specific places, times of day, or species that might be important to scientists or managers without skewing our results.
MRIP Guide to Weighting
We strive to be completely transparent about the methods we use to estimate recreational catch, why we use these methods, and how they work. In this presentation, we explain how we use the weighting process to produce accurate estimates and make the most of our limited sampling resources.
Percent Standard Error
All survey estimates include some amount of statistical error and uncertainty. Being able to decipher this error is critical to understanding a catch estimate.
Every MRIP estimate is made up of two parts: the point estimate and the PSE. The point estimate is the estimated fishing effort, or the number of fish caught at a given place over a specified period. When using MRIP queries to examine the data, you will see a number on a table or a point on a graph that indicates the point estimate. Even though it is a specific number, it’s important to remember that this number is an estimate. It is impossible to have 100 percent certainty with any type of sample survey. To indicate how confident we are about a point estimate, we use the PSE.
The PSE is similar to the margin of error that is frequently used in public opinion surveys. It is the measure of an estimate’s precision. The lower the PSE, the greater the precision. Accurately calculating PSEs is important because fully understanding what we don’t know—and how we can better fill gaps in our knowledge—is essential for making prudent, sustainable fisheries management decisions.
We know that the more data we collect, the higher our precision will be. However, there are trade-offs associated with increasing the number of anglers we sample. In an effort to increase the precision of our data, MRIP has funded several projects that are studying ways to increase precision while balancing trade-offs such as data timeliness and accuracy.
Learn more about these projects
