Data Management Plan
GUID: gov.noaa.nmfs.inport:67299 | Published / External
Data Management Plan
DMP Template v2.0.1 (2015-01-01)Please provide the following information, and submit to the NOAA DM Plan Repository.
Reference to Master DM Plan (if applicable)
As stated in Section IV, Requirement 1.3, DM Plans may be hierarchical. If this DM Plan inherits provisions from a higher-level DM Plan already submitted to the Repository, then this more-specific Plan only needs to provide information that differs from what was provided in the Master DM Plan.
1. General Description of Data to be Managed
Topo-bathy lidar acquisition and processing in the Mariana Islands covering Farallon de Medinilla and Pagan. This product is a classified lidar point cloud data tiles in LAS 1.4 format, delivered in 500m x 500m tiles with FileSourceID set to 0, headers in OGC(2001) WKT, intensity normalized to 16-bit, and linear rescaling. Lidar is clipped to the extent of the area of interest for the topo-bathy data.
Woolpert Inc. (Woolpert) was contracted for a two-part lidar data acquisition and lidar data processing effort in the Commonwealth of the Northern Mariana Islands. Part one required lidar data acquisition, initial data processing, and data coverage verification in the field performed under the United States Geological Survey (USGS). Part two is for the final data processing, derivative lidar products, and QA/QC and is performed under the NOAA Office of Coastal Management (NOAA) Contract.
Woolpert collected lidar using their Hawkeye 4X topo-bathy lidar sensor, to provide high density topographic lidar to meet National Geospatial Program Lidar Base Specification Version 1.3 QL1 standard, while simultaneously acquiring bathymetric lidar data at National Coastal Mapping Strategy 1.0 QL2b standard.
In addition to these lidar point data, the hydro-flattened and topobathy versions of the Digital Elevation Models (DEMs) created from the lidar point data, are also available. These data are available for custom download at the links provided in the URL section of this metadata record.
Notes: Only a maximum of 4000 characters will be included.
Notes: Data collection is considered ongoing if a time frame of type "Continuous" exists.
Notes: All time frames from all extent groups are included.
Notes: All geographic areas from all extent groups are included.
(e.g., digital numeric data, imagery, photographs, video, audio, database, tabular data, etc.)
(e.g., satellite, airplane, unmanned aerial system, radar, weather station, moored buoy, research vessel, autonomous underwater vehicle, animal tagging, manual surveys, enforcement activities, numerical model, etc.)
2. Point of Contact for this Data Management Plan (author or maintainer)
Notes: The name of the Person of the most recent Support Role of type "Metadata Contact" is used. The support role must be in effect.
Notes: The name of the Organization of the most recent Support Role of type "Metadata Contact" is used. This field is required if applicable.
3. Responsible Party for Data Management
Program Managers, or their designee, shall be responsible for assuring the proper management of the data produced by their Program. Please indicate the responsible party below.
Notes: The name of the Person of the most recent Support Role of type "Data Steward" is used. The support role must be in effect.
Programs must identify resources within their own budget for managing the data they produce.
5. Data Lineage and Quality
NOAA has issued Information Quality Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information which it disseminates.
(describe or provide URL of description):
This data was collected by Woolpert, Inc. for the USGS and NOAA Office for Coastal Management (OCM).
- All lidar data were acquired using a HE4X sensor (Figure 4). The HE4X is a latest generation topographic and bathymetric lidar sensor. The system provides denser data than previous traditional bathymetric lidar systems. It is unique in its ability to acquire bathymetric lidar, topographic lidar and 4-band digital camera imagery simultaneously. The HE4X provided up to 500 kHz topographic data and an effective 140 kHz shallow bathymetric data and a 40 kHz deep channel. While not a required deliverable for this survey, 4-band 80 MP digital camera imagery was also collected simultaneously with the sensor's RCD-30 camera and utilized during data editing in some cases. The bathymetric and topographic lasers are independent and do not share an optical chain or receivers, so they are optimized for their specific function. As with any bathymetric lidar, maximum depth penetration is a function of water clarity and seabed reflectivity. The HE4X is designed to penetrate to 3 times the secchi depth. This is also represented as Dmax = 4/K, where K is the diffuse attenuation coefficient, and assuming K is between 0.1 and 0.3, a normal sea state and 15% seabed reflectance. Both the topographic and bathymetric sub-systems use a palmer scanner to produce an elliptical scan pattern of laser points with a degree of incidence ranging from +/-14 degrees (front and back) to +/-20 degrees (sides), providing a 40 degree field of view. This has the benefit of providing multiple look angles on a single pass and helps to eliminate shadowing effects. This can be of particular use in urban areas, where all sides of a building are illuminated, or for bathymetric features such as the sides of narrow water channels, or features on the seafloor such as smaller objects and wrecks. It also assists with penetration in the surf zone where the back scan passes the same ground location a couple of seconds after the front scan, allowing the areas of whitewater to shift.
- Position and orientation data were acquired in the aircraft using a NovAtel SPAN with LCI-100C IMU. All data were post-processed using NovAtel Inertial Explorer software to provide a tightly coupled position and orientation solution. A single base station was used to control trajectory processing providing final trajectories for Saipan and Tinian on NAD83 (MA11), Epoch 2010, located in the Saipan airport. This base station was replaced for each of the three separate collects of the project (Table 11). SPN1, SPN2 and SPN3 were occupied with a Trimble GNSS receiver by Woolpert. Due to the distance of Rota, Aguijan, Farallon de Medinilla, and Pagan from the single base station on Saipan and their remoteness a precise point positioning (PPP) solution was used for them on ITRF2014. To establish a reliable coordinate for SPN1 data were uploaded to the National Geodetic Service (NGS) Online Positioning User Service (OPUS), and for SPN2 and SPN3 Trimble CenterPoint RTX Post-Processing service was used. The average OPUS or RTX coordinate from multiple days of observations was used to process the final trajectories.
- Initial data coverage analysis and quality checks to ensure there were no potential system issues were carried out in the field prior to demobilization of the sensor. Final processing was conducted in Woolpert's offices. In general data were initially processed in Leica's Lidar Survey Studio (LSS) using final processed trajectory information. LAS files from LSS were then imported to a Terrascan project where spatial algorithms were used to remove noise and classify bare earth/ground. Manual review was conducted in both Terrascan and LP360 prior to product creation.
- Lidar processing was conducted using the Leica Lidar Survey Studio (LSS) software. Calibration information, along with processed trajectory information were combined with the raw laser data to create an accurately georeferenced lidar point cloud for the entire survey in LAS v1.4 format. All points from the topographic and bathymetric laser include 16-bit intensity values. During this LSS processing stage, an automatic land/water discrimination is made for the bathymetric waveforms. This allows the bathymetric (green) pulses over water to be automatically refracted for the pulse hitting the water surface and travelling through the water column, producing the correct depth. Another advantage of the automatic land/water discrimination is that it permits calculation of an accurate water surface over smaller areas, allowing simple bathymetric processing of smaller, narrower streams and drainage channels. Sloping water surfaces are also handled correctly. Prior to processing, the hydrographer can adjust waveform sensitivity settings dependent on the environment encountered and enter a value for the refraction index to be used for bathymetry. The index of refraction is an indication of the water type. Values used for sensitivity settings and the index of refraction are included in the LSS processing settings files. A value of 1.34206 was used for the index of refraction, indicating saltwater. In the field, default waveform sensitivity settings were used for processing. In order to determine the optimal waveform sensitivity settings for final processing, sample areas were selected and processed with multiple different settings, to iteratively converge on the best possible settings. This is done by reviewing the processed point cloud and waveforms within sample areas. A sample waveform is provided in Figure 6, while a sample LSS editing screen is provided in Figure 7. Settings affect which waveform peaks are classified as valid seabed, and which peaks are classified as noise. Optimal settings strike a balance between the amount of valid data that is classified as seabed bottom, and the amount of noise that is incorrectly classified due to peaks in the waveforms. Ideally all valid data is selected, while only a small amount of noise remains to be edited out. Once optimal threshold settings were chosen, these were used for the entire project. It is important to note that all digitized waveform peaks are available to be reviewed by the hydrographer; both valid seabed bottom and peaks classed as noise. This allows the hydrographer to review data during TerraScan and LP360 editing for valid data such as objects that may have been misclassified as noise. LSS processing produced LAS files in 1.4 format. Additional QC steps were performed prior to import to TerraScan. Firstly, the derived water surface was reviewed to ensure a water surface was correctly calculated for all bathymetry channels. No significant issues were apparent. Spot checks were also made on the data to ensure the front and back of the scans remained in alignment and no calibration or system issues were apparent prior to further data editing in TerraScan. LSS stores data in multiple LAS files for a single flight line. Each file corresponds to a single .dat file from the raw airborne data. Woolpert merged these multiple files into a single file per flight line and moved data into a standard class definition in preparation for data editing using Woolpert's proprietary scripts within SAFE's FME software. Data produced by LSS for flights over Saipan and Tinian were processed on the NAD83 (MA11) Epoch 2010 datum in UTM 55N Zone with units in meters, and elevations on the ellipsoid also in meters. Data produced for Farallon de Medinilla, Pagan, Aguijan and Rota were processed on the ITRF2014 datum in UTM 55N Zone with units in meters, with elevations on the ellipsoid also in meters.
- After data were processed in LSS and the data integrity reviewed, Aguijan, Rota, Farallon de Medinilla, and Pagan were transformed from the ITRF2014 ellipsoid to the NAD83 (MA11) Epoch 2010 ellipsoid using VDatum. With the entire project now on the correct ellipsoid, data were organized into tiles within a TerraScan project. The tile layout is the same as that provided with the project deliverables. Data classification and spatial algorithms were applied in Terrasolid's TerraScan software. Customized spatial algorithms, such as isolated points and low point filters, were run to remove gross fliers in the topographic and bathymetric data. A grounding algorithm was also run on the topographic data to distinguish between points representing the bare earth, and other valid topo lidar points representing features such as vegetation, buildings, and so forth. Algorithms were run on the entire dataset. Data were reviewed manually to reclassify any valid bathy points incorrectly identified by the automated routines in LSS as invalid, and vice versa. In addition, any topo points over the water were reclassified to correct the ground representation. Manual editing was conducted both in TerraScan and LP360. Steps for manual editing included: - Re-class any topo unclassified laser data and bathy seabed data from the water surface to a water surface class - Review bathymetry in cross section. - Re-class suitable data to Seabed (Class 40). - Re-class any noise in the bathy ground class to bathy noise (Class 45). - Review topo ground points in areas of gaps or spikes. - Add points to ground (Class 2) from the topo laser if points are available to fill gaps in the ground model. - Re-class any noise in the ground class to Topo Unclassified (Class 1) if valid vegetation or other feature, or Noise if the point is not valid (Low Noise (Class 7) or High Noise (Class 18)). - Review topo ground points for bridges and re-class to Bridge Deck (Class 17). - Review bathymetry using imagery and nautical charts and re-class obvious man-made objects to Submerged Object (Class 43). Once editing was completed in TerraScan the islands of Saipan, Tinian, Aguijan, and Rota were vertically transformed to the NMVD03 datum using GEOID12B. Pagan and Farallon de Medinilla were not transformed as they were outside the GEOID12B extents and retained NAD83(MA11) ellipsoid heights.
- Although the bathymetry data includes intensity values, these are raw values. For intensity (reflectance) to correctly represent the reflectance of the seabed, the intensities must be normalized for any losses in signal as the light travels through the water column, so that the intensity value better reflects the intensity of the seabed itself. One of the fundamental issues that exists with reflectance imagery is the variance in return due to water clarity differences occurring spatially along line, and temporally from day to day. This is challenging for any bathymetric lidar sensor. If water clarity is relatively consistent along a line, then it is possible to achieve an overall homogenous reflectance image for an area. To a certain extent, variation in reflectivity intensity can be minimized by limiting the size of flight blocks and trying to ensure similar environmental parameters exist within a single flight block. In other words, where changes in water clarity or environment may be expected, flight blocks should be split to allow different normalization parameters to be used per block for the reflectance processing. Where this is not possible, and water clarity varies significantly along a line, variation in reflective intensity will be seen in the output imagery. While this imagery can still be analyzed and used for manual seabed classification, it prohibits the use of unsupervised, or semi-automated classification. For this survey, cloud shadows (ambient light) had an effect on the resulting reflectance images. Woolpert used proprietary in-house scripts developed in MATLAB to compute project specific correction parameters and normalize the raw intensity data for depth. This provides intensities that more closely represent the reflectance of the actual seabed. Corrected values were used to create 1m reflectance images per flightline using Applied Imager's QT Modeler software. Individual flightline reflectance images were then used in Trimble's OrthoVista software to create a final reflectance image for the entire area. OrthoVista was used to improve radiometric balancing between lines and the seamline editor was used to improve the joins between lines to remove as much line to line edge matching and cloud artifact issues as possible.
- 2022-05-27 00:00:00 - The NOAA Office for Coastal Management (OCM) received the lidar point cloud data in las format from Woolpert, Inc. The data for the islands of Farallon de Medinilla and Pagan were in UTM Zone 55 NAD83(MA11), meters coordinates and ellipsoid elevations in meters. The point classifications were: 1 - Unclassified, 2 - Ground, 7 - Low Noise, 9 - Water, 17 - Bridge Deck, 18 - High Noise, 20 - Ignored Ground, 40 - Bathymetric Point, 41 - Water Surface, 42 - Derived Water Surface, 43 - Submerged Object, 45 - No Bottom At. OCM processed all point classes to the Digital Coast Data Access Viewer (DAV). OCM performed the following processing for Digital Coast storage and provisioning purposes: 1. Internal OCM scripts were run to check the number of points by classification and by flight ID and the gps, elevation, and intensity ranges. 2. Internal OCM scripts were run on the las files to: a. Convert from ellipsoid elevations to WGS84 elevations b. Convert from UTM Zone 55 (NAD83 MA11), meters coordinates to geographic coordinates (ITRF2014) c. Assign the geokeys, to sort the data by gps time and zip the data to database and to the s3 Amazon bucket.
(describe or provide URL of description):
6. Data Documentation
The EDMC Data Documentation Procedural Directive requires that NOAA data be well documented, specifies the use of ISO 19115 and related standards for documentation of new data, and provides links to resources and tools for metadata creation and validation.
- 1.7. Data collection method(s)
- 3.1. Responsible Party for Data Management
- 5.2. Quality control procedures employed
- 7.1.1. If data are not available or has limitations, has a Waiver been filed?
- 7.4. Approximate delay between data collection and dissemination
- 8.3. Approximate delay between data collection and submission to an archive facility
(describe or provide URL of description):
7. Data Access
NAO 212-15 states that access to environmental data may only be restricted when distribution is explicitly limited by law, regulation, policy (such as those applicable to personally identifiable information or protected critical infrastructure information or proprietary trade information) or by security requirements. The EDMC Data Access Procedural Directive contains specific guidance, recommends the use of open-standard, interoperable, non-proprietary web services, provides information about resources and tools to enable data access, and includes a Waiver to be submitted to justify any approach other than full, unrestricted public access.
Notes: The name of the Organization of the most recent Support Role of type "Distributor" is used. The support role must be in effect. This information is not required if an approved access waiver exists for this data.
Notes: This field is required if a Distributor has not been specified.
Notes: All URLs listed in the Distribution Info section will be included. This field is required if applicable.
Data is available online for bulk and custom downloads.
Notes: This field is required if applicable.
8. Data Preservation and Protection
The NOAA Procedure for Scientific Records Appraisal and Archive Approval describes how to identify, appraise and decide what scientific records are to be preserved in a NOAA archive.
(Specify NCEI-MD, NCEI-CO, NCEI-NC, NCEI-MS, World Data Center (WDC) facility, Other, To Be Determined, Unable to Archive, or No Archiving Intended)
Notes: This field is required if archive location is World Data Center or Other.
Notes: This field is required if archive location is To Be Determined, Unable to Archive, or No Archiving Intended.
Notes: Physical Location Organization, City and State are required, or a Location Description is required.
Discuss data back-up, disaster recovery/contingency planning, and off-site data storage relevant to the data collection
Data is backed up to tape and to cloud storage.
9. Additional Line Office or Staff Office Questions
Line and Staff Offices may extend this template by inserting additional questions in this section.