Data Management Plan
GUID: gov.noaa.nmfs.inport:64809 | Published / External
Data Management Plan
DMP Template v2.0.1 (2015-01-01)Please provide the following information, and submit to the NOAA DM Plan Repository.
Reference to Master DM Plan (if applicable)
As stated in Section IV, Requirement 1.3, DM Plans may be hierarchical. If this DM Plan inherits provisions from a higher-level DM Plan already submitted to the Repository, then this more-specific Plan only needs to provide information that differs from what was provided in the Master DM Plan.
1. General Description of Data to be Managed
Woolpert, Inc. was contracted to acquire and process topographic-bathymetric lidar for the islands of Anatahan, Alamagan, Guguan, and Sarigan in response to Hurricane Yutu for Quantum Spatial, Inc. (QSI). Woolpert collected lidar using their Leica HawkEye 4X (HE4X) topo-bathy lidar sensor that consists of a Chiroptera 4X (CH4X) sensor, with an additional Leica 40kHz deep bathymetric channel to provide high density topo lidar. The HE4X is a latest generation topographic and bathymetric lidar sensor. The system provides denser data than previous traditional bathymetric lidar systems. It is unique in its ability to acquire bathymetric lidar, topographic lidar and 4-band digital camera imagery simultaneously. The HE4X provided 300 kHz topographic data, an effective 140 kHz shallow bathymetric data and 40 kHz deep bathymetric data. 4-band 80 MP digital camera imagery was also collected simultaneously with the sensor’s RCD-30 camera.
The bathymetric and topographic lasers are independent and do not share an optical chain or receivers, so they are optimized for their specific function. As with any bathymetric lidar, maximum depth penetration is a function of water clarity and seabed reflectivity. The HE4X is designed to penetrate to 3 times the secchi depth. This is also represented as Dmax = 4/K, where K is the diffuse attenuation coefficient, and assuming K is between 0.1 and 0.3, a normal sea state and 15% seabed reflectance.
Both the topographic and bathymetric sub-systems use a palmer scanner to produce an elliptical scan pattern of laser points with a degree of incidence ranging from +/-14 degrees (front and back) to +/-20 degrees (sides), providing a 40 degrees field of view. This has the benefit of providing multiple look angles on a single pass and helps to eliminate shadowing effects. This can be of particular use in urban areas, where all sides of a building are illuminated, or for bathymetric features such as the sides of narrow water channels or features on the seafloor such as smaller objects and wrecks. It also assists with penetration in the surf zone where the back scan passes the same ground location a couple of seconds after the front scan, allowing the areas of whitewater to shift.
All topo lidar data for this project were collected simultaneous to meet United States Geological Survey, Quality Level 1 (USGS QL1) with a minimum of 8 pts per square meter at an accuracy of 10cm RMSEz. A minimum of 2 points per square meter were acquired for bathymetric lidar data. For practical purposes the survey area was divided into survey blocks in each island, allowing acquisition to be conducted in the most efficient and consistent manner possible.
The data includes topobathy data in an LAS 1.4 format file along with associated bare earth digital elevation models (DEM). The dataset was derived from topobathymetric data in a LAS format 1.4, point data record format 6, with the following classifications in accordance with project specifications and the American Society for Photogrammetry and Remote Sensing (ASPRS) classification standards:
1 - unclassified
2 - ground
7 - noise
40 - bathymetric bottom or submerged topography
41 - water surface
42 - derived water surface
43 - manmade submerged feature
45 - water column
1 Overlap - edge clip
1 Withheld - bathy land
User data values differentiates between NIR and green lasers. A value of of 1 indicates the point is from the NIR laser, and a values of 2-5 indicate the green laser.
Notes: Only a maximum of 4000 characters will be included.
Notes: Data collection is considered ongoing if a time frame of type "Continuous" exists.
Notes: All time frames from all extent groups are included.
Notes: All geographic areas from all extent groups are included.
(e.g., digital numeric data, imagery, photographs, video, audio, database, tabular data, etc.)
(e.g., satellite, airplane, unmanned aerial system, radar, weather station, moored buoy, research vessel, autonomous underwater vehicle, animal tagging, manual surveys, enforcement activities, numerical model, etc.)
2. Point of Contact for this Data Management Plan (author or maintainer)
Notes: The name of the Person of the most recent Support Role of type "Metadata Contact" is used. The support role must be in effect.
Notes: The name of the Organization of the most recent Support Role of type "Metadata Contact" is used. This field is required if applicable.
3. Responsible Party for Data Management
Program Managers, or their designee, shall be responsible for assuring the proper management of the data produced by their Program. Please indicate the responsible party below.
Notes: The name of the Person of the most recent Support Role of type "Data Steward" is used. The support role must be in effect.
Programs must identify resources within their own budget for managing the data they produce.
5. Data Lineage and Quality
NOAA has issued Information Quality Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information which it disseminates.
(describe or provide URL of description):
- 2020-05-04 00:00:00 - The CH4X sensor was mounted in a Leica PAV100 gyro-stabilized mount integrated with a NovAtel SPAN GNSS and LCI-100C IMU. Real time navigation and GNSS/IMU data logging was provided by Leica FlightPro software. Lidar data were logged on the Airborne Hydrography, AB (AHAB) operator console. Physical mounting offsets between the GNSS antenna, IMU and gyro-stabilized mount were determined through a combination of manual measurements and iterative processing in NovAtel Inertial Explorer software. Manual measurements were taken from the GNSS antenna to the reference point on the IMU in the CH4X sensor head. These measurements are added to the known offset between the IMU reference point and the rotation center of the gyro-stabilized mount to calculate the preliminary offset between the GNSS antenna and sensor reference point. This preliminary value was then used to seed the post-processing software which, through an iterative computation, uses the dynamic accelerations and rotations during flight to refine the offsets. Once the solution converges, the final offsets are entered into the flight management software and used in subsequent post-processing of the GNSS/IMU data for final trajectories.
- 2021-05-04 00:00:00 - Field calibration of the HE4X system is carried out to eliminate systematic errors by calculating corrections for boresight errors, scanner angle errors, remaining IMU angle errors and any necessary internal timing errors. a. 2 x Line A over mixed terrain with flat or gentle slopes and features such as peaked roof buildings (1 x each direction) b. 1 x Line B offset +50% from Line A in one direction c. 1 x Line C offset -50% from Line A in the same direction as Line B d. 2 x Line D orthogonal to previous lines (1 x each direction) A set of calibration lines were acquired at 500m and 400m altitude for the Niue calibration; and at 600m, 500m, and 400m for the Saipan calibration. Persistent high clouds in the area prevented collection of calibration lines at higher altitudes. All sets of lines are used to calibrate and verify the topographic lidar, while the 500m and 400m lines are used for the bathymetric lidar. Calibration values are calculated using the automatic calibration routine within the Leica Lidar Survey Studio (LSS) software. This utility first identifies patches or areas of gentle slope within the overlap region of all the lines to use for calibration. Patch selection prevents areas of vegetation, side of cars or buildings, from being used in the calibration process. Next, the utility compares the front side and back side of the elliptical scan within the same line, as well as comparing all lines to each other, to identify suitable calibration parameters such that data within the patches match. The procedure is iterative and continues until the best possible solution is computed. Calibration for each channel (topo and shallow) is done independently. Topo channel calibration was computed using 500m altitude lines. The 600m and 400m lines were then used for verification. Calibration of the shallow channel were computed using 500m altitude. Any lower altitude data were used for verification. At each step of the calibration process, quality assurance is conducted to ensure values being calculated are valid. This is done using the Leica LSS Quality Control Utility. Two types of checks are done; firstly, the front scan is compared to the back scan for every line. Secondly, a single line is chosen as a baseline and is compared to every other line. We would expect the average errors from both of these checks to be small; less than. In addition, the data is visually reviewed. In particular, features are studied to ensure lines from different directions show structures in the same position, in other words, verifying horizontal accuracy is maintained. These tests all provide assurance of relative accuracy. For this project, calibration lines were acquired over Niue International Airport on January 9, 2020, and Garapan, Saipan on July 14, 2020. Results are good and indicate that both calibrations were successful. Values computed were used for all data collected during a given mobilization period. Woolpert acquired a set of ground truth points with Real Time Kinematic (RTK) GNSS within the calibration areas. Ground truth is not used within the automatic calibration routine; however, a comparison to the lidar data was used to verify absolute accuracy. Results show data is well within required accuracy specifications.
- 2020-05-04 00:00:00 - For this project, the flight parameters were used to provide 100% coverage using a 15% sidelap for the topo and 20% sidelap for the bathymetry. During acquisition, flight lines are shown on a pilot’s display, and the aircraft is controlled by the pilot at all times. The HE4X system includes a NovAtel SPAN GNSS system with an LCI-100C IMU for aircraft position and orientation. One IMU is in the main Chiroptera sensor head, which includes the topo channel, shallow channel and RCD30 camera. Information from this IMU are also used in real-time by the PAV100 gyro-stabilized mount to compensate for deviations in pitch and roll. A second IMU is contained within the deep channel sensor head, installed over a second hatch in the aircraft. This head does not include a gyro-stabilized mount. Aircraft bank angles were restricted to 20º to avoid any potential GNSS dropouts. No flights were planned if the PDOP was expected to go above 3.0. Data were monitored for quality during acquisition using the Operators Console running on the AHAB collection computer. The operator monitored system status of the scanners and receivers, waveforms, camera images, data coverage, flight lines and the health of the navigation system. All data were recorded to a removable solid-state hard disk. At the end of the flight the hard disk was removed and taken to the field office where data was copied on to backup disks for transmittal back to the main processing office. Data was reviewed daily in the field for quality and coverage. second hatch in the aircraft. This head does not include a gyro-stabilized mount. Aircraft bank angles were restricted to 20º to avoid any potential GNSS dropouts. No flights were planned if the PDOP was expected to go above 3.0. Data were monitored for quality during acquisition using the Operators Console running on the AHAB collection computer. The operator monitored system status of the scanners and receivers, waveforms, camera images, data coverage, flight lines and the health of the navigation system. All data were recorded to a removable solid-state hard disk. At the end of the flight the hard disk was removed and taken to the field office where data was copied on to backup disks for transmittal back to the main processing office. Data was reviewed daily in the field for quality and coverage.
- 2020-05-04 00:00:00 - Position and orientation data were acquired in the aircraft using a NovAtel SPAN with LCI-100C IMU. All data were post-processed using NovAtel Inertial Explorer software to provide a tightly coupled position and orientation solution. Due to the distance of Alamagan, Anatahan, Guguan, and Sarigan from the single base station on Guam and their remoteness a precise point positioning (PPP) solution was used for them on ITRF2014. Initial data coverage analysis and quality checks to ensure there were no potential system issues were carried out in the field prior to demobilization of the sensor. Final processing was conducted in Woolpert’s offices. In general, data were initially processed in Leica’s Lidar Survey Studio (LSS) using final processed trajectory information. LAS files from LSS were then imported to a Terrascan project where spatial algorithms were used to remove noise and classify bare earth/ground. Manual review was conducted in both Terrascan and LP360 prior to product creation. Final trajectory data were post processed in NovAtel Inertial Explorer. Inertial Explorer accounts for the fixed offset between the reference point and IMU and uses a multi-pass algorithm to compute a tightly coupled solution. A GNSS base station or precise point positioning was used for processing. Average Forward and Reverse Separation RMS for the project was 0.019m in Easting and Northing, and 0.042m in Height.
- 2020-05-04 00:00:00 - Lidar processing was conducted using the Leica Lidar Survey Studio (LSS) software. Calibration information, along with processed trajectory information were combined with the raw laser data to create an accurately georeferenced lidar point cloud for the entire survey in LAS v1.4 format. All points from the topographic and bathymetric laser include 16-bit intensity values. During this LSS processing stage, an automatic land/water discrimination is made for the bathymetric waveforms. This allows the bathymetric (green) pulses over water to be automatically refracted for the pulse hitting the water surface and travelling through the water column, producing the correct depth. Another advantage of the automatic land/water discrimination is that it permits calculation of an accurate water surface over smaller areas, allowing simple bathymetric processing of smaller, narrower streams and drainage channels. Sloping water surfaces are also handled correctly. Prior to processing, the hydrographer can adjust waveform sensitivity settings dependent on the environment encountered and enter a value for the refraction index to be used for bathymetry. The index of refraction is an indication of the water type. A value 1.3423 was used for the index of refraction, indicating salty water. In the field, default waveform sensitivity settings were used for processing. In order to determine the optimal waveform sensitivity settings for final processing, sample areas were selected and processed with multiple different settings, to iteratively converge on the best possible settings. This is done by reviewing the processed point cloud and waveforms within sample areas. Settings affect which waveform peaks are classified as valid seabed, and which peaks are classified as noise. Optimal settings strike a balance between the amount of valid data that is classified as seabed bottom, and the amount of noise that is incorrectly classified due to peaks in the waveforms. Ideally all valid data is selected, while only a small amount of noise remains to be edited out. Once optimal threshold settings were chosen, these were used for the entire project. It is important to note that all digitized waveform peaks are available to be reviewed by the hydrographer; both valid seabed bottom and peaks classed as noise. This allows the hydrographer to review data during Terrascan and LP360 editing for valid data such as objects that may have been misclassified as noise. LSS processing produced LAS files in 1.4 format. Additional QC steps were performed prior to import to Terrascan. Firstly, the derived water surface was reviewed to ensure a water surface was correctly calculated for all bathymetry channels. No significant issues were apparent. Spot checks were also made on the data to ensure the front and back of the scans remained in alignment and no calibration or system issues were apparent prior to further data editing in Terrascan. LSS stores data in multiple LAS files for a single flight line. Each file corresponds to a single .dat file from the raw airborne data. Woolpert merged these multiple files into a single file per flight line and moved data into a standard class definition in preparation for data editing using proprietary scripts within SAFE’s FME software.
- 2020-05-04 00:00:00 - After data were processed in LSS and the data integrity reviewed. Anatahan, Alamagan, Guguan, and Sarigan were transformed from the ITRF2014 ellipsoid to the NAD83(2011) Epoch 2010 ellipsoid using VDatum. With the entire project now on the correct ellipsoid, data were organized into tiles within a Terrascan project. The tile layout is the same as that used for the imagery and is provided with the project deliverables. Data classification and spatial algorithms were applied in Terrasolid’s Terrascan software. Customized spatial algorithms, such as isolated points and low point filters, were run to remove gross fliers in the topographic and bathymetric data. A grounding algorithm was also run on the topographic data to distinguish between points representing the bare earth, and other valid topo lidar points representing features such as vegetation and buildings. Algorithms were run on the entire dataset. Data were reviewed manually to reclassify any valid bathy points incorrectly identified by the automated routines in LSS as invalid, and vice versa. In addition, any topo points over the water were reclassified to Water to correct the ground representation. Manual editing was conducted both in Terrascan and LP360. Steps for manual editing included: • Re-class any topo unclassified laser data and bathy seabed data from the water surface to a water surface class • Review bathymetry in cross section. o Re-class suitable data to Seabed (Class 40). o Re-class any noise in the bathy ground class to bathy noise (Class 45). o Re-class man made submerged objects to Submerged Object (Class 43). • Review topo ground points in areas of gaps or spikes. o Add points to ground (Class 2) if points are available to fill gaps in the ground model. o Re-class any noise in the ground class to Topo Unclassified (Class 1) if valid vegetation or other feature, or Noise if the point is not valid Low Noise (Class 7). Once editing was completed in TerraScan the data was vertically transformed to the GUVD04 datum using GEOID12B. Digital Elevation Models (DEMs) were then created using TerraScan at 1m resolution using the Topo Ground, Seabed, and Submerged Object Classes (Class 2, 40, 43) at a 1m resolution on a 5000m x 5000m tile layout provided in the project deliverables.
- 2020-05-04 00:00:00 - Although the bathymetry data includes intensity values, these are raw values. For intensity (reflectance) to correctly represent the reflectance of the seabed, the intensities must be normalized for any losses in signal as the light travels through the water column, so that the intensity value better reflects the intensity of the seabed itself. One of the fundamental issues that exists with reflectance imagery is the variance in return due to water clarity differences occurring spatially along line, and temporally from day to day. This is challenging for any bathymetric lidar sensor. If water clarity is relatively consistent along a line, then it is possible to achieve an overall homogenous reflectance image for an area. To a certain extent, variation in reflectivity intensity can be minimized by limiting the size of flight blocks and trying to ensure similar environmental parameters exist within a single flight block. In other words, where changes in water clarity or environment may be expected, flight blocks should be split to allow different normalization parameters to be used per block for the reflectance processing. Where this is not possible, and water clarity varies significantly along a line, variation in reflective intensity will be seen in the output imagery. While this imagery can still be analyzed and used for manual seabed classification, it prohibits the use of unsupervised, or semi-automated classification. For this survey, cloud shadows (ambient light) had an effect on the resulting reflectance images. Woolpert used proprietary in-house scripts developed in MATLAB to compute project specific correction parameters and normalize the raw intensity data for depth. This provides intensities that more closely represent the reflectance of the actual seabed. Corrected values were used to create 1m reflectance images per flightline using Applied Imagery’s QT Modeler software. Individual flightline reflectance images were then used in Trimble’s OrthoVista software to create a final reflectance image for the entire area.
- 2020-05-04 00:00:00 - Quality control is carried out through every phase of the project. Several checks were used to ensure data integrity and quality was maintained. Specific statistics were generated from lidar check points and image air targets. The airborne operator monitored system status of the scanners and receivers, waveforms, camera images, data coverage, flight lines and health of the navigation system during data acquisition. Flight logs are maintained during data acquisition. Logs not only track lines acquired, but also any relevant information on weather or water clarity, instances when sensor issues occur, and so on. These logs are a valuable resource during processing. During acquisition, aircraft bank angles were restricted to 20º to avoid any potential GNSS dropouts. No flights were planned if the PDOP was expected to go above 3.0. Separation plots and additional statistics were reviewed for each flight trajectory processed. Throughout data editing adjacent survey lines of data are compared to ensure there are no data busts, or system artifacts. During processing Terrasolid’s TMatch software is run to examine the Delta Z differences between overlapping lines. If differences are greater than 0.02m, then a simple Z correction is applied per flight line to remove any vertical differences between flight lines. TMatch can then be run again once all corrections are applied to ensure adjacent lines agree within specification. This provides a measure of inter-swath accuracy. Due to the rugosity and canopy cover of Anatahan, Alamagan, Guguan, and Sarigan a reliable comparison to adjacent lines was not obtained. Check points were not obtained on Anatahan, Alamagan, Guguan, or Sarigan due to the remoteness of the islands.
- The NOAA Office for Coastal Management (OCM) received files in laz format. The files contained lidar elevation and intensity measurements. The data were in UTM Zone 55 coordinates and ellipsoid elevations in meters. OCM performed the following processing on the data for Digital Coast storage and provisioning purposes: 1. Converted from UTM Zone 55 to geographic coordinates 2. Sorted by gps time
(describe or provide URL of description):
6. Data Documentation
The EDMC Data Documentation Procedural Directive requires that NOAA data be well documented, specifies the use of ISO 19115 and related standards for documentation of new data, and provides links to resources and tools for metadata creation and validation.
- 1.7. Data collection method(s)
- 3.1. Responsible Party for Data Management
- 5.2. Quality control procedures employed
- 7.1.1. If data are not available or has limitations, has a Waiver been filed?
- 7.4. Approximate delay between data collection and dissemination
- 8.3. Approximate delay between data collection and submission to an archive facility
(describe or provide URL of description):
7. Data Access
NAO 212-15 states that access to environmental data may only be restricted when distribution is explicitly limited by law, regulation, policy (such as those applicable to personally identifiable information or protected critical infrastructure information or proprietary trade information) or by security requirements. The EDMC Data Access Procedural Directive contains specific guidance, recommends the use of open-standard, interoperable, non-proprietary web services, provides information about resources and tools to enable data access, and includes a Waiver to be submitted to justify any approach other than full, unrestricted public access.
Notes: The name of the Organization of the most recent Support Role of type "Distributor" is used. The support role must be in effect. This information is not required if an approved access waiver exists for this data.
Notes: This field is required if a Distributor has not been specified.
Notes: All URLs listed in the Distribution Info section will be included. This field is required if applicable.
Data is available online for bulk or custom downloads
Notes: This field is required if applicable.
8. Data Preservation and Protection
The NOAA Procedure for Scientific Records Appraisal and Archive Approval describes how to identify, appraise and decide what scientific records are to be preserved in a NOAA archive.
(Specify NCEI-MD, NCEI-CO, NCEI-NC, NCEI-MS, World Data Center (WDC) facility, Other, To Be Determined, Unable to Archive, or No Archiving Intended)
Notes: This field is required if archive location is World Data Center or Other.
Notes: This field is required if archive location is To Be Determined, Unable to Archive, or No Archiving Intended.
Notes: Physical Location Organization, City and State are required, or a Location Description is required.
Discuss data back-up, disaster recovery/contingency planning, and off-site data storage relevant to the data collection
Data is backed up to tape and to cloud storage.
9. Additional Line Office or Staff Office Questions
Line and Staff Offices may extend this template by inserting additional questions in this section.