Data Management Plan

The National Oceanic and Atmospheric Administration (NOAA) has the statutory mandate to collect hydrographic data in support of nautical chart compilation for safe navigation and to provide background data for engineers, scientific, and other commercial and industrial activities. Hydrographic survey data primarily consist of water depths, but may also include features (e.g. rocks, wrecks), navigation aids, shoreline identification, and bottom type information. David Evans and Associates, Inc. (DEA), in conjunction with Geomatics Data Solutions and Quantum Spatial, Inc., conducted hydrographic lidar survey operations along the New Jersey coast, from Barnegat Bay to Hereford Inlet. Survey H12606 was conducted in accordance with the Statement of Work, Hydrographic Lidar Surveying Services (SOW), June 2013 and Hydrographic Survey Project Instructions, June 18, 2013 for OPR-C308-KRL-13. Nine potential survey areas were identified, however, only three areas were surveyed. Areas 1, 2, and 6 were selected. Data were collected from April 1 - 3, 2014 using the Chiroptera Lidar system.

Original contact information:

Contact Org: Hydrographic Surveys Division, Office of Coast Survey, National Ocean Service, NOAA, U.S. Department of Commerce

Phone: 301-713-3028

Email: HSD.Inquiries@noaa.gov

Process Steps:

• 2014-04-01 00:00:00 - Quality Control Survey Methods & Procedures Mobilization The ChiropteraI System was installed into a Cessna 206 (Tail N7266Z) at Capital City Airport in Frankfort, Kentucky on March 26, 2014. The system can be installed in either a forward or reverse direction, depending on the physical space requirements within the aircraft. The system was installed in the forward configuration for this project. Once the system was installed in the aircraft, an offset survey was performed to derive the offsets from the GNSS antenna to the IMU center in the IMU coordinate reference frame. During this first offset survey, issues keeping the total station level were encountered. This offset dataset was not used for the project. A second offset survey was performed at Atlantic City International Airport (ACY) on March 30, 2014 using a different total station, tribrach and tripod (Figure 5). Measurements were taken of the system case corners, random points on the sides of the system case, and points around the base and top of the GNSS antenna. Since a data logger was not used, measurements were logged to an Excel spreadsheet. Measurements were then processed with an AHAB proprietary script in MatLab to compute the final offsets. In general measurements to the side of the system case are used to help better define the plane of the system case side and therefore improve the corner coordinate measurements. The offset from the corners of the system case to the IMU center is known and fixed. These were used for all data processing including the calibration and reconnaissance flights acquired prior to March 30, 2014.
• 2014-04-01 00:00:00 - B1.b Lidar Calibration Immediately following system installation, a calibration flight was conducted over the airport on March 26, 2014. A second calibration flight was conducted at the end of the survey over ACY on April 3, 2014. A calibration flight consists of about 6 to 12 flight lines with about 50% overlap between adjacent lines as shown in Figure 6. Ideally the calibration lines will cover certain types of features, including a large flat area or an area with a gentle even slope, and buildings with slant roofs. The buildings are acquired in perpendicular directions. The calibration site can also include known ground control points. These can be used during the calibration routine, or for validation. For this project known points were used for validation only. Data from the calibration flights are processed initially using calibration values from a prior installation in Lidar Survey Studio (LSS). Over the course of multiple projects, it has been found that calibration values change very little within the ChiropteraI system, indicating the system to be robust, even during shipping. Processed data is provided to the automatic calibration routine for analysis. The automatic calibration routine sets up a number of constraints and then uses an optimization algorithm to minimize the error estimation. For both calibration sets for H12606 constraints were in the form of patches. Patches are areas of relatively flat ground that are used for calibration. An automated routine was used to select suitable patches of flat ground from a number of lines. Patch locations selected can be reviewed to ensure even distribution over the calibration site before continuing with the calibration analysis. Once patches have been selected, data falling within these patches are then compared in two ways: first within a single line, the front and back half of the elliptical scans are analyzed, and secondly with data from multiple lines. This is done automatically by the optimization algorithm which generates recommended calibration values. Lidar data from the calibration flight are then reprocessed in LSS using the new values and reviewed for accuracy. The accuracy review involves a visual comparison to ensure there are no position misalignments in the horizontal or vertical. Calibration values calculated by the automated routine were deemed acceptable by the hydrographer and used for processing. Calibration values from the initial flight on March 26, 2014 were applied to all lidar data acquired from March 26, 2014 up to April 1, 2014. Calibration values from the flight over ACY on April 3, 2014 were applied to all lidar data acquired from April 2, 2014 up to April 3, 2014. Two calibration files were used for the project due to a firmware upgrade mid-project. After the reconnaissance flight on March 27, 2014 it was noticed that a larger volume of data than normal was being generated by the sensor. Data quality was not affected by this; however a small adjustment was made to the acquisition settings to reduce the number of files being generated by the system. Unfortunately the setting adjustments caused the bathymetric waveforms to be clipped for the second acquisition flight over Area 1 on April 1, 2014. This data was not usable and was not included in any further processing or products generated for H12606. A firmware upgrade was made from CATbtopV25 to CATbtopV31 before flights commenced on April 2, 2014. This required an update to the calibration, and the calibration data were collected on April 3, 2014.
• 2014-04-01 00:00:00 - Reconnaissance Nine potential survey areas were identified by NOAA and provided in OPR-C308-KRL-13_PRF.000. The areas were numbered in order of priority with Area 1 being the highest priority and Area 9 being the lowest priority (Figure 8). As defined in the project instructions, the project was limited to 20 hours of flying. Initially a reconnaissance flight acquired data over all areas to identify those with the best water clarity giving the most chance for success. Reconnaissance flight lines were planned such that the line for each area would cover a range of depths, from land across deeper channels. This would give an indication of depth penetration for the area. Reconnaissance data were acquired on March 27, 2014 and processed that evening for review. Reconnaissance data were acquired using the same survey parameters used during project data acquisition. Analysis of this reconnaissance data, along with the area priority, were used to determine which areas would be the focus for the remaining flight hours. Areas 1 and 2 were initially selected, having the best water clarity and being the highest priority areas. Upon completion of Areas 1 and 2, there were enough flight hours remained to complete a third area. Area 6 was selected, OPR-C308-KRL-13 New Jersey Coast and Vicinity March 2014 Data Acquisition and Processing Report Field Unit: David Evans and Associates, Inc. due to its promising water clarity during the reconnaissance flight and its proximity to the base airport. All areas selected were approved by the NOAA COTR in advance of data collection.
• 2014-04-01 00:00:00 - B1.d Survey All survey data collection was conducted from 400m altitude at around 97 knots. The ChiropteraI simultaneously acquired bathymetric lidar at 35 kHz, topographic lidar at 300 kHz and digital camera imagery at one frame per second. These survey parameters produced a nominal pulse spacing of 0.75 meters for the bathymetric laser, 0.25 meters for the topographic laser and a ground sample distance of 25 centimeters for the rectified imagery mosaic for 100% coverage. The project required 200% bathymetric lidar coverage at a 1m x 1m laser spot spacing. To achieve this, hydrographic lidar flights were planned using the parameters provided in Table 5. All areas were flown to provide 200% coverage as required in the project instructions. Parameters used during survey operations exceeded the project requirements. During data acquisition the operators display was used to select flight lines, which were then displayed on the pilot's flight management system. In addition, the operator monitored system status of the scanners and receivers, waveforms, camera images, data coverage, flight lines and the health of the navigation system. The operator made notes of the general environmental conditions and any issue that occurred during flight, such as line restarts. A flight acquisition log was generated upon return to the office that included all relevant survey information for the flight. Flight logs are provided in the accompanying H12606 DR, Separate I. Aircraft bank angles were restricted to 20 degrees to avoid any potential GNSS dropouts. No flights were planned if the PDOP was expected to go above 3.0. Positioning of the aircraft in real-time was accomplished by the integrated IGI AEROControl IMU and GNSS antenna. No real time corrections were used and no IMU data was used to steer the scan to aid in line-keeping. All lidar system data were logged to AHAB's proprietary format on two ruggedized removable solid state hard drives. All raw position and IMU data were recorded to a removable flash storage card inside the ChiropteraI System in IGI's .c5l format.
• 2014-04-01 00:00:00 - Data Processing Methods & Procedures Data Management Upon return to the field office, data from the aircraft were copied onto two sets of external USB 3.0 hard disks to provide data redundancy. GNSS reference station data were sent via FTP and duplicate copies were also made of this data. On completion of the field work, processing was moved to the Geomatics Data Solutions office in San Diego. All data were stored on NAS and replicated nightly for backup. In addition, a copy of all the raw data was sent to David Evans and Associates office in Vancouver, WA where it was housed on an additional NAS and also subject to nightly backups. Upon completion of data processing a full copy of the data were placed on the DEA NAS, where final S-57 and BASE surface production was conducted. All data were organized according to Geomatics Data Solutions established directory structure for lidar processing. Standardized naming conventions were used throughout to maintain data integrity. Field Processing Initial data processing followed the trajectory and LSS processing steps described in detail in the following sections. However due to the short duration of the project, initial field processing made use of preliminary calibration and trajectory files. Data were reviewed in LSS for data coverage and also to ensure there were no potential system issues. Data were not taken further in the processing work flow during field operations. The only system issue occurred during the second flight on April 1, 2014, when bathymetric waveforms were clipped before the seabed was reached. No data from this flight was used. Feedback was provided to the operator and a system modification was made before the next flight, allowing successful data collection on April 2, 2014. Workflow Overview An overview of the processing workflow is provided in Figure 11. In general data were processed in LSS using final processed trajectory information. LAS files from LSS were then imported to a Terrascan project where spatial algorithms were used to remove noise, mostly on the water surface. Manual editing was also conducted at this stage, with the entire data set being reviewed by a hydrographer. LAS files were exported by flight line from Terrascan and converted to mean lower low water (MLLW) using VDatum. In addition features pertinent to S- 57 development were exported as ASCII XYZ files for import to CARIS Bathy DataBASE. LAS files on MLLW were imported to a Fledermaus project and a Fledermaus PFM file created with a Combined Uncertainty and Bathymetry Estimator (CUBE) surface. This CUBE surface was reviewed and soundings designated as necessary. Designated soundings were applied to the surface and the surface exported as a BAG, which contained a depth and uncertainty layer. The BAG was imported to CARIS Bathy DataBASE and a BASE surface created. The BASE surface was finalized, clipping to the average MHW height for the area.
• 2014-04-01 00:00:00 - Trajectory Processing Airborne GNSS and IMU data were processed along with ground GNSS reference station data in AeroOffice. Airborne and ground GNSS data were processed together using GrafNav. Processing made use of both GPS and GLONASS satellites for positioning. This final GNSS solution was combined with the IMU data, taking in to account the GNSS antenna lever arm, to compute a final trajectory for the center of the IMU. GNSS data were processed in NAD83 (2011) and final trajectory solutions were exported from AeroOffice in a custom ASCII format for use in LSS processing. Final trajectory solutions were also in NAD83 (2011). Reference points established specifically for project H12606 and existing CORS stations were used for trajectory processing as provided in Table 6. During processing QC plots were reviewed both at the GNSS processing stage and also after final IMU processing. The Trajectory Processing Log is provided in the H12606 DR, Separate I. Point OCS_NJ_01 was established for use while flying Area 1, however GLONASS ephemeris data showed GLONASS satellites as turned off part way through April 1, 2014. This affected data acquired at the OCS_NJ_01 base station. Data were subsequently processed with CORS station data from NJGT and NJOC. No degradation in data accuracy occurred.
• 2014-04-01 00:00:00 - Lidar Processing Airborne lidar data were processed using AHAB's Lidar Survey Studio (LSS). During this stage raw airborne data is combined with system offsets, trajectory and calibration information to produce an accurately georeferenced lidar point cloud, with additional attributes associated to each point. During this processing routine LSS automatically discriminates between land and water points from the bathymetric laser. If a point is over water, then the point is automatically refracted to provide the correct final depth. No additional processing is necessary to refract data. Prior to processing the hydrographer can adjust waveform sensitivity settings dependent in the environment encountered and enter a value for the refraction index to be used for bathymetry. For this project, water salinity and temperature were monitored at the USGS water gage (01408167) and the average values during each flight were used, along with the laser wavelength of 532nm, to calculate an index of refraction number for processing of the bathymetric lidar data within each survey area. Values used are included in the LSS processing settings files delivered with the raw data for the project. In the field, default waveform sensitivity settings were used for processing. In order to determine the optimal waveform sensitivity settings for final processing, sample areas were selected and processed with multiple different settings, to iteratively converge on the best possible settings. This is done by reviewing the processed point cloud and waveforms within the sample areas. Settings affect which waveform peaks are classified as valid seabed, and which peaks are classified as noise. Optimal settings strike a balance between the amount of valid data that is classified as seabed bottom, and the amount of noise that is incorrectly classified due to peaks in the waveforms. Ideally all valid data is selected, while only a small amount of noise remains to be edited out. Once optimal threshold settings were chosen for this project, they were used for the entire project. It is important to note that all digitized waveform peaks are available to be reviewed by the hydrographer; both valid seabed bottom and peaks classed as noise. This allows the hydrographer to review data during Terrascan editing for objects that may have been potentially misclassified as noise. LSS processing produced LAS files in 1.2 format. Although LSS is capable of producing and working with LAS 1.4, many third party systems are not. Therefore LAS 1.2 was used for this project. LSS stores data in multiple LAS files for a single flight line. Each file corresponds to a single .dat file from the raw airborne data. LAS data were produced in UTM Zone 18N in meters, with a vertical elevation relative to the NAD83 (2011) ellipsoid. Once data were processed QC steps were performed in LSS prior to import to Terrascan. The derived water surface was reviewed to ensure a water surface was correctly calculated for all channels, pools and bay areas. Portions of three lines in Area 1, and 12 lines in Area 2 had issues calculating a localized water surface. For these locations a mean water surface for the affected line segments was used. Specific line segments affected are noted in the LSS Processing Logs provided with the accompanying H12606 DR, Separate I. Some overlap was processed for each line using both localized and average water surface methods, to review how the use of the average water surface may affect data accuracy. Review indicated that error was not likely to be significant (less than 0.05 meters) due to the calm water conditions in the bay. No overlapping duplicate data was used during further processing; overlapping data was used for QC only. Spot checks were also made on the data to ensure the front and back of the scans remained in alignment and no calibration or system issues were apparent prior to further data editing in Terrascan.
• 2014-04-01 00:00:00 - Lidar Editing Once the data were processed in LSS and the data integrity reviewed, LAS files were copied from the LSS directory for import to Terrascan. Prior to Terrascan import, LAS files were renamed from the extended LSS name to an integer number incrementing from 1. This number was used as the flight line number on import to Terrascan. For all areas, files from the bathy laser had a flight line number between 1 and 2999, while files from the topo laser had a flight line number greater than or equal to 3000. Line renaming was controlled by a batch script. Once data were exported from Terrascan, files were renamed back to their original LSS name, maintaining continuity throughout the project. Terrascan projects were organized by block, with blocks aligned parallel to the general flight line direction. Area 1 contained 93 1km x 1km blocks, Area 2 contained 80 1km x 1km blocks, and Area 6 contained 164 0.5km x 1km blocks. Blocks were smaller for Area 6 due to the increased amount of dense topo laser data. To identify appropriate settings for each algorithm step, sample blocks were tested, and a manual review completed after each step. Algorithms were run on all flight data from Area 1 and Area 2 at once. Algorithms were run on the Area 6 high tide flight independently from the Area 6 low tide flight, due to large differences in the water surface elevation between the two flights. The two flights were then combined prior to manual review. Once the algorithms were run, manual editing began. Steps for manual editing included: During manual editing the entire dataset was reviewed by a hydrographer in 5-meter increments. Orthorectified imagery mosaics were used in MicroStation to assist with editing. An imagery mosaic existed for both the high tide and low tide flights for each area. Permanent piers and docks were maintained in the dataset, while floating or temporary structures were removed. Once manual editing was complete, LAS data were moved to suitable classification levels for use in Fledermaus, as described in Table 8. Data were then exported by flight line and renamed to the original LSS names. At this stage multiple files that made up one flight line were merged together to create a single LAS file for each flight line acquired. Points required for S-57 development were exported in ASCII XYZ files. Accepted data were also exported to LAS files for use in crossline analysis and to calculate the average offset between MLLW and MHW for each area.
• 2014-04-01 00:00:00 - Datum Conversion LAS files were converted to the required MLLW tidal datum using VDatum v3.3 making use of Geoid12A. During this conversion data were automatically clipped to the VDatum spatial extents. In all areas this removed sections of valid data, therefore the NAD83 LAS files are delivered as part of the dataset for H12606. In the case of Areas 1 and 2, data were only removed at the edges of the area. For Area 6 however, data were removed at the edges of the survey area, but also internal to the survey area over low-lying sections of land. VDatum Clipping in Area 6, Line 012_FL15 MLLW and MHW LAS files containing accepted only data were imported into Fledermaus and MLLW and MHW surfaces created. A surface difference was generated to provide the difference between MHW and MLLW across each area. The MHW surface was used to generate the MHW shoreline, exported in DXF format for use in S-57 generation.
• 2014-04-01 00:00:00 - Product Creation LAS files containing all the data were imported into a Fledermaus project for each area. PFM CUBE surfaces were then generated for each area. When generating the PFM, any withheld classes in the LAS files were not used. In addition water surface classes 30 and 31 were excluded. Classes 7 and 20 were brought into the PFM as "manually invalidated", so they could be viewed as rejected in the 3D Editor. The 1m CUBE surface was created from the Class 2 accepted data using the parameters provided in Table 9. A capture distance of 1.41 meters was used in accordance with the NOS HSSD (April 2013) requiring that the maximum prorogation distance be no greater than the grid resolution divided by the sqrt(2). During the PFM build, total horizontal uncertainty (THU) and total vertical uncertainty (TVU) values were assigned to soundings. To calculate the TVU for each area, the standard deviation from the crossline analysis was combined with the datum and transform errors from VDatum in the form: VDatum errors for the New Jersey Coastal Embayment as provided by NOAA (revised December 2013) have a cumulative uncertainty of 10.242 centimeters. Lidar data uncertainty was taken as the StDev from the crossline analysis. Final uncertainty values for the Fledermaus CUBE surface were propagated to the node based on the soundings contributing to each node.
• 2014-04-01 00:00:00 - To assess THU lidar data were acquired over a parking lot in multiple directions using both the bathymetric (green) laser and the topographic (infrared) laser. Intensity images were made for each line. Parking lot line intersections and end points in the georeferenced intensity images were compared to known coordinates acquired independently using Real Time Kinematic (RTK) GNSS. A total of 45 known points were compared to four topographic laser lines and four bathymetric laser lines with a mean difference in position of 0.23 meters and StDev of 0.15 meters. Rather than use the mean of all lines, the values for the line with the maximum uncertainty were used. This provided a mean difference of 0.45 meters and StDev of 0.16. The THU of 0.78 meters was computed in the form: Values used are provided in Table 10. It is important to note that the uncertainty value of the CUBE bin is a propagated uncertainty based on the location and number of soundings contributing to each node. Area 1 Area 2 Area 6 TVU 0.122 0.122 0.120 THU 0.780 0.780 0.780 Once the PFM with CUBE surface was generated, soundings greater than half the allowable error, in this case 25 centimeters, were flagged as suspect. All suspect soundings were reviewed in the 3D editor. Suspect soundings fell into 3 classes: remaining noise at the edge of the swath, valid points on slopes or soundings requiring designation. Remaining noise points were rejected, points on slopes had the suspect flag removed and any points requiring designation were flagged as feature soundings. A point was flagged as a feature sounding if there were no shoaler hypothesis within 20 meters. The CUBE surface was then regenerated. Any areas with multiple hypothesis, typically at the land/water interface, were reviewed and the shoalest hypothesis selected. The suspect filter was then run again, and all suspect flags reviewed as before. Finally feature soundings were applied to the CUBE surface. All feature locations were reviewed to ensure the surface reflected the sounding. The CUBE surface uncertainty layer was also reviewed at this time. The resulting calculated uncertainty values of the majority of nodes in the final surfaces ranges from 0 to 0.20 meters. Higher uncertainty values up to 4.7 meters are located at the edge of harbor and pier walls caused by gridding data over vertical features (Figure 15). These nodes were carefully reviewed in Fledermaus and reviewing these regions in the 3D Editor shows good agreement between survey lines. The high standard deviation and high uncertainty associated with these nodes is considered an artifact of gridding data over a steep and irregular seafloor. As a result, all data are considered within specification.
• 2014-04-01 00:00:00 - Finally, the CUBE surface was exported to BAG format, named according to the required convention specified in the NOS HSSD: BAG data were converted to .csar format using Safe FME software and a *.csar surface created Due to datum issues between software the CARIS shifts the BAG surface by up to 2 meters if opened with other data. The BAG opens correctly in other software. The .csar surfaces were imported to CARIS Bathy DataBASE and finalized, clipping to the average MHW height for each area.
• 2015-12-10 00:00:00 - The NOAA Office for Coastal Management (OCM) received LAS point clouds from NOAA NGDC in Boulder, CO in November of 2015. These files were incorrectly compressed to LAS 1.2 format from LAS 1.4 therefore some values were corrupt. A 2nd version came in the Spring of 2016 and were not compressed. These files were in LAS 1.4 and referenced to UTM coordinates in meters and NAVD88 vertical coordinates also in meters. OCM performed the following processing for data storage and Digital Coast provisioning purposes: 1. A series of reclassification were applied to meet NOAA OCM LAS 1.2 Class scheme: Class 7 to 18 - rejected during editing Class 20 to 30 - s-57 objects Class 30 to 27 - bathy water surface Class 31 to 9 - derived water surface Class 128 to 17 - line run in/out Class 129 to 1 - unclassified Class 130 to 12 - topo ground - not used Class 133 to 25 - water surface - not used Class 146 to 7 - noise Class 157 to 14 - bathy land - not used 2. LAS files were transformed to geographic coordinates (decimal degrees) and ellipsoidal heights (meters). All classes above are available via FTP versions but through the Data Access Viewer, only certain classes will be made available. See information for details in Viewer.

This data can be obtained on-line at the following URL: https://coast.noaa.gov/dataviewer/#/lidar/search/where:ID=4984

The data set is dynamically generated based on user-specified parameters.;