Data Management Plan
DMP Template v2.0.1 (2015-01-01)Please provide the following information, and submit to the NOAA DM Plan Repository.
Reference to Master DM Plan (if applicable)
As stated in Section IV, Requirement 1.3, DM Plans may be hierarchical. If this DM Plan inherits provisions from a higher-level DM Plan already submitted to the Repository, then this more-specific Plan only needs to provide information that differs from what was provided in the Master DM Plan.
1. General Description of Data to be Managed
Lidar (Light detection and ranging) discrete-return point cloud data are available in the American Society for Photogrammetry and Remote Sensing (ASPRS) LAS format. The LAS format is a standardized binary format for storing 3-dimensional point cloud data and point attributes along with header information and variable length records specific to the data. Millions of data points are stored as a 3-dimensional data cloud as a series of x (longitude), y (latitude) and z (elevation) points.
Notes: Only a maximum of 4000 characters will be included.
Notes: Data collection is considered ongoing if a time frame of type "Continuous" exists.
Notes: All time frames from all extent groups are included.
Notes: All geographic areas from all extent groups are included.
(e.g., digital numeric data, imagery, photographs, video, audio, database, tabular data, etc.)
(e.g., satellite, airplane, unmanned aerial system, radar, weather station, moored buoy, research vessel, autonomous underwater vehicle, animal tagging, manual surveys, enforcement activities, numerical model, etc.)
2. Point of Contact for this Data Management Plan (author or maintainer)
Notes: The name of the Person of the most recent Support Role of type "Metadata Contact" is used. The support role must be in effect.
Notes: The name of the Organization of the most recent Support Role of type "Metadata Contact" is used. This field is required if applicable.
3. Responsible Party for Data Management
Program Managers, or their designee, shall be responsible for assuring the proper management of the data produced by their Program. Please indicate the responsible party below.
Notes: The name of the Person of the most recent Support Role of type "Data Steward" is used. The support role must be in effect.
Programs must identify resources within their own budget for managing the data they produce.
5. Data Lineage and Quality
NOAA has issued Information Quality Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information which it disseminates.
(describe or provide URL of description):
- 2005-01-28 00:00:00 - The points are generated as Terrascan binary Format using Terrapoint's proprietary Laser Postprocessor Software. This software combines the Raw Laser file and GPS/IMU information to generate a point cloud for each individual flight. All the point cloud files encompassing the project area were then divided into 2 kilometer by 2-kilometer tiles. The referencing system of these tiles is based upon the project boundary minimum and maximums. This process is carried out in Terrascan. The bald earth is subsequently extracted from the raw LiDAR points using Terrascan in a Microstation environment. The automated vegetation removal process takes place by building an iterative surface model. This surface model is generated using three main parameters: Building size, Iteration angle and Iteration distance. The initial model is based upon low points selected by a roaming window and are assumed to be ground points. The size of this roaming window is determined by the building size parameter. These low points are triangulated and the remaining points are evaluated and subsequently added to the model if they meet the Iteration angle and distance constraints (fig. 1). This process is repeated until no additional points are added within an iteration. There is also a maximum terrain angle constraint that determines the maximum terrain angle allowed within the model. Quality Control Once the data setup has taken place the manual quality control of the surface occurs. This process consists of visually examining the LiDAR points within Terrascan and correcting errors that occurred during the automated process. These corrections include verifying that all non ground elements, such as vegetation and buildings are removed from the ground model and that all small terrain undulations such as road beds, dykes, rock cuts and hill tops are present within the model. This process is done with the help of hillshades, contours, profiles and crosssections. To correct misclassifications, a full suite of Terrascan and custom in-house data tools are used. Deliverables All LiDAR Data Products were delivered on DVD-ROM. Three copies were provided. All products other than hill-shade data were provided in 1k and 2k tiles with a 25-meter buffer. Hill-shades were delivered in three large areas. Full Feature or All Return Point Data Data delivered in ASCII, comma delimited files with one record per return as defined in Table 4. The records are ordered sequentially according to Easting with no duplicate records. The individual returns are classified into the categories as defined in Table 5. If a return cannot be reliably classified as "Vegetation" or "Building/structure", it shall be assigned as "Not ground". This product was generated using an in house custom utility. The process involved extracting a Terrascan Binary file packed with the scan angle and the extra precision required for the for a unique time stamp. - Bare Earth Point Data Data delivered in ASCII, fixed length, formatted files with one record per return as defined in Table 4 in the attached report. The records are ordered sequentially according to Easting with no duplicate records. The individual returns are classified according to the categories defined. This product was generated using an in house custom utility. The process involved extracting a Terrascan Binary file packed with the scan angle and the extra precision required for the for a unique time stamp. Bare Earth and Full Feature Hill-Shade Image Data delivered in GeoTIFF format with a TFW file. The image resolution is 1m. This product was generated in ArcView.
- 2005-01-28 00:00:00 - Contour Data Data is delivered at 50cm nominal contour interval, 2.5m labeled index contours in AutoCAD format. This product was generated using a combination of TerraModeler and Microstation. Data Field 1: Time GPS time Reported to nearest microsecond Data Field 2: x,y location Geographic location of return NAD83-92, to nearest 0.01m Data Field 3: Elevation Elevation of return NAVD-88, to nearest 0.01m Data Field 4: Return classification Return classification of this return First or last return Data Field 5: Off Nadir Angle Angle between nadir and transmitted pulse Reported to nearest 0.01 degrees Data Field 6: Return Intensity Intensity of return Data Field 7: Classification Code Classification of Return Please note that all products, other than the hillshade products were delivered in 2 kilometer tiles and 1 kilometer tiles. Return Classification System Code Description Notes 2 Ground or water -Bare Earth surface 5 Vegetation -Non-ground points, 1.51 m up to 60 m above coincident ground
- 2005-01-28 00:00:00 - Problems, Resolutions and Conclusions Boresight Our laser encountered some technical difficulties during acquisition. The laser manufacturer, Riegl, was unable to pin point the problem on the bench due to the small magnitude of the error. Riegl anticipated this result due to the limits of testing within a lab setting. As a precaution to eliminate any possible source of error, the laser angle encoder unit and the bearings were replaced. As an additional precaution, an extra screw was added during the inspection to improve the mounting of encoder mechanism. The unit belt was eliminated as the source of the problem. As a result of this malfunction, the boresight values for each flight had to be thoroughly reviewed. This manual process took approximately three weeks to resolve for the appropriate boresight corrections. The majority of the flights required a single roll correction of 0.097 degrees while others required an adjustment of 0.068 or 0.142 degrees. Certain flights required two-roll corrections. Unfortunately, this boresight problem caused by equipment malfunction was only discovered after initially delivering the project - this was principally due to a shortcoming in our QC methodology that did not reveal mechanical problems such as those encountered with our scanning laser. It should be noted that the laser deployed for this project had previously operated without error on approximately 200 missions. We therefore consider the problems encountered with the South Bay project to be an outlier. Nevertheless, our processing flow has now been modified to identify such outliers by performing more quality control checks on the data prior to entering production. These include: profiles, full feature hillshades, GPS checks and distance grids. The distance grids are quite beneficial as they visually display relative elevation error between flightlines. Please note that we attempted to perform distance grids on all the tiles, but were successful on only 127 of 145 tiles due to lack of overlap in open water or tiles containing a great deal of flightlines causing the application to crash. Close attention was paid to these tiles in particular during the second round of production as to ensure the quality was good.
- 2005-01-28 00:00:00 - The distance images were captured as screengrabs from the application and are located on the accompanying disk. The late discovery of this problem proved to be special challenge, as the processing team had spent the majority of the initial filtering run extracting the buildings from the point cloud. To deliver this project in a rapid manner, we opted to maintain the building classifications as to minimize the amount of reprocessing. Therefore we could not reprocess the point cloud with the boresight compensation values. An additional problem ensued; two GPS week rollovers occurred during acquisition. Due to software limitations we are unable to store GPS week, only GPS time, therefore, there is a strong likelihood of having multiple identical GPS times within the dataset. This situation leads to the possibility of associating the LiDAR data to the wrong trajectory and applying an incorrect roll compensation factor based upon aircraft position and required correction. To overcome these obstacles we developed a new procedure using existing software and custom applications. Once applied, these corrections were verified with the aforementioned distance grids. This issue is being addressed in future projects by allowing the LiDAR operator to tag flight lines with an attribute during acquisition. As a result of applying the roll compensation the overall quality of the dataset increased significantly allowing the project to meet the set accuracy guidelines. Upon the discovery of this problem by the USGS a few selected distance grids were generated. These grids indicated an average error of 30 centimeters while a few areas indicated errors of 45-50 centimeters. Once the boresight corrections were applied, all but a few areas are now within the range of +/- 15 centimeters. A few areas remain as outliers in the 15 to 30 cm range. A benefit of the correction was the elimination of "false vegetation". The error was more evident in flat open areas of the mud flats. The error manifested itself to look like isolated areas of low scrub. Once corrected these were reduced significantly to meet specification or eliminated all together. Vegetation removal also proved to be tricky as quite often the bulrush directly adjoining dikes masked themselves as extensions to the dikes. Pickle weed and other forms of low vegetation shorter than 20 cm also tended blend very well into the mudflats. Particular attention had to be paid to these areas and the assistance of the USGS proved to be a valuable asset in discerning valid ground from vegetation.
- 2016-02-10 00:00:00 - The NOAA Office for Coastal Management (OCM) received the files in laz format from USGS via an FTP online repository. The files contained lidar elevation and intensity measurements. The data were in State Plane California Zone 3 and Zone 4, NAVD88 (orthometric) heights in meters. The California Coastal Project was divided into two projects: State Plane Zone 3 and State Plane Zone 4 respectively. OCM performed the following processing for data storage and Digital Coast provisioning purposes: 1. The data were converted from state plane coordinates to geographic coordinates. 2. The data were converted from NAVD88 (orthometric) heights in meters to GRS80 (ellipsoid) heights in meters using Geoid 09. 3. The LAS Noise class was dropped and Class 4 (medium vegetation scrutinized). All Class 4 points are considered to include all vegetation and man made objects.
(describe or provide URL of description):
6. Data Documentation
The EDMC Data Documentation Procedural Directive requires that NOAA data be well documented, specifies the use of ISO 19115 and related standards for documentation of new data, and provides links to resources and tools for metadata creation and validation.
- 1.6. Type(s) of data
- 1.7. Data collection method(s)
- 3.1. Responsible Party for Data Management
- 4.1. Have resources for management of these data been identified?
- 4.2. Approximate percentage of the budget for these data devoted to data management
- 5.2. Quality control procedures employed
- 7.1. Do these data comply with the Data Access directive?
- 7.1.1. If data are not available or has limitations, has a Waiver been filed?
- 7.1.2. If there are limitations to data access, describe how data are protected
- 7.4. Approximate delay between data collection and dissemination
- 8.1. Actual or planned long-term data archive location
- 8.3. Approximate delay between data collection and submission to an archive facility
- 8.4. How will the data be protected from accidental or malicious modification or deletion prior to receipt by the archive?
(describe or provide URL of description):
7. Data Access
NAO 212-15 states that access to environmental data may only be restricted when distribution is explicitly limited by law, regulation, policy (such as those applicable to personally identifiable information or protected critical infrastructure information or proprietary trade information) or by security requirements. The EDMC Data Access Procedural Directive contains specific guidance, recommends the use of open-standard, interoperable, non-proprietary web services, provides information about resources and tools to enable data access, and includes a Waiver to be submitted to justify any approach other than full, unrestricted public access.
Notes: The name of the Organization of the most recent Support Role of type "Distributor" is used. The support role must be in effect. This information is not required if an approved access waiver exists for this data.
Notes: This field is required if a Distributor has not been specified.
Notes: All URLs listed in the Distribution Info section will be included. This field is required if applicable.
This data can be obtained on-line at the following URL:
The data set is dynamically generated based on user-specified parameters.;
Notes: This field is required if applicable.
8. Data Preservation and Protection
The NOAA Procedure for Scientific Records Appraisal and Archive Approval describes how to identify, appraise and decide what scientific records are to be preserved in a NOAA archive.
(Specify NCEI-MD, NCEI-CO, NCEI-NC, NCEI-MS, World Data Center (WDC) facility, Other, To Be Determined, Unable to Archive, or No Archiving Intended)
Notes: This field is required if archive location is World Data Center or Other.
Notes: This field is required if archive location is To Be Determined, Unable to Archive, or No Archiving Intended.
Notes: Physical Location Organization, City and State are required, or a Location Description is required.
Discuss data back-up, disaster recovery/contingency planning, and off-site data storage relevant to the data collection
9. Additional Line Office or Staff Office Questions
Line and Staff Offices may extend this template by inserting additional questions in this section.