Data Management Plan (Deprecated)

A Raster having 20 m resolution with decimal values was assembled from 18.6 billion bathymetric soundings that were obtained from the National Center for Environmental Information (NCEI) https://www.ncei.noaa.gov. Bathymetric soundings extends from Kuril-Kamchatka Trench in the Bearing Sea along the Aleutian Trench to the Gulf of Alaska, and in the Arctic Ocean from Prince Patrick Island to the International Date Line. Bathymetric soundings were scrutinized for accuracy using statistical analysis and visual inspection with some imputation. Editing processes included: deleting erroneous and superseded values, digitizing missing values, and referencing all data sets to a common, modern datum.

Alaska and surrounding waters

Lineage Statement:
-Currently, our process keeps a record of the survey from which that point originated. Some older data does not have this level of metadata. Servers are crawled for relevant data, and a list of download URLS to data files is returned. Data files are then retrieved using custom python scripts. Raw data is downloaded from online NCEI web servers. Data is converted to CSV or XYZ files Points missing one or more of their XYZ points values are removed and archived. Data is evaluated as a component of the bathymetry map to identify outliers (instances where data point(s) are not consistent with the expected variability of the surrounding environments) using a variety of statistical and manual methods. K-Natural Neighbors (KNN) - Python and SciPy Percentiles with Standard Deviations - Python and SciPy Slope and Neighbors - ArcGIS Models Manual/visual selection Human Imputation of points to make them consistent with surrounding terrain tracklines and satellite altimetry. Upon integration into the dataset, between 0-25% of the deepest and shallowest points are immediately removed from each trackline based on the StdDev of each induvial trackline. All data is archived; a dataset with the outliers could be built within 7 weeks. The percentage of points removed (R) is determined by a non-linear function of the datasets standard deviation (s), and can be seen below: R=-4.746813 +30.059/(1+(s/9.584625)^0.9983 ) This function was derived using a best-fit curve tool, which was instructed to return a naturally-logarithmic function which was equal to ~25 when s=0, and which decreased asymptotically to 0 as s grew larger. The function was then tailored to have what the developer felt was a reasonable slope The logic here is that datasets with low standard deviations would be relatively flat and featureless. Since they have a lower level of topographic complexity, they can undergo a higher rate of removal while still retaining the essential topographic character of the surface they represent. Data is then organized into a Kd-Tree structure in which data points are organized based on their values with the data sorted between levels in the tree (i.e., the first level is split along the x axis, the next level is split along the y axis, the next along the z axis, and then the fourth along the x axis again. The result was a tree which can be searched in O (log(n)) time, and which was optimized for quick spatial searches, critical for the next step. K-Nearest-Neighbors (KNN) statistical model is used on the data. This uses the value of each data point’s K spatially nearest neighbors (k-value) to produce an ‘expected value.’ The expected value is then subtracted from the point’s observed value, and the absolute value of this difference is the point’s ‘residual.’ After calculating the residuals for all of the points in the data set, we remove the 5% of points with the highest residuals. After these steps are completed, the remaining data are converted to Feature Classes. This data structure is composed of not only the raw data, but also a host of metadata calculated from the raw data, such as vessel name and tracklines number. Spatial indexes are added to the data to optimize operations. All internal data is point data and is stored in Alaska Albers project. ArcMap and Arc Pro Slope and Neighbor Outlier Tools. The ArcMap and ArcPro function analyzes each data point based on the slope of the rendered terrain polygon and the point’s immediate adjacent neighbors. If a sufficient portion of these slopes exceeded a manually pre-defined threshold, the point is flagged as a potential outlier but not removed. After this function has identified all potential outliers, the set is visually reviewed and flagged. Flagged points are manually removed from the terrain but stored as an independent shapefile and thus no data removed from the active dataset were truly deleted.

Used K-natural neighbors, Percentiles, and ArcGIS slope tools to location and remove outliers.

By Raster format download

NA