Skip to content

ODSS Operations Guide

How-Tos

Managing ODSS Services

If you need to stop/start/restart the ODSS services, you first need access to the server normandy8.mbari.org. Once you can ssh into the server, first switch to the odssadm account by running

su odssadm

(you will need to get the password from Information Engineering). Once you have switched to the odssadm account, type

cd

This will put you in the odssadm home directory.

All the ODSS services are managed by systemctl and are a mix of podman containers and local NodeJS processes. To get the stats of the ODSS services, you can run:

systemctl --user status container*

And that will show you the list of the backing services of the ODSS which are:

container-erddap.service
container-mapserver.service
container-mbaritracking.service
container-mongodb.service
container-odss-utils.service
container-thredds.service
container-tracking-clients.service

The ODSS server itself is a service named

odss-server.service

and you can check the status of that service using systemctl --user status odss-server.service:

[odssadm@normandy8 user]$ systemctl --user status odss-server.service
● odss-server.service - The ODSS web application server
Loaded: loaded (/home/odssadm/.config/systemd/user/odss-server.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-10-02 07:47:40 PDT; 1 day 8h ago
    Docs: https://odss.mbari.org
Main PID: 5702 (node)
CGroup: /user.slice/user-11290.slice/user@11290.service/odss-server.service
        └─5702 /usr/bin/node /home/odssadm/odss/webapp/server/app.js

You can manage the services using systemctl --user [start|stop] service-name.

Most of the time when the system reboots, the services should all start by themselves, but if not, you can manually start/stop them.

Note

All the services except for the main ODSS server are running in podman containers. You should be able to see the list of the containers by running podman ps and you should see all the services running. The ODSS server itself is a NodeJS process running directly on the host and you can see if it's active by running ps -ef | grep node and you should see a process named /usr/bin/node /home/odssadm/odss/webapp/server/app.js.

How to pull platform positon data from the ODSS

A common request is to get location data directly from the ODSS. The position are stored in the Tracking Database, but the locations can also be requested from the ODSS through a simple URL call. In order to use this service, first bring up the ODSS.

Figure1

Open the data pane on the left side of the ODSS and in the Platforms section, scroll to the platform you are interested in getting data from. To the right of the platform name, click on the squiggly line with dots.

Figure2

this will open a new window and take note of the 'platformID' from the URL in the web page address (see red box in image below). It is probably easiest to copy everything after 'platformID=', but before the first '&'. This is the ID of the platform you will want to query for location data from.

Figure3

Next, you can create a URL and make requests for platform location data by using the following URL form:

https://odss.mbari.org/odss/tracks?platformID=xxxxxxxxxxxxxxxxxxxxxxxxx&returnSRS=4326&lastNumberOfFixes=yy&returnFormat=html

Note:

  1. The lastNumberOfFixes can be increased to go farther back in time.
  2. If you want to specify a time window, you can use startDate and endDate instead of the lastNumberOfFixes. The format for the dates is ISO and is of the form YYYY-MM-DDTHH:mm:ssZ. Here is an example of a request for data from a 1 hour window of time https://odss.mbari.org/odss/tracks?platformID=54065b5560d0e168c88d403e&returnSRS=4326&startDate=2023-08-03T12:00:00Z&endDate=2023-08-03T13:00:00Z&returnFormat=html
  3. The returnFormat can be any one of:
    1. HTML
    2. CSV
    3. JSON
    4. KML
    5. KMLTrack

How to add WMS Map from ERDDAP server to the ODSS

While there are plans to change the ODSS in the future to make this easier, right now, if you want to add a WMS layer from an ERDDAP server to the ODSS. You would take the following steps (I am going to use Coastwatch's Pseudo-Nitzschia layer in this example).

  1. Go to the ERDDAP server and find the data set you want to add (must have WMS capability)

    https://coastwatch.pfeg.noaa.gov/erddap/info/index.html?page=1&itemsPerPage=1000

  2. Click on the WMS of data set of interest (example is charmForecast1day_LonPM180)

    https://coastwatch.pfeg.noaa.gov/erddap/wms/charmForecast1day_LonPM180/index.html

  3. Sometimes they have multiple layers available. In this case:

    pseudo_nitzschia particulate_domoic cellular_domoic chla_filled r555_filled r488_filled

  4. Figure out which layer you want (example is pseudo_nitzschia)

  5. Now to set this up, there are 4 pieces that need to be put in place:
    1. Create a mapserver mapfile template that will be used to generate the mapserver file
    2. Update the mapdatasets.json file to start downloading subsets of the ERDDAP source data and generated the mapserver mapfiles
    3. Update the datasets.xml file in the ERDDAP server to point to the new dataset that is being downloaded.
    4. Create a layer in the ODSS MongoDB database to point to this new layer.

  6. For the first step, let's create the mapserver template file.

    1. I ssh into the odss server and cd to /data/mapserver/mapfiles.
    2. I either cd into an existing directory, or create a new directory if one is needed. These directories are just to help organize all the different mapserver files. For this example, the coastwatch directory already existed, so I cd'd into it.
    3. For this example, I copied a working mapfile template that was from a similarly structured ERDDAP data set using cp .chl1day.layer.map .charm1day.layer.map
    4. I then edited the .charm1day.layer.map file to look like this:
      LAYER
    NAME 'charmForecast1day:pseudo_nitzschia'
    TYPE RASTER
    STATUS ON

    CLASS
    NAME   "charmForecast1day:pseudo_nitzschia"
    KEYIMAGE   "legend-icons/charmForecast1day0.0pseudo_nitzschia.png"
    END

    CONNECTION 'http://odss-test.shore.mbari.org/erddap/wms/charmForecast1day/request?TIME=%TIME%&ELEVATION=%ELEVATION%'

    CONNECTIONTYPE WMS

    PROJECTION
        'init=epsg:4326'
    END

    METADATA
    'wms_title'           'charmForecast1day:pseudo_nitzschia'
    'wms_name'            'charmForecast1day:pseudo_nitzschia'
    'wms_onlineresource'  'http://localhost/cgi-bin/mapserv?map=/data/mapserver/mapfiles/odss.map'
    'wms_server_version'  '1.1.0'
    'wms_srs'             'EPSG:4326'
    'wms_format'          'image/png'
    'wms_transparent'     'true'
    END

    VALIDATION
    'default_time'        'default_start_time'
    'default_elevation'   '0.0'
    'TIME'          '.'
    'ELEVATION'     '.'
    END
END

  7. I edited the /data/mapserver/mapfiles/odss.map file and added the include statement INCLUDE "coastwatch/charm1day.layer.map"

  8. For the second step, I cd'd to the /data/utils directory and edited the mapdatasets.json file and added the following dataset (note the 'flagKey' is left empty for now)

    {
    "minlegend": "",
    "maxlegend": "",
    "layeroffsethours": 0,
    "incrhours": 24,
    "legendprefix": "",
    "haslegend": true,
    "numdims": 3,
    "download": true,
    "outdir": "/data/erddap/data/coastwatch",
    "url": "http://coastwatch.pfeg.noaa.gov/erddap/griddap/charmForecast1day_LonPM180.nc?",
    "mapfile": "/data/mapserver/mapfiles/coastwatch/charm1day.layer.map",
    "lonwrap": false,
    "datatype": "pseudo_nitzschia",
    "erddapid": "charmForecast1day",
    "erddapidsrc": "charmForecast1day_LonPM180",
    "flagKey": "",
    "startdelayhours": 24,
    "timeformat": "%Y-%m-%dT:%H:%M:%SZ",
    "querytimeformat": "%Y-%m-%dT12:00:00Z",
    "latMin": 31.3
}
    1. Now in order for the scripting container to pick up those changes, it needs to be restarted using: systemctl --user stop container-odss-utils and then systemctl --user start container-odss-utils. After restarting the container, wait until the script runs (at every quarter of the hour).

    2. Once the download script runs and there is a NetCDF file in the /data/erddap/data/coastwatch directory, we need to add the data set to ERDDAP. Do this by editing the /data/erddap/content/erddap/datasets.xml file and adding it. One thing to note, is that you can generate a starting entry for the datasets file by doing the following:

      1. podman exec -it erddap /bin/bash
      2. cd webapps/erddap/WEB-INF/
      3. bash GenerateDatasetsXml.sh -verbose
      4. EDDGridFromNcFilesUnpacked
      5. /data/erddap/data/coastwatch/charmForecast1day
      6. .*\.nc
      7. /data/erddap/data/coastwatch/charmForecast1day/charmForecast1day_20211013_120000.nc

    3. This will generate a bunch of stuff and among that will be the base dataset XML snippet. If you clean it all up to it's minimum you get something like this:

      <dataset type="EDDGridFromNcFilesUnpacked" datasetID="charmForecast1day" active="true">
    <reloadEveryNMinutes>10080</reloadEveryNMinutes>
    <updateEveryNMillis>10000</updateEveryNMillis>
    <fileDir>/data/erddap/data/coastwatch/charmForecast1day/</fileDir>
    <fileNameRegex>.*\.nc</fileNameRegex>
    <recursive>true</recursive>
    <pathRegex>.*</pathRegex>
    <metadataFrom>last</metadataFrom>
    <matchAxisNDigits>20</matchAxisNDigits>
    <fileTableInMemory>false</fileTableInMemory>
    <addAttributes>
    </addAttributes>
    <axisVariable>
        <sourceName>time</sourceName>
        <destinationName>time</destinationName>
        <addAttributes>
        </addAttributes>
    </axisVariable>
    <axisVariable>
        <sourceName>latitude</sourceName>
        <destinationName>latitude</destinationName>
        <addAttributes>
        </addAttributes>
    </axisVariable>
    <axisVariable>
        <sourceName>longitude</sourceName>
        <destinationName>longitude</destinationName>
        <addAttributes>
        </addAttributes>
    </axisVariable>
    <dataVariable>
        <sourceName>pseudo_nitzschia</sourceName>
        <destinationName>pseudo_nitzschia</destinationName>
        <dataType>float</dataType>
        <addAttributes>
            <att name="long_name">Probability of Pseudo-nitzschia &gt; 10,000 cells/L</att>
        </addAttributes>
    </dataVariable>
</dataset>

  9. Once this is saved, you can restart the ERDDAP container using systemctl --user stop container-erddap and systemctl --user start container-erddap

  10. After it starts up if you visit the base ERDDAP url, it will kick off the indexing of the datasets. It should then eventually show up in the ERDDAP catalog. Also, you should get an email that contains a summary of the data sets. You will need to look at the bottom of that summary and grab the 'flagKey' for the newly added dataset.
  11. Edit the /data/utils/mapdatasets.json file and add the flag key to the JSON for the new dataset and restart the utils to pick up the changes using systemctl --user stop container-odss-utils and then systemctl --user start container-odss-utils

  12. Now in MongoDB, go to layers collection, copy an existing working layer and create new document. For this example, it ended up being

    {
    "_id" : ObjectId("616858f97e53d4155e1bb42e"),
    "legendurl" : "/cgi-bin/mapserv?MAP=/data/mapserver/mapfiles/odss.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getlegendgraphic&FORMAT=image%2Fpng&LAYER=charmForecast1day:pseudo_nitzschia&RULE=charmForecast1day:pseudo_nitzschia",
    "name" : "Pseudo-Nitzschia CHARM 1-day Forecast",
    "options" : {
        "timeformat" : "UTC:yyyy-mm-dd'T'12:00:00'Z'",
        "hoursbetween" : 24.0,
        "minhoursback" : 24.0
    },
    "params" : {
        "map" : "/data/mapserver/mapfiles/odss.map",
        "service" : "WMS",
        "version" : "1.1.1",
        "request" : "GetMap",
        "srs" : "EPSG:4326",
        "exceptions" : "application/vnd.ogc.se_iniimage",
        "layers" : "charmForecast1day:pseudo_nitzschia",
        "elevation" : "0.0",
        "transparent" : "true",
        "bgcolor" : "0x808080",
        "format" : "image/png"
    },
    "type" : "wms",
    "url" : "/cgi-bin/mapserv?",
    "views" : [
        {
            "name" : "Public",
            "folder" : "Biology"
        }
    ]
}

  13. It should then show up in the ODSS and the layer and the legend should work.

  14. WHEW!!!

How to add Mapserver Geometry to the ODSS

Often times, people want to add a fixed location or geometry to the ODSS and this can be done by creating a 'layer' using something like KML. This section documents how to create a simple layer that can be added to the ODSS.

  1. First, ssh into the ODSS machine ssh user@odss.mbari.org
  2. The easiest thing to do is use an existing file that is something similar to what you want to add. For this example, I will be creating a location for the AUV docking buoy using the file that is used as the fixed location for the MARS node. Once, ssh'd in, I changed directory to /data/mapserver/mapfiles. For this instance, I created a new directory called buoys.
  3. I changed into the buoys directory and then copied the example files using cp ../mars/mars.kml ./auv-dock.kml and cp ../mars/mars.layer.map ./auv-dock.layer.map.

  4. I then edited the two auv-dock files to update with the information that I wanted for the new layer (updated location, name, etc.). The final auv-dock.kml looked like:

    <?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">
    <Folder>
        <name>AUV Dock Location</name>
        <open>1</open>

        <Document>
            <name>point</name>
            <Placemark>
                <name>AUV Dock</name>
                <Point>
                    <coordinates>-121.861932,36.777049,0</coordinates>
                </Point>
            </Placemark>
        </Document>
    </Folder>
</kml>

  5. And the final auv-dock.layer.map looked like:

    LAYER
    NAME 'auvdock'
    TYPE POINT

    CONNECTIONTYPE OGR
    CONNECTION '/data/mapserver/mapfiles/buoys/auv-dock.kml'

    DATA "point"
    LABELITEM "name"

    CLASS
    NAME 'AUV Dock'
    STYLE
        SYMBOL "circle"
        SIZE 8
        OUTLINECOLOR 0 0 0
        COLOR 229 114 50
    END
    LABEL
        FONT "sans"
        TYPE truetype
        POSITION AUTO
        SIZE 8
        COLOR 255 255 255
        OUTLINECOLOR 229 114 50
        FORCE TRUE
        PARTIALS FALSE
    END
    END

    PROJECTION
    'proj=longlat'
    'datum=WGS84'
    'no_defs'
    END

    METADATA
    'wms_title'           'auvdock'
    'wms_name'            'auvdock'
    'wms_onlineresource'  'http:/localhost/cgi-bin/mapserv?map=/data/mapserver/mapfiles/odss.map'
    'wms_server_version'  '1.1.0'
    'wms_srs'             'EPSG:4326'
    'wms_format'          'image/png'
    'wms_transparent'     'true'
    END

END

  6. After getting those two files set up, I then needed to add them to the master odss.map file. I edited the /data/mapserver/mapfiles/odss.map file and added the following section near the bottom:

    # Layers for various fixed buoys
INCLUDE "buoys/auv-dock.layer.map"

  7. That is all that needs to be done to add the layer to the mapserver so that it can be served up by Mapserver. Now, however, it needs to be added to the ODSS catalog in MongoDB. There are different ways to do this, but ultimately, the document looked like this:

    {
    "_id":{"$oid":"6241f065c48be703760a61fd"},
    "name":"AUV Dock",
    "type":"wms",
    "url":"/cgi-bin/mapserv?",
    "views":[
        {
            "_id":{
                "$oid":"6241f0c13d9fd5357886587b"
            },
            "name":"Public",
            "folder":"Locations"
        }],
    "params":{
        "map":"/data/mapserver/mapfiles/odss.map",
        "service":"WMS",
        "version":"1.1.1",
        "request":"GetMap",
        "srs":"EPSG:4326",
        "exceptions":"application/vnd.ogc.se_iniimage",
        "layers":"auvdock",
        "transparent":true,
        "bgcolor":"0x808080",
        "format":"image/png"
    },
    "options":{
        "timeformat":"UTC:yyyy-mm-dd'T'00:00:00'Z'",
        "hoursbetween":1,
        "minhoursback":1
    },
    "__v":1,
    "_ver":2
}

  8. The layer then shows up in the ODSS under the 'Layers->Location' folder and can be turned on/off.

Warning

In order for the layers to show up in the ODSS, the following must be identical: 'layers' name in MongoDB entry; the 'wms_title' in the .layer.map file; the 'wms_name' in the .layer.map file; and the top leve 'NAME' in the .layer.map file. Also note that the text in the tag in under the tag in the KML is what will show up on the map label in the ODSS.

How to add a NOAA Chart to the ODSS

NOAA has a large number of nautical charts that are very useful in the ODSS. There are a few steps that need to be done to add one of these layers. They are raster images and the first step is to find the one you are looking for. You can find them here to download:

Raster Navigational Charts

Usually, you likely want one of the west coast ones. For this example, you click on the CA link to download a .zip file. You then unzip the file which will create a BSB_ROOT directory. Each of the subdirectories represent one chart. For this example, we will add the chart around Santa Barbara that is numbered 18720. Assuming you have the GDAL tools installed on your machine (I used homebrew to install on my Mac), you can convert the default .KAP files to tiled GeoTIFFs using:

gdal_translate -of GTiff -co "TILED=YES" 18720_1.KAP 18720_1_tiled.tif

This creates a tiled GeoTIFF file, but it still needs to be converted to the 4326 projection which can be done using:

gdalwarp -t_srs EPSG:4326 18720_1_tiled.tif 18720_1_tiled_wgs.tif

The 4326 projection file then needs to be copied to the ODSS data directory. It is easiest to do this by mounting the ODSS Atlas share to you machine and then navigating to the /data/mapserver/mapfiles directory. The subdirectory you place the file in will sort of depend on the purpose of the file and is really a matter of convenience for organizing files. For this example, I chose the already existing noaacharts directory and placed the 18720_1_tiled_wgs.tif file in that directory. I then created a new file in the same directory and named it 18720.layer.map file and put the following contents:

LAYER
    NAME 'NOAANavigationalChart:18720Public'
    TYPE RASTER
    STATUS ON

    DATA '/data/mapserver/mapfiles/noaacharts/18720_1_tiled_wgs.tif'

    PROJECTION
    'init=epsg:4326'
    END

    #DEBUG 5

    METADATA
    'wms_name'            '18746'
    'wms_server_version'  '1.1.1'
    'wms_srs'             'EPSG:4326'
    'wms_format'          'image/png'
    'wms_transparent'     'true'
    END

END

This file describes the data in the NOAA chart file so that mapserver knows how to interpret the tif file. Now, in order to add this file to the mapserver configuration, you need to edit the odss.map file that is in the /data/mapserver/mapfiles directory and add a line like this:

  INCLUDE "noaacharts/18720.layer.map"

This then tells mapserver to include the layer defined in that file. The last step is to register this new layer with the ODSS. Right now, the easiest way to do this is to add a new document to the MongoDB ODSS database. I use a tools called Studio 3T to interact with the MongoDB (connecting up is not described here). Whatever way you choose to do this, you need to add a new document to the ODSS database in the layers collection that looks something like this:

{
    "_id" : ObjectId("628293673a969c667a83f128"),
    "name" : "SB NOAA Nav Chart",
    "type" : "wms",
    "url" : "/cgi-bin/mapserv?",
    "params" : {
        "map" : "/data/mapserver/mapfiles/odss.map",
        "layers" : "NOAANavigationalChart:18720Public",
        "transparent" : "true",
        "singleTile" : "true"
    },
    "options" : {

    },
    "__v" : NumberInt(5),
    "views" : [
        {
            "_id" : ObjectId("6282e18ebc5c1015fc97958b"),
            "name" : "Public",
            "folder" : "2022 ECOHAB"
        }
    ],
    "_ver" : NumberInt(4)
}

Please note the "_ver", "__v", and the "_id" are added by the MongoDB server, they are not added by you. Also, the ObjectId for the view will need to be taken from your database instance (you can also add the layer to a view through the ODSS interface directly).