Client Guide

This is a document that contains helpful tips if you want to interact with the Tracking DB to either get data into the system, or out of the system.

Querying for Positions from the ODSS

Below is a diagram of the tracking data messages flow.

Tracking Message Flow

All of the situational awareness data flow via email processing and the Advanced Message Queuing Protocol into the PostGIS MBARItracking database. Access to this database is provided through the Open Geospatial Consortium standard WMS and WFS protocols. These are fine for delivering rendered maps and GML formatted responses to clients. Server-side editing of Mapserver configuration files is required to customize WMS output. For example, simply changing a symbol size or color in a WMS response requires a change to a mapserver file on the web server. Though we haven't fully explored the WFS capabilities of the Mapserver we expect that similar configurations are required to customize the WFS response. These standards-based protocols are certainly useful, but many clients have a need to simply retrieve data directly from the database and format the data as they see fit. Because the database schema is managed in a GeoDjango project the programming environment to efficiently interact with this database is Python. To expose the information from the database in a more "raw" fashion to other environments another web service besides WMS and WFS is needed.

The solution is to provide a REST-style web service that has a simple request syntax and simple responses provided in a requested format. Django provides a web service framework to deliver this solution. It has been implemented as part of the ODSS project and it provides full visibility into the 3 tables of the Tracking database. All of the timestamped position data from the Position table can be requested with various constrains including:

by platform name
by platform type
the last number of reports
the last reports from the last number of seconds, minutes, hours, days, or weeks
since a specified time
between specified times

In addition a 'stride' option can be specified for any of these position data reports to decimate the response, reducing the number of reports that a client must handle. A 'count' option can also be applied to any of these queries so that a client can adjust the stride option in order to maintain responsiveness. The service does not page output. If the client requests a year's worth of data from a platform that reports once a minute then over half a million records will be returned. It is expected that the client will first perform a count operation and then adjust further requests with stride or finer 'between' requests rather than overwhelm the server (and client) with too much data to process.

In order to expose the data to other environments some rudimentary web services have been set up to directly access the data in the database.

The code (and documentation)

For complete documentation please see the urls.py and views.py source code in SVN.

The urls.py file contains the regular expressions for matching the URL request string.

Example requests

(Note: these services are provided in addition to the existing WMS and WFS services provided by the Minnesota Map Server.)

Here are some example calls of this service (they should be somewhat self explanatory):

https://odss.mbari.org/trackingdb/platformTypes (to get a listing of the platform 'types' that are in the tracking database)

https://odss.mbari.org/trackingdb/platformNames (to get a full list of all the names of the various platforms that are in the database. Note that the numbers that make up the majority of the list are identifiers from AIS beacons on ships in range of our AIS antenna on Mount Toro)

https://odss.mbari.org/trackingdb/platformAssociations.csv (to download a list of platform names and their associated platform types in CSV file format)

The various formats (for last, since, and between) for the position requests are:

position/<platformName>/last/<number>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

position/<platformName>/last/<number><s,m,h,d,w (for seconds, minutes, hours, days, weeks)>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

position/<platformName>/since/<isoStartDateTime>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

position/<platformName>/between/<isoStartDateTime>/<isoEndDateTime>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

positionOfType/<platformTypeName>/last/<number>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

positionOfType/<platformTypeName>/last/<number><s,m,h,d,w (for seconds, minutes, hours, days, weeks)>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

positionOfType/<platformTypeName>/since/<isoStartDateTime>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

positionOfType/<platformTypeName>/between/<isoStartDateTime>/<isoEndDateTime>[/stride/<strideValue>]/data.[csv|html|xkml|kml]

(For the .xkml and .kml formats you can specify 'googlemap' instead of 'data' to get a simple non-timestamped LineString KML representation of the track.)

In order to pass the arguments for the query through the URL the ISO-8601 format has no '-'s or ':'s. The format is '%Y%m%dT%H%M%S'.

KML output

You can replace the format specification, e.g. 'html' or 'csv' with 'kml' to get timestamp tagged KML with the proper MIME type to open the tracks directly in KML to any KML-capable viewer, such as Google Earth. If you specify 'xkml' you will get the kml but with a MIME type of text/html so that the KML will appear in the browser. This is useful for debugging. You may also specify 'googlemap' instead of 'data' in a kml request in order to get a simple LineString KML element without timestamps. This KML response works well for feeding a GoogleMap (or other map) display of the data. For example, for the last example above, you get the KML by using https://odss.mbari.org/trackingdb/position/367766000/last/1d/stride/10/data.kml

Sending Tracking Data to the TrackingDB

Via Email

Warning

Please note that for this to work, the email messages must be in 'plain text' format.

One of the easiest ways to do this is to send an email to one of the following email addresses depending on the type of platform you are sending messages for. This method should not be used for locations that are published frequently (see AMQP method below if that is the case).

For moorings, send the messages to mooringtrack@mbari.org
For ships, send the messages to shiptrack@mbari.org shiptrack@mbari.org
For AUVs, send the messages to auvtrack@mbari.org
For drifters, send the messages to driftertrack@mbari.org
For gliders, send messages to glidertrack@mbari.org

The data is included in the SUBJECT of the message and is of the following format:

platform_name,timestamp,longitude,latitude

where:

platform_name is the name you want the platform to be represented by in the tracking db. NO SPACES are allowed in the name
timestamp is the timestamp of the location fix and is represented as epoch seconds (can have decimal for sub second)
longitude is the longitude of the location fix in decimal degrees.
latitude is the latitude of the location fix in decimal degrees.

An example message might be:

my_platform,1616194802,-121.78619,36.80296

Via AMQP

The messages that come in via email are converted to messages that are then published to an AMQP broker and are fine for slow updates. If you are trying to push messages at a rate that is higher than resonable for email (more than 1 per minute) you should think about publishing those locations to AMQP directly. The messages that come in must be of the format:

platform_type,platform_name,epoch_seconds,latitude,longitude,source,mmsi,imei,iso_datetime

where:

platform_type: is the class of platform that you fall under and is one of the following: auv, ship, drifter, glider, or mooring.
platform_name: is the name you want the platform to be represented by in the tracking db. NO SPACES are allowed in the name
epoch_seconds: is the timestamp of the location fix and is represented as epoch seconds (can have decimal for sub second)
latitude: is the latitude of the location fix in decimla degrees
longitude: is the longitude of the location fix in decimal degrees
source: is a name that you can give to your source of this locations. It's ususally the name of the software that is publishing the messages and helps in identifying where the messages are coming from on the server side.
mmsi: is the Martime Mobile Service Identity for the agent responsible for the location fix
imei: is the International Mobile Equipment Identity number for the agent responsible for the location fix
iso_datetime: is the ISO 8601 formatted timestamp for readability.

Note

Even though you specify the platform_type in the message itself, in your client software you need to push the message to the correct exchange. The exchange names are exactly the same as the type names and are one of the following: auv, ship, drifter, glider, or mooring.

NodeJS Example

For NodeJS, you must have a base install of NodeJS first, then you want to create a new project with

npm init

After filling out the init prompts, you will have a package.json file in your directory. Then run:

npm install amqp

Then edit a new file called index.js and put in the following

    // Require the AMQP library
    var amqp = require('amqp');

    // Read in the environment variables
    var AMQP_HOST = process.env.AMQP_HOST
    var AMQP_VHOST = process.env.AMQP_VHOST
    var AMQP_USERNAME = process.env.AMQP_USERNAME
    var AMQP_PASSWORD = process.env.AMQP_PASSWORD

    var connection = amqp.createConnection(
        {
            host: AMQP_HOST,
            port: '5672',
            login: AMQP_USERNAME,
            password: AMQP_PASSWORD,
            connectionTimeout: 0,
            authMechanism: 'AMQPLAIN',
            vhost: AMQP_VHOST,
            ssl: { enabled: false }
        }
    );

    // Grab the current date and time
    const now = new Date()
    const secondsSinceEpoch = Math.round(now.getTime() / 1000)

    // Construct the message
    var shipMessage = 'ship,testship,' + secondsSinceEpoch + ',36.80296,-121.78619,tracking-db-client-test-node,,,' + now.toISOString();

    // Wait for connection to become established.
    connection.on('ready', function () {
        console.log("Connection ready ... ");

        // Grab the exchange
        var exchange = connection.exchange('ships', {
            type: 'fanout',
            durable: true,
            autoDelete: false,
            confirm: true
        });

        // Handle the open event
        exchange.on('open', function () {
            console.log("Exchange open");
            exchange.publish('', shipMessage, {}, function (err) {
                if (err) {
                    console.log('Error publishing message!');
                    console.log(err);
                    return;
                } else {
                    console.log("published OK");
                }
                connection.end();
                connection.destroy();
            });
        });
    });

    // Handle an error
    connection.on('error', function (error) {
        console.log("Error on connection", error);
    });

Python Example

The first thing you need to do is get the proper Python and supporting libraries installed. For this example, I am starting with a bare 2.7.15 install and then you need to install the amqp library by using:

pip install amqplib

Once that is done, you an use the following example code to publish a message

    # First import the necessary libraries
    import os
    import datetime
    import amqplib.client_0_8 as amqp

    # Grab the necessary configuration from the environment
    AMQP_HOST = os.getenv('AMQP_HOST')
    AMQP_VHOST = os.getenv('AMQP_VHOST')
    AMQP_USERNAME = os.getenv('AMQP_USERNAME')
    AMQP_PASSWORD = os.environ.get('AMQP_PASSWORD')
    print('Will send messages to host ' + AMQP_HOST + ' to virtual host ' + AMQP_VHOST + ' as user ' + AMQP_USERNAME)

    # Grab the current datetime
    now = datetime.datetime.now()
    now_iso = now.isoformat()
    now_ts = now.timestamp()

    # Construct a message body
    msg_body = 'ship,testship,' + str(now_ts) + ',36.80296,-121.78619,tracking-db-client-test,,,' + now_iso
    print(msg_body)

    conn = amqp.Connection(AMQP_HOST, userid=AMQP_USERNAME, password=AMQP_PASSWORD, virtual_host=AMQP_VHOST)

    ch = conn.channel()

    msg = amqp.Message(msg_body, content_type='text/plain')

    ch.basic_publish(msg, 'ships')

    ch.close()
    conn.close()