Skip to content

Waveglider Data Extraction

In order to extract data related to the Wavegliders, we leverage the data service from Liquid Robotics known as their 'Data Service'. It is basically a set of SOAP calls to retrieve JSON payloads that contain the data we are interested in extracting.

The extraction code for this lives in the BitBucket repository here and is currently running on the machine named wg-data.shore.mbari.org. You can consult the main README.md for notes on development and deployment of the data extraction service.

Operational Notes

The extraction service is currently running under the 'kgomes' account, but is located in a spot where others can edit the configuration. There is really no code checkout since the entire stack is being run in Docker. If you change into the /opt/wg-portal directory, you will see three files:

  1. .env: This is the environment configuration file for the application configuration for Docker. It contains usernames and passwords for the docker images.
  2. docker-compose.yml: This is the Docker compose files that tells Docker how to configure and run the images that make up the application. There are basically two containers.
    1. The first container is a Postgres database which is used to store all the data that we extract from the data service. The data for the postgres database is actually stored on the host in the /data/postgis directory which is then mounted into the Postgres docker container.
    2. The second container is the waveglider extraction Python code that is connecting to the Liquid Robotics Data Service and then writing the data to the Postgres DB. The Python code also publishes the locations of the Wavegliders to the ODSS. The only external data for the extraction image is the logs which are written to /data/logs.
  3. wg-portal.json: This file configures the extraction process and how it pulls data from the data service.

In order to configure how the extraction runs, you will need to edit the /opt/wg-portal/wg-portal.json file.

Renaming Wavegliders in the ODSS

The Liquid Robotics Data Service has no concept of the names we want to use for our data systems here at MBARI. For that reason, we 'map' the names to something that is more meaningful to us. That happens in the wg-portal.json file, but once the image starts up, that same data gets written to the database table gliders. Basically there is a vehicle id (vid) and then a corresponding mapped name that we call tracking_alias. In order to rename a vehicle once data is already flowing, you need to stop the docker service, edit the wg-portal.json file and update the database so they match, then restart the docker containers. As an example, in November of 2023, we wanted to make the following changes:

  1. wgSV3006 needed to be renamed to wgHansen
  2. wgSV3110 needed to be renamed to wgTiny
  3. wgSV3172 needed to be renamed to wgDevTest

First I queried the tracking database to see what data has been logged for these vehicles to date.

  1. For wgSV3006, it had a tracking database ID of 16190.
    1. In the gliders table on wg-data.shore.mbari.org, there were two numbers associated with the tracking alias wgSV3006 and they were:
      1. 1338599832
      2. 2027666087
    2. This tracking name alias was also listed in the wg-portal.json file as well
    3. In the tracking database, the data from this vehicle goes back to October 27, 2023 so I think we should move all of that data to wgHansen as well.
  2. For wgSV3110, it had a tracking database ID of 15281
    1. In the gliders table on wg-data.shore.mbari.org, there was one number associated with the tracking alias wgSV3110 and it was:
      1. 466563798
    2. There was already an association between 466563798 and 'wgTiny' in the wg-portal.json so no change made
    3. There was no data in the tracking database for this controller so just renaming should suffice.
  3. For wgSV3172, it had a tracking database ID of 15870
    1. In the gliders table on wg-data.shore.mbari.org, there were two numbers associated with the tracking alias wgSV3172 and they were:
      1. 1993437788
      2. 163770185
    2. This tracking name alias was also listed in the wg-portal.json file as well
    3. In the tracking database, there was a period of position publishing in July that should be moved to the new name.

So, in order to do this move. I took the following steps.

  1. ssh'd into the wg-data.shore.mbari.org machine
  2. cd /opt/wg-portal
  3. docker ps (to make sure I can see the running containers)
  4. docker compose stop wg-portal
  5. Edited the wg-portal.json file:
    1. Changed the name 'wgSVS3006' to 'wgHansen' (two places)
    2. Left the '466563798' association to 'wgTiny' that was already there alone
    3. Changed the name 'wgSV3172' to 'wgDevTest' (two places)
  6. In the postgres database on wg-data.shore.mbari.org, I changed the tracking alias:
    1. I changed 'wgSV3006' to 'wgHansen' in two places
    2. I changed 'wgSV3110' to 'wgTiny' in two places
    3. I changed 'wgSV3172' to 'wgDevTest' in two places
  7. Restarted the wg-portal using docker compose start wgportal
  8. After the first set of publishing happened, there new positions the tracking database associated with the platform 'wgHansen' with an id of 15156
  9. So, in order to move all the data in the tracking database from 'wgSV3006' to 'wgHansen', I needed to update all the positions with platform_id of 16190 to the ID 15156. I did this using DataGrip. Now all the data from wgSV3006 is now associated with wgHansen in the ODSS and new positions are flowing in as wgHansen.
  10. Since no data has been flowing for what is now wgDevTest, there is no entry in the tracking database for that platform so there is nowhere to move the data from wgSV3172 to. There was only a couple of days of test data, so I am not going to worry about moving it.