Skip to content

ODSS Test Installation

CentOS 8

This page documents the steps taken during the installation and migration of the ODSS from the machine odss-test to the machine odss-test8. This was necessary because odss-test was CentOS 6 which was end of life so we needed to move to CentOS 8. In addition, we moved the operational deployment of the ODSS to a docker-based deployment to make management easier. Docker is actually 'podman' on CentOS 8 so that was part of the migration as well (moving from Docker to Podman).

Preparation

Before starting this work, I wanted to make sure I knew what the existing URLs were so that we could make sure they still worked after migration. Right now, the relative URLs that are operating are:

  1. / -> redirects to /odss
  2. /odss is the web application (proxypass :3000/odss)
  3. /odss/elevation (just part of the ODSS application)
  4. /odss/services/catalog/datasource (just part of the ODSS application)
  5. /odss/tracks is the tracking DB pulls (just part of the ODSS application)
  6. /tilecache/ (alias defined in /etc/httpd/custom/tilecache.conf)

Basically, there are several components that need to be installed for the ODSS to operate:

  1. THREDDS server
    1. URL: /thredds (proxpass from :8280/thredds)
    2. This is linked to the /data/thredds directory by running THREDDS with a property:
      1. -Dtds.content.root.path=/data (it must add thredds automagically)
    3. This means, in theory, I could copy the /data/thredds data over, run THREDDS (same version) in docker, point it to a directory in the thredds image and then mount the external directory to that location.
    4. Need to check and see if there are any scripts that are copying remote data down
  2. ERDDAP server
    1. URL: /erddap (proxypass from :8180/erddap)
    2. This is linked to the /data/erddap directory by running erddap with a property:
      1. -DerddapContentDirectory=/data/erddap/content/erddap/
    3. So this means, in theory, that I could just run a docker version of ERDDAP that is the same as the one running now, then use this property to point to a directory in the docker image where I have this same data directory mounted. This means, I might be able to just copy over the content from the current test machine to the new one.
    4. Need to check and see if there are any scripts that are copying remote data down
  3. Mapserver (with tilecache)
    1. /mapserver is the URL that responds to this and I think it is a CGI-exec
  4. MongoDB
  5. TrackingDB (with consumers)
    1. On ships, this need push to Winfrog
  6. On ships, script that pulls select AIS data
  7. On ships, script that pulls tracking data and builds TimeZero files
  8. Sync scripts between ships and shore

From a data storage perspective, I have a /data directory which has the following subdirectories:

  1. canon (these are the ‘repos’ that show up in the data sections. In the views, the repoBasePath is ‘canon/xxx’)
  2. logs
  3. mapserver
  4. mongo
  5. thredds
  6. tilecache-2.11
  7. tmp

I would really like to use docker for all this, which for CentOS 8 is podman. Fortunately, I have all the instructions that I kept for the esp-portal which was all podman stuff.

Steps

  1. Submitted a ticket to create a new VM named odss-test8.shore.mbari.org. Here is the help ticket response:

    VM deployed
podman installed and tested
Installed:
    apr-1.6.3-9.el8.x86_64
    apr-util-1.6.1-6.el8.x86_64
    apr-util-bdb-1.6.1-6.el8.x86_64 
    apr-util-openssl-1.6.1-6.el8.x86_64
    centos-logos-httpd-80.5-2.el8.noarch  
    httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64
    httpd-filesystem-2.4.37-21.module_el8.2.0+382+15b0afa8.noarch 
    httpd-tools-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64
    mod_http2-1.11.3-3.module_el8.2.0+307+4d18d695.x86_64

Enabled and started the httpd service [WHD #29331]
Changed /usr/lib/systemd/system/http.service to start after network is online [WHD #29403]
Added ports 80,443,8080,8081,8082,5432,5984 to the firewall [WHD #29469]
Installed git [WHD #29431]
Gave kgomes the same sudo privs as on esp-portal
I created the local podman_user account using the UID of the AD DockerOdssTest user (11618).
Ran the command "loginctl enable-linger kgomes" to enable Kevin to start / stop user level services.

  2. Had Pat add an ‘odssadm’ local account so I can run everything under that. I switched over to that account using ‘sudo -u odssasdm -i’.

  3. In order to get the user level systemctl to work correctly, in the .bashrc file for odssadm, I added:

    export XDG_RUNTIME_DIR=/run/user/$(id -u)

  4. The first thing to do is to set up a pod that all of the individual services will be deployed into. One thing you need to know beforehand is the ports that will need to be open outside the pod. For the ODSS, you basically need:

    1. 5432 for the tracking DB and this should be available to anybody coming from a 134.89.* address.
    2. 27107 for MongoDB and again, this should be available to anybody coming from a 134.89.* address.
    3. 8082 for the ERDDAP service. This will be made available to the world on /erddap through apache. I used 8082 for ERDDAP due to the fact that mapserver will only work on 80 or 8080 so I needed to move ERDDAP to 8082.
    4. 8081 for the THREDDS service. This will be made available to the world on /thredds through apache.
    5. 8080 for the mapserver service. This one might be a bit tricky. Normally, you would be calling /cgi-bin/mapserver since it’s an executable, but with a docker image you should be able to map the 80 port to something like /mapserver, but that means I have to change all the URLs in mapserver configs and in MongoDB. Joe successfully mapped it to /cgi-bin/mapserver. One thing to note is that I used port 8080 on this as that is the only port available other than 80 which does not work in this configuration.
    6. 3000 for the ODSS itself.

  5. To set up the pod, I ran:

    podman pod create -p 5432:5432 -p 27017:27017 -p 8080:8080 -p 8081:8081 -p 8082:8082 -p 3000:3000 --name=odss

  6. That returned and ID of:

    63be67052a4e08b732820e25f101a8cc1b82f4c7a4d350af661dad9e665a22a9

  7. Next, I needed to create the base level directory for all the data that will be stored and manipulated by the various containers. I created the /data directory.

  8. Now, for this to have the least amount of downtime, the MongoDB and PostGIS tracking DB should be migrated last.

  9. ERDDAP

    1. The first thing to do was to move the data for ERDDAP from the current server to the new one. In order to do that, I ssh’d into odss-test and cd’d to /data and then created (using sudo) the /data/tmp/erddap-backup directory and then ran:

      tar -cvf tmp/erddap-backup/erddap-backup.tar erddap

    2. This created a very large tar file in the /data/tmp/erddap-backup directory.

    3. Over on the new server, I scp’d that file over using:

      scp odssadm@odss-test.shore.mbari.org:/data/tmp/erddap-backup/erddap-backup.tar /data/erddap/bigParentDirectory/erddap-backup.tar

    4. Then, on the new server side, I extracted the tar contents using tar -xvf

    5. Now, I am changing the directory structure a bit to keep things cleaner and separated better since I know a little more about ERDDAP now. The top level structure on the new server will be /data/erddap with following subdirectories:
      1. content: This contains the erddap/(datasets.xml|setup.xml) files
      2. data: This contains all the actual data files that we use for our ERDDAP server
      3. bigParentDirectory: This is where all the other files go and will be where I tell ERDDAP to look for this in the setup.xml file.
    6. After moving the data files to the locations where I want them, I need to edit the /data/erddap/content/erddap/setup.xml file and change the bigParentDirectory to point to /data/erddap/bigParentDirectory and changed the baseUrl to point to odss-test8 instead of odss-test for now for testing.

    7. Now, everything should be set up to run the ERDDAP docker container. In order to do this, I needed to override the server.xml file for the tomcat server so that I could run the ERDDAP server on port 8082. I created a erddap-server.xml file in the odss-components directory in the odssadm home. That file looks like the following

      <?xml version="1.0" encoding="UTF-8"?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector server="Apache" secure="true" port="8082" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               relaxedPathChars='[]|'
               relaxedQueryChars='[]|{}^&#x5c;&#x60;&quot;&lt;&gt;'
    />
    <Connector server="Apache" secure="true" port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"><CredentialHandler className="org.apache.catalina.realm.MessageDigestCredentialHandler" algorithm="SHA" /></Realm>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

    8. In order to create a new image with this, I needed a new Dockerfile so I created a file named ERDDAP-Dockerfile and entered the following in the file:

      # This creates a Docker image from ERDDAP that overrides the server.xml file

# First define the base image that this will build from
FROM axiom/docker-erddap:2.02

# Now overwrite the server.xml file so that we can change the port
ADD erddap-server.xml /usr/local/tomcat/conf/server.xml

    9. I had all I needed to create the new image by running:

      podman build -t mbari/odss-erddap -f ERDDAP-Dockerfile

    10. Then I could create the container using

      podman create --security-opt label=disable --name erddap --pod odss -v /data/erddap/content/erddap:/usr/local/tomcat/content/erddap -v /data/erddap/bigParentDirectory:/data/erddap/bigParentDirectory -v /data/erddap/data:/data/erddap/data mbari/odss-erddap

    11. This returned an ID of:

      b33be9838229da602e1e54fe04db2ce261f5a001221074ec15a739c876560b3f

    12. Now, to run this as a service, we need to create the systemd file by running:

      podman generate systemd --name erddap > ~/.config/systemd/user/container-erddap.service

    13. I then edited the .service file so it looks like:

      # container-erddap.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 12:19:44 PST 2020
# Edited by kgomes 2020-11-30: Updated description, changed Restart to always

[Unit]
Description=Container that runs the ERDDAP server for the ODSS
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start erddap
ExecStop=/usr/bin/podman stop -t 10 erddap
ExecStopPost=/usr/bin/podman stop -t 10 erddap
PIDFile=/run/user/11775/overlay-containers/b33be9838229da602e1e54fe04db2ce261f5a001221074ec15a739c876560b3f/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    14. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-erddap.service

    15. You can then start and stop the service using:

      systemctl --user start container-erddap.service
systemctl --user stop container-erddap.service

    16. And check the status of the service using:

      systemctl --user status container-erddap.service

    17. I got an error because the container could not seem to write to the /data/erddap/bigParentDirectory so I opened it up wide open using ‘chmod a+w -R /data/erddap/bigParentDirectory’. That worked, but probably not ideal and will probably have issues on the machine with the mount. We will see. One option is to put the bigParentDirectory on local disk, that should improve performance too (lucene).

    18. It also failed because the setup.xml’s aren’t correct because of the version update. The first error I got was: ‘adminInstitutionUrl is null or ''’. So I edited the setup.xml file. Hmmm, interesting, I don’t have permissions to do that, I wonder if that’s related to the earlier error. I am going to run chown on all files in the /data/erddap directory. I stopped the container, and ran the chown. Hmm … something is very funky with the interaction with the container and the files on the host. It’s like the container can’t create directories, but it took ownership of the files and now I can’t edit them on the host side. Strange. If I get inside the container, it’s the tomcat user that has ownership of those files now. So, it looks like podman changes the ownership of the mounts to the tomcat user, but it could not create directories? Strange. OK, was able to edit it from kgomes sudo and got past that to hit that Bitstream Vera Sans was not installed. I changed that to DejaVu Sans to just see if I can get this to work. OK, that seemed to be the magic combination, but then I started bumping into https redirects. Joe had set up a redirect to send all http traffic to https and the certificate was valid for odss-test8.shore.mbari.org so there is a mismatch between the Apache server and the baseUrl of the ERDDAP server. I need to change the baseUrl to be https://odss-test8.shore.mbari.org, but there is no proxy pass in place yet to send /erddap traffic to :8080/erddap. In the end, I added this to the end of the admin section:
      <adminInstitutionUrl>https://www.mbari.org</adminInstitutionUrl>

    and changed the font entry to be:

        <fontFamily>DejaVu Sans</fontFamily>

    1. Joe enabled SSL and put the proxies in place and it WORKED! The SSL steps were:

      yum install mod_ssl
upload ssl certs to /usr/local/keystore
vim ssl.conf -- modify values to reflect the certs
service httpd restart
verify local firewall open for 443, 80
vim /etc/httpd/conf.d/redirect.conf

<VirtualHost *:80>
ServerName odss-test8.shore.mbari.org
RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

service httpd restart
The config file is called odss.conf and looks like:
<VirtualHost *:80>
ServerName odss-test8.shore.mbari.org
RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<Location /erddap>
  ProxyPass        http://127.0.0.1:8082/erddap
  ProxyPassReverse http://127.0.0.1:8082/erddap
</Location>

<Location /thredds>
  ProxyPreserveHost On
  ProxyPass        http://127.0.0.1:8081/thredds
  ProxyPassReverse http://127.0.0.1:8081/thredds
</Location>

<Location /odss>
  ProxyPass        http://127.0.0.1:3000/odss
  ProxyPassReverse http://127.0.0.1:3000/odss
</Location>

<Location /cgi-bin/mapserv>
  Proxypass        http://127.0.0.1:8080/
  ProxyPassReverse http://127.0.0.1:8080/
</Location>

    2. I will need to do this set of steps again as I’m not sure if the --security-opt label=disable arguments on the pod create did anything or if it was my permission wrangling that made a difference.

  10. THREDDS

    1. There is a Docker image for THREDDS that used for this work. THREDDS is more straightforward than ERDDAP because it has a data directory that everything is contained within. So, in theory, I should just be able to copy over the THREDDS directory and start the THREDDS container with the data directory set to that location. This is done by mounting the host directory to /usr/local/tomcat/content/thredds.

    2. So, I first logged into odss-test8 and scp’d the /data/thredds directory from odss-test to odss-test8 (I did this using the odssadm account for ownership reasons).

      scp -r odssadm@odss-test.shore.mbari.org:/data/thredds /data/thredds

    3. NOTE, it’s probably a good idea to clean out the log files before copying

    4. Since I am deploying these images into a pod and the port mapping in the pod is expecting 8081, I need to change the server.xml file on the THREDDS server to use port 8081. I just created a matching server.xml file, change the port number, and then built a new image that has this file in it. Here is the condensed version of the server.xml file:

      <?xml version="1.0" encoding="UTF-8"?>
<Server port="-1" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
    <Resource digest="SHA" name="UserDatabase" auth="Container"
            type="org.apache.catalina.UserDatabase"
            description="User database that can be updated and saved"
            factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
            pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
    <Connector server="Apache" secure="true" port="8081" protocol="HTTP/1.1"
            connectionTimeout="20000"
            redirectPort="8443"
            relaxedPathChars='[]|'
            relaxedQueryChars='[]:|{}^&#x5c;&#x60;&quot;&lt;&gt;' />
    <Engine name="Catalina" defaultHost="localhost">
    <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
                resourceName="UserDatabase">
                        <CredentialHandler className="org.apache.catalina.realm.MessageDigestCredentialHandler" algorithm="sha-512" />
        </Realm>
    </Realm>
    <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
            prefix="localhost_access_log" suffix=".txt"
            pattern="%h %l %u %t &quot;%r&quot; %s %b" />
    </Host>
    </Engine>
</Service>
</Server>

    5. Notice the port was changed from 8080 to 8081. Then, I used a the following Dockerfile to build a new image:

      # This creates a Docker image from THREDDS that overrides the server.xml file

# First define the base image that this will build from
FROM unidata/thredds-docker:4.6.15

# Now overwrite the server.xml file so that we can change the port
ADD thredds-server.xml /usr/local/tomcat/conf/server.xml

    6. Once I had these files on the new server, I ran:

      podman build -t mbari/odss-thredds -f THREDDS-Dockerfile

    7. Now, in order to create the THREDDS pod, I ran:

      podman create --security-opt label=disable --name thredds --pod odss -v /data/thredds:/usr/local/tomcat/content/thredds mbari/odss-thredds

    8. This returned an id of:

      47d2d370d536d89027691c9784904188cbd216f3b3622dca8b9305fc50e423b4

    9. Now, to generate the start up script, I ran:

      podman generate systemd --name thredds > ~/.config/systemd/user/container-thredds.service

    10. I then edited the service file to convert it to:

      # container-thredds.service
# autogenerated by Podman 2.1.1
# Fri Dec  4 08:35:20 PST 2020
# Updated by kgomes 2020-11-30: Updated description and changed Restart to always

[Unit]
Description=ODSS Test THREDDS Server
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start thredds
ExecStop=/usr/bin/podman stop -t 10 thredds
ExecStopPost=/usr/bin/podman stop -t 10 thredds
PIDFile=/run/user/11775/overlay-containers/47d2d370d536d89027691c9784904188cbd216f3b3622dca8b9305fc50e423b4/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    11. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-thredds.service

    12. You can then start and stop the service using:

      systemctl --user start container-thredds.service
systemctl --user stop container-thredds.service

    13. And check the status of the service using:

      systemctl --user status container-thredds.service

    14. So this seemed to work. One thing that is a little odd is that on the THREDDS catalog page, the catalog URL is shown with 127.0.0.1 as the host and 8081 as the port. I can’t seem to find a way to change that to being just odss-test8.shore.mbari.org, but the URLs and link within the pages seem to resolve OK. It’s annoying, but I don’t see it breaking anything.

  11. Mapserver

    1. For the mapserver component, we want to use the camptocamp version of mapserver as it seems to be the most downloaded version. Important notes:
      1. You can put .env and .sh files in the image and they will be sourced and executed.
      2. If run as root, it will be on port 80, if not it will be on port 8080. I tried to change it to port 8082 but that failed so I thought I would try to create an image that runs under a different user so that it will switch to 8080 automatically.

    2. So, one thing I found through MUCH pain was that the default EPSG file for mapserver did not have the Google projection in it which was causing all my layers to fail. To fix this, I took a copy of the epsg file from the running mapserver container located at /usr/share/proj/epsg and put in the odss-components directory. I then edited the file and added the following line:

      # Google Earth / Virtual Globe Mercator
<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +no_defs

    3. Then to create an image with my customizations, I created a file in the odss-components directory named mapserver-Docker file with the following contents:

      # This creates a Docker image from camptocamp/mapserver that overrides the start-server file

# First define the base image that this will build from
FROM camptocamp/mapserver:7.6

# Add the 'epsg' file with the Google projection
ADD espg /usr/share/proj/epsg

# Create a new user
RUN useradd odssadm && \
    echo 'odssadm:xxxxxxxxxxxxxx' | chpasswd && \
    usermod -g root odssadm

    4. Then to build the image, I ran:

      podman build -t mbari/odss-mapserver -f mapserver-Dockerfile

    5. Before starting it up, I needed to move the mapserver files over by running (as odssadm):

      scp -r odssadm@odss-test.shore.mbari.org:/data/mapserver /data/mapserver

    6. Now to create the image that will be added to the odss pod, I ran:

      podman create --security-opt label=disable --name mapserver --pod odss -u odssadm -v /data/mapserver:/data/mapserver mbari/odss-mapserver

    7. That returned an ID of:

      6a813908e56fb399198d40afa5b096fd7c2026532150491bde057506931b7f77

    8. Then, I created the systemd file using:

      podman generate systemd --name mapserver > ~/.config/systemd/user/container-mapserver.service

    9. I then edited the generated file to look like:

      # container-mapserver.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 12:40:59 PST 2020
# Updated by kgomes 2020-11-30: Updated description and changed Restart to always

[Unit]
Description=Container to run the mapserver component of the ODSS
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start mapserver
ExecStop=/usr/bin/podman stop -t 10 mapserver
ExecStopPost=/usr/bin/podman stop -t 10 mapserver
PIDFile=/run/user/11775/overlay-containers/1ff12bc70643628b6142fcc192e17ff8266867beec6db7ad8f017dbf20873fbb/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    10. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-mapserver.service

    11. You can then start and stop the service using:

      systemctl --user start container-mapserver.service
systemctl --user stop container-mapserver.service

    12. And check the status of the service using:

      systemctl --user status container-mapserver.service

    13. I can’t really validate if this works until I migrate the MongoDB, Tracking and launch ODSS. After staring the ODSS, I did see that the NOAA nav chart worked!!!!!! Hell yeah! Some things are broken and not rendering as layers, but it matches exactly with what is running on odss-test. If I can fix one of them, I think I will be good.

  12. MongoDB

    1. The ODSS itself uses a catalog that is served by MongoDB. So we need to migrate the data from the old MongoDB to the new. I am going to try to use mongodump and mongorestore to see if that will bridge the version gap.
    2. Before starting, I created the /data/mongodb directory on odss-test8.

    3. The first thing is to start up MongoDB on the new server using podman. In order to do this, I first needed to create the container. I did this by running:

      podman create --security-opt label=disable --name mongodb --pod odss -e MONGO_INITDB_ROOT_USERNAME=mongoadm -e MONGO_INITDB_ROOT_PASSWORD=xxxxx -v /data/mongodb:/data/db mongo:4.4

    4. Which returns the ID:

      a719d660d162e9d98f80af4bb1ea28b9bbe0a82e2172705a39024152d68db02b

    5. Then I have to create the systemd file using:

      podman generate systemd --name mongodb > ~/.config/systemd/user/container-mongodb.service

    6. I then edited that file to change it to:

      # container-mongodb.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 12:45:46 PST 2020
# Updated by kgomes 2020-11-30: Updated description and changed Restart to always

[Unit]
Description=Container to run mongodb component of the ODSS
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start mongodb
ExecStop=/usr/bin/podman stop -t 10 mongodb
ExecStopPost=/usr/bin/podman stop -t 10 mongodb
PIDFile=/run/user/11775/overlay-containers/a719d660d162e9d98f80af4bb1ea28b9bbe0a82e2172705a39024152d68db02b/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    7. Then, I needed to enable service using:

      systemctl --user daemon-reload
systemctl --user enable container-mongodb.service

    8. You can then start and stop the service using:

      systemctl --user start container-mongodb.service
systemctl --user stop container-mongodb.service

    9. And check the status of the service using:

      systemctl --user status container-mongodb.service

    10. After starting the service, I connected to the container using podman ps to get the container ID, then jumped into a bash shell:

      podman ps
podman exec -it ba9e2c9e0af3 /bin/bash

    11. In another window, I logged into the old odss-test as odssadm. I then ran:

      mkdir /tmp/mongobackup
cd /tmp/mongobackup
sudo mongodump -u odssadm -p xxxxxxxxxxx --db odss --out /tmp/mongobackup/odss.backup
tar -cvf odss.backup.tar odss.backup

    12. Then, over on odss-test8, I changed into the /data/mongodb directory and ran:

      scp odssadm@odss-test.shore.mbari.org:/tmp/mongobackup/odss.backup.tar .

    13. That copied the file to odss-test8, but I need to be working in the mongodb container so I went back to the terminal that was the bash shell inside the mongodb container. I wanted to then move the odss.backup.tar file out of the data directory to prevent any cluttering so I ran:

      mv /data/db/odss.backup.tar ~/odss.backup.tar

    14. I then untarred the file using:

      cd
tar -xvf odss.backup.tar

    15. I then ran:

      cd odss.backup
mongorestore --db odss --username mongoadm --authenticationDatabase=admin odss

    16. This created the odss database in the local mongodb instance. So I am really close, but I need to create a user that can have full rights to the DB and can then be used by the ODSS application. Traditionally, I’ve used the username odssadm. So, to create that user and give it full permissions on the odss database, I entered the mongo shell as user mongoadm and then ran:

      use odss
db.createUser({user:"odssadm",pwd:"xxxxxx",roles:[{role:"dbOwner",db:"odss"}]})

    17. I did have to put in a ticket to add port 27017 to the firewall so I could get to it from my Studio 3T application. But, I could after that. I think that is all I need for MongoDB

  13. Tracking database migration

    1. Now, the tricky part of all this is the tracking database migration. The data from the current database has to be migrated over but this takes a REALLY long time. So, the overview of steps to take for the migration should be:
      1. The first thing to do is manually create the queues in RabbitMQ and bind them to the proper exchanges. This will start the accrual of tracking messages. The key here is to create them and then quickly stop the consumers on the production machine. This will then allow both servers queues to start accumulating the same messages.
      2. Backup the old database.
      3. Restart the consumers on the old machine.
      4. Create the database on the replacement machine, then restore the old machines DB.
      5. Restart the consumers on the new machine. This should mean that both tracking databasese are identical at this point.
      6. When we are ready to cut over the new machine to the old machine's name, stop both consumers.
      7. Rename the old machine
      8. Rename the new machine to the old name
      9. Start the consumers on the new machine which should start draining queue from the old named queues.
      10. Delete the new machines previous queues.
    2. So, first I set up the PostGIS server in the pod so that I had a place to migrate the tracking database to.
    3. I needed to create a host directory where the files would be written, so I created /data/mbaritracking/postgresql/data directory first.

    4. To create the PostGIS pod, I ran:

      podman create --name mbaritracking --pod odss -e POSTGRES_PASSWORD=xxxxxxx -v /data/mbaritracking/postgresql/data:/var/lib/postgresql/data postgis/postgis:13-3.0

    5. This returned an ID of:

      c49093a6a0405124fd0b35e007e2187164a064c19cc89d255dc23909ea619ed3

    6. This just creates the container, but does not run it yet. In order to get the container to run as a service, I had to create the systemd file. I first created the ~/.config/systemd/user directory. Then, Podman helps you create the systemd file by running:

      podman generate systemd --name mbaritracking > ~/.config/systemd/user/container-mbaritracking.service

    7. I then had to make a couple of small changes, but the final systemd file was:

      # container-mbaritracking.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 12:51:33 PST 2020
# Updated by kgomes 2020-11-30: Updated description and changed Restart to always

[Unit]
Description=Container that runs the PostGIS database for MBARITracking
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start mbaritracking
ExecStop=/usr/bin/podman stop -t 10 mbaritracking
ExecStopPost=/usr/bin/podman stop -t 10 mbaritracking
PIDFile=/run/user/11775/overlay-containers/c49093a6a0405124fd0b35e007e2187164a064c19cc89d255dc23909ea619ed3/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    8. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-mbaritracking.service

    9. You can then start and stop the service using:

      systemctl --user start container-mbaritracking.service
systemctl --user stop container-mbaritracking.service

    10. And check the status of the service using:

      systemctl --user status container-mbaritracking.service

    11. Next, I logged into the RabbitMQ admin page and manually created 6 queues that were connected to the vhost 'trackingvhost'. The queue names were:

      1. odss-test8_persist_ais (bound to ais/ais)
      2. odss-test8_persist_auvs (bound to auvs/auvs)
      3. odss-test8_persist_drifters (bound to drifters/drifters)
      4. odss-test8_persist_gliders (bound to gliders/gliders)
      5. odss-test8_persist_moorings (bound to moorings/moorings)
      6. odss-test8_persist_ships (bound to ships/ships)
    12. I then ssh'd into the old machine (odss-test) and edited the crontab to stop the consumers from restarting
    13. I then got a process listing for all the python consumers. This gave me the process IDs that I would need to kill
    14. For each one of the queues, I would go to the queue in the web admin page and then fill out the information to bind it to the various exchanges. Then, as soon as I created the binding, I would kill the maching consumer on the old machine. I did this for each queue so that the messages that started accruing were the same for both old and new machine.
    15. On odss-test, I ran 'su postgres' to switch to the postgres account.
    16. I thought it would be a good idea to clean up the DB as much as possible before migration, so I ran the ‘vacuumdb mbaritracking’ (from the postgres account).

    17. Then, you want to use pg_dump to backup the database using the following form (from the postgres account):

      pg_dump -Fc -b -b -f /data/tmp/pgbackup/mbaritracking.backup mbaritracking

    18. I backed it up to /data because it had room. Once this dump completes, restart the consumers on odss-test machine by editing the cron job and letting it restart the consumers. You should see the queues connected to odss-test drain.

    19. I then SCP'd the backup file over to the replacment machine and put it in the host machine directory where the postgres data was mounted so that the container could get access to it. Note that on odss-test8, the /data/mbaritracking/postgresql/data directory has restrictive permissions put on it by the container. I simply ran 'sudo chmod g+wrx /data/mbaritracking/postgresql/data' from the kgomes account so that odssadm had permissions to write the file. Then I ran scp odssadm@odss-test.shore.mbari.org:/data/tmp/pgbackup/mbaritracking.backup . from that directory as user odssadm.

    20. Before restoring the DB, it needs to be created. The best thing is to connect to the container using podman exec -it 7927d6d472e3 /bin/bash and then switch to the postgres account using su postgres and then run psql. After in psql, run the following statements:

      CREATE USER odssadm WITH PASSWORD 'xxxxxxx';
CREATE DATABASE mbaritracking with OWNER=odssadm TEMPLATE=template_postgis;
\c mbaritracking
GRANT ALL ON ALL TABLES IN SCHEMA PUBLIC TO odssadm;

    21. Then, you want to exit the psql utility and navigate to the location of the postgis_restore.pl script using cd /usr/share/postgresql/13/contrib/postgis-3.0/. You can then restore the database to the new server using:

      perl postgis_restore.pl /var/lib/postgresql/data/mbaritracking.backup | psql mbaritracking 2> ~/errors.txt

    22. This will take a REALLY LONG TIME, but once it’s complete, the database should be available on the PostGIS server. I exited the postgres account and then exited the mbaritracking container, back to the odss-test8 host.

    23. Now, it is time to create the python consumers that will read the messages off RabbitMQ and then will stuff them into the tracking DB. There are quite a few environment variables for this container, so I thought it was easier to create them in a file and then point to the file when I create the pod. In the /home/odssadm/odss-components directory, I created a file named .env-tracking and put the following in that file:

      # These are the environment variable settings for the tracking database AMQP clients
DJANGO_SETTINGS_MODULE=settings
TRACKING_ADMIN_NAME="Kevin Gomes"
TRACKING_ADMIN_EMAIL=kgomes@mbari.org
TRACKING_HOME=/opt/MBARItracking
TRACKING_DATABASE_ENGINE=django.contrib.gis.db.backends.postgis
TRACKING_DATABASE_NAME=mbaritracking
TRACKING_DATABASE_USER=odssadm
TRACKING_DATABASE_PASSWORD=xxxxxxx
TRACKING_DATABASE_HOST=localhost
TRACKING_DATABASE_PORT=5432
TRACKING_SECRET_KEY="dumb-key"
AMQP_QUEUE_TYPE=ships,ais,drifters,auvs,moorings,gliders
AMQP_HOSTNAME=messaging.shore.mbari.org
AMQP_PORT=5672
AMQP_USERNAME=tracking
AMQP_PASSWORD=xxxxxxxxx
AMQP_VHOST=trackingvhost
MBARI_TRACKING_HOME_HOST=/data/mbaritracking
MBARI_TRACKING_HOST_NAME=odss-test8
PYTHONPATH=/opt/MBARItracking/amqp

    24. I also manually created the /data/mbaritracking/client/logs directory that will be used to write the logs from the consumers.

    25. That should be all I need to start the consumers, so I then ran:

      podman create --name tracking-clients --pod odss --env-file /home/odssadm/odss-components/.env-tracking -v /data/mbaritracking/client:/data mbari/tracking:1.0.3

    26. This returned an ID of:

      3beb2c6e7b758cc1beaf9e3c0a96d56d327a9ec97edfbce64eed20fc4bed8d9b

    27. This just creates the container, but does not run it yet. In order to get the container to run as a service, I had to create the systemd file. I first created the ~/.config/systemd/user directory. Then, Podman helps you create the systemd file by running:

      podman generate systemd --name tracking-clients > ~/.config/systemd/user/container-tracking-clients.service

    28. I then had to make a couple of small changes, but the final systemd file was:

      # container-tracking-clients.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 12:55:04 PST 2020
# Updated by kgomes 2020-11-30: Updated the description and set Restart to always

[Unit]
Description=This is the container service that runs a series of AMQP consumers to process platform locations into tracking database
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start tracking-clients
ExecStop=/usr/bin/podman stop -t 10 tracking-clients
ExecStopPost=/usr/bin/podman stop -t 10 tracking-clients
PIDFile=/run/user/11775/overlay-containers/3beb2c6e7b758cc1beaf9e3c0a96d56d327a9ec97edfbce64eed20fc4bed8d9b/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    29. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-tracking-clients.service

    30. You can then start and stop the service using:

      systemctl --user start container-tracking-clients.service
systemctl --user stop container-tracking-clients.service

    31. And check the status of the service using:

      systemctl --user status container-tracking-clients.service

    32. Damn, that actually worked ... I am shocked. OK, so I have odss-test8 running in parallel with odss-test and both tracking DBs are being updated with identical information so that is GREAT!

  14. ODSS

    1. OK, now the final piece of the puzzle (well almost) is to get the ODSS container running. I need a couple of locations on odss-test8, one to house the repository files and one for the server log files. The repository files already exist on the odss-test, so I need to copy them over and I did that by copying /data/canon directory to /data/repo/canon. I had to create a /data/tmp/repotar (using sudo, then chowning it to odssadm) and then ran tar -cvf /data/tmp/repotar/canon.tar /data/canon. That created the tar file on odss-test and then I scp'd that tar file over to odss-test8 using scp odssadm@odss-test.shore.mbari.org:/data/tmp/repotar/canon.tar /data/repo. I then untarred those files using tar -xvf canon.tar in the /data/repo directory. That copies all the old repository files. I then created a /data/server/logs directory to hold the server logs. It turns out I did not have enough disk space to do this, so I copied a couple of directories over while I waited on the trouble ticket to increase disk space.

    2. Once that was done, I could create the container by running:

      podman create --name odss-server --pod odss -e ODSS_MONGODB_HOST=localhost -e ODSS_MONGODB_USERNAME=odssadm -e ODSS_MONGODB_PASSWORD=xxxxxx -e ODSS_TRACKING_DATABASE_HOST=localhost -e ODSS_TRACKING_DATABASE_USERNAME=odssadm -e ODSS_TRACKING_DATABASE_PASSWORD=xxxxxxxxxx -v /data/repo:/data/repo -v /data/server/logs:/data/logs mbari/odss:1.0.0

    3. This returned an ID of:

      35aa2e6266b76808a91e2bfae88337160c5a72702ae31c731f04a83c16d64f90

    4. This just creates the container, but does not run it yet. In order to get the container to run as a service, I had to create the systemd file. Podman helps you create the systemd file by running:

      podman generate systemd --name odss-server > ~/.config/systemd/user/container-odss-server.service

    5. I then had to make a couple of small changes, but the final systemd file was:

      # container-odss-server.service
# autogenerated by Podman 2.1.1
# Mon Nov 30 13:07:38 PST 2020
# Updated by kgomes 2020-11-30: Updated description and change Restart to always

[Unit]
Description=This is the container which has the ODSS server application
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
ExecStart=/usr/bin/podman start odss-server
ExecStop=/usr/bin/podman stop -t 10 odss-server
ExecStopPost=/usr/bin/podman stop -t 10 odss-server
PIDFile=/run/user/11775/overlay-containers/35aa2e6266b76808a91e2bfae88337160c5a72702ae31c731f04a83c16d64f90/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

    6. I then enabled the service using:

      systemctl --user daemon-reload
systemctl --user enable container-odss-server.service

    7. You can then start and stop the service using:

      systemctl --user start container-odss-server.service
systemctl --user stop container-odss-server.service

    8. And check the status of the service using:

      systemctl --user status container-odss-server.service

    9. IT WORKS! There was some kind of oddity with the MongoDB server stopping, which then caused the ODSS not to work. I spent some time flailing around on that for awhile.

    10. Lastly, I put a index.html page at /var/www/html that had this so user's would be redirected to the ODSS properly (I had to take ownership of html directory and then put the ownership back after):

      <html>
<title>Redirecting to ODSS Test</title>
<head>
<meta HTTP-EQUIV="REFRESH" content="0; url=https://odss-test8.shore.mbari.org/odss/">
</head>
<body>
<p>Redirecting...</p>
</body>
</html>

    11. ODSS Planning

      1. In order to practice for installation of the planning component, I wanted to install Carlos' planning tool on odss-test8 as practice.
      2. I first had IS install NodeJS (12) and Quasar CLI.

      3. I cd'd to the /home/odssadm directory and ran:

        git clone https://kgomes@bitbucket.org/mbari/odss-planning.git

      4. That created an odss-planning directory, I cd'd into that directory and ran:

        npm install @quasar/cli
npm install
/home/odssadm/odss-planning/node_modules/@quasar/cli/bin/quasar build
tar zcf ../odss-planning.tgz *

      5. That created a tar gzipped file of a static website

      6. I moved it to the /tmp directory
      7. I logged in as kgomes
      8. I changed ownership on the /var/www/html directory to kgomes
      9. I then created a odss-planning directory in /var/www/html
      10. Then I copied the /tmp/odss-planning.tgz file into that directoy and unzipped it using tar xf odss-planning.tgz
      11. Lastly, I then deleted both odss-planning.tgz files (/tmp and /var/www/html/odss-planning)
      12. AND IT WORKED, I can see it in the odss tab on odss-test8!

I think that is all the moving parts!!

Upgrade the ODSS

When new releases of the ODSS docker image are published to DockerHub, these are the steps to take to upgrade the image. In general terms, you need to stop the current services (and thus, container), disable the current container-odss-server.service, remove the container, image and service files. Then you create the new container version and service file and re-enable them. Here is an example of an upgrade to v1.0.4:

  • SSH into the test server using ssh username@odss-test8.shore.mbari.org

  • Then switch over to the odssadm account using

    [username@odss-test8 ~]$ sudo -u odssadm -i
[odssadm@odss-test8 ~]$

  • Stop the ODSS container using systemctl

    systemctl --user stop container-odss-server.service

  • Next, disable the systemctl startup using

    [odssadm@odss-test8 ~]$ systemctl --user disable container-odss-server
Removed /home/odssadm/.config/systemd/user/default.target.wants/container-odss-server.service.
Removed /home/odssadm/.config/systemd/user/multi-user.target.wants/container-odss-server.service.

  • Now remove the container using podman

    [odssadm@odss-test8 ~]$ podman rm odss-server
35aa2e6266b76808a91e2bfae88337160c5a72702ae31c731f04a83c16d64f90

  • Remove the service definition file as well by running

    rm ~/.config/systemd/user/container-odss-server.service

  • This basically removes the ODSS server completely. Now it's time to install the newer version which is basically like installing it from scratch. Here is the process:

    podman create --name odss-server --pod odss -e ODSS_MONGODB_HOST=localhost -e ODSS_MONGODB_USERNAME=odssadm -e ODSS_MONGODB_PASSWORD=xxxxxx -e ODSS_TRACKING_DATABASE_HOST=localhost -e ODSS_TRACKING_DATABASE_USERNAME=odssadm -e ODSS_TRACKING_DATABASE_PASSWORD=xxxxxxxxxx -v /data/repo:/data/repo -v /data/server/logs:/data/logs mbari/odss:1.0.4

  • Now create the service file using

    podman generate systemd --name odss-server > ~/.config/systemd/user/container-odss-server.service

  • Edit the newly created file (container-odss-server.service) and change the description to something more useful and the restart to always.

  • I then enabled the service using:

    systemctl --user daemon-reload
systemctl --user enable container-odss-server.service

  • You can then start and stop the service using:

    systemctl --user start container-odss-server.service