Skip to content

Dockerizing MBARItracking

After dockerizing MBARItracking amqp on my local machine, I committed the changes to SVN repo. I added some Docker related files, but the files of most concern are amqp/settings.py and amqp/consumer.py. I tried to change those files so they would use environment variables first (to enable Docker), but fall back to old methods so they should just work on existing deployments.

ODSS-Test

First, I wanted to test the upgrade on odss-test:

  1. When I first logged into the test machine, I cd'd into dev/MBARItracking and ran svn status and got:

    M       amqp/settings.py
M       amqp/runPersistMonitor.sh
M       amqp/persistWatchdog.py
M       amqp/tracking.wsgi

  2. To make sure I don't lose any local changes, I first made a copy of the entire dev/MBARItracking directory to a ~/backup/2018-09-18 directory.

  3. I then did a SVN diff on each of those files

    1. For amqp/settings.py

      [odssadm@odss-test MBARItracking]$ svn diff amqp/settings.py
Index: amqp/settings.py
===================================================================
--- amqp/settings.py    (revision 13106)
+++ amqp/settings.py    (working copy)
@@ -19,7 +19,7 @@
# Then load those sensitive settings from a local file with tight filesystem permissions.
from os.path import expanduser
##execfile(expanduser('~/.django-mbaritracking-settings'))    ## From mod_wsgi this expands to /var/www
-execfile(expanduser('/home/trackadm/.django-mbaritracking-settings'))
+execfile(expanduser('/home/odssadm/.django-mbaritracking-settings'))

from base64 import b64decode
DATABASE_PASSWORD = b64decode(DATABASE_PASSWORD)
@@ -37,7 +37,8 @@
        'HOST': DATABASE_HOST,
        'NAME': DATABASE_NAME,
        'USER': DATABASE_USER,
-       'PASSWORD': DATABASE_PASSWORD
+       ##'PASSWORD': DATABASE_PASSWORD
+       'PASSWORD': 'water4u!'
    }
}

@@ -99,7 +100,7 @@
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
-   '/home/trackadm/dev/MBARItracking/amqp/tracking_tmpl',
+   '/home/odssadm/dev/MBARItracking/amqp/tracking_tmpl',
)

INSTALLED_APPS = (

    2. For amqp/runPersistMonitor.sh

      [odssadm@odss-test MBARItracking]$ svn diff amqp/runPersistMonitor.sh
Index: amqp/runPersistMonitor.sh
===================================================================
--- amqp/runPersistMonitor.sh    (revision 13106)
+++ amqp/runPersistMonitor.sh    (working copy)
@@ -4,13 +4,13 @@
# Django-enabled persist monitoring job from cron.

# Settings
-export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/freetds/lib
-export PATH=/opt/python/bin:/usr/pgsql-9.0/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/trackadm/bin
+export LD_LIBRARY_PATH=/usr/local/lib
+export PATH=/usr/pgsql-9.1/bin:/usr/local/bin:/bin:/usr/bin
export DJANGO_SETTINGS_MODULE=settings
-export HOME=/home/trackadm
-export PYTHONPATH=/home/trackadm/dev/MBARItracking/amqp
+export HOME=/home/odssadm
+export PYTHONPATH=/home/odssadm/dev/MBARItracking/amqp
export PQN_=`hostname --short`_

# We may want to redirect output to to a log file...
-echo /home/trackadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
-/home/trackadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
+echo /home/odssadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
+/home/odssadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_

    3. For amqp/persisteWatchdog.py

      [odssadm@odss-test MBARItracking]$ svn diff amqp/persistWatchdog.py
Index: amqp/persistWatchdog.py
===================================================================
--- amqp/persistWatchdog.py    (revision 13106)
+++ amqp/persistWatchdog.py    (working copy)
@@ -214,14 +214,14 @@

    en = qn.split('_')[-1]

-    base = '/home/trackadm/dev/MBARItracking/amqp'
+    base = '/home/odssadm/dev/MBARItracking/amqp'
    if qn.find('persist_ais') != -1:
        outFile = '/dev/null'
    else:
-        outFile = '/home/trackadm/logs/nonAISPersistConsumers.out'
+        outFile = '/data/logs/tracking/nonAISPersistConsumers.out'

    # Use -u option for unbuffered stdout
-    cmd = "/opt/python/bin/python -u %s/consumer.py --en %s --et fanout --qn %s --vh trackingvhost --persist >> %s 2>&1 &" % (base,
+    cmd = "/usr/bin/python -u %s/consumer.py --en %s --et fanout --qn %s --vh trackingvhost --persist >> %s 2>&1 &" % (base,
        en, qn, outFile)
    print "Executing command: %s" % cmd
    os.system(cmd)
@@ -262,12 +262,12 @@
    # Restart (if the proper users) any processes that are missing from the process table
    mqList = missingQueues(pqn)
    if mqList:
-        if getpass.getuser() == 'trackadm':
+        if getpass.getuser() == 'odssadm':
            for mq in mqList:
                print "Consumer on queue %s is not running.  Restarting it." % mq
                startConsumer(mq)
        else:
-            print "Only the trackadm user can restart the consumers."
+            print "Only the odssadm user can restart the consumers."

    print ""    # Blank line

    4. For amqp/tracking.wsgi

      [odssadm@odss-test MBARItracking]$ svn diff amqp/tracking.wsgi
Index: amqp/tracking.wsgi
===================================================================
--- amqp/tracking.wsgi    (revision 13106)
+++ amqp/tracking.wsgi    (working copy)
@@ -15,9 +15,9 @@
sys.path.append('/opt/python/lib/python2.6/dist-packages') # cautionary to load django
sys.path.append('/opt/python/lib/python2.6') # to load os

-sys.path.append('/home/trackadm/dev/MBARItracking') # The app - in order to find amqp.settings
-sys.path.append('/home/trackadm/dev/MBARItracking/amqp') #
-sys.path.append('/home/trackadm/dev/MBARItracking/amqp/MBARItracking') # Where the model is, so that import models as m works
+sys.path.append('/home/odssadm/dev/MBARItracking') # The app - in order to find amqp.settings
+sys.path.append('/home/odssadm/dev/MBARItracking/amqp') #
+sys.path.append('/home/odssadm/dev/MBARItracking/amqp/MBARItracking') # Where the model is, so that import models as m works

  4. Now to do the actual upgrade, I did the following:

    1. Stopped the runPersistMonitor cron job by commenting out the crontab
    2. Did a ps -ef | grep python to get listing of current consumer.py processes
    3. Killed each one of them
    4. Then from the dev/MBARItracking directory I ran 'svn update' which did not work as the ssh key was out of date for kahuna.
    5. I edited the ~/.ssh/known_hosts file and deleted the kahuna entry and retried (had to login using my kgomes account)

    6. It updated a bunch of files and stopped for conflicts on amqp/settings.py, amqp/runPersistMonitor.sh, amqp/persistWatchdog.py, amqp/tracking.wsgi. Here is the output:

      U   scripts/plot_platforms.pl
U   scripts/waveglider.py
A   scripts/sample_tethysNewData_2016.msg
U   scripts/auv.py
U   scripts/dispatchGliderMessage.py
U   scripts/remove_points_files.sh
A   scripts/shout_header.template
A   scripts/wgetPositions.sh
A   scripts/grabAISpositions.sh
A   scripts/ship.py
U   scripts/dispatchMessage.py
U   scripts/glider.py
U   scripts/parseLogrEmail.py
U   scripts/mooring.py
U   scripts/grabROVVideo.pl
U   scripts/grabOASISpos.pl
U   scripts/aisWatchdog.py
U   scripts/drifter.py
U   scripts/drifterMessage.py
U   scripts/processSent.py
U   scripts/dorado.py
U   scripts/grabAISpositions.py
U   scripts/rabbitmqWatchdog.py
A   scripts/sample_tethysNewData.msg
A   scripts/sample_tethysNewData_2012.msg
U   web/MBARItrackingKML.cgi
U   amqp/latlon.py
U   amqp/consumer.py
U   amqp/views.py
Conflict discovered in 'amqp/settings.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

    7. I used tc (their-conflict) for all conflicts.

    8. I then ran a diff on the backup files and the updated files:

      [odssadm@odss-test MBARItracking]$ !diff
diff ~/backup/2018-09-18/MBARItracking/amqp/settings.py amqp/settings.py
1a2
> import os
6,22c7,40
< # Contact info - should be set in local settings file
< ADMIN_NAME = ''
< ADMIN_EMAIL = ''
<
< # Exposing database details is a security hole, so leave them blank here.
< DATABASE_ENGINE = ''
< DATABASE_NAME = ''
< DATABASE_USER = ''
< DATABASE_PASSWORD = ''
< DATABASE_HOST = ''
< DATABASE_PORT = ''
< SECRET_KEY = ''
<
< # Then load those sensitive settings from a local file with tight filesystem permissions.
< from os.path import expanduser
< ##execfile(expanduser('~/.django-mbaritracking-settings'))    ## From mod_wsgi this expands to /var/www
< execfile(expanduser('/home/odssadm/.django-mbaritracking-settings'))
---
> # See if we can get our variables from the environment first
> ADMIN_NAME = None
> ADMIN_EMAIL = None
> # I define this one with a default so it will continue to exist with existing installs that
> # are not using the Docker version.
> TRACKING_HOME = '/home/odssadm/dev/MBARItracking'
> DATABASE_ENGINE = None
> DATABASE_NAME = None
> DATABASE_USER = None
> DATABASE_PASSWORD = None
> DATABASE_HOST = None
> DATABASE_PORT = None
> SECRET_KEY = None
>
> try:
>   ADMIN_NAME = os.environ["TRACKING_ADMIN_NAME"]
>   ADMIN_EMAIL = os.environ["TRACKING_ADMIN_EMAIL"]
>   TRACKING_HOME = os.environ["TRACKING_HOME"]
>   DATABASE_ENGINE = os.environ["TRACKING_DATABASE_ENGINE"]
>   DATABASE_NAME = os.environ["TRACKING_DATABASE_NAME"]
>   DATABASE_USER = os.environ["TRACKING_DATABASE_USER"]
>   DATABASE_PASSWORD = os.environ["TRACKING_DATABASE_PASSWORD"]
>   DATABASE_HOST = os.environ["TRACKING_DATABASE_HOST"]
>   DATABASE_PORT = os.environ["TRACKING_DATABASE_PORT"]
>   SECRET_KEY = os.environ["TRACKING_SECRET_KEY"]
> except Exception, e:
>   print 'Could not find all the variables in the environment, will look for django settings file'
>   from os.path import expanduser
>   ##execfile(expanduser('~/.django-mbaritracking-settings'))    ## From mod_wsgi this expands to /var/www
>   execfile(expanduser('/home/odssadm/.django-mbaritracking-settings'))
>
>
> # Set the version of PostGIS
> POSTGIS_VERSION = (2, 4, 4)
25c43
< DATABASE_PASSWORD = b64decode(DATABASE_PASSWORD)
---
> DATABASE_PASSWORD = DATABASE_PASSWORD
97c115
< ROOT_URLCONF = 'amqp.urls'
---
> ROOT_URLCONF = 'urls'
103c121
<   '/home/odssadm/dev/MBARItracking/amqp/tracking_tmpl',
---
>   TRACKING_HOME + '/amqp/tracking_tmpl',

    9. It looks like that all worked and I should be able to just run it.

    10. I brought up Aqua Data Studio and the RabbitMQ web page to look at the queues. As expected the queue were stockpiling some messages.
    11. I ran a query to find the last 10 W_FLYER positions.

    12. I then ran amqp/runPersistMonitor.sh to try and start everything up. It failed with:

      sh: /home/odssadm/dev/MBARItracking/amqp/..//venv-tracking/bin/activate: No such file or directory

    13. This is because odss-test is not using a virtual environment. In order to keep the source code as uniform as possible, I decided it was better to create a venv-tracking to replicate an environment that the code is expecting. To do this, I simply ran:

      virtualenv --system-site-packages venv-tracking

    14. Thjs created a venv-tracking directory with all the system modules already installed. I then ran amqp/runPersistMonitor.sh and it worked!!! Queues drained and entries were updating in the databse.

One thing to note here is that the amqp/tracking.wsgi file is quite a bit different than the backed up one so I suspect I may have broken something on the odss-test machine, but not sure how to test it. Again, I did not test the tracking.wsgi part, but maybe later.

ODSS Production

So that took care of odss-test. Now let's take a look at odss on normandy (production).

So the first thing I noticed is that venv-tracking does exist on normandy so I should not have to edit runPersistMonitor.sh or persistWatchdog.py.

  1. I ran an svn status and got:

    [odssadm@normandy MBARItracking]$ svn status
?       venv-tracking
M       amqp/tracking.wsgi

  2. So then I did an svn diff on amqp/tracking.wsgi and got:

    (venv-tracking)[odssadm@normandy MBARItracking]$ svn diff amqp/tracking.wsgi
Index: amqp/tracking.wsgi
===================================================================
--- amqp/tracking.wsgi    (revision 13519)
+++ amqp/tracking.wsgi    (working copy)
@@ -50,7 +50,7 @@

# Configure logging settings for Apache logs
import logging
-log_level=logging.DEBUG
+log_level=logging.INFO
logger = logging.getLogger('')
logger.setLevel(log_level)
handler = logging.StreamHandler(sys.stderr)

  3. So I'm thinking that this installation is better aligned with the source repo, so it should be pretty straightforward.

  4. I created backup of dev/MBARItracking in the ~/backups/2018-09-18 directory
  5. Turned off the cron job
  6. I killed all the python jobs.
  7. I could not check out the SVN update, Mike has to ... on hold and restarted everything
  8. Mike just did the SVN update
  9. I am going on a leap of faith and going to kill the python consumers and run runPersistMonitor
  10. Thank God it looks like it worked!!!

Note

Something was very wonky and it looks like queries were failing due to the data outside the 180/-180/90/-90 thing, going to clean out those points. That was not it. OK, so what was the very wonky thing? Not anything I had done (with respect to MBARItracking that is). It was actually the JavaScript that was pulling tracking data for the main www.mbari.org website (WordPress). It was supposed to be only doing it for pages that had the map on it, but it was doing it for every single page! A web crawler started indexing our main website last night which was causing every single page to make a request to the ODSS and basically overwhelmed the server. Anyway, I got that fixed and the requests dropped off dramatically so I am back to the upgrade. After Mike did the SVN update, I thought it had broken everything so I copied the backup into the original location and went back to the original code. This did not fix the issue and it did appear to work before rolling back so I am going to put the new stuff back in place.

  1. Disabled cron job
  2. Kill python scripts that were running.
  3. Removed the MBARItracking directory and renamed the new stuff back to MBARItracking from MBARItracking-new.
  4. Ran runPersistMonitor
  5. Re-enabled cron

That seems to now work OK and the server is still responsive. WHEW!

Rachel Carson (Zuma)

The Rachel Carson is dockside, so I think I will do the upgrade there too. This one will be a little different in that I need to make sure I have the Winfrog stuff configured correctly. That means I need the Winfrog IP address, I need to extract out the name to socket numbers from consumer.py to the portLookup.py file. So, here we go:

  1. ssh to zuma
  2. zuma does not have venv-tracking set up, so I need to do that first.
  3. I ran 'virtualenv --system-site-packages venv-tracking'
  4. Created a 2018-09-19 directory in ~/backup and copied the entire ~/dev/MBARItracking to that directory

  5. When I run an SVN status from ~/dev/MBARItracking, I get:

    ?       venv-tracking
M       amqp/latlon.py
M       amqp/consumer.py
M       amqp/views.py
M       amqp/settings.py
M       amqp/tracking_tmpl/position.kml
M       amqp/runPersistMonitor.sh
M       amqp/persistWatchdog.py
M       amqp/tracking.wsgi

  6. Then running an svn diff on each, I get:

    1. For amqp/latlon.py

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/latlon.py
Index: amqp/latlon.py
===================================================================
--- amqp/latlon.py    (revision 13106)
+++ amqp/latlon.py    (working copy)
@@ -112,7 +112,8 @@
        d = dict(latd = latd, latm = latm, NS = self._NS_hemisphere,
                lond = lond, lonm = lonm, EW = self._EW_hemisphere,
        utc = time.strftime('%H%M%S',time.gmtime()))
-       s = '$GPGLL,{latd}{latm},{NS},{lond}{lonm},{EW},{utc},A'.format(**d)
+       ##s = '$GPGLL,{latd}{latm},{NS},{lond}{lonm},{EW},{utc},A'.format(**d)
+    s = '$GPGLL,{latd:02d}{latm:09.6f},{NS},{lond:03d}{lonm:09.6f},{EW},{utc},A'.format(**d)
        ##print '{0}*{1}\n'.format(s, nmeachecksum(s))
        return '{0}*{1}\n'.format(s, nmeachecksum(s))

    2. For amqp/consumer.py

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/consumer.py
Index: amqp/consumer.py
===================================================================
--- amqp/consumer.py    (revision 13106)
+++ amqp/consumer.py    (working copy)
@@ -102,12 +102,18 @@
            print "consumer.sendGGLtoTCP(): Sent to %s:%s: %s" % (TCP_HOST, TCP_PORT, gllStrNoWhite)

            tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+               print "socket created"
        tcp.settimeout(1)
+               print "socket timeout set to 1"
            tcp.connect((TCP_HOST, TCP_PORT))
+               print "socket connected"

            tcp.sendall(gllStr)
+               print "message sent"
        tcp.shutdown(socket.SHUT_RDWR)
+               print "socket shutdown"
            tcp.close()
+               print "socket closed"

    def sendGGLtoUDP(self, esecs, lat, lon, DigiOneSP_IP, UDP_PORT = 2101):
@@ -137,22 +143,56 @@
        # If running on malibu (on the Western Flyer) send positions to the Winfrog computer by platform name
        # Platform names are case sensitive to what they are know by in the MBARI Tracking database
        ##winfrogHost = 'dogfish.shore.mbari.org'
-        winfrogHost = '134.89.20.18'
+        ##winfrogHost = '134.89.20.18'      # Flyer's Winfrog master
+        winfrogHost = '134.89.22.33'        # Carson's Winfrog master
        portLookup = {
                'tethys':    31401,
-                'daphne':    31402,
+#               'daphne':    31402,
                'dorado':    31403,
-                'ESP':      31404,
-                'ZEPHYR':    31405,
-                'waveglider':    31406,
+                'Mack':     31404,
+                'Bruce':    31405,
                'stella101':    31101,
                'stella103':    31103,
                'stella104':    31104,
                'stella108':    31108,
                'stella110':    31110,
                'stella111':    31111,
+                               'stella113':    31130,
+                               'stella114':    31131,
+                               'stella115':    31132,
+                               'stella116':    31133,
+                               'stella117':    31134,
+                               'stella118':    31135,
+                               'stella119':    31136,
+                               'stella120':    31137,
+                               'stella121':    31138,
+                               'stella122':    31139,
+                'stella201':     31112,
+                'stella202':    31113,
+                'stella203':    31114,
+                'stella204':    31115,
+                'stella205':    31116,
+                'wgOA':     31117,
+                'wgTex':    31118,
+                               'wgTiny':       32130,
+                               'wgTinyToo':    32131,
+                'wgSparky':    32134,
+                               'R_CARSON':     32132,
+                               'wgHermes':     32133,
+                'L_662':    31119,
+                'NPS_G29':    31120,
+                'NPS_G34':    31221,
+                'oa1':      31222,
+                'oa2':      31223,
+                'm1':       31224,
+                'M46044':    31225,
+                'Martin':    31226,
+                'paragon':    31227,
+                'LST_3049':    31228,
+                'LST_3050':    31229,
                }
-        hostLocalToWinfrog = 'malibu'       # The host on which we send messages to Winfrog
+        ##hostLocalToWinfrog = 'malibu'     # The host on which we send messages to Winfrog
+        hostLocalToWinfrog = 'zuma'     # The host on which we send messages to Winfrog
        ##print "self.hostName = %s" % self.hostName
        if self.hostName.startswith(hostLocalToWinfrog):
            ##print "Running on computer local to Winfrog: %s" % hostLocalToWinfrog

    3. For amqp/views.py

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/views.py
Index: amqp/views.py
===================================================================
--- amqp/views.py    (revision 13106)
+++ amqp/views.py    (working copy)
@@ -24,7 +24,7 @@
from django.http import HttpResponse, HttpResponseBadRequest
from datetime import datetime, timedelta
import time
-import models as m
+from MBARItracking import models as m
import csv
import collections

    4. For amqp/settings.py

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/settings.py
Index: amqp/settings.py
===================================================================
--- amqp/settings.py    (revision 13106)
+++ amqp/settings.py    (working copy)
@@ -19,10 +19,10 @@
# Then load those sensitive settings from a local file with tight filesystem permissions.
from os.path import expanduser
##execfile(expanduser('~/.django-mbaritracking-settings'))    ## From mod_wsgi this expands to /var/www
-execfile(expanduser('/home/trackadm/.django-mbaritracking-settings'))
+execfile(expanduser('/home/odssadm/.django-mbaritracking-settings'))

from base64 import b64decode
-DATABASE_PASSWORD = b64decode(DATABASE_PASSWORD)
+DATABASE_PASSWORD = DATABASE_PASSWORD

ADMINS = (
@@ -76,7 +76,7 @@
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
-ADMIN_MEDIA_PREFIX = '/media/'
+STATIC_URL = '/media/'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
@@ -99,7 +99,7 @@
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
-   '/home/trackadm/dev/MBARItracking/amqp/tracking_tmpl',
+   '/home/odssadm/dev/MBARItracking/amqp/tracking_tmpl',
)

INSTALLED_APPS = (

    5. For amqp/tracking_tmpl/position.kml

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/tracking_tmpl/position.kml
Index: amqp/tracking_tmpl/position.kml
===================================================================
--- amqp/tracking_tmpl/position.kml    (revision 13106)
+++ amqp/tracking_tmpl/position.kml    (working copy)
@@ -21,6 +21,12 @@
                {% if pList.0.7 = 'ship' %}<href>http://dods.mbari.org/images/{{ pList.0.0 }}.png</href>{% endif %}
                <scale>0.70</scale>
            </Icon>
+           {% if pList.0.7 = 'mooring' %}<color>ff00ffff</color>{% endif %}
+           {% if pList.0.7 = 'drifter' %}<color>ff00ffff</color>{% endif %}
+           {% if pList.0.7 = 'glider' %}<color>ff2d32e7</color>{% endif %}
+           {% if pList.0.7 = 'auv' %}<color>ff00ffff</color>{% endif %}
+           {% if pList.0.7 = 'ais' %}<color>fff0f0f0</color>{% endif %}
+           {% if pList.0.7 = 'ship' %}<color>ff0000ff</color>{% endif %}
        </IconStyle>
        <LabelStyle>
            <scale>0</scale>

    6. For amqp/runPersistMonitor.sh

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/runPersistMonitor.sh
Index: amqp/runPersistMonitor.sh
===================================================================
--- amqp/runPersistMonitor.sh    (revision 13106)
+++ amqp/runPersistMonitor.sh    (working copy)
@@ -4,13 +4,13 @@
# Django-enabled persist monitoring job from cron.

# Settings
-export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/freetds/lib
-export PATH=/opt/python/bin:/usr/pgsql-9.0/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/trackadm/bin
+export LD_LIBRARY_PATH=/usr/local/lib
+export PATH=/usr/pgsql-9.1/bin:/usr/local/bin:/bin:/usr/bin
export DJANGO_SETTINGS_MODULE=settings
-export HOME=/home/trackadm
-export PYTHONPATH=/home/trackadm/dev/MBARItracking/amqp
+export HOME=/home/odssadm
+export PYTHONPATH=/home/odssadm/dev/MBARItracking/amqp
export PQN_=`hostname --short`_

# We may want to redirect output to to a log file...
-echo /home/trackadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
-/home/trackadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
+echo /home/odssadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_
+/home/odssadm/dev/MBARItracking/amqp/persistWatchdog.py --monitor --pqn $PQN_

    7. For amqp/persistWatchdog.py

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/persistWatchdog.py
Index: amqp/persistWatchdog.py
===================================================================
--- amqp/persistWatchdog.py    (revision 13106)
+++ amqp/persistWatchdog.py    (working copy)
@@ -105,7 +105,9 @@
    Returns a list of queue names that do not have a consumer running on them.
    '''

-    rQs = [    'persist_ais', 'persist_auvs', 'persist_drifters', 'persist_gliders', 'persist_moorings', 'persist_ships']
+    ##rQs = [    'persist_ais', 'persist_auvs', 'persist_drifters', 'persist_gliders', 'persist_moorings', 'persist_ships']
+    # Remove check on ais on zuma for CANON-ECOHAB March 2013
+    rQs = [    'persist_auvs', 'persist_drifters', 'persist_gliders', 'persist_moorings', 'persist_ships']
    requiredQs = []
    for q in rQs:        # Prepend any prepend queue name string
        requiredQs.append(pqn + q)
@@ -214,14 +216,14 @@

    en = qn.split('_')[-1]

-    base = '/home/trackadm/dev/MBARItracking/amqp'
+    base = '/home/odssadm/dev/MBARItracking/amqp'
    if qn.find('persist_ais') != -1:
        outFile = '/dev/null'
    else:
-        outFile = '/home/trackadm/logs/nonAISPersistConsumers.out'
+        outFile = '/data/logs/tracking/nonAISPersistConsumers.out'

    # Use -u option for unbuffered stdout
-    cmd = "/opt/python/bin/python -u %s/consumer.py --en %s --et fanout --qn %s --vh trackingvhost --persist >> %s 2>&1 &" % (base,
+    cmd = "/usr/bin/python -u %s/consumer.py --en %s --et fanout --qn %s --vh trackingvhost --persist >> %s 2>&1 &" % (base,
        en, qn, outFile)
    print "Executing command: %s" % cmd
    os.system(cmd)
@@ -262,12 +264,12 @@
    # Restart (if the proper users) any processes that are missing from the process table
    mqList = missingQueues(pqn)
    if mqList:
-        if getpass.getuser() == 'trackadm':
+        if getpass.getuser() == 'odssadm':
            for mq in mqList:
                print "Consumer on queue %s is not running.  Restarting it." % mq
                startConsumer(mq)
        else:
-            print "Only the trackadm user can restart the consumers."
+            print "Only the odssadm user can restart the consumers."

    print ""    # Blank line

    8. For amqp/tracking.wsgi

      (venv-tracking)[odssadm@zuma MBARItracking]$ svn diff amqp/tracking.wsgi
Index: amqp/tracking.wsgi
===================================================================
--- amqp/tracking.wsgi    (revision 13106)
+++ amqp/tracking.wsgi    (working copy)
@@ -6,6 +6,7 @@

import os
import sys
+import site

# This is a hack to avoid the error seen in Apache's error_log:
#   ImportError: No module named _functools
@@ -15,12 +16,25 @@
sys.path.append('/opt/python/lib/python2.6/dist-packages') # cautionary to load django
sys.path.append('/opt/python/lib/python2.6') # to load os

-sys.path.append('/home/trackadm/dev/MBARItracking') # The app - in order to find amqp.settings
-sys.path.append('/home/trackadm/dev/MBARItracking/amqp') #
-sys.path.append('/home/trackadm/dev/MBARItracking/amqp/MBARItracking') # Where the model is, so that import models as m works
+##sys.path.append('/home/trackadm/dev/MBARItracking') # The app - in order to find amqp.settings
+##sys.path.append('/home/trackadm/dev/MBARItracking/amqp') #
+##sys.path.append('/home/trackadm/dev/MBARItracking/amqp/MBARItracking') # Where the model is, so that import models as m works

+project_path=os.path.dirname(__file__)

+# Can add the site-packages of our virtualenv as a site dir
+# For now add the project paths so that all imports can be seen (arghhh, should do a better job in the app with paths :-\)
+site.addsitedir(os.path.join(project_path, '../'))      # to see settings
+site.addsitedir(project_path)                   # to see MBARItracking.models

+prev_sys_path=sys.path[:]
+
+sys.path[:0] = [sys.path.pop(pos) for pos, p in enumerate(sys.path) if p not in prev_sys_path]
+del prev_sys_path
+print sys.path
+
+
+
os.environ['DJANGO_SETTINGS_MODULE'] = 'amqp.settings'

import django.core.handlers.wsgi

  7. Now that I have a backup copy and a grab of all the actual diffs, I can continue

  8. I edited the crontab to stop the runPersistMonitor
  9. Killed all the python consumers and started to see messages accumulate on messaging.shore.mbari.org
  10. had to remove the kahuna entry from .ssh/known_hosts

  11. Then ran svn update and chose their-conflict for all conflict resolution:

    (venv-tracking)[odssadm@zuma MBARItracking]$ svn update
The authenticity of host 'kahuna.shore.mbari.org (134.89.12.199)' can't be established.
RSA key fingerprint is 2b:ee:e1:de:ee:95:45:d8:bd:9c:b1:4a:f2:ba:d0:98.
Are you sure you want to continue connecting (yes/no)? yes
kgomes@kahuna.shore.mbari.org's password:
U   scripts/plot_platforms.pl
U   scripts/waveglider.py
A   scripts/sample_tethysNewData_2016.msg
U   scripts/auv.py
U   scripts/dispatchGliderMessage.py
U   scripts/remove_points_files.sh
A   scripts/shout_header.template
A   scripts/wgetPositions.sh
A   scripts/grabAISpositions.sh
A   scripts/ship.py
U   scripts/dispatchMessage.py
U   scripts/glider.py
U   scripts/parseLogrEmail.py
U   scripts/mooring.py
U   scripts/grabROVVideo.pl
U   scripts/grabOASISpos.pl
U   scripts/aisWatchdog.py
U   scripts/drifter.py
U   scripts/drifterMessage.py
U   scripts/processSent.py
U   scripts/dorado.py
U   scripts/grabAISpositions.py
U   scripts/rabbitmqWatchdog.py
A   scripts/sample_tethysNewData.msg
A   scripts/sample_tethysNewData_2012.msg
U   web/MBARItrackingKML.cgi
G   amqp/latlon.py
Conflict discovered in 'amqp/consumer.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: tc
G   amqp/consumer.py
G   amqp/views.py
Conflict discovered in 'amqp/settings.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: tc
G   amqp/settings.py
U   amqp/README
G   amqp/tracking_tmpl/position.kml
Conflict discovered in 'amqp/runPersistMonitor.sh'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: tc
G   amqp/runPersistMonitor.sh
Conflict discovered in 'amqp/persistWatchdog.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: tc
G   amqp/persistWatchdog.py
A   amqp/env.template
A   amqp/portLookup.py
A   amqp/docker
A   amqp/docker/postgis
A   amqp/docker/postgis/Dockerfile
A   amqp/docker/postgis/tracking-init.sh
A   amqp/Dockerfile
A   amqp/COPYING
A   amqp/docker-compose.yml
Conflict discovered in 'amqp/tracking.wsgi'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: tc
G   amqp/tracking.wsgi
U   amqp/urls.py
A   amqp/run.sh
Updated to revision 13793.

  12. Now, I need to edit the consumer.py to make sure the winfrog IP address is correct which for the Carson is 134.89.22.33. For this, I just edited the entry in the consumer.py script to give it the default since it's not defined in the environment anywhere. Note this will be blow away on a SVN update

  13. Also, I need to update the portLookup.py dictionary with the values from the old consumer.py
  14. Started everything and it all looks good.
  15. I enabled the crontab and killed the python consumers so I could clean out the log files and test the crontab.
  16. the cron kicked correctly and all is well.
  17. lastly I put in a reminder to do this on the Flyer (malibu) when they return.