Documentation previously maintained as a google doc.

MBARI TethysDash Operation¶

Basic troubleshooting¶

Assuming you already have some basic understanding of how the various components are set up on the okeanids and tethysdash2 machines, and about the associated user accounts (see here for an overview), let's start with some troubleshooting, as this might likely be a reason for you to look into this documentation.

Should I reboot?¶

IS has granted tethysadmin sudo privilege to reboot the tethysdash2 machine (which is the primary server for the TethysDash system).

Note

In general, however, a reboot of the machine (tethysdash2) should not be necessary. In terms of "restarts," a first try would be to relaunch the dockerized TethysDash system itself, see below. Of course, rebooting the machine is a possible last resort.

TethysDash logs¶

Basic troubleshooting can start by looking at the various log files:

As tethysadmin@tethysdash2

Inspect the docker logs in case of issues with basic launching or execution of the containers:

docker logs --tail=100 -f tethysdash

(The --tail option is just a convenience to avoid printing out the whole log output.)

Or, for the database container (rarely inspected):

docker logs --tail=100 -f tdpostgres

The TethysDash log files themselves are kept under ~/TethysDash/td_logs/:

cd ~/TethysDash/

tail -f td_logs/main.log    # the main log

tail -f td_logs/ws.log      # websocket interactions log

Okeanids log¶

Very rarely needed, on okeanids, you can inspect the usual Apache log files under /var/log/httpd/*.

NOTE: We rarely have to look here for troubleshooting purposes; only when setting up or adjusting the proxy passes to tethysdash2.

Inspecting memory/cpu use¶

Besides the usual linux based tools, (e.g., top), you can also use ctop, which is already installed on tethysdash2:

ctop

Note

If at all possible, please capture information about memory/cpu consumption (e.g., a screenshot like the above) before restarting TethysDash or rebooting the machine, which may eventually help diagnose problems.

Restarting the TethysDash container¶

cd ~/TethysDash/
source setenv.sh
docker compose restart tethysdash

Relaunching the whole TethysDash system (including the database container)¶

cd ~/TethysDash/
source setenv.sh
docker compose down

then, after a few moments:

docker compose up -d

docker logs --tail=100 -f tethysdash

tail -f td_logs/main.log

Adjusting TethysDash log levels¶

As a TethysDash system admin user, and along with inspection of the logs as indicated above, you can also open this page in Dash4:

https://okeanids.mbari.org/dash4/#/admin/dev

to change the logging level associated to java classes in the TethysDash backend code, and see the effect immediately in the generated log.

/LRAUV is a critical share (on tethysdash2) used by the TethysDash system.

Try:

mount -v -a

See IS Ticket 36309.

For some reason, the entries in /etc/fstab did not mount successfully upon reboot. Not sure why, but a simple mount -v -a seemed to fix the problem. We coordinated in #lrauv-dev on slack.