Troubleshooting

Service monitoring¶

Various tools are enabled for this purpose. See Service monitoring.

Should I reboot?¶

IS granted tethysadmin sudo privilege to reboot the tethysdash2 machine (which is the primary server for the TethysDash system).

In general, please first consider doing various inspections as indicated below, and then eventually restart the TethysDash container, and, as a last resort, reboot the machine.

TethysDash logs¶

Basic troubleshooting can start by looking at the various log files:

As tethysadmin@tethysdash2

Inspect the docker logs in case of issues with basic launching or execution of the containers:

docker logs --tail=300 -f tethysdash

The TethysDash log files themselves are kept under ~/TethysDash/td_logs/:

cd ~/TethysDash/

tail -f td_logs/main.log    # the main log

tail -f td_logs/ws.log      # websocket interactions log

Postgres logs

Rarely needed to be inspected:

docker logs --tail=100 -f tdpostgres

Okeanids logs

On okeanids the Apache log files are under /var/log/apache2/. Typically, we only inspect these logs when setting up or adjusting the proxy passes to tethysdash2, but rather rarely for TethysDash troubleshooting purposes.

Restarting the TethysDash container¶

cd ~/TethysDash && source setenv.sh
docker compose restart tethysdash

Relaunching the whole TethysDash system (including the database container)¶

cd ~/TethysDash/
source setenv.sh
docker compose down

then:

docker compose up -d
docker logs --tail=100 -f tethysdash
tail -F -n300 td_logs/main.log

Adjusting TethysDash log levels¶

As a TethysDash system admin user, and along with inspection of the logs as indicated above, you can also open this page in Dash4:

https://okeanids.mbari.org/dash4/#/admin/dev

to change the logging level associated to java classes in the TethysDash backend code, and see the effect immediately in the generated log.

`/LRAUV` share not getting mounted?¶

/LRAUV is a critical share (on tethysdash2) used by the TethysDash system.

(Sept 2023) Per IS Ticket 36309, try:
```
mount -v -a
```
For some reason, the entries in /etc/fstab did not mount successfully upon reboot. Not sure why, but a simple mount -v -a seemed to fix the problem. We coordinated in #lrauv-dev on slack.

(Dec 2025) Migration to Ubuntu

/LRAUV was being mounted as read-only. Fix by Peter:

modify /etc/fstab, to add rw to line for LRAUV share

atlasnfs.shore.mbari.org:/ifs/mbariarchive/LRAUV   /LRAUV   nfs  defaults,rw  0 0

run systemctl daemon-reload
run mount -a

Need to restore a database backup?¶

This has never been needed, but see this section for relevant notes.