Navigation Collector module

File: backend/navproc_collector.py

Pulls navigation and sensor data from a ship's NATS JetStream KV store and writes it into Redis so the dive detector and the dashboard can read it without a direct NATS connection. The set of keys fetched is controlled by a TOML allow-list rather than discovered dynamically.

Configuration file

The collector reads its key allow-list from backend/collector_keys.toml at startup:

keys = [
    "ROV.CTD.INWATER",
    "ROV.MBARI.DEPTH",
    "SHIP.GPS.LAT",
    "SHIP.GPS.LON",
    "ROV.POSITION.LAT",
    "ROV.POSITION.LON",
]

The path can be overridden with --keys-config. Parsing uses Python's stdlib tomllib (3.11+) — no extra dependency required. Missing or malformed config raises at startup so the collector fails fast rather than silently mirroring nothing.

`load_collector_keys`

load_collector_keys(path: Path | str = DEFAULT_KEYS_CONFIG) -> list[str]

Reads the TOML file and returns the top-level keys array. Raises FileNotFoundError if the file is missing and ValueError if keys is not a list of strings.

Class: `ShipWatcher`

ShipWatcher(ship_name: str, domain: str)

Async NATS client that connects to the ship's coredata server and fetches the configured allow-list of keys on demand.

Constructor parameters

Parameter	Type	Description
`ship_name`	`str`	Ship identifier, e.g. `"rcsn"`
`domain`	`str`	NATS domain, e.g. `"rc"`

The NATS server URL is constructed as:

nats://coredata-{ship_name}.{domain}.mbari.org:4222

Method: `get_keys`

async get_keys() -> list[str]

Returns every key name present in the ship's JetStream KV bucket. This method is kept for ad-hoc debugging but is no longer called from the main loop — the allow-list replaces it for production polling.

Method: `watch_keys`

async watch_keys(keys: list[str]) -> dict[str, str]

Fetches the current value for each key in keys. Per-key fetches are wrapped individually, so a missing or malformed key on one entry logs a warning and the loop continues with the rest of the batch.

Values that parse as int are stored as int; otherwise float; otherwise the key is skipped (with a debug log) rather than guessing how to encode a non-numeric value.

Class: `ShipCache`

ShipCache(bucket_name: str, redis_host: str, redis_port: int)

Redis writer. Accepts the dict produced by ShipWatcher.watch_keys() and stores each entry under a normalised key name.

Key normalisation

NATS key names use dot-separated UPPERCASE segments. Redis keys use colon-separated lowercase, prefixed with the bucket name:

NATS key	Redis key (bucket=`rcsn`)
`ROV.CTD.INWATER`	`rcsn:rov:ctd:inwater`
`ROV.MBARI.DEPTH`	`rcsn:rov:mbari:depth`
`SHIP.GPS.LAT`	`rcsn:ship:gps:lat`
`ROV.POSITION.LAT`	`rcsn:rov:position:lat`

TTL

Every key is written with a 600-second TTL. If the collector stops, keys expire within ten minutes and downstream services see the values as missing.

Method: `store_data`

store_data(data: dict[str, str]) -> None

Writes all key-value pairs to Redis with the normalised naming scheme and TTL.

Main loop

main() runs an async polling loop:

keys = load_collector_keys(keys_config)        # once at startup
await ship_watcher.connect()
while True:
    if keys:
        data = await ship_watcher.watch_keys(keys)
        if data:
            ship_cache.store_data(data)
    await asyncio.sleep(poll_interval)

The pre-existing get_keys() call has been removed from the hot path — the allow-list approach saves a NATS round-trip per poll AND ensures unused keys aren't pulled into Redis.

CLI reference

usage: navproc_collector.py [options]

  --ship SHIP             Ship name / NATS bucket identifier (e.g. rcsn)
  --domain DOMAIN         NATS domain (default: rc)
  --redis-host HOST       Redis hostname (default: localhost)
  --redis-port PORT       Redis port (default: 6379)
  --poll-interval SECS    Collection frequency in seconds (default: 5)
  --keys-config PATH      TOML allow-list (default: backend/collector_keys.toml)
  --debug 0|1             Enable verbose logging (default: 0)

Example

uv run backend/navproc_collector.py \
  --ship rcsn \
  --domain rc \
  --redis-host redis \
  --poll-interval 10

Navigation Collector module

Configuration file

load_collector_keys

Class: ShipWatcher