# BubbleCam User Guide A self-contained underwater stereo camera system on a Raspberry Pi Zero 2 W, built to record a seafloor bubble plume for ~72 hours unattended (Milos, Greece), with no network in the field. This guide explains the hardware, the software architecture, how to operate it, and — importantly — *why* things are built the way they are, including the failures that shaped the design. For install steps, SD-card partitioning, and troubleshooting recipes, see [README.md](../README.md). This document is the mental model. --- ## 1. What the system does Flip a physical switch: the camera records 3200x1200 stereo video (1600x1200 per eye, 15fps MJPEG) in crash-safe hourly files, an LED ring lights the scene when it's dark, and a potted RGB indicator tells a diver at a glance whether recording is genuinely happening. Alongside video, a CSV logs temperature, humidity, pressure, supply voltage/current, and light state every 10 seconds. An e-paper screen shows live status and camera previews during topside checkout. Everything of interest is written to a Windows-readable exFAT partition: pull the SD card, plug it into a laptop, and the videos, sensor CSVs, and a human-readable event log are right there. A layered watchdog reboots the system out of otherwise-fatal failures (including a dead SD card) automatically. The design goal everywhere: **a failure costs minutes, never the mission.** ## 2. Hardware map | Component | Connection | Notes | |---|---|---| | MMlove stereo USB camera (OG02B10) | micro-USB OTG port (UVC) | Emits hardware-encoded MJPEG; the Pi never encodes | | BME280 (temp/humidity/pressure) | I2C1, addr 0x76 | pins 3/5 (SDA/SCL) | | DS3231 RTC | I2C1, addr 0x68 (`UU` = kernel-owned) | Field timekeeping; no NTP underwater. Carries EEPROM at 0x57 | | INA219 power monitor | I2C1, addr 0x40 | High-side, in series with the system 5V feed | | Waveshare 2.13" e-paper HAT (V4) | SPI0 + GPIO17/24/25 | 250x122, 1-bit | | WS2812B chain: BR indicator + 40-LED ring | GPIO13 (pin 33) → 3.3→5V level shifter | See wiring below | | Record switch | GPIO26 (pin 37) ↔ GND (pin 39) | Internal pull-up; polarity configurable | | Blue Robotics Lumen | GPIO18 PWM (when attached) | **See PWM conflict warning, §8** | **LED chain wiring** (order is load-bearing, see §8): ``` GPIO13 (pin 33) → level shifter LV→HV → BR indicator GREEN (data in) BR indicator WHITE (data out) → ring DIN ``` The Blue Robotics indicator datasheet requires data within 0.5V of its 5V supply — the level shifter is mandatory, never raw 3.3V GPIO. A 300–470Ω series resistor at the indicator's data input and a 500–1000µF cap across the ring's 5V/GND are recommended hardening for deployment. **Storage layout** (2TB SanDisk Extreme microSD, MBR): p1 FAT32 512MB `/boot/firmware` (holds the editable config) · p2 ext4 32GB root · p3 exFAT ~1.9TB `/data` — Windows-readable, MBR type 0x07. ## 3. Software architecture Everything lives in `/opt/bubblecam` (mirrored in this repo under `opt/bubblecam/`). One systemd service per concern; services never call each other. They communicate only through **state files**: each service atomically writes exactly one JSON file under `/run/bubblecam/` (tmpfs) with its status and a heartbeat timestamp, and anyone may read anyone's. A service whose heartbeat goes stale is treated as faulty — a crashed *or wedged* process betrays itself by silence. | Service | Script | Role | Mission-critical? | |---|---|---|---| | `bubblecam-camera@primary` | `bin/capture_supervisor.py` | Runs and verifies the capture pipeline | Yes (while switch says record) | | `bubblecam-sensors` | `bin/sensor_logger.py` | BME280 + INA219 + light state → CSV every 10s | Yes | | `bubblecam-watchdog` | `bin/watchdog.py` | Detects faults, restarts/reboots to recover | Yes (self-monitoring) | | `bubblecam-record-switch` | `bin/record_switch.py` | Polls GPIO26, starts/stops camera units | — | | `bubblecam-led` | `bin/led_status.py` | Status indicator patterns + illumination ring | No (healed if wedged) | | `bubblecam-epaper` | `bin/epaper_display.py` | Status screen + camera preview when idle | No (healed if wedged) | | `bubblecam-lumen` | `bin/lumen_control.py` | Dusk/dawn calc; Lumen PWM when enabled | No (healed if wedged) | | `bubblecam-rtc-sync` | (oneshot) | Seeds system clock from DS3231 at boot | — | Shared library (`bubblecam/`): `config.py` (TOML + defaults), `state_lib.py` (atomic state files, staleness logic), `events.py` (mission event log), `bounded_io.py` (time-limited storage access — see §7), `reboot_guard.py` (persistent reboot-attempt counter), and `hardware/` wrappers, every one of which has a `BUBBLECAM_SIM=1` mode so the entire system runs on a dev machine with zero hardware. ## 4. The recording pipeline The Pi never encodes video. The camera's own silicon compresses each frame to JPEG (~400KB at 3200x1200); the Pi's job is moving and slicing bytes: ``` camera (MJPEG in hardware) → USB/UVC → v4l2-ctl --stream-mmap=3 → pipe → ffmpeg -c copy -f segment → hourly cam_primary_.mjpeg files on /data ``` Three deliberate choices: - **`v4l2-ctl` as the capture source, not ffmpeg.** ffmpeg's V4L2 reader hardcodes 32 capture buffers = ~246MB at this resolution, which OOM-kills a 512MB Pi. `v4l2-ctl --stream-mmap=3` does it in ~23MB (`[camera].capture_buffers`). - **Raw `.mjpeg` files** — bare concatenated JPEGs, no container. Power loss costs at most the final partial frame; every earlier frame is independently recoverable. Playback needs a framerate hint: `ffplay -f mjpeg -framerate 15 -i file.mjpeg`, or losslessly rewrap: `ffmpeg -f mjpeg -framerate 15 -i file.mjpeg -c copy out.mkv`. - **Verified progress, not liveness.** The supervisor reports `running` only when the newest file *actually grows*. That's what the indicator's green really means. Measured rates: ~5.5–8 MB/s depending on scene complexity (dark scenes compress smaller), i.e. roughly 20–29 GB/hour. Budget against the ~1.9TB partition accordingly. ## 5. Controls and displays **Record switch (GPIO26 ↔ GND):** polarity set by `[record_switch].record_when_grounded`. Currently `true`: closed = record, open = stop. NOTE: with this polarity a broken switch wire silently stops recording; consider `false` (fail-toward-recording) before a sealed deployment. The daemon debounces (two consecutive 1s polls) and *reconciles every poll*, so a camera unit that systemd restarted behind its back is corrected within a second. **Status indicator** (the potted BR pixel — the diver's interface): | Pattern | Meaning | |---|---| | Red blink (0.5s/2.5s) | Powered, not recording | | Solid green | Recording confirmed; first 10 min of this session | | Green flash (0.2s per 10s) | Still recording (power-saving heartbeat) | | Blue blink (0.5s/5s) | Mission-critical fault; auto-recovery in progress (or exhausted) | | Dark | No power, or LED service down | **Illumination ring** (40 LEDs): policy `[led].ring_mode` — currently `night_and_recording` (lit only when recording AND dark per the dusk/dawn calculation). Brightness `ring_white_brightness` (128 ≈ 1.2A; 255 ≈ 2.4A — mind the supply). Turns on/off via a 2s soft ramp so the shared 5V rail never sees a load step. **E-paper:** while recording, a data screen every 10s (UTC, recording state, BME280, SD usage, CPU temp). While idle, alternates data with a live dual-eye camera preview every 5s (grabbed at 640x240 — never the recording resolution; see §8). ## 6. The data that comes back Everything on the exFAT partition, readable directly on Windows: - `/data/video/cam_primary_.mjpeg` — hourly video segments. - `/data/sensors/sensors_.csv` — 10s samples: BME280, lumen brightness %, INA219 volts/amps/watts. Blank cells mean "unknown", never a fake zero. - `/data/bubblecam/events.log` — **read this first.** One timestamped line per major event: boots (annotated when they follow a recovery reboot), switch flips, recording confirmed/stopped, hourly rollovers, ring/light transitions, dusk/dawn, storage failures, watchdog actions. A healthy night is a metronome of `segment rollover` lines; trouble writes its own story. (A dead SD card can't record its own failure — that appears instead as a gap followed by `system up -- after recovery reboot`.) - `/data/bubblecam/reboot_attempts.count` — present only if the watchdog escalated to reboots during an unresolved failure window. Pull the card only when powered off (`sudo poweroff`, wait for the activity LED to stop) or with recording stopped via the switch. ## 7. The reliability machinery Layered, because each layer covers the one below it failing: 1. **Per-service**: systemd `Restart=on-failure`; the capture supervisor exits nonzero when progress stalls, forcing a clean pipeline rebuild. 2. **Watchdog process** (`watchdog.py`): polls state-file heartbeats. A mission-critical service continuously faulty for 120s → graceful reboot, capped at 3 attempts per failure window (counter persists on /data, resets after 5 healthy minutes). A switch-stopped camera is *not* a fault. 3. **Storage probe**: every 60s the watchdog write-probes /data. A wedged SD card (the system's one recurring real-world failure — see §8) fails the probe, and because nothing can be gracefully saved on dead storage, escalation goes **directly to a kernel reset** via `/proc/sysrq-trigger` — a pure syscall requiring nothing from disk. Every filesystem access on this path runs through `bounded_io.run_with_timeout()` (abandonable worker threads), because a dead card makes ordinary reads block forever in uninterruptible sleep — and even *exec'ing systemctl* needs the disk. 4. **Wedged-but-alive healing**: non-critical services (LED, e-paper, lumen) whose heartbeat goes stale while systemd still says "active" get a `systemctl restart` (5-min cooldown per service). 5. **Hardware watchdog**: `RuntimeWatchdogSec=15s` catches a dead kernel; `RebootWatchdogSec=2min` catches a shutdown that hangs unmounting. Quieted background writers (install.sh): zram writeback, apt-daily, man-db, dpkg-backup, e2scrub timers — the SD card serves capture, CSVs, and the journal, nothing else. ## 8. Hard-won hardware knowledge (do not relearn these) - **The PWM conflict (worst one).** rpi_ws281x on GPIO13/19 programs the same PWM silicon the kernel's sysfs PWM driver uses for the Lumen on GPIO18. One sysfs write while LEDs are rendering freezes the LED service solid (alive per systemd, heartbeat dead). Current config is safe ONLY because `[lumen].enabled = false` keeps the kernel driver's hands off the hardware (the service still computes dusk/dawn). **Before enabling a real Lumen: move LED data to physical pin 40 and set `[led].gpio_pin = 21`** — that's the PCM peripheral, which shares nothing with PWM. - **The SD wedge.** Twice, hours into recording, the entire MMC device stopped responding (journal dies first, buffered video drains ~3 min, machine hangs; no kernel log possible). Card is genuine, passed a 210GB isolation soak — the trigger involves concurrent camera USB traffic and/or the since-disabled zram writeback (the journal's last entry before one wedge). First clean 16h+ run followed those fixes. Recovery layer 3 exists because of this failure. - **The BR indicator quirks.** (a) It displays colors RGB-ordered despite its datasheet saying GRB — `[led].indicator_order = "rgb"` applies the swap; verify with a red frame if hardware changes. (b) Its DATA OUT repeats the full input stream instead of consuming one word, so anything chained after it re-displays the indicator's own word. It must sit FIRST with the ring after it (also convenient: the bench ring's own DOUT pad is dead). - **Memory is the scarcest resource.** 512MB total, no swap-to-disk. ffmpeg's 32-buffer V4L2 default, and an e-paper preview once grabbing frames at full recording resolution, each OOM-killed the system — and the OOM killer's favorite victims are NetworkManager/wpa_supplicant, producing "recording works but the Pi vanished from Wi-Fi". `gpu_mem=16` reclaims 48MB; the e-paper service is memory-capped (`MemoryMax=150M`) and marked the preferred OOM victim, while sensors are protected. - **Bookworm/Trixie platform trivia** that cost real hours: config.txt appends after the stock `[cm4]/[cm5]` sections are silently ignored (the block must start with `[all]`); `/dev/i2c-1` needs the `i2c-dev` module, not just `dtparam=i2c_arm=on`; `hwclock` moved to `util-linux-extra`; the right BME280 library is `RPi.bme280`, not `bme280`; Windows won't mount an MBR partition typed 0x83 no matter what filesystem is inside; `StartLimitIntervalSec` under `[Service]` is silently ignored (belongs in `[Unit]`). ## 9. Everyday operations ```bash # Health at a glance systemctl --failed; tail -8 /data/bubblecam/events.log for f in /run/bubblecam/*.json; do echo "== $f"; cat "$f"; echo; done # Is the LED/e-paper loop alive (not just "active")? Heartbeat must be fresh: cat /run/bubblecam/led.json; date +%s # Live power/system draw tail -3 "$(ls -t /data/sensors/*.csv | head -1)" # Stop services for bench work (NOT a glob -- rtc-sync refuses manual stop): sudo systemctl stop bubblecam-epaper.service bubblecam-camera@primary.service \ bubblecam-record-switch.service bubblecam-sensors.service \ bubblecam-lumen.service bubblecam-watchdog.service bubblecam-led.service ``` Config changes: edit `/boot/firmware/bubblecam.toml` (or from a laptop via the card's boot partition), then restart the affected service — config is read once at service start, no live reload. Bench test scripts (`led_ring_test.py`, `epaper_test.py`, `ina219_test.py`) live in `/opt/bubblecam/bin/` and want the services stopped first. Dev-machine testing: `pip install -r requirements-dev.txt && pytest` (124 tests, no hardware needed). `BUBBLECAM_SIM=1` runs any service against simulated hardware. ## 10. Pre-deployment checklist 1. 72-hour continuous bench soak, wall power, deployment config — the events log is the qualification record. 2. Pull-power-mid-recording test: yank power while recording, confirm the truncated segment plays to the cut. 3. Storage budget vs. real scene brightness (20–29 GB/hour observed range × mission hours vs 1.9TB). 4. Real dive-site GPS in `[lumen]` (dusk/dawn timing) — still the Milos island-center placeholder. 5. Decide `[record_switch].record_when_grounded` (fail-safe vs intuitive). 6. If deploying the Lumen: pin-40/GPIO21 migration FIRST (§8). 7. Series resistor + bulk cap on the LED chain (§2). 8. Sync clocks (`date`, `hwclock -w`), then `sudo bubblecam-arm` on the final SSH session: pre-seal checks, then kills the radios.