# BubbleCam Headless Raspberry Pi Zero 2 W software for an underwater stereo bubble-plume camera (Milos, Greece deployment). **Start with [docs/USER_GUIDE.md](docs/USER_GUIDE.md)** -- the system's mental model: hardware map, service architecture, the recording pipeline, the reliability machinery, and the hard-won hardware knowledge (PWM conflicts, SD wedges, memory limits) that shaped the design. This README is the install/operations reference. ## Hardware summary | Signal | Pin | Function | |---|---|---| | I2C1 SDA/SCL | pin3 / pin5 (GPIO2/GPIO3) | DS3231 RTC (0x68) + BME280 (0x76) | | LED_DATA (WS2812B) | pin40 (GPIO21, PCM) | Status indicator + illumination ring chain (via level shifter). PCM on purpose: on a PWM pin, rpi_ws281x and the Lumen's kernel PWM driver clobber each other and the LED service freezes. | | PWM_LUMEN | pin12 (GPIO18, PWM0) | Blue Robotics Lumen dimming | | e-Paper SPI0 | SCLK=pin23, MOSI=pin19, CE0=pin24 | Waveshare 2.13" HAT (250x122) | | RECORD_SWITCH | pin37 (GPIO26), internal pull-up | Physical record start/stop (polarity set by `[record_switch].record_when_grounded`) | | INA219 | I2C1 (0x40) | Power monitor, in series with a 5V feed (optional) | | Camera | OTG micro-USB (UVC) | MMLove stereo camera | 2TB microSD (see "SD card partitioning" below), single Pololu 5V/2.5A buck, 10S4P alkaline pack. ## Repository layout This repo mirrors the real deployment paths so `install.sh` is a straight copy: ``` opt/bubblecam/bin/*.py -> /opt/bubblecam/bin/ (entry point scripts) opt/bubblecam/bubblecam/ -> /opt/bubblecam/bubblecam/ (shared library + hardware wrappers) etc/systemd/system/*.service -> /etc/systemd/system/ etc/tmpfiles.d/bubblecam.conf -> /etc/tmpfiles.d/ boot/firmware/bubblecam.toml -> /boot/firmware/bubblecam.toml (hand-editable config) usr/local/sbin/bubblecam-arm -> /usr/local/sbin/bubblecam-arm (pre-seal script) ``` ## One-time setup (with internet access, before deployment) 1. Flash Raspberry Pi OS Lite (64-bit, Bookworm) and boot it headless (enable SSH + Wi-Fi via Raspberry Pi Imager's advanced options, or a `firstrun`/`userconf.txt` + `wpa_supplicant.conf` drop-in). 2. Clone/copy this repo onto the Pi. 3. **Probe the camera's real format before anything else** -- plug it into the OTG port and run: ```bash v4l2-ctl --list-devices v4l2-ctl --list-formats-ext -d /dev/video0 ``` Set `[camera].mode` in `boot/firmware/bubblecam.toml` accordingly (`sbs_mjpeg` | `sbs_h264` | `dual_node_mjpeg` | `dual_node_h264`) before running the installer. 4. Confirm `opt/bubblecam/bubblecam/hardware/epaper.py`'s `EPD_DRIVER_MODULE` matches your e-Paper HAT's hardware revision (check the sticker on the HAT PCB) -- the Waveshare driver itself is installed from their git repo by `install.sh`. 5. Run the installer: ```bash sudo ./install.sh sudo reboot ``` 6. After reboot, verify hardware bring-up: ```bash i2cdetect -y 1 # expect 0x68 (RTC) and 0x76 (BME280) systemctl status 'bubblecam-*' journalctl -u bubblecam-camera@primary.service -f ``` ## SD card partitioning (2TB card, Windows-readable data partition) The Pi Zero 2 W's Wi-Fi is far too slow to pull days of video, so the workflow is: shut the Pi down, pull the card, and read the data partition directly in a laptop's card reader. Windows can't read ext4, so `/data` lives on its own exFAT partition (readable on Windows/macOS/Linux) while the OS stays on ext4. Target layout (MBR -- the Pi bootloader requires it; a 2TB card is ~1.86TiB, safely under MBR's 2TiB limit): | # | Type | Size | Purpose | |---|---|---|---| | 1 | FAT32 | 512MB | `/boot/firmware` (also holds `bubblecam.toml`) | | 2 | ext4 | to 32GB | `/` root | | 3 | exFAT | rest (~1.9TB) | `/data` (video + sensor CSVs), Windows-readable | Procedure: 1. Flash Raspberry Pi OS Lite (64-bit) with Raspberry Pi Imager (set hostname/user/Wi-Fi/SSH in its customization dialog). 2. Before first boot, open the card's boot partition and edit `cmdline.txt`: delete the ` init=/usr/lib/raspi-config/init_resize.sh` token at the end of the line. This stops the root partition auto-expanding to fill all 2TB on first boot. 3. Boot the Pi, SSH in, then grow root to 32GB and create the data partition (interactive `parted` -- answer Yes to the "partition in use" warning): ```bash sudo parted /dev/mmcblk0 # (parted) resizepart 2 32GB # (parted) mkpart primary 32GB 100% # (parted) quit sudo resize2fs /dev/mmcblk0p2 ``` 4. Set the new partition's MBR type to 7 (Windows "IFS"): parted tags it as type 0x83 "Linux" by default, and Windows refuses to mount 0x83 partitions no matter what filesystem is inside. Metadata-only, safe: ```bash sudo sfdisk --part-type /dev/mmcblk0 3 7 ``` 5. Format the data partition as exFAT and mount it at /data: ```bash sudo apt install -y exfatprogs sudo mkfs.exfat -L BUBBLEDATA /dev/mmcblk0p3 sudo mkdir -p /data # uid/gid 1000 = the login user, so manual ffmpeg test captures and # file cleanup work without sudo. Services run as root and write # regardless. Ownership is meaningless once the card is read on Windows. echo 'LABEL=BUBBLEDATA /data exfat defaults,nofail,uid=1000,gid=1000 0 0' | sudo tee -a /etc/fstab sudo systemctl daemon-reload && sudo mount /data ``` 6. Continue with the normal one-time setup above (`install.sh` creates `/data/video` and `/data/sensors`; the shipped `[paths]` config already points there, so no config change is needed). To retrieve data: `sudo poweroff`, wait for the green activity LED to stop, pull the card. In Windows the `BUBBLEDATA` partition mounts like any USB drive with `video/` and `sensors/` folders. **Never pull the card while the Pi is powered** -- a mid-write removal can corrupt the current video segment (everything already closed is crash-safe Matroska and survives). ## Record switch A physical two-position switch between GPIO26 (physical pin 37) and GND (pin 39, adjacent) starts and stops camera recording -- no SSH needed. Polarity is configurable via `[record_switch].record_when_grounded`: - **true (current setting)**: switch closed = RECORD, open = stop. The intuitive "flip on to record" feel. Trade-off: a broken or unplugged switch wire mid-mission reads "stop" and silently ends recording. - **false**: open/broken = RECORD, closed = stop. Fail-safe -- a wiring fault cannot end the mission. Worth considering for the real sealed deployment. `bubblecam-record-switch.service` polls the pin (debounced) and starts/stops the `bubblecam-camera@*` capture units to match; the watchdog and status LED both understand that a switch-stopped camera is not a fault (no error blink, no recovery reboot), while sensors keep logging regardless. The e-paper display follows along: while recording it shows only the data screen; while stopped it alternates between the data screen and a live camera preview. ## Configuration Edit `/boot/firmware/bubblecam.toml` (readable/writable from a laptop's SD card reader with no SSH needed) then reboot -- there is no live-reload by design. See the comments in that file for every tunable (Lumen brightness, segment lengths, LED timing, watchdog thresholds, dusk definition, GPS coordinates). **Set the real dive-site GPS coordinates before deployment** -- the shipped values are an island-center placeholder. ## Clock sync There's no internet in the field, so don't rely on NTP. From your laptop, over the last SSH session before sealing: ```bash ssh pi@bubblecam.local "sudo date -u -s '$(date -u +%Y-%m-%dT%H:%M:%S)' && sudo hwclock -w" ``` This pushes your laptop's current UTC time to the Pi's system clock, then writes it to the DS3231 hardware RTC. On every boot, `bubblecam-rtc-sync.service` reads it back (`hwclock -s`) before anything time-dependent (sensors, dusk/dawn calc, watchdog) starts. ## Pre-deployment arming Run this over your last SSH session before sealing the enclosure: ```bash sudo bubblecam-arm ``` It checks all services report healthy, checks free disk space and clock sanity, then (after explicit confirmation) disables Wi-Fi/Bluetooth and stops mDNS. This is the last thing you should do before sealing -- it ends the SSH session itself. ## Status LED reference (for divers) Status shows on the Blue Robotics RGB Indicator (the last pixel of the GPIO13 chain); the 40-LED ring ahead of it is camera illumination and lights **white while recording** (`[led].ring_white_brightness`), independent of the status color. | Indicator pattern | Meaning | |---|---| | Blinking RED (0.5s on / 2.5s off) | Powered on, not recording | | Solid GREEN | Recording confirmed (camera + sensors both running), first 10 minutes of this recording session | | Brief GREEN flash (0.2s every ~10s) | Still recording, past the 10-minute window -- minimal-power heartbeat | | Blinking BLUE (brief pulse every ~5s) | A mission-critical fault (camera or sensor logging) -- the system will attempt an automatic recovery reboot (up to 3 times) before settling into this state permanently | | Fully dark | No power, or the LED service itself is down | ## Architecture notes - **State sharing**: every service writes exactly one file under `/run/bubblecam/*.json` (tmpfs, atomic writes); other services only read. See `opt/bubblecam/bubblecam/state_lib.py`. - **Camera capture**: `ffmpeg -c copy` only -- never re-encodes. MJPEG modes segment into raw `.mjpeg` files (bare concatenated JPEG frames: inherently crash-safe -- truncation only ever costs the final frame -- and trivially parseable); H.264 modes use crash-safe Matroska (`.mkv`). Never MP4 (moov-atom-at-close corruption on sudden power loss). To play a raw `.mjpeg` (no embedded framerate -- supply the configured one): `ffplay -f mjpeg -framerate 120 -i cam_primary_X.mjpeg`, or losslessly rewrap for normal players: `ffmpeg -f mjpeg -framerate 120 -i cam_primary_X.mjpeg -c copy out.mkv`. - **Error recovery**: `bubblecam-watchdog.service` escalates a persistently-erroring mission-critical service to a full Pi reboot, capped at 3 attempts (counter on `/data/bubblecam/reboot_attempts.count`, survives reboot) so a permanently broken device can't crash-loop for the rest of the mission. - **Everything hardware-facing has a simulate mode**: set `BUBBLECAM_SIM=1` to run any service against fake sensors/LED/PWM/e-paper, for dev-machine testing with no physical hardware attached. ## Testing without hardware ```bash export BUBBLECAM_SIM=1 export BUBBLECAM_STATE_DIR=/tmp/bubblecam-run export BUBBLECAM_CONFIG=/tmp/bubblecam-test.toml # optional, falls back to defaults python3 opt/bubblecam/bin/led_status.py # or sensor_logger.py, lumen_control.py, etc. ``` Each service will print `[SIM ...]` lines instead of touching real hardware and write its normal state file to `$BUBBLECAM_STATE_DIR`, which you can inspect with `cat`. ## Camera bitrate calibration The camera is a confirmed MMlove USB Stereo Camera Module (OG02B10 sensor), UVC, single combined side-by-side MJPEG frame -- `[camera].mode = "sbs_mjpeg"` is settled, not a guess. What's still open is the exact resolution/framerate, because MJPEG has no temporal compression (unlike the H.264 assumption in early planning), so bitrate isn't tunable to a target the way a CBR H.264 encoder would be -- it's roughly proportional to resolution x framerate at whatever JPEG quality the firmware fixes. `bubblecam.toml` ships with `resolution = "3200x1200"`, `framerate = 15` -- full sensor resolution (1600x1200 per eye). **Measured on the real hardware**: ~400KB/frame, **~6 MB/s**, ~21 GB/hour, so the 1.9TB card holds roughly **88 hours**. The camera's own format list confirms 15fps is natively supported at this resolution (`v4l2-ctl --list-formats-ext`), so the rate is honored rather than silently renegotiated. Two hardware ceilings this sits under, worth re-checking if you change resolution or framerate: - **SD write throughput**: measured ~20 MB/s (`dd if=/dev/zero of=/data/x bs=1M count=512 conv=fsync`). Capture at ~6 MB/s leaves comfortable headroom; pushing past roughly half the card's ceiling backs up the write queue, which stalls the whole system and drops Wi-Fi. - **Capture memory**: the driver reports 7.68MB per V4L2 buffer at 3200x1200. See `[camera].capture_buffers` -- and the note in capture_supervisor.py about why the MJPEG path uses `v4l2-ctl` instead of letting ffmpeg open the device. **If you change resolution or framerate**, re-run the measurement: ```bash # Stop everything first so the device is free and memory is at maximum. # NB: not `systemctl stop 'bubblecam-*'` -- see the rtc-sync note below. sudo systemctl stop bubblecam-epaper.service bubblecam-camera@primary.service \ bubblecam-record-switch.service bubblecam-sensors.service \ bubblecam-lumen.service bubblecam-watchdog.service bubblecam-led.service # Capture 150 frames (10s at 15fps) the same way the recorder does. v4l2-ctl -d /dev/video0 \ --set-fmt-video=width=3200,height=1200,pixelformat=MJPG \ --set-parm=15 --stream-mmap=3 --stream-count=150 \ --stream-to=/data/video/cal.mjpeg ls -l /data/video/cal.mjpeg # bytes / 150 = per-frame; x framerate = MB/s ``` Use `v4l2-ctl`, not a bare `ffmpeg -f v4l2 ...`: ffmpeg allocates 32 capture buffers, which at 7.68MB each is ~246MB and gets OOM-killed on a 512MB board. That is a property of the measuring tool, not the camera. Multiply MB/s by 3600 for GB/hour, then against your mission duration, and compare to the ~1.9TB partition and the card's ~20 MB/s write ceiling. If it overshoots either, step down to `2560x720` (still 1280x720 per eye) or lower the framerate. The camera's full native mode list: `3200x1200@60` | `2560x720@60` | `1600x600@120` | `1280x480@120` | `640x240@120`. ## Troubleshooting: everything stops at once, no error in the logs Symptom: video and sensor writes both stop at the same moment, the journal's last entry is a few minutes *before* the last written data, and the Pi is unresponsive but never rebooted. This is storage failing underneath a running system. Processes touching the card block in uninterruptible I/O and never return; the kernel stays healthy enough that the hardware watchdog keeps being petted, so nothing resets. No log line explains it because writing that line needs the same dead card. Observed once for real: hung at 06:48 after ~5.5 hours of recording and sat there 8.5 hours. It didn't self-recover because `reboot_guard`'s counter lived on `/data` -- the watchdog's first escalation step read a file on the filesystem that had just died, and blocked forever. The rescue path depended on the thing that failed. Now guarded: all counter access is time-bounded (`bubblecam/bounded_io.py`) and fails toward rebooting; the watchdog write-probes `/data` every 60s and treats a wedged partition as mission-critical; and if a graceful reboot stalls (systemd blocking on unmounting the hung filesystem) it escalates to a kernel-level reset via sysrq after 60s. To isolate a suspect card from everything else, soak-test writes alone with no camera or services involved: ```bash dd if=/dev/zero of=/data/soak.bin bs=1M count=200000 conv=fdatasync status=progress rm /data/soak.bin ``` If that dies partway, the card (or its power) is the problem, not the software. Attach an HDMI monitor before a long soak: kernel I/O errors print to the console even when they can't be written to disk. ## Troubleshooting: stopping services Use explicit unit names, not a glob: ```bash sudo systemctl stop bubblecam-epaper.service bubblecam-camera@primary.service \ bubblecam-record-switch.service bubblecam-sensors.service \ bubblecam-lumen.service bubblecam-watchdog.service bubblecam-led.service ``` `systemctl stop 'bubblecam-*'` also matches `bubblecam-rtc-sync.service`, a boot-time oneshot ordered `Before=basic.target`. A stop job on it stalls until timeout and takes the rest of the transaction down with it -- the e-paper service stays running and keeps holding the camera, which then makes any manual `ffmpeg`/`v4l2-ctl` capture hang with no error. The unit now sets `RefuseManualStop=yes` so this fails loudly instead of silently. ## Troubleshooting: Pi runs but never joins Wi-Fi Symptom: the Pi boots, the e-paper flashes on its normal refresh cycle, but it never appears on the network and SSH is impossible. Check the previous boot's log for the OOM killer: ```bash journalctl -b -1 --no-pager | grep -i 'oom-killer' ``` The Zero 2 W has 512MB and `install.sh` disables swap. When something exhausts RAM, the kernel's OOM killer picks victims by size -- and NetworkManager/wpa_supplicant are large enough to be attractive targets. The machine keeps running, so lightweight services (e-paper, sensors) look healthy, while the network is gone for good until reboot. This happened once for real: the e-paper preview was grabbing frames at the full recording resolution (3200x1200), and decoding a 3.8-megapixel frame every few seconds to draw a 250x122 monochrome image exhausted memory. Guards now in place: previews capture at `epaper.preview_resolution` (default 640x240), and `bubblecam-epaper.service` carries `MemoryMax=150M` plus `OOMScoreAdjust=500` so a runaway preview kills only itself. If you raise any capture resolution, keep the preview resolution small. ## Verification checklist (with real hardware, before the actual deployment) 1. `i2cdetect -y 1` shows 0x68 and 0x76; e-paper responds; WS2812B lights via a manual test. 2. Run the camera bitrate calibration test above and finalize `[camera].resolution`/`framerate`. 3. Run `capture_supervisor.py` manually, pull power mid-segment, confirm the `.mkv` is still playable up to the truncation point. 4. Full boot: confirm LED RED -> GREEN timing reflects genuinely confirmed recording (kill ffmpeg mid-run, confirm the LED escalates rather than staying green). 5. Force an error, confirm the blink pattern and the watchdog's reboot-then-give-up behavior over 3+ induced failures. 6. Set the RTC across a simulated day/night boundary, confirm Lumen PWM engages/disengages at the right transition and brightness. 7. Full-duration soak test on the real battery pack before the actual deployment -- watch card fill rate against the storage budget. ## Open items to resolve before deployment 1. ~~Final resolution/framerate within the storage budget~~ -- MEASURED: 3200x1200@15fps runs 20-29 GB/hour depending on scene brightness (MJPEG frame size tracks scene complexity). 72h at the observed 16h average (19.6 GB/h) is ~1.41TB against the 1.9TB partition; re-check against real underwater scene brightness, and drop to 10fps if tight. 2. ~~USB2 sustained throughput~~ -- CONFIRMED: ~6-8 MB/s sustained for 16h+ continuous capture. 3. ~~WS2812B (`rpi_ws281x`/PWM1) + Lumen (sysfs/PWM0) coexistence~~ -- RESOLVED: they conflict (both program the same PWM peripheral; the LED service froze solid at the Lumen dusk transition). Current config is safe via `[lumen].enabled = false` (no Lumen hardware attached; the service computes dusk/dawn only). BEFORE enabling a real Lumen, move the LED data wire to physical pin 40 and set `[led].gpio_pin = 21` (PCM peripheral, no shared silicon). 4. Real Milos dive-site GPS coordinates -- the shipped placeholder (~36.7°N, 24.42°E) is considered close enough; swap in the real fix if available. 5. 72-hour qualification soak + pull-power-mid-recording playability test (see the USER_GUIDE pre-deployment checklist). 6. Decide `[record_switch].record_when_grounded` polarity for the sealed deployment (currently true = closed-to-record; a broken switch wire then reads "stop").