J~j8dZddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z ej deeejjddlmZmZmZmZmZdZdZdd d d Zd Zd Zefd'dZd(dZd)dZ d)dZ!d*dZ"d+dZ#d,d!Z$d-d"Z%d-d#Z&d.d%Z'e(d&krej)e'dSdS)/aError escalation watchdog with a capped, graceful auto-reboot. Polls the mission-critical services' state files (camera + sensors -- deliberately NOT lumen/epaper, matching led_status.py's reasoning: a display or light glitch isn't mission failure and doesn't warrant a disruptive reboot). A per-service systemd restart (Restart=on-failure) is given first crack at self-healing; only once a service has remained continuously in error for `watchdog.escalation_after_seconds` does this watchdog escalate to a full Pi reboot. Reboots are capped at `watchdog.max_reboot_attempts` (persisted on /data, survives the reboot it's counting) so a permanently broken device can't send the Pi into a reboot-crash-loop for the rest of the mission, burning battery and SD wear for no benefit. Once the cap is hit, this watchdog gives up rebooting and reports its own state as "error" -- which is what makes led_status.py leave the LED in its blinking-BLUE fault indication for the remainder of the mission, since watchdog is itself one of led_status.py's monitored mission-critical services. If this process itself dies, its state file goes stale, which state_lib.has_error() also treats as an error -- so a watchdog crash surfaces the same way a watchdog-detected fault would, rather than silently going unsupervised. ) annotationsN)Path)DictListOptional) bounded_ioconfigevents reboot_guard state_lib<zbubblecam-led.servicezbubblecam-epaper.servicezbubblecam-lumen.service)ledepaperlumenxi, stale_afterfloatcg}tD]G\}}tj|}|,tj||s|||fH|S)z(name, unit) pairs for non-critical services whose heartbeat exists but has gone stale. Services with no state file at all are skipped -- never-started is a choice (e.g. lumen deliberately disabled), not a fault this remediation should fight.)NON_CRITICAL_UNITSitemsr read_stateis_freshappend)rstalenameunitstates IC:\Users\jvoigtlander\Desktop\BubbleCamCode\opt\bubblecam\bin\watchdog.pystale_noncritical_unitsrAsl E(..00'' d$T**  Y%7{%K%K  LL$ & & & LreturnboolcHtjddkS)N BUBBLECAM_SIM1)osenvirongetr r _simulater*Ns :>>/ * *c 11r cfgdict List[str]cV|dd}|drddgSdgS)Ncameramode dual_node camera_left camera_rightcamera_primary startswith)r+r0s rcamera_service_namesr7Us8 x= D {##/~..  r cn|dd}|drddgndg}d|DS)Nr/r0r1leftrightprimarycg|]}d|d S)zbubblecam-camera@z.servicer)).0instances r z(camera_systemd_units.._s$ M M Mx 2 2 2 2 M M Mr r5)r+r0 instancess rcamera_systemd_unitsrA\sF x= D%)__[%A%AR!! {I M M9 M M MMr cmdlisttimeoutNonec:tjfd|dS)a,Run a command, abandoning it (not just killing it) on timeout. On a machine whose storage has died, merely exec'ing systemctl can block forever in uninterruptible sleep -- the binary has to be paged in from the dead device. Python's subprocess timeout can't recover from that (it SIGKILLs then wait()s, and a D-state child never reaps), so the whole call runs on a throwaway thread instead. Observed for real: the watchdog froze inside a systemctl call mid-escalation and the Pi hung 12.5 hours with the fix one line away. c0tjdS)NFcheck) subprocessrun)rBsrz"_run_abandonable..ms s%(H(H(Hr rDN)rrun_with_timeout)rBrDs` r_run_abandonablerObs, H H H HRYZZZZZZr cdgt|zD]7}trtd|#tdd|gd8dS)aAsk ffmpeg (via each capture_supervisor) to stop gracefully so the current video segment gets finalized before we reboot out from under it. The record-switch daemon is stopped first: otherwise its reconcile loop would see stopped-but-switch-says-record units and restart them inside the pre-reboot grace window, un-finalizing the very segment this stop exists to protect. It comes back with everything else after the reboot.zbubblecam-record-switch.servicez[SIM WATCHDOG] systemctl stop systemctlstoprrMN)rAr*printrO)r+rs rstop_camera_servicesrTpsu336J36O6OOCC ;;  9499 : : : +vt4cBBBBB CCr video_dirrcvtrdSdfd }ttj|dS)aTime-bounded write probe of the data partition. A dead/wedged SD card doesn't announce itself: writes just never return, and every service that touches the card silently stops making progress while the kernel stays healthy enough that the hardware watchdog never fires. Observed in a real soak -- the Pi hung for 8.5 hours with no log line explaining it, because writing that log line needed the same dead card. This probe is what turns that silent hang into a reboot. Tr!r"c(dz }t|d5}|d|tj|dddn #1swxYwY|dS)Nz.bubblecam-storage-probewokT)openwriteflushr&fsyncfilenounlink) probe_pathfrUs r_probezstorage_healthy.._probes!;; *c " " !a GGDMMM GGIII HQXXZZ  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! tsAA33A7:A7F)defaultr!r")r*r"rrN)rUrbs` rstorage_healthyresT{{t  +FEBBB C CCr cjtrtddStddgddS)Nz[SIM WATCHDOG] systemctl rebootrQrebootrM)r*rSrOr)r rtrigger_rebootris@{{ /000k8,b999999r ctrtddS tdd5}|dddddS#1swxYwYdS#t$rt jddgd YdSwxYw) aReset via the kernel directly, bypassing systemd's shutdown sequence. Only reached when a graceful reboot didn't happen within FORCED_REBOOT_AFTER_SECONDS -- which on this hardware means systemd is blocked unmounting a filesystem whose device stopped responding. Data loss is not a concern at that point: the card already isn't accepting writes, and every completed segment is crash-safe by design. z&[SIM WATCHDOG] forced reboot via sysrqNz/proc/sysrq-triggerrXbrgz-fFrH)r*rSrZr[OSErrorrJrK)ras r force_rebootrms{{ 67776 ' - -  GGCLLL                   666$'u55555556s4A!A A!AA!AA!!"BBintc  tj}|d}|d}|d|d}|d}|d}|d}t|}t|dd }ii} d} d } d } t jdd t j} tj dd | rd| dndz tj | z tkr| }t|} } |r| stj ddn| r|stj ddtD]\}}| |}||z t kr+| |<tj d|d|t#rt%d|kt'dd|gdt j|}|r|ngdgz}g}|dgzD]]}||vrAt j||r,|||G|d^| s,d|dnddfd|D}|rdfd|D}t j|r:t jddd|d |!tj dd|d"|n4d|v}t jddd#|!|s#t7|tj|t j}t jddd$|d%|d&|!tj dd'|rd(nd)d*|d%|d&||r@t#s#t=tjd+nft=nWt?t#s;tjt@t=tjd+d} n|r0t jdd d,d-|!d} nMt jdd | } n3| z |kr*t jd.krt j!tj| )/Nwatchdogstale_after_secondsescalation_after_secondsmax_reboot_attemptshealthy_reset_after_seconds poll_secondsgraceful_stop_grace_secondspathsrUTgrunningz system upz -- after recovery reboot (z attempt(s) on record)z2STORAGE PROBE FAILED -- /data not accepting writeszstorage recoveredz heartbeat stale -- restarting z![SIM WATCHDOG] systemctl restart rQrestartr rMsensorsstoragec2g|]}|z k|Sr)r))r=r error_sinceescalation_afternows rr?zmain..s5   3T9J3JN^3^3^D3^3^3^r z; c3<K|]}|d|z ddVdS)z errored z.0fsNr))r=rr~rs r zmain..sBppTX$OO{47H1HOOOOppppppr errorzreboot attempts exhausted (z); giving up: )detailz) -- giving up: zescalating reboot: zrebooting now (attempt /z): z REBOOTING (forcedgracefulz , attempt rhz monitoring, not yet escalating: z, r)"r loadr7rr write_stater read_countr logtime monotonicSTORAGE_PROBE_INTERVAL_SECONDSrerr(%NON_CRITICAL_RESTART_COOLDOWN_SECONDSr*rSrOrecording_requested has_error setdefaultrpopjoin exhaustedrTsleep incrementrmriFORCED_REBOOT_AFTER_SECONDSreset)r+ watchdog_cfgr max_attemptshealthy_reset_afterrugraceful_stop_grace camera_namesrUnoncritical_restarted_at healthy_since storage_okstorage_checked_at boot_attemptswas_okrrlastrcritical_servicescurrently_erroringescalate_candidatesrstorage_failedattemptr~rrs @@@rmainrs3 +--Cz?L45K#$>? 56L&'DE/L&'DE',,LS\+.//I$&K13%)MJ *i000 +--M Jz;O\dKmKKKKbd{!n # #'E E EF(33J!$  ???$ 2 MM  %  !IY$))DV:W:WYY    !MM  !*i 8 8 8$ # }$(;;; @W@Y@Y\]@]@]"$$$ <   w{!r __main__)rrrd)r+r,r!r-)rBrCrDrr!rE)r+r,r!rE)rUrr!r")r!rE)r!rn)*__doc__ __future__rr&rJsysrpathlibrtypingrrrpathinsertstr__file__resolveparent bubblecamrr r r r rrr NON_CRITICAL_STALE_AFTER_SECONDSrrr*r7rArOrTrerirmr__name__exitr)r rrs12#"""""  ''''''''''33ttH~~--//6=>>???IIIIIIIIIIIIII "$ ! #( & $' ),%2R     2222NNNN [ [ [ [ C C C CDDDD2::::6666&T!T!T!T!n z CHTTVVr