Rich : MARS Datamanagement Overview This page last changed on Jul 26, 2012 by rich. Pismo as ssdsadmin mars to ssds architecture has three major subsystems. 1 Basic data acquisition happens on the mars pmacs server 2 The mars2ssds "stuff" runs on pismo, basically there are three main tasks that scrape data via web requests from the pmacs server archives, break them into pseudo 'ssds devices' and publish to rabbitMQ as ssds protocol buffers. (psc2rabbit, onewire2rabbit, and npc2rabbit tasks in /opt/ssds/mars2ssds) 3 Qmonitor (/opt/ssds/qmonitor) run perodically by cron, (/u/ssds/crontab.pismo) looks at rabbitmq messaging server, whenever it detects an 'active' device queue with no consumer, it starts a consumer for it, that pulls protocol buffer packets and pushes them to sql server dione device base tables in SSDS_Data. (Note: from home dir /u/ssdsadmin see special files crontab.pismo and .netrc Both mars2ssds and qmonitor projects can be checked out from svn repository in projects of those same names. The cronjob and .netrc are also tucked away for lack of a better place, in the qmonitor/bin dir, although they need to be deployed to /u/ssdsadmin on pismo.) view on ssds http://new-ssds.mbari.org:8080/ssds/siamRawDataStep1.jsp Mars devices are ssds device 1782 thru 1784 1772-1779 are MARS External Load 1-8 1780 Groundfault 1781 Onewire (HA7) 1782 PSC 1783 LVC 1784 MVC
    On Jul 25, 2012, at 8:19 PM, Rich Schramm wrote:

    three separate projects are checked in to svn:  mars2ssds, qmonitor and mars2web
    mars2web stuff goes to pmacs user acct on pmacs server. (see README.txt in the mars2web/bin dir)
    *** everything else below refers to user ssdsadmin on pismo for the mars2ssds and qmonitor stuff ***

    also the cronjob.pismo and the .netrc files are in svn in the qmonitor/bin directory

    everything should restart on reboot or if the database is rebooted etc. - needs further testing to verify

    run logs are in /opt/ssds/mars2ssds/log and /opt/ssds/qmonitor/log and are set to rotate.

    TODO:
    1)npc2rabbit has the external loads 1-8 turned off. We need to decide on in-row-storage and then run the bulk loads
    of legacy data (by running npc2server.py), then turn on the real-time loads (uncomment the lines in npc2rabbit.py)

    Future enhancement:
    1)Would be 'nice' if onewire/psc/npc2rabbit.py would query for the max(timestampseconds)
    for a device and pickup processing from there.... instead of attempting to reload
    all records from midnight of the day it is started.  very minor issue as the insert just ignores records already loaded
Attachments: MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) MARS DM Architecture Overview (text/xml) Mars Data Management Architecture (text/xml) Mars Data Management Architecture (text/xml)

Document generated by Confluence on Feb 04, 2026 08:50