There are some data management products for MUCE 2009 that are already available or planned; these are described in Section 2. There are several other products that have been requested, but there is more requested than time to complete the work. These are described in Section 3, and our goal is to prioritize them..
Mike McCann in Support Engineering routinely generates QC-level access to data and plots for many of the instruments logging to SSDS. The MUCE 2009 pageshows the current status of these QC products for the instruments in the lab.
This page includes access to each instrument's raw data stream(s), netCDF files containing that data, log files of the processing results, and QC plots for the full deployment and last 7 days of data. (All these data products will be 'reset' to 0 when the mooring is deployed.)
As it stands, Mike intends to provide QC plots and netCDF files for all the data-producing devices except the McLane Profiler[1510] and the Workhorse ADCP [1549] data stream (the summary stream for that is available, though), and hasn't entirely given up on providing those last two if processed data and time allows. (But the Medusa plots depend on good XML metadata from Mark/John.)
Examples of the (nicer-looking) 7-day plots are below, click on them to get a bigger version. (The third example is from an M0-deployed ADCP; you can see more deployments at the NetCDF files and plots index page.)
Kevin Gomes (now leading the Support Engineering software team) is producing a set of pages that will display plots from MUCE 2009 that are the equivalent of the CIMT Web Pages. As you can see at that page, the plots are organized by domain, and often include scientific data that has been post-processed (if not actually quality controlled). These pages will also refer to other pages, even off-site, if that provides the best version of the mooring's results.
The organization of this deployment's page will be the same as for the CIMT, except it will not include meteorological data. The summary page will be rewritten to describe this experiment. Comparing each page's content to the CIMT content, we have:
The vertical profiler could contribute to these pages, depending on what post-processing is possible on its data.
The following requests were identified in previous MUCE meetings and in conversations with Jim Barry, Bill Ussler, Charlie Paull, Andy Hamilton, and Mike McCann.
The McLane Vertical Profiler produces a binary data record which must be unpacked from SIAM/SSDS records and merged into a file, which is then processed by McLane third-party software. This was manually done before, as a one-off confirmation, but that does not scale. Principal components of this request include:
There already exist Matlab processes for the last step which could be modified for use in a workflow, and steps 2 and 3 are relatively straightforward given ASCII data. Step 1 will require either automatic execution of PC-based software, or rewrite of that software for another operating system (from info collected so far, this looks like it may be somewhat complex, but not out of the question).
Estimated time: (1) 3-8 days; (2) 4-8 days (3) Support Eng.? (4) 10 days or more
The following requests have been made for changes to the QC plots for the ADCPs:
Here are a few example plots from other deployments, for reference (click to enlarge). Note the AquaDopp plot uses summary data.
Estimated time: 2 to 4 days, depending on subsequent requests.
Another set of requests focus on the plotting capabilities. Currently available plotting for SSDS data is limited to pre-selected time intervals (e.g., 2 hours, 7 days) -- the post-processing software pre-generates the plots for faster access. (This is especially important for plots with lots of data.) This approach limits plotting features; current plots can't support zooming in and out, scaling either axis, or combining data from multiple instruments. (The principle so far has been to support those capabilities by providing netCDF files with the data, which the user can then look at in their own plotting utility.)
So requests for MUCE include:
Estimated time: (1) Depends on the availability and cost of suitable web plotting service software. (Likely to be at least 10 days, may approach 30 days if software doesn't effectively integrate with our site.) (2) Depends on what kind of blending is required; if it's blending identical record types this might only be 5-10 days.
A final set of requests are in the category of domain-specific analyses (though many of these may be reusable on other projects). These include the following:
Estimated time: Unfortunately I am not conversant with the data formats and algorithms involved for most of these steps, so it is hard to provide time estimates. A starting guess might be 5 to 15 days each, depending on whether the algorithms are in hand and the results are saved to or indexed in the SSDS.
An application that monitors SSDS data would be useful for both science and engineering results. As Tom notes, an SSDS application that scans incoming SIAM packets for error messages can detect important messages. In particular, if an instrument port goes into the overcurrent condition, no science data is generated—and the data will be lost until manual remedial action is taken. If no one is watching the telemetry stream, the overcurrent condition can be missed and a lot of data lost.
On the science side, individual data elements could be monitored for science thresholds and triggers. This would be useful for detecting events of scientific interest (turbidity, temperature extremes).
For the example above, Tom says automated detection of overcurrent is easy—check every incoming SIAM message packet for the word "OVRCURR". If detected, send email to the appropriate person. Then an operator can log onto the node via satellite and fix the problem.
Estimated time: A very simple test could be implemented within a day or even a few hours, but would only address one parameter. A simple testing framework might take 5 days, and could do multiple tests, but would not be particularly powerful (e.g., limited notification capabilities). More comprehensive systems are available and would be capable of more tests and more notification schemes (mail, cell phone text message, even phone alerts)—these would be more like 15 or 20 days.