This list no longer updated This task list is being kept for posterity sake and that it has some desired tasks that were shelved for a later date. The task that are currently being worked on are managed in the project plan located here

The current work (tasks) for SSDS are basically targeted at these main areas:

1. Cleanup Configuration Management
2. Internal Application Consolidation
3. Metadata Integrity Checking\-Repairing\-Enhancing
4. Improve Testing
5. Enhance Access Interfaces
6. Improve Data Ingest Mechanisms
7. Documentation
8. Prepare for Opening to Community

900823 SSDS Hardening Original Tasks List (Proposal)

1. Cleanup Configuration Management (4 days - 3 KG, 1 MM)
   1. (2 days) Finish build overhaul (done on Mac) to make sure SSDS can be built easily (common sense defaults) and that all components work with JBoss/MySQL combination.
   2. (.5 day) Setup javadoc deployment as part of build task
   3. (.5 day) Verify that wrapper generator unit test are on during test target of build.
   4. (.5 day) Create some template startup scripts and document
2. Internal application Consolidation (10 days SE - 5 KG, 5 MM, 10 days I.S.)
   1. (1 day) Look at SSDS_Metadata database for all urls that point to ssds.shore.mbari.org and see if we can point those to new-ssds or DODS.
   2. (1 day) Look at SSDS_Metadata database for all urls that point to dods.shore.mbari.org and see if we can point those to dods.mbari.org.
   3. (.5 day) Shutdown web server on predator (dods too).
   4. (.5 day) Look at all web applications on predator and see if we simply CNAME ssds.shore.mbari.org to new-ssds, that will work OK.
   5. (.5 day) Shutdown jboss on predator.
   6. (.5 day) Plan shutdown time for predator.
   7. (1 day) Backup files off predator. (should we migrate UpdateBot and Graphing stuff to elvis?).
   8. (0 day for SSDS-I.S. request) Have Pat upgrade predator to RHE.
   9. (.5 day) Reinstall updateBot and graphing software and restart.
   10. (2 day) Figure out what ssds.mbari.org is right now, move content to web application and CNAME ssds.mbari.org to new-ssds.mbari.org (or rename new-ssds.mbari.org to ssds.mbari.org and CNAME new-ssds to ssds.mbari.org)
   11. (.5 day - I.S.) Remove Microsoft SQL Server on SSDSPub
   12. (.5 day) Remove data directories on SSDPub
   13. (.5 day) Clean everything up and look at making SSDSPub just a Tomcat installation to house web applications
   14. (.5 day) Verify data replication from tornado to ssdspub is turned off (remove data from ssdspub too)
   15. (.5 day) Verify the GetOriginalDataServlet on ssds.shore and ssdpsub simply forward on to the same servlet on new-ssds
   16. (.5 day) Remove the SSDS database from Solstice (backup first)
   17. (.5 day) Remove the SSDS database from Fog (backup first)
   18. (.5 day) Backup and remove all DTS's except on Fog for Solstice-SSDS_Metadata->Fog-SSDS_Metadata
3. Prepare for opening to community (3 days - 3 KG)
   1. (.5 day) Put Copyright in all SSDS source code and zip up and make externally available.
   2. (2 days) Setup Source on public repository
4. Metadata Integrity Checking/Repairing/Enhancing (33 days - 11 KG, 10 MM, 12 RS)
   1. (3 days) Have updateBot check all resources and if they do not have a ResourceType, use the file extension to map it to a MIMEType and create a ResourceType (this will allow queries for specific resources)
   2. (1 day) Have updateBot crawl all resources and update contentLength if not specified.
   3. (10 days) Lookup standard variables and standardUnits from UCAR and have UpdateBot mechanism add them to those without (for NetCDF, try to pull StandardVariable and update SSDS).
   4. (2 days) Implement a mechanism that updates the true end date/time on the DataContainer for data streams to show the latest data for each stream (should also update the number of records).
   5. (5 days) Look into having SSDS create "README" type files in the same location as certain DataContainers.
      1. These could/should be in FGDC format
   6. (12 days) Refactor and reinstate the SQL integrity checks Rich wrote.
5. Enhance Access Interfaces (42 days - 20 KG, 22 MM)
   1. (2 days) Develop web page to allow administrators to configure plot creation
   2. (5 days) Finish implementing all DAOs
      1. Make sure all methods have associated count method
      2. Make sure all methods have boolean option for return full graph
      3. Make sure all methods have capability to specify a sort by field
      4. Verify returned DataContainer collections should be sorted by start date as default
      5. Verify implemented query for DataContainer by DataContainerGroup
   3. (.5 day) Verify PC02 plots are working after M0 turnaround
   4. (.5 day) Add links to CVS XML on device pages
   5. (1 day) Need a mechanism for users to upload and store "Resources" in SSDS (some secure/backed up network location) and link them to other resource aware objects.
   6. (.5 day) In Explorer, truncate long deployment names
   7. (5 days) Implement more queries in Explorer
      1. Desire in long term for common 'status reporting' system for instruments (something like an ops portal)
      2. Find all post products from deployment
      3. Find all resources of certain types (graphics, log files, calibration files, etc.)
      4. "I want the Air Temperature (StandardVariable.name) from the DataProducer named "M1" from such and such to such and such a time."
      5. Given a deployment, can I have a query mechanism that can automatically walk up the tree to find the parent DataProducer of type Deployment?
      6. Given a deployment, can I walk up the parent tree to look for a parent with a certain role?
      7. "Give me all QC'd data and realtime data up to now". This would need to pull the best QC'd data for some sampled parameter and then integrate it with the realtime, un-QC'd data from streaming data.
   8. (5 days) Migrate HOOVES to new architecture
   9. (22 days) Add HOOVES improvements
      1. Full edit pages for deployment information
      2. Tree structure for dataset variables that are functions of depth
      3. SGT-driven contour plots of ZT data (e.g. ADCP, CTD String)
      4. Faster variable list generation by using DODS rather than netCDF API
      5. More consistent use of resourceType contentType info (MIME types)
      6. Top-level data set display for platform level deployment nodes
      7. Additional queries:
         1. by standard variable name
         2. by lat/lon rubber band box via mini maplet gui interface
      8. Fix Bugs:
         1. Window sizing on startup
         2. thread/hash problem with multiple plots
         3. Numerics not showing for some data sets
6. Develop admin application to edit all metadata objects and their relationships (11 days - 11 KG)
   1. One function should be able to change the start time on a DataProducer and have an option to update all the child deployment (deep update) to that same start time.
   2. Build web pages that allow user to send messages to different topics in the ingest component
   3. Build web page that will allow a user to load data packets into the database from the file storage mechanism. (user selects device and time window).
   4. Replace instrument monitoring to read open deployments from SSDS and have configuration options.
7. Improve Data Ingest Mechanisms (8 days - 8 KG)
   1. (.5 day) Try to change OASIS to make mooring turns less painful (documentation basically)
   2. (5 days) Build non-JMS mechanism for users to send data/metadata to SSDS.
   3. (1 day) Put true exception handling in SQLIngest to throw messages when the DB can't be reached.
   4. (1 day) Change PacketSQLOutput/Input to work with any database (not just MS SQL)
8. Improve Testing (5 days - 3 KG, 2MM)
   1. (.5 day) Verify (unit tests) that the RecordDescription level parse regular expression works
   2. (.5 day) Make sure XML dates that are coming out of XMLBuilder/ObjectBuilder cycle can be parsed by the XMLDateFormat.
   3. (.5 day) Check Object Builder/XML builder to make sure it handles URI/URL/UriString/DODSUrl/X/Y/Zoffsets correctly.
   4. (.5 day) Write valid unit test for Object and XMLBuilders
   5. (.5 day) Make sure object Schema/ObjectBuilder/XMLBuilder only use one DataContainer/DataProducer for output, input, consumer, tags.
   6. (.5 day) Write tests for ResourceBLOB->ObjectBuilder for byte array and verify that it is working correctly.
   7. (2 days) Improve Wrapper tests
9. Documentation (6 days - 3 KG, 3 MM)
   1. (.5 day) Put UML diagram of data model on developer section of web app.
   2. (.5 day) Finish documenting data packet structure on web pages.
   3. (5 days) Document Explorer, Admin app and HOOVES

Tasks that were descoped from 900823 SSDS Hardening Proposal

1. Setup SPLUNK to monitor SSDS system logs (JBoss, Apache, UNIX system)
2. Can we setup SSDS so that if a certain process/data container were deemed bad, that you could prevent any reprocessing of that data using that process from happening?
3. Mike M had new idea about extending our model classes to implement remote lazy loading ... interesting idea and I would like to add that functionality later.
4. Can I embed the business logic documentation as JavaDoc tags in the source code and then export that somewhere? This is specifically related to the EJB services.
5. Add end of line terminator as separator in parsing packet records (not files)
6. Implement fixed format (length) parsing
7. Implement way to handle multi-record data
8. Allow user's to "annotate" DataContainer by storing comments
9. Design architecture for plug-in type functionality for servlet type access (i.e. inline filters for data like calculate salinity on the fly)
   1. Add a standard capability to SSDS to do things like mx+b, quadratics, using multiple variables in calculations (these should be able to be defined in the XML and SSDS do them automatically)
10. Refactor navigation menus and scheme for SSDS site and post processed pages (left menu)
    1. Need to add rest of SSDS web page content to main SSDS pages
11. Web pages to help with automated workflows
12. Create web page to do variable/variable plots (with some query capability)
13. Create page to merge data from different sources (could we use Francisco's help/resources:By Platform,By Instrument,By Variable,By Lat/Lon Box,map with AUV track and moorings/instruments,data from AUV/Moorings/Ships merged by time (plots and raw data)
14. For application that show lots of graphics, implement thumbnails (click to enlarge)
15. SIAMRawDataAccess Page -> The list of instruments and parents doesn't show what is current. Maybe I can put a time/date of last update next to the instrument.
16. Build web form to help developers build URLs that they can use in their code to access data from SSDS
17. Somehow need to tell SSDS user's that they need to specify an acknowledgement of MBARI when users utilize the data from SSDS.
18. Implement mechanism (web page, email notification) that allows valid users to approve or map additional relationship and then notify the user of that change so they can change their source. This should be tied into UpdateBot so that it knows what associations it can make between RecordVariable and StandardVarible, for example.
    1. StandardVariables
    2. StandardUnits
    3. StandardKeywords
    4. StandardDomain
    5. StandardReferenceScale
    6. DeviceType
    7. ResourceType
    8. DataProducerGroup
    9. DataContainerGroup
19. GoogleMaps/GoogleEarth/Worldwind integration
20. Instead of using command line argument for timezone->UTC, look at [http://www.hibernate.org/100.html Hibernate code] for UTC dates.
21. Put statistics on plots (min/max of past 24 hrs, latest read value), use max, mins to drive axis range (no autoranging)
22. Where data are dense enough to make for easy viewing, use scatter instead of line plots, leaving gaps for missing data (if lines must be used, put breaks where data gaps are)
23. Setup DeviceQCPlotCreator to create plots with multiple lines on one chart (and separate axes).
24. Make any direction plot (wind, heading, etc.) plot as points, not lines
25. Put nominal lattitude and longitude in plot titles
26. Have capability to turn on/off autoscale on plots and specify range
27. Create servlet that dynamically generates schema and puts in tokens from class constants and database entries. For example, DeviceTypes, ResourceTypes, StandardXXXXXs
28. Build application to allow users to add QC flags and comments to data packets in SSDS_Data
29. Convert ingest to pull metadata revision number from XML instead of metadata sequence number from packet
30. Add mechanism to notify users of new data arrival (method TBD)
    1. Look into NRSS (RSS for data streams)
31. In PacketOutputManager, implement a cleanup watchdog (You could do this by keeping another hashmap that had the key deviceID_metadataRevision_recordType_platformID and then a value of last time accessed. Have the watchdog loop through those and PacketOutputs that have not been accessed in some interval get closed and removed from the packetOutputs hashmap)
32. Add software to crawl SQL packets, read in corresponding GPS data and attach to packet
    1. (5 days) Remove Deployment info from PUCK XML and move all to new schema and validate (due to the amount of work to do this, it will be done on an as needed basis)
33. Could we move applications on SSDSPub to another machine with Tomcat and CNAME ssdspub to that machine?
34. Follow up on PUCK configuration tool (ACE)
35. Build services to read data from DataContainers that are files through the query interface (not just from packets). (This is really valuable, but not REALLY needed)
36. Look into implementing paging in services (Hibernate supports this).
37. Load historical OASIS data into SSDS (data and metadata). This is important but too big for this, separate project
    1. Figure out how we will integrate data files offloaded from instruments when they are recovered. (generalize to any data file and this might be a location where the file can be place with appropriate metadata and it will be loaded into SSDS. SSDS can even do some thinking like with NetCDF files). IMPORTANT EVEN THOUGH DESCOPED