IN13B-1209
Exploring the Ocean Remotely and in Real-Time With HiSeasNet
HiSeasNet is a satellite communications network providing continuous Internet connectivity for oceanographic research ships and platforms throughout the Pacific, Atlantic, and Indian oceans. With the addition of new vessels and satellites in the network, HiSeasNet now uses five satellite beams to extend campus networks out to 13 ships at sea as well as a seismic observatory on South Georgia Island in the Southern Ocean. This year (2007) HiSeasNet has also supported ship operations in the Indian Ocean using a ground station in Germany. Employing a variety of networking technologies, HiSeasNet has allowed scientists to conduct videoconference outreach programs, maintain limited long-term real-time data collection, connect campus phones to shipboard phones through the ship's PBX using Voice over IP (VoIP) protocols, and direct multi-ship research from shore and other ships. Standard Internet protocols available include FTP, https, and ssh. With still more bandwidth available for data, this system can provide new opportunities for enhancing ship-based ocean exploration as well as supporting the long-term data collection projects such as those proposed in the NSF's Ocean Observatories Initiative program. The highest data rate achieved to date was 19 Mbps from the R/V Thomas Thompson off Vancouver. This involved a temporary conversion of the 2.4m shipboard antenna to Ku-Band for relaying HD video from the seafloor to the Internet ashore for real-time imagery of vent activity on the seafloor. The R/V Roger Revelle and R/V Melville include NSF Real-time Observatory and Data management Network (ROADNet) Points- of-Presence aboard ship for buffering and delivering shipboard data to shore continuously and in near-real-time. ROADNet is providing the baseline technology for the OOI Cyberinfrastructure Implementing Organization – the extension of this capability to more of the UNOLS fleet can provide near-real-time data for the OOI for non- proprietary data (e.g. meteorology and acoustic Doppler current profiler) to shore for general access and use. http://www.hiseasnet.net
IN13B-1210
Applications of Server Clustering Technology in Sensor Networks
The Array Network Facility is charged with the acquisition and processing of seismic data from the Earthscope USArray experiment. High resolution data from 400 seismic sensors is streamed in near real-time to the ANF at UCSD in La Jolla, CA where it is automatically processed by machine and reviewed by analysts before being externally distributed to other data centers, including the IRIS Data Management Center. Data streams include six channels of 24- bit seismic data at 40 samples per second and over twenty channels of state-of-heath data at 1 sample per second per station. The sheer volume of data acquired and processed overwhelms the capabilities of any one affordable server system. Due to the relatively small buffers on-site (typically four hours) at the seismic stations, it is vital that the real-time systems remain online and acquiring data around the clock in order to meet data distribution requirements in a timely manner. Although the ANF does not have a 24x7x365 operations staff, the logistical difficulty in retrieving data from often remote locations after it expires from the on-site buffers requires the real- time systems to automatically recover from server failures without immediate operator intervention. To accomplish these goals, the ANF has implemented a five node Sun Solaris Cluster with acquisition and processing tasks shared by a mixture of integer and floating point processing units (Sun T2000 and V240/V245 systems). This configuration is an improvement over the typical regional network data center for a number of reasons: - By implementing a shared storage architecture, acquisition, processing, and distribution can be split between multiple systems working on the same data set, thus limiting the impact of a particularly resource-intensive task on the acquisition system. - The Solaris Cluster software monitors the health of the cluster nodes and provides the ability automatically fail over processes from a failed node to a healthy node. - Redundant power and networking connections reduce the chances of a single hardware failure taking down an entire node. - By allowing processing systems to be transferred between cluster nodes, Sun Cluster provides the ability to take servers down for both planned and unplanned maintenance. - This is not a "grid system", there is no single controlling node to fail and take down the entire cluster. http://anf.ucsd.edu
IN13B-1211
Lessons Learned From 104 Years of Mobile Observatories
As the oceanographic community ventures into a new era of integrated observatories, it may be helpful to look back on the era of "mobile observatories" to see what Cyberinfrastructure lessons might be learned. For example, SIO has been operating research vessels for 104 years, supporting a wide range of disciplines: marine geology and geophysics, physical oceanography, geochemistry, biology, seismology, ecology, fisheries, and acoustics. In the last 6 years progress has been made with diverse data types, formats and media, resulting in a fully-searchable online SIOExplorer Digital Library of more than 800 cruises (http://SIOExplorer.ucsd.edu). Public access to SIOExplorer is considerable, with 795,351 files (206 GB) downloaded last year. During the last 3 years the efforts have been extended to WHOI, with a "Multi-Institution Testbed for Scalable Digital Archiving" funded by the Library of Congress and NSF (IIS 0455998). The project has created a prototype digital library of data from both institutions, including cruises, Alvin submersible dives, and ROVs. In the process, the team encountered technical and cultural issues that will be facing the observatory community in the near future. Technological Lessons Learned: Shipboard data from multiple institutions are extraordinarily diverse, and provide a good training ground for observatories. Data are gathered from a wide range of authorities, laboratories, servers and media, with little documentation. Conflicting versions exist, generated by alternative processes. Domain- and institution-specific issues were addressed during initial staging. Data files were categorized and metadata harvested with automated procedures. With our second-generation approach to staging, we achieve higher levels of automation with greater use of controlled vocabularies. Database and XML- based procedures deal with the diversity of raw metadata values and map them to agreed-upon standard values, in collaboration with the Marine Metadata Interoperability (MMI) community. All objects are tagged with an expert level, thus serving an educational audience, as well as research users. After staging, publication into the digital library is completely automated. The technical challenges have been largely overcome, thanks to a scalable, federated digital library architecture from the San Diego Supercomputer Center, implemented at SIO, WHOI and other sites. The metadata design is flexible, supporting modular blocks of metadata tailored to the needs of instruments, samples, documents, derived products, cruises or dives, as appropriate. Controlled metadata vocabularies, with content and definitions negotiated by all parties, are critical. Metadata may be mapped to required external standards and formats, as needed. Cultural Lessons Learned: The cultural challenges have been more formidable than expected. They became most apparent during attempts to categorize and stage digital data objects across two institutions, each with their own naming conventions and practices, generally undocumented, and evolving across decades. Whether the questions concerned data ownership, collection techniques, data diversity or institutional practices, the solution involved a joint discussion with scientists, data managers, technicians and archivists, working together. Because metadata discussions go on endlessly, significant benefit comes from dictionaries with definitions of all community-authorized metadata values. http://gdccoll.ucsd.edu:8080/digarch
IN13B-1212
A web-based modular framework for real-time monitoring of large scale sensor networks
The Antelope Real Time System (ARTS) is an integrated combination of protocols, acquisition systems and applications designed for real-time data collection and analysis from an array of deployed field sensors. Historically these were seismic sensors, however the open architecture of the ARTS facilitated development of acquisition protocols for a diverse group of sensors, including data streams from hf radar, meteorological instrumentation and cameras. In parallel with the expansion of data-type ingestion, a web-based interface to the ARTS was developed in PHP, a popular HTML embedded scripting language. The application-driven development of web-based software to Antelope-stored data has risen exponentially over the last four years, from simple database interactions to web-based AJAX applications similar in look and feel to desktop software. As the web-based applications have grown in complexity, the architecture around their development has matured into an extensible framework with "plug'n'play" capabilities. Their modular design has allowed multiple institutions to deploy the same web-based applications, tailored for their specific requirements. Examples include the NSF Earthscope USArray Transportable Array, ROADNet's Realtime Imagebank, the broadband seismic network monitoring of the University of Nevada Reno and University of California San Diego, and monitoring of the downhole arrays maintained by the University of California Santa Barbara. The success of these deployments suggest that such a framework could be applicable to other large scale sensor networks, including the developing Ocean Observatories project. http://anf.ucsd.edu
IN13B-1213 INVITED
Cyberinfrastructure Lessons from Antelope-based Installations
This work reviews over a decade of experience with numerous seismological and multidisciplinary operations based on the Antelope Environmental Monitoring System from Boulder Real-Time Technologies, Inc., deriving key considerations that inform the development of cyberinfrastructure frameworks for small, medium, and large-scale near-real-time operations. Among these: that core construction quality is more important than any individual ideas implemented, though those when subordinate nevertheless enable the implementation of sophisticated features; likewise core construction quality is more important than any particular user-visible graphics, web pages or GUIs, though the latter are nevertheless important in order to deliver results, without which the core systems serve little purpose; that separation of content and presentation is a key aspect for maintainability and growth; that requirements engineering is critical for successful tool design; that it is critical to have a central development framework that provides organization and tool sets; that one must have feedback mechanisms to catch real-time system errors; that it is critical to take an aggressive approach to addressing those errors at their source after a committed hunt to understand them at their deepest layer; that long-term growth and open architecture is furthered by an ability to integrate foreign technology frameworks that offer significant advances and features; that the platform must have the ability to support integration with legacy systems; and that the system must have the ability to support wide datalogger variety, with the understanding that most umbrella formalisms built into acquisition frameworks prove breakable by manufacturers as the support mission expands. Selected case studies will be discussed. At best as the software architecture evolves, it becomes a maintainable, scalable, and runnable platform that explains, quite literally codifies, with increasing levels of sophistication the community's evolving understanding of the monitoring challenges being addressed. The observations presented are expected to be pertinent to the upcoming Ocean Observatories Initiative.
IN13B-1214
iRODS: A Distributed Data Management Cyberinfrastructure for Observatories
Large-scale and long-term preservation of both observational and synthesized data requires a system that virtualizes data management concepts. A methodology is needed that can work across long distances in space (distribution) and long-periods in time (preservation). The system needs to manage data stored on multiple types of storage systems including new systems that become available in the future. This concept is called infrastructure independence, and is typically implemented through virtualization mechanisms. Data grids are built upon concepts of data and trust virtualization. These concepts enable the management of collections of data that are distributed across multiple institutions, stored on multiple types of storage systems, and accessed by multiple types of clients. Data virtualization ensures that the name spaces used to identify files, users, and storage systems are persistent, even when files are migrated onto future technology. This is required to preserve authenticity, the link between the record and descriptive and provenance metadata. Trust virtualization ensures that access controls remain invariant as files are moved within the data grid. This is required to track the chain of custody of records over time. The Storage Resource Broker (http://www.sdsc.edu/srb) is one such data grid used in a wide variety of applications in earth and space sciences such as ROADNet (roadnet.ucsd.edu), SEEK (seek.ecoinformatics.org), GEON (www.geongrid.org) and NOAO (www.noao.edu). Recent extensions to data grids provide one more level of virtualization - policy or management virtualization. Management virtualization ensures that execution of management policies can be automated, and that rules can be created that verify assertions about the shared collections of data. When dealing with distributed large-scale data over long periods of time, the policies used to manage the data and provide assurances about the authenticity of the data become paramount. The integrated Rule-Oriented Data System (iRODS) (http://irods.sdsc.edu) provides the mechanisms needed to describe not only management policies, but also to track how the policies are applied and their execution results. The iRODS data grid maps management policies to rules that control the execution of the remote micro-services. As an example, a rule can be created that automatically creates a replica whenever a file is added to a specific collection, or extracts its metadata automatically and registers it in a searchable catalog. For the replication operation, the persistent state information consists of the replica location, the creation date, the owner, the replica size, etc. The mechanism used by iRODS for providing policy virtualization is based on well-defined functions, called micro-services, which are chained into alternative workflows using rules. A rule engine, based on the event-condition-action paradigm executes the rule-based workflows after an event. Rules can be deferred to a pre-determined time or executed on a periodic basis. As the data management policies evolve, the iRODS system can implement new rules, new micro-services, and new state information (metadata content) needed to manage the new policies. Each sub- collection can be managed using a different set of policies. The discussion of the concepts in rule-based policy virtualization and its application to long-term and large-scale data management for observatories such as ORION and NEON will be the basis of the paper.
IN13B-1215
A Prototype for a Distributed and Automatic Visualization Pipeline for Oceanographic Datasets.
The Laboratory for the Ocean Observatory Knowledge INtegration Grid (LOOKING) is a NSF research project focused on the identification, synthesis and assembly of existing and emerging concepts and technologies into a coherent viable cyberinfrastructure design for ocean observatories. One of the goals of the project is to prototype an automated pipeline for continuously generating visualization products (time variant geometric representations and rendered image sequences) from streaming and regenerating data sources. Current work involves remote visualization of NASA JPL's Our Ocean Data Assimilation of the Central California region on a continuous basis. The prototype uses OPeNDAP as the data retrieval mechanism to fetch netcdf formatted data for specific variables or time steps. A geometry conversion engine transforms this data into 3D geometric models (e.g. isosurfaces for scalar data like temperature and salinity or streamlines for ocean currents) using the Visualization Toolkit (VTK) and delivers a 3D scene graph that can be imported into the end user's choice of visualization software for viewing the scene. Our current preferred 3D viewer is ossimPlanet (a 3D Geospatial viewer built using OpenSceneGraph, libwms and OSSIM) embedded inside the COVISE framework for interactive exploration in a georeferenced framework. http://www.lookingtosea.ucsd.edu
IN13B-1216
An Automated Visual Event Detection System for Cabled Observatory Video
The permanent presence of underwater cameras on oceanic cabled observatories, such as the Victoria Experimental Network Under the Sea (VENUS) and Eye-In-The-Sea (EITS) on Monterey Accelerated Research System (MARS), will generate valuable data that can move forward the boundaries of understanding the underwater world. However, sightings of underwater animal activities are rare, resulting in the recording of many hours of video with relatively few events of interest. The burden of video management and analysis often requires reducing the amount of video recorded and later analyzed. Sometimes enough human resources do not exist to analyze the video; the strains on human attention needed to analyze video demand an automated way to assist in video analysis. Towards this end, an Automated Visual Event Detection System (AVED) is in development at the Monterey Bay Aquarium Research Institute (MBARI) to address the problem of analyzing cabled observatory video. Here we describe the overall design of the system to process video data and enable science users to analyze the results. We present our results analyzing video from the VENUS observatory and test data from EITS deployments. This automated system for detecting visual events includes a collection of custom and open source software that can be run three ways: through a Web Service, through a Condor managed pool of AVED enabled compute servers, or locally on a single computer. The collection of software also includes a graphical user interface to preview or edit detected results and to setup processing options. To optimize the compute-intensive AVED algorithms, a parallel program has been implemented for high-data rate applications like the EITS instrument on MARS. http://www.mbari.org/AVED/
IN13B-1217
Interacting With A Near Real-Time Urban Digital Watershed Using Emerging Geospatial Web Technologies
The value of real-time hydrologic data dissemination including river stage, streamflow, and precipitation for operational stormwater management efforts is particularly high for communities where flash flooding is common and costly. Ideally, such data would be presented within a watershed-scale geospatial context to portray a holistic view of the watershed. Local hydrologic sensor networks usually lack comprehensive integration with sensor networks managed by other agencies sharing the same watershed due to administrative, political, but mostly technical barriers. Recent efforts on providing unified access to hydrological data have concentrated on creating new SOAP-based web services and common data format (e.g. WaterML and Observation Data Model) for users to access the data (e.g. HIS and HydroSeek). Geospatial Web technology including OGC sensor web enablement (SWE), GeoRSS, Geo tags, Geospatial browsers such as Google Earth and Microsoft Virtual Earth and other location-based service tools provides possibilities for us to interact with a digital watershed in near-real-time. OGC SWE proposes a revolutionary concept towards a web-connected/controllable sensor networks. However, these efforts have not provided the capability to allow dynamic data integration/fusion among heterogeneous sources, data filtering and support for workflows or domain specific applications where both push and pull mode of retrieving data may be needed. We propose a light weight integration framework by extending SWE with open source Enterprise Service Bus (e.g., mule) as a backbone component to dynamically transform, transport, and integrate both heterogeneous sensor data sources and simulation model outputs. We will report our progress on building such framework where multi-agencies" sensor data and hydro-model outputs (with map layers) will be integrated and disseminated in a geospatial browser (e.g. Microsoft Virtual Earth). This is a collaborative project among NCSA, USGS Illinois Water Science Center, Computer Science Department at UIUC funded by the Adaptive Environmental Infrastructure Sensing and Information Systems initiative at UIUC.
IN13B-1218
Cyberinfrastructure for the NSF Ocean Observatories Initiative
The Internet today is vastly different than the Internet that we knew even five years ago and the changes that will be evident five years from now, when the NSF Ocean Observatories Initiative (OOI) prototype has been installed, are nearly unpredictable. Much of this progress is based on the exponential growth in capabilities of consumer electronics and information technology; the reality of this exponential behavior is rarely appreciated. For example, the number of transistors on a square cm of silicon will continue to double every 18 months, the density of disk storage will double every year, and network bandwidth will double every eight months. Today's desktop 2TB RAID will be 64TB and the 10Gbps Regional Scale Network fiber optical connection will be running at 1.8Tbps. The same exponential behavior characterizes the future of genome sequencing. The first two sequences of composites of individuals' genes cost tens of millions of dollars in 2001. Dr. Craig Venter just published a more accurate complete human genome (his own) at a cost on the order of $100,000. The J. Craig Venter Institute has provided support for the X Prize for Genomics offering $10M to the first successful sequencing of a human genome for $1,000. It's anticipated that the prize will be won within five years. Major advances in technology that are broadly viewed as disruptive or revolutionary rather than evolutionary will often depend upon the exploitation of exponential expansions in capability. Applications of these ideas to the OOI will be discussed. Specifically, the agile ability to scale cyberinfrastructure commensurate with the exponential growth of sensors, networks and computational capability and demand will be described. http://lookingtosea.ucsd.edu/
IN13B-1219
Discovery and Access to ARM Data from Statistical Views of Measurement Results: A new View Into a Long-term Data Collection
Atmospheric Radiation Measurement (ARM) Program has been establishing a climatic collection of observations about radiative flux transfer and cloud formation over the past 10-14 years. This DOE sponsored program provides climate modelers information to improve the parameterization of the models. The ARM Archive has been providing data access and discovery through various user interfaces based on inventories of data availability (from 3.3 million data files and 100's of measurement types). A new interface is being developed so that users can discover and access data based on the observed results by viewing hierarchical statistical summaries. The display of statistical views allows the users to include an overview of the measured results in their decisions for the selection of long-term collection of ARM data. This approach provides the various modeling communities with products that are easily accessible and display insights needed for research on model parameterization and validation. The prototype of statistical views currently consists of pre-computed products for nested time ranges (whole period of record; annual; seasonal; and monthly - as appropriate). For each time range and measurement, a variety of simple statistics are computed. Graphs of the statistical distribution of measurement (e.g., histograms) are also linked to the statistical results. The graphs are available through a web based interface. Users select a location and measurement and then drill down through times scales ranging from the full period of record to individual months. While viewing the graphs displayed by the user interface, users will be able to extract the data behind the statistical graphs, or the measurement data used for the statistics, or order the ARM data files.
IN13B-1220
Using RBNB Data Turbine for Observational Cyberinfrastructure
The vision of large-scale sensor-based Observing Systems to address the National Research Council's "Grand Challenges" for environmental science relies on robust cyberinfrastructure. Environmental science and engineering communities are now actively engaged in the early planning and development phases of the next generation of large-scale sensor-based observing systems. Streaming data middleware is a layer above the network that provides several key functions: robustness, device abstraction, scalability, buffering and routing. All of these assist in the design, deployment and management of large sensor networks. RBNB (Ring Buffer Network Bus) DataTurbine is a newly open-sourced system for streaming data middleware that is being used in a variety of projects. Formerly a successful commercial streaming data product with a track record of performance in NSF and NASA projects, RBNB DataTurbine provides an excellent basis for developing robust streaming data middleware. We are working on several efforts to deploy, improve and adapt DataTurbine for observational cyberinfrastructure. Performance metrics, documentation, instrument interfaces, web access to live data, system health monitoring, database backends as well as scalability experiments are all underway. Additionally, we are attempting to build a community of developers and users around the project to provide for the kind of longevity required in long-term projects. In this paper we discuss our results to date with RBNB DataTurbine on a variety of projects from ecological to earthquake engineering.
IN13B-1221
Cyberinfrastructure and Software Middleware Efforts Applied to Ocean Observing Systems
Engineers and scientists at the Monterey Bay Aquarium Research Institute (MBARI), Scripps Institute of Oceanography (SIO), National Center for Supercomputing Applications (NCSA), and the University of Washington (UW) have spent time building technologies to enable science on ocean observatories. From our experience building observing technologies we have learned many valuable lessons about supporting long-term observations consisting of large numbers of heterogeneous instruments. This collaboration is now applying those technologies and lessons learned to the next generation of cyberinfrastructure. We will discuss the various cyberinfrastructure technologies, how we applied them to deployed instrumentation and how we are using that existing infrastructure to develop and test other technologies to meet the scalability issues associated with large scale ocean observatories.