IN43C-01 INVITED
NOAA's Global Earth Observation - Integrated Data Environment (GEO-IDE)
The international Group on Earth Observation (GEO) and the U.S. coordination group, USGEO, have identified nine societal benefit areas that require environmental data of a wide range of types and from many diverse sources. GEO has called on the nations of the world to ensure that the relevant data that they hold is made accessible and useful to these applications. In response, nations and their environmental agencies are addressing the challenges associated with data integration of these distributed and diverse data types. The National Oceanic and Atmospheric Administration (NOAA) holds extremely large collections of data describing the physical and biological properties of the Earth's environment. To date, the data collections and the systems that support them have been acquired by individually funded and managed programs with differing requirements, standards, interfaces and conventions, mirroring the data integration issues faced at the national and international level. The Global Earth Observation – Integrated Data Environment (GEO-IDE) has been initiated by NOAA to address these issues for its own interdisciplinary applications as well as those of the the broader national and international iniatives. The concept and initial plans for GEO-IDE have been developed by the Data Management Integration Team (DMIT), a group of data management professionals representing all NOAA's Line Offices, Goal Teams and the office of the CIO. The goal of GEO-IDE is to define an architecture and the associated processes necessary to establish the required standards and guidelines that allow NOAA's data providers to make their products available as a set of interoperable services. GEO-IDE is addressing the integration of existing data services while at the same time providing guidance to future data system development activities. It is intended to meet an important NOAA need while also supporting NOAA's contribution to USGEO and GEO.
IN43C-02
NASA's Earth Science Gateway within the GEOSS Architecture Framework and in Support of Distributed Global Systems
Progress continues within the arena of interoperability towards greater discovery, access, and use of scientific data regarding improved societal benefit and decision solutions. The Group on Earth Observation System of Systems has developed multiple pilot projects in which many of these maturing and emerging technologies are being interconnected, tested, and implemented as operational systems. Within this network of components are data stores, registries, catalogs, portals, models, work-process flows, satellites, UAVs, and many more components. The pilots tackle the intricate process of ensuring these components properly work together within the framework of open-standards based interoperability, and enabling the vision of a distributed, comprehensible, global system of scientific tools and data for the lay-person as well as the researcher. This paper will concentrate on the NASA Earth Science Gateway (http://esg.gsfc.nasa.gov) view of interconnections to the registries, catalogs, models, and so on, that support this System of Systems. This paper covers what standards were used, how components can be connected and tested, where difficulties emerged, where we have seen return on investment, and how this level of interoperability is progressing.
IN43C-03
Web Services at the National Oceanic and Atmospheric Administration (NOAA) National Climatic Data Center (NCDC)
NOAA's National Climatic Data Center (NCDC) currently archives over 1.5 petabytes of climatological data from various networks and sources including in-situ, numerical models, radar and satellite. Access to these datasets is evolving from interactive web interfaces utilizing database technology to standardized web services in a Service Oriented Architecture (SOA). NCDC is currently offering several web services using Simple Object Access Protocol (SOAP), XML over Representational State Transfer (REST/XML), Open Geospatial Consortium (OGC) Web Map Service (WMS) / Web Feature Service (WFS) / Web Coverage Service (WCS) and OPeNDAP web service protocols. These services offer users a direct connection between their client applications and NCDC data servers. In addition, users may embed access to the services in custom applications to efficiently navigate and subset data in an automated fashion. NCDC currently provides gridded numerical model data through a THREDDS Data Server and GrADS Data Server which offers OPeNDAP and WCS access. In-situ network metadata are available through WMS and WFS while the corresponding time-series data are accessible through SOAP and REST web services. These in-situ services are a part of the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) WaterOneFlow services, a consolidated access system for hydrologic data, and comply with the WaterOneFlow specifications. NCDC's Severe Weather Data Inventory (SWDI), which provides user access to archives of several datasets critical to the detection and evaluation of severe weather, is also accessible through REST/XML services. Providing cataloging, access and search capabilities for many of NCDC's datasets using community driven standards is a top priority for the ever increasing data volumes being archived at NCDC. Providing interoperable access is critical to supporting data stewardship across multiple scientific disciplines and user types. This demonstration will showcase NCDC's latest work towards standardized web services with both server and client examples. http://www.ncdc.noaa.gov/ws
IN43C-04
Encoding Observational Point Data in WCS: Measuring Performance of XML versus binary NetCDF.
We examine the possibility of using the Web Coverage Service (WCS) protocol to deliver observational point data. An important implementation question is whether to encode the data in XML or binary. We implement a prototype server and measure the performance of returning the data in XML using a profile of the Geography Markup Language (GML) or returning the data in a netCDF binary file. We may also discuss related encoding issues, including XML variants and the use of the netCDF CF Conventions. http://www.unidata.ucar.edu/staff/caron/presentations/wcsobs/
IN43C-05 INVITED
SeaDataNet Pan-European infrastructure for Ocean & Marine Data Management
SEADATANET is developing a Pan-European data management infrastructure to insure access to a large number of marine environmental data (i.e. temperature, salinity current, sea level, chemical, physical and biological properties), safeguard and long term archiving. Data are derived from many different sensors installed on board of research vessels, satellite and the various platforms of the marine observing system. SeaDataNet allows to have information on real time and archived marine environmental data collected at a pan-european level, through directories on marine environmental data and projects. SeaDataNet allows the access to the most comprehensive multidisciplinary sets of marine in-situ and remote sensing data, from about 40 laboratories, through user friendly tools. The data selection and access is operated through the Common Data Index (CDI), XML files compliant with ISO standards and unified dictionaries. Technical Developments carried out by SeaDataNet includes: A library of Standards - Meta-data standards, compliant with ISO 19115, for communication and interoperability between the data platforms. Software of interoperable on line system - Interconnection of distributed data centres by interfacing adapted communication technology tools. Off-Line Data Management software - software representing the minimum equipment of all the data centres is developed by AWI "Ocean Data View (ODV)". Training, Education and Capacity Building - Training ‘on the job' is carried out by IOC-Unesco in Ostende. SeaDataNet Virtual Educational Centre internet portal provides basic tools for informal education http://www.seadatanet.org
IN43C-06
Data and Semantic Interoperability for the Oceans Sensor Web
Ocean observing systems incorporate a spectrum of sensors and data. Making the data available to any interested scientist is important: data sharing and experimental reproducibility are hallmarks of the scientific process. However, different groups may represent, transport, store and distribute their data in different ways, leading to difficulties in sharing these data. OOSTethys, an open-source community effort with involvement from six regional associations and two major research institutes, is exploring the best mechanisms to make ocean data and metadata interoperable by advancing and influencing standards from the Open Geospatial Consortium (OGC), World Wide Web Consortium (W3C) and OASIS. Our strategy to address these challenges has been to envision a service oriented architecture (SOA) which comprises data providers, registries, semantic mediators, aggregators and visualizers. For each component of the system we select the most appropriate standard(s) and create cookbooks and tools to support its implementation. This improves accessibility for data providers with limited time and limited budgets for information technology projects. For example, we have created cookbooks and toolkits in Perl, Java and Python to facilitate implementation of OGC Sensor Observation Services (SOS). The implementation includes publishing metadata in SensorML, and making data available via Geographic Markup Language (GML) records conforming to the Observation and Measurements specifications. A semantic mediator implemented as a web service uses Semantic Web technologies to solve semantic incompatibilities, and enables proper categorization of the different services. Our initial results are positive: in addition to several national demonstrations of data interoperability, the cookbooks have been used to bring more than 60 oceanographic platforms online, and we have at least 7 data consumers relying on web services for their own oceanographic applications. OOSTethys work is an essential part of the OGC's Ocean Science Interoperability Experiment, which seeks more participation from the Geospatial community to advance data interoperability in Ocean Observing Systems. http://www.oostethys.org/
IN43C-07
Information Viewpoints and Geoscience Service Architectures
When dealing with earth science data, different use-cases may require different views of the underlying information. At a basic level, data generation and initial assimilation generally involves dealing with event-based types and different granularity than data organized for processing, often on a grid, while further downstream the desired result of most scientific exercises is an interpretation view characterized by high semantic content and small size. The stages often map reasonably well onto the basic meta-models of Observation, Coverage and Feature provided by the OGC/ISO 19100 framework, and the matching service interfaces (SOS, WCS, WFS). However, on closer inspection of common use-cases, the vision of the Observation viewpoint as most primitive and the Feature viewpoint as most evolved does not consistently stand up. Furthermore, common discovery and access routes may involve traversing associations between instances using different viewpoints. These considerations lead to information (and thus service) composition arrangements with a variety of data flows. For example, an observation service may obtain its result data from a coverage service, while another coverage may be composed from multiple atomic observations; observations are often discovered through their association with a sampling-feature such as a cruise or borehole, or with a sensor platform such as a specific satellite whose description is available from a strongly governed register. The relationship of service instances to data stores (or other sources) is also not one-to-one, as multiple views of the same data are frequently involved. Useful service profiles may thus imply specific service architectures, and requirement to transform between viewpoints becomes almost ubiquitous. Adherence to a sound underlying meta-model for both data and services is a key enabler.
IN43C-08
Processes for Achieving Interoperability in GEOSS
GEOSS, the Global Earth Observing System of Systems, is being built from existing systems and initiatives, with an emphasis on the creation of synergies among GEOSS component that provide increased benefits to society. The goal is to leverage existing programs and established standards wherever possible, and to broaden convergence of systems based on agreed interoperability arrangements. This talk will describe the specific approaches that GEOSS has proposed for achieving interoperability among its component systems and will give an overview of the GEOSS Interoperability Process Pilot Project (IP3). The IP3 was conceived as a way to exercise the process that has been defined for reaching interoperability arrangements. We describe the phases and status of the IP3, which begins with identification of the system components, and the standards, interface protocols and interoperability agreements currently in use by these systems. This information is captured in web-accessible catalogs and registries that are part of the core GEOSS architecture. Four systems/disciplines were initially identified as sources for the pilot project, covering weather and climate, seismology, biodiversity, and water cycle. This selection was based on the desire to have participation from diverse disciplines and the commitments of representatives from the disciplines to actively support the process. Systems contributed to GEOSS are built to serve particular needs, but those systems should also be designed or adapted so their inputs and outputs support interoperability with other systems. Consequently, we focus on interoperability situations that are surfaced by actual requirements to interface with other GEOSS- affiliated systems through what are termed GEOSS interoperability arrangements. Use case scenarios were developed that required the exchange of data and information between the identified systems. In designing interfaces to support interoperability among two or more component systems of GEOSS, the objective is to deploy a solution that not only solves the interoperability requirement, but does so by taking full advantage of existing international standards. The IP3 has provided valuable practical experience which is helping to refine the process for reaching interoperability arrangements for GEOSS. This includes working with the nascent Standards and Interoperability Forum, the GEOSS mechanism for facilitating interoperability. This project is also contributing to the design of the GEOSS Standards and Interoperability Registry, which is the repository of information on the interoperability arrangements employed among GEOSS components, and has provided initial content for the GEOSS Components and Services Registers. http://www.ieee-earth.org/
IN43C-09
GENESIS SciFlo: Choreographing Interoperable Web Services on the Grid using a Semantically-Enabled Dataflow Execution Environment
The General Earth Science Investigation Suite (GENESIS) project is a NASA-sponsored partnership between the Jet Propulsion Laboratory, academia, and NASA data centers to develop a new suite of Web Services tools to facilitate multi-sensor investigations in Earth System Science. The goal of GENESIS is to enable large-scale, multi-instrument atmospheric science using combined datasets from the AIRS, MODIS, MISR, and GPS sensors. Investigations include cross-comparison of spaceborne climate sensors, cloud spectral analysis, study of upper troposphere-stratosphere water transport, study of the aerosol indirect cloud effect, and global climate model validation. The challenges are to bring together very large datasets, reformat and understand the individual instrument retrievals, co-register or re-grid the retrieved physical parameters, perform computationally-intensive data fusion and data mining operations, and accumulate complex statistics over months to years of data. To meet these challenges, we have developed a Grid computing and dataflow framework, named SciFlo, in which we are deploying a set of versatile and reusable operators for data access, subsetting, registration, mining, fusion, compression, and advanced statistical analysis. SciFlo leverages remote Web Services, called via Simple Object Access Protocol (SOAP) or REST (one-line) URLs, and the Grid Computing standards (WS-* & Globus Alliance toolkits), and enables scientists to do multi- instrument Earth Science by assembling reusable Web Services and native executables into a distributed computing flow (tree of operators). The SciFlo client & server engines optimize the execution of such distributed data flows and allow the user to transparently find and use datasets and operators without worrying about the actual location of the Grid resources. In particular, SciFlo exploits the wealth of datasets accessible by OpenGIS Consortium (OGC) Web Mapping Servers & Web Coverage Servers (WMS/WCS), and by Open Data Access Protocol (OpenDAP) servers. SciFlo also publishes its own SOAP services for space/time query and subsetting of Earth Science datasets, and automated access to large datasets via lists of (FTP, HTTP, or DAP) URLs which point to on-line HDF or netCDF files. Typical distributed workflows obtain datasets by calling standard WMS/WCS servers or discovering and fetching data granules from ftp sites; invoke remote analysis operators available as SOAP services (interface described by a WSDL document); and merge results into binary containers (netCDF or HDF files) for further analysis using local executable operators. Naming conventions (HDFEOS and CF-1.0 for netCDF) are exploited to automatically understand and read on-line datasets. More interoperable conventions, and broader adoption of existing converntions, are vital if we are to "scale up" automated choreography of Web Services beyond toy applications. Recently, the ESIP Federation sponsored a collaborative activity in which several ESIP members developed some collaborative science scenarios for atmospheric and aerosol science, and then choreographed services from multiple groups into demonstration workflows using the SciFlo engine and a Business Process Execution Language (BPEL) workflow engine. We will discuss the lessons learned from this activity, the need for standardized interfaces (like WMS/WCS), the difficulty in agreeing on even simple XML formats and interfaces, the benefits of doing collaborative science analysis at the "touch of a button" once services are connected, and further collaborations that are being pursued.