HR: 1340h
AN: IN13B-1214    [Abstracts]
TI: iRODS: A Distributed Data Management Cyberinfrastructure for Observatories
AU: * Rajasekar, A
EM: sekar@sdsc.edu
AF: San Diego Supercomputer Center, University of Clifornia, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, United States
AU: Moore, R
EM: moore@sdsc.edu
AF: San Diego Supercomputer Center, University of Clifornia, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, United States
AU: Vernon, F
EM: flvernon@ucsd.edu
AF: Scripps Institution of Oceanography, University of Clifornia, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, United States
AB: Large-scale and long-term preservation of both observational and synthesized data requires a system that virtualizes data management concepts. A methodology is needed that can work across long distances in space (distribution) and long-periods in time (preservation). The system needs to manage data stored on multiple types of storage systems including new systems that become available in the future. This concept is called infrastructure independence, and is typically implemented through virtualization mechanisms. Data grids are built upon concepts of data and trust virtualization. These concepts enable the management of collections of data that are distributed across multiple institutions, stored on multiple types of storage systems, and accessed by multiple types of clients. Data virtualization ensures that the name spaces used to identify files, users, and storage systems are persistent, even when files are migrated onto future technology. This is required to preserve authenticity, the link between the record and descriptive and provenance metadata. Trust virtualization ensures that access controls remain invariant as files are moved within the data grid. This is required to track the chain of custody of records over time. The Storage Resource Broker (http://www.sdsc.edu/srb) is one such data grid used in a wide variety of applications in earth and space sciences such as ROADNet (roadnet.ucsd.edu), SEEK (seek.ecoinformatics.org), GEON (www.geongrid.org) and NOAO (www.noao.edu). Recent extensions to data grids provide one more level of virtualization - policy or management virtualization. Management virtualization ensures that execution of management policies can be automated, and that rules can be created that verify assertions about the shared collections of data. When dealing with distributed large-scale data over long periods of time, the policies used to manage the data and provide assurances about the authenticity of the data become paramount. The integrated Rule-Oriented Data System (iRODS) (http://irods.sdsc.edu) provides the mechanisms needed to describe not only management policies, but also to track how the policies are applied and their execution results. The iRODS data grid maps management policies to rules that control the execution of the remote micro-services. As an example, a rule can be created that automatically creates a replica whenever a file is added to a specific collection, or extracts its metadata automatically and registers it in a searchable catalog. For the replication operation, the persistent state information consists of the replica location, the creation date, the owner, the replica size, etc. The mechanism used by iRODS for providing policy virtualization is based on well-defined functions, called micro-services, which are chained into alternative workflows using rules. A rule engine, based on the event-condition-action paradigm executes the rule-based workflows after an event. Rules can be deferred to a pre-determined time or executed on a periodic basis. As the data management policies evolve, the iRODS system can implement new rules, new micro-services, and new state information (metadata content) needed to manage the new policies. Each sub- collection can be managed using a different set of policies. The discussion of the concepts in rule-based policy virtualization and its application to long-term and large-scale data management for observatories such as ORION and NEON will be the basis of the paper.
DE: 0430 Computational methods and data processing
DE: 0910 Data processing
DE: 3315 Data assimilation
DE: 4260 Ocean data assimilation and reanalysis (3225)
DE: 4894 Instruments, sensors, and techniques
SC: Earth and Space Science Informatics [IN]
MN: 2007 Fall Meeting