HR: 1340h
AN: IN23A-0951    [Abstracts]
TI: Techniques for Efficiently Managing Large Geosciences Data Sets
AU: * Kruger, A
EM: anton-kruger@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AU: Krajewski, W F
EM: witold-krajewski@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AU: Bradley, A A
EM: allen-bradley@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AU: Smith, J A
EM: jsmith@princeton.edu
AF: Princeton University, Department of Civil and Environmental Engineering C-319G E-Quad, Princeton, USA 08544,
AU: Baeck, M L
EM: mlbaeck@princeton.edu
AF: Princeton University, Department of Civil and Environmental Engineering C-319G E-Quad, Princeton, USA 08544,
AU: Steiner, M
EM: msteiner@ucar.edu
AF: National Center for Atmospheric Research, Research Applications Laboratory Boulder Colorado, Boulder, CO 80307, United States
AU: Lawrence, R E
EM: ramon.lawrence@ubc.ca
AF: UCAR Office of Programs, Unidata Program Center P.O. Box 3000, Boulder, CO 80307, United States
AU: Ramamurthy, M K
EM: mohan@unidata.ucar.edu
AF: National Climatic Data Center, Federal Building 151 Patton Avenue, Asheville, NC 28801, United States
AU: Weber, J
EM: jweber@ucar.edu
AF: National Climatic Data Center, Federal Building 151 Patton Avenue, Asheville, NC 28801, United States
AU: Delgreco, S A
EM: stephen.a.delgreco@noaa.gov
AF: University of British Columbia Okanagan, Psychology and Computer Science Unit Irving K. Barber School of Arts and Sciences SCI 263 3333 University Way, Kelowna, BC V1V 1V7, Canada
AU: Domaszczynski, P
EM: piotr-domaszczynski@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AU: Seo, B
EM: bongchul-seo@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AU: Gunyon, C A
EM: charles-gunyon@uiowa.edu
AF: The University of Iowa, IIHR-Hydroscience & Engineering 107 SHL, Iowa City, IA 52242, United States
AB: We have developed techniques and software tools for efficiently managing large geosciences data sets. While the techniques were developed as part of an NSF-Funded ITR project that focuses on making NEXRAD weather data and rainfall products available to hydrologists and other scientists, they are relevant to other geosciences disciplines that deal with large data sets. Metadata, relational databases, data compression, and networking are central to our methodology. Data and derived products are stored on file servers in a compressed format. URLs to, and metadata about the data and derived products are managed in a PostgreSQL database. Virtually all access to the data and products is through this database. Geosciences data normally require a number of processing steps to transform the raw data into useful products: data quality assurance, coordinate transformations and georeferencing, applying calibration information, and many more. We have developed the concept of crawlers that manage this scientific workflow. Crawlers are unattended processes that run indefinitely, and at set intervals query the database for their next assignment. A database table functions as a roster for the crawlers. Crawlers perform well-defined tasks that are, except for perhaps sequencing, largely independent from other crawlers. Once a crawler is done with its current assignment, it updates the database roster table, and gets its next assignment by querying the database. We have developed a library that enables one to quickly add crawlers. The library provides hooks to external (i.e., C-language) compiled codes, so that developers can work and contribute independently. Processes called ingesters inject data into the system. The bulk of the data are from a real-time feed using UCAR/Unidata's IDD/LDM software. An exciting recent development is the establishment of a Unidata HYDRO feed that feeds value-added metadata over the IDD/LDM. Ingesters grab the metadata and populate the PostgreSQL tables. These and other concepts we have developed have enabled us to efficiently manage a 70 Tb (and growing) data weather radar data set.
DE: 0525 Data management
DE: 0530 Data presentation and visualization
DE: 0599 General or miscellaneous
DE: 1719 Hydrology
SC: Earth and Space Science Informatics [IN]
MN: 2007 Fall Meeting