HR: 0800h
AN: IN11B-0469    [Abstracts]
TI: A System for Scripted Data Analysis at Remote Data Centers
AU: * Wang, D L
EM: wangd@uci.edu
AF: Department of Electrical Engineering and Computer Science University of California, Irvine, Engineering Tower, Irvine, CA 92697,
AU: Zender, C S
EM: zender@uci.edu
AF: Department of Earth System Science University of California, Irvine, Croul Hall, Irvine, CA 92697,
AU: Jenks, S F
EM: stephen.jenks@uci.edu
AF: Department of Electrical Engineering and Computer Science University of California, Irvine, Engineering Tower, Irvine, CA 92697,
AB: Terascale data reduction and analysis still remain elusive for most geoscientists even as large scale geophysical models and observing systems begin to produce petascale datasets. Massive amounts of geophysical netCDF data remain underutilized due to scientists' limited bandwidth and computational capacity. Compounding the problem are the unique characteristics of geoscientists' data analyses which are poorly matched to traditional grid technologies. Our system, the Script Workflow Analysis for MultiProcessing (SWAMP) allows scientists to efficiently leverage grid resources for parallel computation through a rich scripting syntax compatible with most existing analysis scripts. Although shell script languages are impossible to optimize for all cases, data analysis scripting is well-defined sufficiently to allow inference of dependencies and use of advanced compilation techniques. SWAMP compiles scripts to detect potential parallelism and to dynamically schedule for parallel execution on resources allocated by an industry-standard grid scheduling engine. By its unique focus on I/O and bandwidth considerations rather than raw computational power, our system more efficiently handles the characteristic I/O-boundedness of geoscience workflows. Although scientists' scripts exhibit significant available parallelism, such parallelism is poorly exploited by conventional means that manage I/O and data locality as afterthoughts, if at all. In contrast, SWAMP eliminates unnecessary data movement to achieve its parallel efficiency in two ways: by providing a computational interface integrated with data service, and by scheduling computation to avoid I/O bottlenecks in disks and networks. This efficiency and performance is designed to integrate with scientists' analysis scripts-- many existing scripts require no modification to work with SWAMP. We test our system in uniprocessor, multiprocessor, and clustered environments, with both simple and terascale test cases. Benchmarks and other workflow statistics quantify not only the real performance benefits but also the reasons that render traditional methods unsuitable. We show that SWAMP typically modestly increases the data source's computational load while significantly reducing bandwidth in nearly all cases, making it suitable for installation in small lab groups as well as large data centers. Local computation and bandwidth requirements are thus drastically reduced, freeing the scientist to perform exploratory analysis and discovery in wider scopes and finer resolutions, and reducing time to discovery.
UR: http://code.google.com/p/swamp/
DE: 0520 Data analysis: algorithms and implementation
DE: 0525 Data management
SC: Earth and Space Science Informatics [IN]
MN: 2007 Fall Meeting