HR: 0800h
AN: IN21B-0478    [Abstracts]
TI: Parallel IDL and Python for Earth and Space Science Data Analysis
AU: * Fillmore, D
EM: fillmore@txcorp.com
AF: Tech-X Corporation, 5621 Arapahoe Ave Suite A, Boulder, CO 80306, United States
AU: Galloy, M
EM: mgalloy@txcorp.com
AF: Tech-X Corporation, 5621 Arapahoe Ave Suite A, Boulder, CO 80306, United States
AU: Messmer, P
EM: messmer@txcorp.com
AF: Tech-X Corporation, 5621 Arapahoe Ave Suite A, Boulder, CO 80306, United States
AB: The large amount of data collected or generated in the Earth and space sciences, such as datasets from space-based Earth and solar observation missions or climate models, poses a significant computational challenge. While data analysis problems often could greatly benefit from parallel computing, widely used tools like IDL (Interactive Data Language) and Python offer only limited support for cluster computing. Users therefore have to develop implementations of taskfarms, an often lengthy, error prone and unportable process. We present a taskfarming support tool simplifying the process of performing loosely coupled data analysis in parallel. The user sends individual tasks to a centralized server and distributed clients, running persistent IDL or Python sessions, fetch these tasks, execute them and return for new tasks. Examples are given in which we have integrated the task farm system with the solar image processing suite Solar Soft (SSW) in IDL and the Climate and Data Analysis Tools (CDAT) in Python.
DE: 9820 Techniques applicable in three or more fields
SC: Earth and Space Science Informatics [IN]
MN: 2007 Fall Meeting