IN21B-0477
ShakeOut and its Effects in Los Angeles and Oxnard Areas
Three-dimensional simulations of earthquakes have given a deeper understanding of wave propagation and site effects in urban regions. In this work we study the impact of a potential major earthquake on the San Andreas Fault with significant seismic hazard in the Greater Los Angeles Basin. We present results for the ShakeOut simulation---a rupture beginning near Salton Sea, California, heading 270 km northwest along the fault, that produces a Mw 7.8 earthquake in a geographical region which includes all major populated areas of Southern California and northern Mexico, in a 600 km by 300 km by 80 km volume, for a maximum frequency of 1.0 Hz and a minimum shear wave velocity of 200 m/s. For the material model, we use a discretized version of SCEC's CVM4 velocity model, called CVM-Etree. The simulation was performed at the Pittsburgh Supercomputing Center using Hercules, a finite element octree-based, parallel software developed by the Quake Group at Carnegie Mellon University. Hercules implements a highly efficient end-to-end algorithm for solving the wave field in highly heterogeneous media due to kinematic faulting. We verify our results by comparing synthetic seismograms computed with a parallel finite difference code by Robert Graves (URS) for a similar scenario earthquake, for a maximum frequency of 0.5 Hz and minimum shear wave velocity of 500 m/s. We focus our analysis of the results of the 1.0 Hz ShakeOut simulation on the Los Angeles Basin area, and the Santa Clara River Valley and Oxnard Plain. We examine the site effects present in these two areas and their proneness to capture and amplify seismic waves due to their geological features. Results show a direct correlation between the amplification levels and the local soil and basin profiles. http://www.cs.cmu.edu/~{}quake
IN21B-0478
Parallel IDL and Python for Earth and Space Science Data Analysis
The large amount of data collected or generated in the Earth and space sciences, such as datasets from space-based Earth and solar observation missions or climate models, poses a significant computational challenge. While data analysis problems often could greatly benefit from parallel computing, widely used tools like IDL (Interactive Data Language) and Python offer only limited support for cluster computing. Users therefore have to develop implementations of taskfarms, an often lengthy, error prone and unportable process. We present a taskfarming support tool simplifying the process of performing loosely coupled data analysis in parallel. The user sends individual tasks to a centralized server and distributed clients, running persistent IDL or Python sessions, fetch these tasks, execute them and return for new tasks. Examples are given in which we have integrated the task farm system with the solar image processing suite Solar Soft (SSW) in IDL and the Climate and Data Analysis Tools (CDAT) in Python.
IN21B-0479
Interactive Visualization and Monitoring of Large-Scale 3-D Mantle Convection Runs
With the imminent arrival of petascale computing in the United States by 2011, new strategies for visualizing and monitoring high-resolution numerical simulations on massively parallel computers are needed to overcome the extreme data and resource requirements. We have employed a visualization system consisting of 14 powerful Dell workstations, each with a multi-terabyte disk, connected via a high-speed network with a bandwidth on the order of a few gigabits per second to a locally situated massively parallel system with approximately 2,000 processing elements. This system has been constructed at the Laboratory of Computational Sciences and Engineering at the University of Minnesota. Near real-time interactive analysis of 3-D mantle convection using around 10 million grid points has been carried out using a client-server application capable of streaming gigabytes of simulated data to a remote Powerwall with 13 million pixels. Concurrently, we have constructed a web-portal that allows a user to monitor the same run at home or in a hotel room, using a laptop. In our case, interactive computing takes on the meaning of performing such runs for a limited duration of time, say 1 to 2 hours. This calls for a balance between grid resolution and the number of processing elements required to provide the level of interactivity needed to achieve one to a few frames per second. Our mode of operation represents a new paradigm in numerical modeling that supports a trend toward both real-time visualization and monitoring of high-resolution models and a consequent reduction in storage of raw output data, since the interactive periods are by definition short. Using this interactive strategy periodically we can facilitate long heroic runs extending over a few days.
IN21B-0480
Simulation of mode conversion process from Upper-Hybrid wave to LO-mode wave in plasmasphere
The mode conversion occurs where one propagating mode is linearly coupled to other propagation mode. There are various phenomena related to the mode conversion process such as terrestrial continuum radiation, Auroral kilometric radiation, and Jovian decametric emissions. However, we often find observational results related to the mode conversion process in the region where theoretical assumptions are not valid; e.g., where WKB approximation might violated. To discuss these phenomena, we should evaluate the conversion process quantitatively by numerical experiments. For this purpose, we use Electron Hybrid code which is originally developed by Katoh (2003). In this scheme hot electrons are treated as particles while cold electrons are treated as a fluid. In order to establish a realistic model, we use Akebono satellite observation of plasmasphere. We assume two-dimensional simulation system where the uniform magnetic field B0 is assumed to be in x-y plane. The wave vector was introduced to be aligned the x-axis direction making an oblique propagating to the external magnetic field. The plasma density gradient was set perpendicular to external magnetic field B0 within the present study. Several simulations with different wave normal angle have been performed, with the same conditions such as, plasma frequency, steepness of density gradient and angular frequency in order to looking for the most efficient wave normal angle for mode conversion. The size of the simulation box used in the present study was determined depend on the wave normal angle. Based on the simulation results, we discussed how the wave coupling occurs in magnetized cold plasma by performing FFT analyses on wave electric field to examine spatial distribution of frequency/wave number spectra and by considering the polarization of wave modes propagating in the simulation system. Thus, the simulation results showed the generation of electromagnetic LO-mode wave through the mode conversion process from Upper Hybrid wave into LO-mode wave quantitatively. We also evaluated the efficiency of mode conversion depending on the wave normal angle.
IN21B-0481
Pakal A New Algorithm to Solve the Radiative Transfer Equation
We present a new algorithm, called "PAKAL", to solve the radiative transfer equation using a cellular automaton and an expert system. Pakal are integrated by four main modules: i) geometry model, ii) radiative transfer equation numerical model, iii) emission model and iv) numerical methods. In this work we present the algorithm applied to 3D geometry. We have solved the radiative transfer equation using radio thermal emission and we used two different numerical methods to recreate the quiet Sun radio thermal emission. Pakal is able to generate multi-frequency images in 2D which may be compared against observations. In this work, we present the convergence analysis and comparetion between the numerical models.
IN21B-0482
A parallel Atmosphere-Ocean Global Circulation Model of intermediate complexity for Earth system climate research
We present the evolution of an Earth System model of intermediate complexity featuring an ocean global circulation model to include a fully coupled 3D primitive equations atmospheric model. The original Earth System climate model, UVic ESCM (Weaver et al. 2001), uses an ocean global circulation model coupled to a one layer atmospheric energy-moisture balance model. It also comprises a viscous-plastic rheology sea ice model, a mechanical land ice model, land surface, oceanic and terrestrial carbon models and a simple 3D marine ecosystem model (Schmittner et al. 2005). A spectral atmospheric, model, PUMA (Fraedrich et al. 2005), was coupled to the UVic ESCM to provide an atmosphere with nonlinear dynamics in target resolutions of T21, T31 and T42, as required. The coupling with the atmosphere, which involves data transfer, preprocessing and interpolation, is done through the OASIS3 coupler. During a run there are 2 + 2N parallel processes: the UVic ESCM, the Oasis3 coupler and the PUMA model with its domain split across 2N processes. The choice of N allows to balance more or less complex configurations of UVic model (e.g. higher level marine ecosystem model or number of biogeochemical tracers) with the atmospheric model at different resolutions, in order to maintain computational efficiency. The relatively simple parameterizations make this new atmosphere-ocean global circulation model much faster than a state-of-the-art Atmosphere-Ocean Global Circulation Model, and so optimally geared for decadal to millennial scale integrations. The latter require special care with the conservation of fluxes during coupling. A second order conservative interpolation method was applied (Jones 1999) and this is compared with the use of typical non-conservative methods.
IN21B-0483
Mapping PetaSHA Applications to TeraGrid Architectures
The Southern California Earthquake Center (SCEC) has a science program in developing an integrated cyberfacility – PetaSHA – for executing physics-based seismic hazard analysis (SHA) computations. The NSF has awarded PetaSHA 15 million allocation service units this year on the fastest supercomputers available within the NSF TeraGrid. However, one size does not fit all, a range of systems are needed to support this effort at different stages of the simulations. Enabling PetaSHA simulations on those TeraGrid architectures to solve both dynamic rupture and seismic wave propagation have been a challenge from both hardware and software levels. This is an adaptation procedure to meet specific requirements of each architecture. It is important to determine how fundamental system attributes affect application performance. We present an adaptive approach in our PetaSHA application that enables the simultaneous optimization of both computation and communication at run-time using flexible settings. These techniques optimize initialization, source/media partition and MPI-IO output in different ways to achieve optimal performance on the target machines. The resulting code is a factor of four faster than the orignial version. New MPI-I/O capabilities have been added for the accurate Staggered-Grid Split-Node (SGSN) method for dynamic rupture propagation in the velocity-stress staggered-grid finite difference scheme (Dalguer and Day, JGR, 2007), We use execution workflow across TeraGrid sites for managing the resulting data volumes. Our lessons learned indicate that minimizing time to solution is most critical, in particular when scheduling large scale simulations across supercomputer sites. The TeraShake platform has been ported to multiple architectures including TACC Dell lonestar and Abe, Cray XT3 Bigben and Blue Gene/L. Parallel efficiency of 96% with the PetaSHA application Olsen-AWM has been demonstrated on 40,960 Blue Gene/L processors at IBM TJ Watson Center. Notable accomplishments using the optimized code include the M7.8 ShakeOut rupture scenario, as part of the southern San Andreas Fault evaluation SoSAFE. The ShakeOut simulation domain is the same as used for the SCEC TeraShake simulations (600 km by 300 km by 80 km). However, the higher resolution of 100 m with frequency content up to 1 Hz required 14.4 billion grid points, eight times more than the TeraShake scenarios. The simulation used 2000 TACC Dell linux Lonestar processors and took 56 hours to compute 240 seconds of wave propagation. The pre-processing input partition, as well as post-processing analysis has been performed on the SDSC IBM Datastar p655 and p690. In addition, as part of the SCEC DynaShake computational platform, the SGSN capability was used to model dynamic rupture propagation for the ShakeOut scenario that match the proposed surface slip and size of the event. Mapping applications to different architectures require coordination of many areas of expertise in hardware and application level, an outstanding challenge faced on the current petascale computing effort. We believe our techniques as well as distributed data management through data grids have provided a practical example of how to effectively use multiple compute resources, and our results will benefit other geoscience disciplines as well.
IN21B-0484
Development of a Web Based Simulating System for Earthquake Modeling on the Grid
Existing cyberinfrastructure-based information, data and computational networks now allow development of state- of-the-art, user-friendly simulation environments that democratize access to high-end computational environments and provide new research opportunities for many research and educational communities. Within the Geosciences cyberinfrastructure network, GEON, we have developed the SYNSEIS (SYNthetic SEISmogram) toolkit to enable efficient computations of 2D and 3D seismic waveforms for a variety of research purposes especially for helping to analyze the EarthScope's USArray seismic data in a speedy and efficient environment. The underlying simulation software in SYNSEIS is a finite difference code, E3D, developed by LLNL (S. Larsen). The code is embedded within the SYNSEIS portlet environment and it is used by our toolkit to simulate seismic waveforms of earthquakes at regional distances (<1000km). Architecturally, SYNSEIS uses both Web Service and Grid computing resources in a portal-based work environment and has a built in access mechanism to connect to national supercomputer centers as well as to a dedicated, small-scale compute cluster for its runs. Even though Grid computing is well-established in many computing communities, its use among domain scientists still is not trivial because of multiple levels of complexities encountered. We grid-enabled E3D using our own dialect XML inputs that include geological models that are accessible through standard Web services within the GEON network. The XML inputs for this application contain structural geometries, source parameters, seismic velocity, density, attenuation values, number of time steps to compute, and number of stations. By enabling a portal based access to a such computational environment coupled with its dynamic user interface we enable a large user community to take advantage of such high end calculations in their research and educational activities. Our system can be used to promote an efficient and effective modeling environment to help scientists as well as educators in their daily activities and speed up the scientific discovery process. http://portal.geongrid.org/synseis/
IN21B-0485
The SCEC Petascale Cyberfacility for Physics-based Seismic Hazard Analysis (PetaSHA): Accelerating SCEC Research Using High Performance Computing
The SCEC Community Modeling Environment (SCEC/CME) collaboration is extending SCEC's program of seismic hazard research using high performance computing with the NSF-funded Petascale Cyberfacility for Physics-based Seismic Hazard Analysis (PetaSHA) Project. The SCEC PetaSHA project is a collaboration of geoscientists and computer scientists that integrate geophysical numerical modeling codes with leading-edge cyberinfrastructure to perform seismic hazard research at large-scales and high-resolution using national academic supercomputing facilities. The PetaSHA computational capabilities are organized around the development of robust, re-usable, well-validated simulation systems we call computational platforms. Researchers on the PetaSHA Project are currently developing the DynaShake Platform (dynamic rupture simulations), the TeraShake Platform (wave propagation simulations), the CyberShake Platform (physics-based probabilistic seismic hazard analysis), the BroadBand Platform (deterministic and stochastic modeling of high frequency synthetic waveforms), the Full 3D Tomography (F3DT) Platform (improvements in structural representations), as well as using and extending the OpenSHA Platform (Probabilistic Seismic Hazard Analysis). We will describe several current PetaSHA research projects including the application of the DynaShake Platform to dynamic rupture modeling of the ShakeOut source, the use of the TeraShake Platform, including the URS- Graves, SDSU-Olsen and CMU-Hercules Anelastic Wave Propagation codes, to model 1Hz ShakeOut simulations, the use of the CyberShake Platform to investigate physics-based PSHA hazard curves, and the use of the F3DT Platform to produce an improved structural model for a large region in southern California. http://www.scec.org/petasha
IN21B-0486
The Rigorous Calculation of the Covariance Matrix for Arbitrarily Large Inverse Problems
The outline of an approach for computing the covariance matrix of a large, dense linear system will be presented. The creation of such matrices often arises in the Earth sciences, in particular when the error characteristics of a complex inverse problem are desired. For extremely large problems, such as those exceeding the memory limits of the machine being used, approximation techniques are often employed; however, doing so inherently introduces errors into the final covariance matrix. To avoid these errors, an approach has been developed by which the covariance matrix of a nearly arbitrarily sized dense linear problem can be computed on a machine of nearly any size. The concept involves the use of out-of-core computations, in which the problem is stored on disk and processed incrementally. While the idea of out-of-core computing has been around for decades, the specific approach we have developed involves the use of tiles as opposed to the more traditional slab-based approach. The tile-based method will be shown to provide both high performance and scalability on a range of problem sizes, number of processors, and machine types. A specific application involving the computation of the Earth's gravitational potential field will be illustrated.
IN21B-0487
Verification of SORD, and Application to the TeraShake Scenario
The Support Operator Rupture Dynamics (SORD) code provides a highly scalable (up to billions of nodes) computational tool for modeling spontaneous rupture on a non-planar fault surface embedded in a heterogeneous medium with surface topography. SORD successfully performs the SCEC Rupture Dynamics Code Validation Project tests, and we have undertaken further dynamic rupture tests assessing the effects of distorted hexahedral meshes on code accuracy. We generate a family of distorted meshes by simple shearing (applied both parallel and normal to the fault plane) of an initially Cartesian mesh. For shearing normal to the fault, shearing angle was varied, up to a maximum of 73-degrees. For SCEC Validation Problem 3, grid-induced errors increase with mesh-shear angle, with the logarithm of error approximately proportional to angle over the range tested. At 73-degrees, RMS misfits are about 10% for peak slip rate, and 0.5% for both rupture time and total slip, indicating that the method--which up to now we have applied mainly to near-vertical strike-slip faulting-- also is capable of handling geometries appropriate to low-angle surface-rupturing thrust earthquakes. The SORD code was used to reexamine the TeraShake 2 dynamics simulations of a M7.7 earthquake on the southern San Andreas Fault. Relative to the original (Olsen et al, 2007) TeraShake 2 simulations, our spontaneous rupture models find decreased peak ground velocities in the Los Angles basin, principally due to a shallower eastward connecting basin chain in the SCEC Velocity Model Version 4 (used in our simulations) compared to Version 3 (used by Olsen et al.). This is partially offset by including the effects of surface topography (which was not included in the Olsen et al. models) in the simulation, which increases PGV at some basin sites by as much as a factor of two. Some non-basin sites showed comparable decreases in PGV. These predicted topographic effects are quite large, so it is important to quantify SORD accuracy in the presence of non-planar free surface geometry. We test the case of a semi-circular canyon to an incident P wave, and find close agreement with boundary element methods, for surface amplification at wavelengths comparable to the canyon width.
IN21B-0488
Community Needs Assessment and Portal Prototype Development for an Arctic Spatial Data Infrastructure (ASDI)
As the creation and use of geospatial data in research, management, logistics, and education applications has proliferated, there is now a tremendous potential for advancing science through a variety of cyber-infrastructure applications, including Spatial Data Infrastructure (SDI) and related technologies. SDIs provide a necessary and common framework of standards, securities, policies, procedures, and technology to support the effective acquisition, coordination, dissemination and use of geospatial data by multiple and distributed stakeholder and user groups. Despite the numerous research activities in the Arctic, there is no established SDI and, because of this lack of a coordinated infrastructure, there is inefficiency, duplication of effort, and reduced data quality and search ability of arctic geospatial data. The urgency for establishing this framework is significant considering the myriad of data that is being collected in celebration of the International Polar Year (IPY) in 2007-2008 and the current international momentum for an improved and integrated circum-arctic terrestrial-marine-atmospheric environmental observatories network. The key objective of this project is to lay the foundation for full implementation of an Arctic Spatial Data Infrastructure (ASDI) through an assessment of community needs, readiness, and resources and through the development of a prototype web-mapping portal. http://www.arcus.org
IN21B-0489
Introduction to Globus (tutorial)
The Globus Project is developing fundamental technologies needed to build Grids --persistent environments that enable software applications to integrate instruments, displays, computational and information. This paradigm has become common in many application communities that use distributed resources in a coordinated manner. In this session we will walk through a set of use cases that will demonstrate the use of many of the components which make up Globus, a set of open source software that includes services and libraries for security, resource management, monitoring and discovery, file transfer and data management. We summarize future plans, and give details on how others can contribute to this effort. We will then open up the session to community concerns and needs through discussion with the audience members. http://www.globus.org