Data Replication and Job Scheduling on Cluster Grid for Data-Intensive Applications

Abstract

In data grids, distributed scientific and engineering applications often require access to large amount of data (terabytes or petabytes). Data access time depends on bandwidth, especially in cluster grid. Network bandwidth within a grid cluster is larger than across clusters. In a communications environment, the major bottleneck to support fast data access in Grids is the high latencies of Wide Area Networks (WANs). Effective scheduling in such network architecture can reduce the amount data transfer across WANs by dispatch a job to where the needed data is present. In another alternative, data replication mechanism generates multiple copies of the existing data to reduce the access opportunity from remote site. To avoid WAN bandwidth bottleneck in a cluster grid, we develop a job scheduling policy, called HCS (Hierarchical Cluster Scheduling), and dynamic data replication strategy, called HRS (hierarchical Replication Strategy) to improve the latency of data accesses. We use simulation studies to evaluate various data access situations. The simulation results show that HCS and HRS successfully reduces data access time and the amount of inter-communications in comparing to other methods in cluster grid.