Job Management

With the exponential growth of supercomputers in parallelism, applications are growing more diverse, including traditional large-scale HPC MPI jobs, and ensemble workloads such as finer-grained many-task computing (MTC) applications. Delivering high throughput and low latency for both workloads requires developing a distributed job management system that is magnitudes more scalable than today's centralized ones. In this paper, we present a distributed job launch prototype, SLURM++, which is comprised of multiple controllers with each one managing a partition of SLURM daemons, while ZHT (a distributed key-value store) is used to store the job and resource metadata. We compared SLURM++ with SLURM using micro-benchmarks of different job sizes up to 500 nodes, with excellent results showing 10X higher throughput. We also studied the potential of distributed scheduling through simulations up to millions of nodes.

With the exponentially growth of distributed computing systems in both flops and cores, scientific applications are growing more diverse with a variety of workloads. These workloads include traditional large-scale High Performance Computing MPI jobs, and ensemble workloads, such as Many-Task Computing workloads comprised of extremely large number of tasks of finer granularity, where tasks are defined on a per-core or per-node level, and often execute in milliseconds to seconds. Delivering high throughput and low latency for these heterogeneous workloads requires developing distributed job management system that is magnitudes more scalable and available than today’s centralized batch-scheduled job management systems. In this paper, we present a distributed job launch prototype SLURM++, which extends the SLURM resource manager by integrating the ZHT zero-hop distributed key-value store for distributed state management. SLURM++ is comprised of multiple controllers with each one managing several SLURM daemons, while ZHT is used to store all the job metadata and the SLURM daemons’ state. We compared SLURM with our SLURM++ prototype with a variety of microbenchmarks of different job sizes (small, medium, and large) at modest scales (500-nodes) with excellent results (10X higher job throughput). Scalability trends shows expected performance to be many orders of magnitude higher at tomorrow’s extreme scale systems. Keywords-job management systems; job launch; distributed scheduling; key-value stores

Job Management Systems (JMSs) efficiently schedule and monitor jobs in parallel and distributed computing environments. Therefore, they are critical for improving the utilization of expensive resources in high‐performance computing systems and centers, and an important component of Grid software infrastructure. With many JMSs available commercially and in the public domain, it is difficult to choose an optimum JMS for a given computing environment. In this paper, we present the results of the first empirical study of JMSs reported in the literature. Four commonly used systems, LSF, PBS Pro, Sun Grid Engine/CODINE, and Condor were considered. The study has revealed important strengths and weaknesses of these JMSs under different operational conditions. For example, LSF was shown to exhibit excellent throughput for a wide range of job types and submission rates. Alternatively, CODINE appeared to outperform other systems in terms of the average turn‐around time for small jobs, and PBS appeared to excel in terms of turn‐around time for relatively larger jobs. Copyright © 2004 John Wiley & Sons, Ltd.

High Performance Computing is characterized by the latest technological evolutions in computing architectures and by the increasing needs of applications for computing power. A particular middleware called Resource and Job Management System (RJMS), is responsible for delivering computing power to applications. The RJMS plays an important role in HPC since it has a strategic place in the whole software stack because it stands between the above two layers. However the latest evolutions in hardware and applications layers have provided new levels of complexities to this middleware. Issues like scalability, management of topological constraints, energy efficiency and fault tolerance have to be particularly considered, among others, in order to provide a better system exploitation from both the system and user point of view. This dissertation provides a state of the art upon the fundamental concepts and research issues of Resources and Jobs Management Systems. It provides a multi-level comparison (concepts, functionalities, performance) of some Resource and Jobs Management Systems in High Performance Computing. An important metric to evaluate the work of a RJMS on a platform is the observed system utilization. However studies and logs of production platforms show that HPC systems in general suffer of significant un-utilization rates. Our study deals with these clusters' un-utilization periods by proposing methods to aggregate otherwise un-utilized resources for the benefit of the system or the application. More particularly this thesis explores RJMS level mechanisms: 1) for increasing the jobs valuable computation rates in the high volatile environments of a lightweight grid context, 2) for improving system utilization with malleability techniques and 3) providing energy efficient system management through the exploitation of idle computing machines. The experimentation and evaluation in this type of contexts provide important complexities due to the inter-dependency of multiple parameters that have to be taken into control. In this thesis we have developed a methodology based upon real-scale controlled experimentation with submission of synthetic or real workload traces.

We propose a multi-tiered architecture for middleware-independent Grid job management. The architecture consists of a number of services for well-defined tasks in the job management process, offeri ...

The last decades have been characterized by an ever growing requirement in terms of computing and storage resources. This tendency has recently put the pressure on the ability to efficiently manage the power required to operate the huge amount of electrical components associated with state-of-the-art high performance computing systems. The power consumption of a supercomputer needs to be adjusted based on varying power budget or electricity availabilities. As a consequence, Resource and Job Management Systems have to be adequately adapted in order to efficiently schedule jobs with optimized performance while limiting power usage whenever needed. We introduce in this paper a new scheduling strategy that can adapt the executed workload to a limited power budget. The originality of this approach relies upon a combination of speed scaling and node shutdown techniques for power reductions. It is implemented into the widely used resource and job management system SLURM. Finally, it is validated through large scale emulations using real production workload traces of the supercomputer Curie.

This paper compares three most common job management systems and their workings with Sun HPC ClusterTools 3.1. Various aspects such as installation, customization, scheduling and resource control issues are discussed. The three chosen systems are: Load Sharing Facility (LSF), Portable Batch System (PBS) and COmputing in DIstributed Networked Environment (CODINE)/ Global Resource Director (GRD). We give a brief overview of each product but mainly focus on integrating these job management systems with Sun HPC ClusterTools. We provide useful guidelines to Sun HPC ClusterTools users when using job management systems with Sun HPC ClusterTools. We further demonstrate how to use the job management systems in support of commercial MPI applications with HPC ClusterTools.

State Machine, ASM for short, is a powerful computational model [34], which has been successfully applied in several cases for modeling critical, complex systems, both in industry and in academia: wide discussion about ASM application is in [7]. It simulates every algorithm’s behavior through a step-by-step way: each step computes a set of updates with given transition rules. After the completion of a step, all updates are committed simultaneously. The concept of abstract state in ASM extends the usual notion of state occurring in Finite State Machines: it is an arbitrarily simple or complex structure, i.e. a collection of domains, with arbitrary functions and relations defined on them. On the other hand, rules are basically nested if-then-else clauses with a set of functions in their bodies. In each state, all conditions guarding rules are checked, that is all rules whose conditions are evaluated to true are simultaneously executed, so determining the state transition. All the parameters are stored in a set of so called locations and at each time the particular configuration of parameters values determines the current state of the ASM. The transition from one state to another is described through a set of formulas, in the form: {if conditioni then updatesi} i = 1,..,n where each conditioni (the guard of the i-th rule) is an arbitrary first-order formula, whose interpretation can be true or false, and each updatesi is a finite set of assignments f(t1,...,tn) = t whose execution is to be understood as changing the value of the function f at the given parameters ti, that leads to a new value of the parameter t. The parameters are stored in a set of locations and the configuration of parameter values at each step determines the current state of the ASM. For the unambiguous determination of a next state, it is necessary that updates are consistent. An update set is consistent if it contains no pair of updates which assign different values to the same location; otherwise, the update set is inconsistent. The concepts related to modeling monolithic systems through ASMs can be extended to distributed systems. The case of job management in Grid systems is a typical case of asynchronous distributed systems. It is distributed because its overall behavior is the composition of different, independent, remote elements, each operating on its own. It is asynchronous because all involved logical components operate and communicate concurrently, each according to its internal behavior. So, job EMSs in Grid can be modeled by a Distributed Asynchronous ASM asyncASM [7], [21]. Essentially, asyncASMs generalize simple ASMs to an arbitrary finite number of independent agents [23], each executing an ASM in its own local state. Formally, an asyncASM is given by a family of pairs (a, ASM(a)) of pairwise different agents, elements of a possibly dynamic finite set Agent, each executing its ASM, ASM(a). In this sense, each agent a executes its own program, operating 298 Informatica 37 (2013) 295–306 A. Bianchi et al. on its own states, so determining a partial view of the entire system. The relation between global and local states is supported by the reserved keyword self, used to denote the specific agent executing a rule, and to store information relevant to itself. A new agent can be introduced into the asyncASM at any time by extending the set Agent. An ASM-based process for developing complex systems is presented in [7]: it allows capturing system model requirements and refining them through intermediate models to any desired level of detail in a validated and verifiable code. In the present work, modeling and implementation activities have been carried out with the support of the coreASM framework [8], [9], [22]. It follows mathematical definition of ASMs and inherits several typical features of the ASM modeling. Its main purpose is to make ASM-based models executable. To this end, the framework includes some language constructs aimed at making easy the development, as, for instance, forAll, which allows executing all rules satisfying a given guard condition; choose, aimed at expressing non-determinism in the choice of a rule to execute when a condition is satisfied; seqblock/endseqblock are the delimiters of block, whose rules must be executed sequentially; par/endpar are the delimiters of block, whose rules must be executed concurrently; enqueue/dequeue are the operators for adding / removing elements to a queue. 4 Modeling the Grid job EMSs In order to model the job EMSs, firstly the Grid capabilities and features are informally described, then they are abstracted and formally defined, finally the asyncASM is created in the coreASM environment [8]. 4.1 Informal description of a Grid system

From its conception the job management system has been distributed to increase scalability and robustness. The system consists of several applications (called ProdAgents) which manage Monte Carlo, reconstruction and skimming jobs on collections of sites within different Grid environments (OSG, NorduGrid, LCG) and submission systems such as GlideIn, local batch, etc... Production of simulated data in CMS mainly takes place on so called Tier2s (small to medium size computing centers) resources. Approximately ~50% of the CMS Tier2 resources are allocated to running simulation jobs. While the so-called Tier1s (medium to large size computing centers with high capacity tape storage systems) will be mainly used for skimming and reconstructing detector data. During the last one and a half years the job management system has been adapted such that it can be configured to convert Data Acquisition (DAQ) / High Level Trigger (HLT) output from the CMS detector to the CMS data format and manage the real time data stream from the experiment. Simultaneously the system has been upgraded to facilitate the increasing scale of the CMS production and adapting to the procedures used by its operators. In this paper we discuss the current (high level) architecture of ProdAgent, the experience in using this system in computing challenges, feedback from these challenges, and future work including migration to a set of core libraries to facilitate convergence between the different data management projects within CMS that deal with analysis, simulation, and initial reconstruction of real data. This migration is important, as it will decrease the code footprint used by these projects and increase maintainability of the code base.

One important component of grid software infrastructure and parallel systems management is the Job Management System (JMS). With many JMSs available commercially and in public domain, it is difficult to choose the most efficient JMS for a given computing environment. All previous comparisons of JMSs had only a conceptual character. In this paper, we present the results of the first empirical study of JMSs reported in the literature. Two most popular commercial systems, LSF and PBS Pro, were included in our study. The study has revealed important strengths and weaknesses of these JMSs under different operational conditions. For example, LSF was shown to exhibit excellent throughput for a wide range of job types and submission rates. On the other hand, PBS appeared to excel in terms of turn-around time. Whenever possible, our study have tried to identify and explain the reasons behind the observed behavior of investigated JMSs.