Carnegie Mellon University School of Computer Science

High Performance Computing (HPC)

SCS Computing Facilities (SCSCF) supports privately owned High Performance Computational (HPC) Clusters for various research and instructional groups. 

The HPC service is designed to minimize support costs and efforts while maximizing the available resources. 

In order to support many clusters effectively, we've established the following practices:

  • common cluster architecture
  • standardized middleware and OS
  • tiered support model for resources

Cluster architecture

The HPC clusters that SCSCF manages consist of the following common set of components:

Head node (required)

  • used to install and configure all additional cluster resources
  • provides shared storage across cluster
  • defines network layout for cluster
  • acts as a login node for users
  • acts as resource scheduler for the entire cluster

Cluster Devices (Storage nodes / NAS servers) (optional)

  • provide additional storage space for the cluster
  • often used to isolate data or improve performance

Compute nodes

  • fungible resources - interchangeable - no persistent data on these machines
  • scalable - easy to add in order to increase computational capacity
  • inexpensive (relatively) - these do not need to be as reliable (or as expensive) as the head node because they are interchangeable and do not store data locally

Cluster Compute nodes (GPU nodes/distributed storage)

  • fungible resources - interchangeable - persistent data replicated across multiple machines
  • scalable - easy to add in order to increase computational capacity
  • inexpensive (relatively) - these do not need to be as reliable as the head node.  Typically contain scarce resources (GPUs)

Cluster Rack / Power

  • customer owned
  • hosts all cluster components (rack)
  • eliminates per-system machine room charges
  • UPS required for critical systems (head node, storage servers)
  • No UPS for GPU servers

Network

  • Private gigabit Ethernet (required)
  • management of cluster resources
  • used to deploy nodes
  • provides access to nodes and data across cluster
  • nat access between cluster nodes and public internet
  • Fast networking (optional)
  • faster data access
  • reduced latency (MPI jobs)
  • 10Gbit Ethernet supported
  • Infiniband supported

Operating Systems and Middleware:

We have standardized on the Rocks cluster distribution as the middleware we support. Rocks is built on top of the CentOS or Redhat Enterprise Linux operating system.

We currently support both Rocks/Centos 6 and Rocks/Centos 7.  The Rocks cluster distribution significantly improves our ability to manage existing cluster resources and to deploy and customize new resources in a consistent and reliable manner.

Job schedulers

For resource allocation and task distribution and scheduling within clusters, SCS supports the following job schedulers:

Running a job scheduler on the cluster allows tasks to be distributed to nodes continuously as resources become available permitting better resource utilization.  It also allows jobs to continue running even when individual compute nodes are unavailable.  Job schedulers also allow fair sharing and prioritization policies to be applied across the cluster.  This allows researchers to safely share resources with the knowledge that they can access their resources when they need to.

OS/application containers:

SCSCF does not support docker on the HPC clusters we manage.  There are a number of compatibility and security issues with running docker in a shared environment which preclude us from support it.  We do support singularity [https://www.cs.cmu.edu/~help/singularity] for running containers on HPC clusters.

Tiered support levels

High Priority (cluster infrastructure):

This includes resources that are critical to the operation of the cluster as a whole.  This generally means the cluster's head node, login nodes, and storage servers.  We work to respond quickly when there are problems with these systems. 

Elevated Priority (distributed storage and scarce or complex resources):

These systems contain scarce resources (persistent data, GPUs, etc) that require higher levels of attention when they experience problems.  We prioritize these systems at a higher level when troubleshooting but do NOT treat the availability of individual nodes in this tier as urgent.

Normal Priority (standard compute resources)

These systems are the regular compute nodes in a cluster.  Cluster compute nodes are fungibly interchangeable resources and should not contain any persistent data.  Support of these systems is best effort as time permits.