Daily Bulletin

Where to run small, CPU-only jobs?

April 11, 2024

CISL recommends running small jobs that use only CPUs on the Casper cluster’s high-throughput computing (HTC) nodes. Casper has 64 HTC nodes specifically for running small batch jobs. It also has:

  • Shared resources rather than dedicated Derecho nodes.
  • More available memory, plus NVMe swap for overflow and local storage.
  • Direct PBS cross-submission from Derecho. 

See Starting Casper Jobs for how to use the NVMe features, and Portable Batch System (PBS) Job Scripts for an example of how to run a job on the HTC nodes.

If you have many small, single core jobs, it can also be advantageous to “pack” them together to run concurrently.  We support this use case through the  use of “command files”—even on Derecho.  The basic idea is to create a single file listing all the commands you want to run concurrently and launch them in parallel. If you have hundreds of such jobs (or more), you will be able to keep full Derecho nodes busy using this approach. See this page or more information.