Daily Bulletin

Where to run small, CPU-only jobs?

November 16, 2023

CISL recommends running small jobs that use only CPUs on the Casper cluster’s high-throughput computing (HTC) nodes. Casper has 64 HTC nodes specifically for running small batch jobs. It also has:

  • Shared resources rather than dedicated Derecho nodes.
  • More available memory, plus NVMe swap for overflow and local storage.
  • Direct PBS cross-submission from Derecho. 

See Starting Casper jobs with PBS for an example of how to run a job on the HTC nodes as well as how to use the NVMe features.

If you have many small, single core jobs, it can also be advantageous to “pack” them together to run concurrently.  We support this use case through the  use of “command files” - even on Derecho.  The basic idea is to create a single file listing all the commands you want to run concurrently and launch them in parallel. If you have hundreds of such jobs (or more), you will be able to keep full Derecho nodes busy using this approach. See this page or more information.

© 2024 UCAR
Postal Address: P.O. Box 3000, Boulder, CO 80307-3000 • Shipping Address: 3090 Center Green Drive, Boulder, CO 80301
National Science Foundation
This material is based upon work supported by the National Center for Atmospheric Research, a major facility sponsored by National Science Foundation and managed by the University Corporation for Atmospheric Research. Any opinions, findings and conclusions or recommendations expressed in this material do not necessarily reflect the views of the National Science Foundation.