Daily Bulletin

Upcoming major change to PBS job behavior on Cheyenne

September 22, 2021

CISL is developing and testing peer-scheduling capability between Cheyenne and Casper, through which users will be able to submit jobs from Cheyenne to Casper, from Casper to Cheyenne, and craft job-dependency rules between jobs on both systems. More details on how to use this capability will come in future updates and documentation.

All job submissions on Cheyenne currently are configured to inherit environment settings (e.g., variables and loaded modules) from the user’s submission shell session, which is typically the login environment. Because of incompatibilities between software on Cheyenne and Casper, this configuration is not compatible with peer-scheduling. Therefore, the default behavior on Cheyenne will be changed on October 20 to not propagate users’ submission-environment settings to PBS jobs.

For many workflows, this change will have no impact, but some users may want to set variables that are passed into job environments during iterative development cycles. For such use cases, individual variables can be forwarded, or explicitly defined, using the “-v” argument to qsub. Users who want to continue propagating their entire submission environment after this change can do so using the “-V” qsub argument or PBS script directive.