To ensure your job is scheduled efficiently, you must explicitly define its core resource requirements. Leaving these to Slurm’s defaults can lead to unexpected behavior like indefinitely pending jobs.
Specify the Node Count: Always provide the exact number of nodes via #SBATCH --nodes. If omitted, Slurm may miscalculate the required resources based on a single task per node. This can cause the scheduler to request an impossible or highly suboptimal number of nodes, preventing your job from ever starting.
Specify a Time Limit: Always set a realistic wall-clock time limit via #SBATCH --time. Your choice directly impacts Slurm’s backfilling mechanism and queue wait times for all users. A precise time estimate is crucial—especially for smaller jobs, as it allows them to fill small gaps in the schedule and start much faster.
Memory is NOT a Consumable Resource: On HSUper, memory (RAM) is not tracked or restricted by the Slurm scheduler. This means that configuration flags like --mem or --mem-per-cpu are completely ignored, and multiple jobs assigned to the same node must share the available RAM freely.
Critical Risk of OOM Crashes: If your job consumes too much memory, it can lead to Out-Of-Memory (OOM) errors that kill your process or destabilize other users’ jobs running on the same node. If your application is highly memory-intensive, you should ensure exclusive node access by using exclusive partitions or by adding
#SBATCH --exclusiveto your script. This guarantees your job has access to the node’s entire RAM without interference.Please note that compute nodes utilize RAM-backed file systems (e.g., for local storage or temporary directories). Consequently, residual data or processes from previous jobs may slightly reduce the available memory compared to a freshly booted node. As a best practice, please consider using the available environment variables regarding temporary files.
For more complex workflows (e.g., specific constraints, GPU allocations, or job dependencies), you can explore the extensive set of available configuration flags.
man sbatch (for batch scripts) or man salloc (for interactive allocations) on a login node.HSUper uses a tree-based network topology (topology/tree). Slurm is fully aware of the switch layout and automatically attempts to allocate nodes that are network-wise as close together as possible to minimize communication latency.
Specify the maximum number of leaf switches your job is allowed to span. This gives the scheduler more flexibility while ensuring high network performance:
#SBATCH --nodes=4
#SBATCH --switches=1
This example requests 4 nodes, strictly limited to a single leaf switch (1). If this configuration cannot be fulfilled within 5 minutes, the job will fall back and start using any available nodes to avoid infinite queue wait times.
Slurm allows to define a maximum wait time for the
--switchconstraint. The maximum wait time value is hard-capped to 5 minutes on HSUper. Please contact support if this wait time limit is not sufficient for your use case.
You can read more about this in the official Slurm –switches documentation.
If your application absolutely requires sequential hostnames (NOT nodes connected to the same switch!), you can enforce this. Be aware that this strictly limits the scheduler and can significantly increase your queue wait time:
#SBATCH --nodes=4
#SBATCH --contiguous
Also, please note that this will usually select nodes connected to different leaf switches due to our topology.
You can find more information in the official Slurm –contiguous documentation.