Software should be compiled/installed on the login nodes – manually or using USER-SPACK.
Container (for Apptainer/Singularity) may be prepared on your local computer (or the login node) and later uploaded to the cluster.
Jobs running on compute nodes should only contain the execution of the already set up environment. All environment preparations (download of additional software packages and data from the Internet) should be done beforehand.
There is a special dev partition (see Partitions) meant to be used for testing purposes. Feel free to use that partition to test different job settings etc.