Contemporary research pipelines often employ Python - either for data analysis or even for data generation (e.g., simulation) itself. On HSUper, Python interpreters and corresponding packages can be installed via Spack or via miniconda, where the latter will be most suitable for typical users. This article briefly describes how to configure miniconda3 on HSUper and provides an example installation of typical deep learning libraries to a user-defined environment.
miniconda3 is management system for your scientific Python stack. It is available via the module system and the initial setup can be performed via
ml gcc miniconda3
conda init
Note that you need to close and re-open your SSH connection to HSUper in order for the changes to take effect.
Typically, conda-users define a conda-environment for each project. Such an environment contains a dedicated python interpreter and all the respective libraries. Environments do not interfere with each other (i.e., you can install different versions of the same library to different environments without causing any problems).
As an example, consider a project in which you perform image classification using the PyTorch library. Assume further, that your specific workflow is only compatible with (slightly outdated) Python v3.11. At the start of the project, you then define a new conda-enviroment entitled img_class_project with the correct Python version by pasting the following command into your terminal:
conda create -n "img_class_project" python=3.10.0
After confirming the installation, you can now switch into your newly created enviroment by using
conda activate img_class_project
Note that you can deactivate your currently active enviroment with the command
conda deactivate
Let us again assume the image classification project from above and switch into the newly created environment as described before. Typically, users want to install packages using pip. To install pip into your currently active environment execute the command
conda install pip
You can then install any pip-compatible package, e.g.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Note again, that the respective package will only be installed into your current environment. Other environments will remain unaffected.
Using conda together with the job scheduler SLURM on HSUper is fairly trivial. Yet, users must distinguish between regular jobs (i.e., using jobscripts) and interactive jobs.
Users can use their regular jobscripts as described in the documentation e.g.,
#!/bin/bash
#SBATCH --job-name=conda_tutorial
#SBATCH --partition=small_gpu
#SBATCH --nodes=1
#SBATCH --time 1-00:00:00
#SBATCH --gpus=1
python main.py
SLURM will execute the respective job in the conda environment that is active at time of job submission i.e., when using sbatch.
Unlike with jobscripts, SLURM does not transfer the current conda environment into an interactive job. Users must hence activate the desired environment again after ssh’ing to the allocated interactive node. Consider the following example
salloc --time=10:00 --partition=dev
#...
# e.g. ssh node002, if this node was allocated for the interactive job
conda activate img_class_project
# execute your python commands e.g., python main.py