🪴 Anil's Garden

Conda

12 Dec 20247 min read

Conda

Recipes and snippets for Conda, the open source package and environment management system widely used with Python, based on a tour of Conda from the command line.

Getting Miniconda

Just in case you ever need a convenience script to download Miniconda, in this case for Linux…

download_miniconda.sh

Resources

Common Issues

Using Conda inside a shell script

TLDR; insert eval "$(conda shell.bash hook)" into the header of your (Bash) script following option (3). Option (1) is good too but requires the user (person running the script) to correctly use source or . where (1) works when the user executes the script via bash <my_script.sh>.

The conda activate command may not work inside a shell script due to the way shell scripts handle environment variables. When you run a shell script, it spawns a new shell, and any changes to the environment (like activating a conda environment) are local to that shell and do not affect the parent shell or any other shells.

There are a few ways to make conda activate work inside a shell script:

  1. Use source or . before conda activate: This is the most common way to make conda activate work in a shell script. The source command or . (dot) operator tells the shell to read and execute commands from the file in the current shell environment, rather than spawning a new shell. Here’s an example:

    source /home/username/miniconda3/bin/activate myenv

    Replace /home/username/miniconda3/bin/activate with the path to your conda’s activate script, and replace myenv with the name of your conda environment Source 4.

  2. Use conda run: The conda run command allows you to run a command in a specific conda environment. It’s not exactly the same as activating the environment, but it can be used to run a command in a specific environment without having to activate the environment first:

    conda run -n myenv python myscript.py

    Replace myenv with the name of your conda environment, and replace myscript.py with the command you want to run Source 5.

  3. Use eval with conda shell.bash hook: The conda shell.bash hook command generates a script that sets up the conda environment for the bash shell. The eval command can execute this script in the current shell:

    eval "$(conda shell.bash hook)"
conda activate myenv

    Replace myenv with the name of your conda environment Source 22.

Remember that the conda activate command only works in conda version 4.4 and later. If you’re using an older version of conda, you might need to use source activate on Linux and macOS, or activate on Windows instead Source 0.

Common Conda Commands

You can find out what Conda - in my case Miniconda - is capable of by executing conda --help at the command line. You’ll get a list of commands and we’ll take them as they come. I frequently use the long-form of arguments here to make things more self-evident.

clean - Remove unused packages and caches

This is a useful command which will free space on the filesystem by deleting packages or the cached artifacts of packages (like tarballs) that are left over after their installation.

For example, if I run conda clean --dry-run --tarballs on my machine, I see the following output

Cache location: /home/anil/miniconda3/pkgs
Will remove the following tarballs:

/home/anil/miniconda3/pkgs
--------------------------
ca-certificates-2022.3.29-h06a4308_1.conda     122 KB
sqlite-3.39.2-h4ff8645_0.tar.bz2             1.5 MB
numpy-1.23.1-py38h6c91a56_0.conda             11 KB
xz-5.2.5-h516909a_1.tar.bz2                  343 KB
blas-1.0-mkl.tar.bz2                           1 KB
zstd-1.5.2-h6239696_4.tar.bz2                448 KB
pytorch-mutex-1.0-cuda.tar.bz2                 3 KB
torchvision-0.12.0-py38_cu113.tar.bz2       27.7 MB
libstdcxx-ng-9.3.0-hd4cf53a_17.conda         3.1 MB
cffi-1.15.1-py38h4a40e3a_0.tar.bz2           229 KB
_libgcc_mutex-0.1-conda_forge.tar.bz2          3 KB
_libgcc_mutex-0.1-main.conda                   3 KB
libwebp-base-1.2.3-h166bdaf_2.tar.bz2        392 KB
...
---------------------------------------------------
Total:                                      2.35 GB

DryRunExit: Dry run. Exiting.

A whopping 2.35 GB would be freed up if I deleted the tarballs that I no longer need for function of the Python packages in my environments.

The output from conda clean --dry-run --all is identical.

Instead, if I check what can be cleaned up from package caches, via the command conda clean --dry-run --packages I see that 164.8 MB are free-able via removal of unused packages from writable package caches:

WARNING: /home/anil/.conda/pkgs does not exist
Cache location: /home/anil/miniconda3/pkgs
Will remove the following packages:
/home/anil/miniconda3/pkgs
--------------------------

libcblas-3.9.0-12_linux64_mkl                 42 KB
giflib-5.2.1-h36c2ea0_2                      312 KB
libblas-3.9.0-12_linux64_mkl                  43 KB
numpy-1.23.1-py38h6c91a56_0                   29 KB
openjpeg-2.4.0-hb52868f_1                    1.5 MB
libzlib-1.2.12-h166bdaf_2                    127 KB
_openmp_mutex-4.5-1_gnu                       93 KB
pyopenssl-22.0.0-pyhd8ed1ab_0                246 KB
numpy-base-1.23.1-py38ha15fc14_0            29.4 MB
...
numpy-1.23.3-py38h3a7f9d9_0                 29.4 MB
freetype-2.10.4-hca18f0e_2                   2.8 MB

---------------------------------------------------
Total:                                     164.8 MB

Finally, I have no index cache to free up as noted by the fact that conda clean --dry-run --index-cache returns nothing.

compare - Compare packages between conda environments

The usage of conda compare is

conda compare [-h] [--json] [-v] [-q] [-n ENVIRONMENT | -p PATH] file

Note that this takes the required positional argument file, which is the path to the environment file that is to be compared against. This constrains you to comparing a target environment passed either by --name or --prefix against an environment YAML file.

Comparing Two Environments Directly

What if you want to compare if two environments on your machine are identical? You have to export one of these environments to a YAML and compare the other against it.

Let’s assume we have two environments bleep and blorp which we have created and installed a few things into…

We have the following in bleep

# requirement_bleep.txtlibrosasoundfilesoxkaldiioscipyg2p-enpydubmatplotlib
conda create --yes --name bleep python=3.8 &&    conda activate bleep &&    pip install --requirement requirements_bleep.txt

…and we have the following in blorp

# requirement_blorp.txt
librosa
soundfile
sox
kaldiio
scipy
g2p-en
pydub
matplotlib

# these weren't in bleep
numpy
black==19.10b0
wandb

conda create --yes --name blorp python=3.8 &&    conda activate blorp &&    pip install --requirement requirements_blorp.txt

Now let’s say we want to compare the two against each other. We can use a small snippet like the following which

  • activates the base conda environment (just so we get used to using the explicit references to environments via names or prefixes)
  • makes ./temp/
  • exports the blorp conda environment to ./temp/blorp.yaml
  • compares the target conda environment bleep to blorp via ./temp/blorp.yaml
  • deletes ./temp/blorp.yaml
  • deletes ./temp if it is empty (or else tells you what else has been left lying around in there)
conda activate basemkdir -p ./tempconda env export --name blorp --file ./temp/blorp.ymlconda compare --name bleep ./temp/blorp.ymlrm ./temp/blorp.ymlrm -r ./temp/

The output we get from running conda compare --name bleep ./temp/blorp.yml is

attrs not found
black not found
docker-pycreds not found
gitdb not found
gitpython not found
pathspec not found
pathtools not found
promise not found
protobuf not found
psutil not found
pyyaml not found
sentry-sdk not found
setproctitle not found
shortuuid not found
smmap not found
toml not found
typed-ast not found
wandb not found

If we were to run it the other way round, comparing an export of bleep to blorp as the target environment via the following sequence of commands

conda activate basemkdir -p ./tempconda env export --name bleep --file ./temp/bleep.ymlconda compare --name blorp ./temp/bleep.ymlrm ./temp/bleep.ymlrm -r ./temp/

We get the output

Success. All the packages in the specification file are present in the environment with matching version and build string.

Be aware that conda will return success if all the packages in the spec. file are present in the target environment with the matching version and build string, including if there are additional packages in the target environment, as is clearly the case in the above example. You can see this another way by running conda activate bleep && pip list | wc -l which returns 59 against conda activate blorp && pip list | wc -l which returns 77.

If you get the rather scary warning that pip’s dependency resolver does not currently take into account all the packages that are installed check out What does the error message about pip –use-feature=2020-resolver mean?.

install - Installs a list of packages into a specified conda environment

Note the option for using copies when installing packages instead of hard- or soft-linking.

Package Linking and Install-time Options:
  --copy                Install all packages using copies instead of hard- or soft-linking.
  -m, --mkdir           Create the environment directory if necessary.
  --clobber             Allow clobbering of overlapping file paths within packages, and suppress related warnings.

Graph View

Backlinks

  • No backlinks found