1.6. Miniconda: Python(s) in a convenient setup

1.6.1. Overview

This page is neither a recommendation to use Conda, nor is it a full tutorial on using it. It is just meant to be a helpful starting point to Conda, if it sounds appealing to you. Examples here are just for managing a few Python needs for AFNI.

Python is increasingly used in the imaging world. Everyone and their sheepadoodle seems to be writing in it and coming across it in some part of their daily work. Part of its appeal is that there exist many useful modules for carrying out tasks (plotting, machine learning, statistical modeling, writing in XKCD font, etc.).

Along with the flexibility and broad appeal of Python comes a need for managing and maintaining dependencies: making sure the necessary dependencies are installed, adding/updating modules, and using the requisite version of Python. For example, there were major changes between Python 2 and 3 (just try evaluating 1/2 in each case), and occasionally across sub-versions code can break; we can only wait and see what will happen with Python 4.

Anyway, managing potentially multiple versions of Python, as well as diferent sets of dependencies can be tricky. There are different ways of doing this, but through personal experience I have found using “Conda” to be (by far) the easiest and most straightforward. We can basically set up one or more environments (or envs) with specific Python versions, modules and dependencies, and we can swap back and forth between them fairly easily—they are each separate and distinct, and don’t get in each other’s way. We can set one to be “on” by default, not really think about it often, and then just change to another environment if we need to. Nice.

We will use the specific installation form of Conda called “Miniconda”, because it starts of being light-weight, and you can then add whatever you need. Some official reference pages for this are:
https://docs.conda.io/projects/conda/en/latest/user-guide/index.html
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

These notes apply to either Linux or Mac OS. Miniconda can also be installed on Windows directly, but since AFNI doesn’t run there, who cares?

These notes are only for setting up Conda for a single user. Instructions for installing Conda for multiple users is described on the conda website here.

Very important note: you should not need your sudo password to install Conda on your system, nor to run later “conda” commands. Using sudo with conda will just cause headaches later, so don’t do it.

1.6.2. Set up Conda (verbose)

Download+install miniconda

  1. To download+install, copy+paste:


    • ... for Linux:

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b

    • ... for Mac (intel chip):

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b

    • ... for Mac (M1 chip):

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh -b

    The curl program should be installed on your computer after following AFNI setup instructions. You can also copy+paste the link into a browser and download it that way.

    NB: The -b option runs the installer in “batch mode”, saying yes to all prompts: install into home directory, accept licenses. To go through those options manually, don’t include that option.


  2. To initialize conda in shells, copy+paste:

    ~/miniconda3/bin/conda init bash tcsh zsh

    This puts conda setup text into each of the above shell’s ~/.*rc files (the relative path to conda must be given when using the batch mode installation in the previous step).

    NB: Conda/miniconda gets setup with path names hardwired into its files, so you cannot just move your “miniconda3” directory and update these path locations later and still have it work.


  3. Make updates known to terminal

    Do one of the following:

    1. Open a new terminal. You should now see a text string like “(base)” to the left of your terminal prompt. (Below, we show you can optionally disable that text.)


    2. Source your shell’s ~/.*rc file (to know your current shell, type echo $0):


      • ... for bash:

        source ~/.bashrc

      • ... for tcsh or csh:

        source ~/.cshrc

      • ... for zsh:

        source ~/.zshrc

    You should see a string “(base)” string stuck before your terminal prompt now. (Below, you can optionally disable that text.)

    Type conda -V to see the version number. NB: It should be at least 4.6.

Update conda version (opt)

When you run some Conda commands, you might receive promptings to update your Conda version. Sometimes explicit commands for copy+pasting are even provided. Go ahead and do so. The following command can also be used to update your Conda (or to check that you are update, at least):

conda update -n base -c defaults conda

To check your current Conda version, type:

conda --version

Load/exit basic conda environments

To see the list of currently available conda environments, type:

conda env list

The name of each available environment appears in the first column (its file location appears in the second). The currently active environment has an asterisk * after its name (one might not be loaded).

To load or “activate” an environment in that list called ENV_NAME, type:

conda activate ENV_NAME

To exit or “deactivate” the current environment, type:

conda deactivate

Make new envs from cmd line: AFNI minimal Python

There are many aspects to creating a new environment. We only provide the most basic here. For example, conda can manage much more complicated environments, beyond loading just Python+modules.

Here is an example of creating a new environment from a command line, one that packages Python version 3.9 and a few useful modules (whose unspecified version numbers will be whatever conda decides, with Matplotlib being at least 2.2.3):

conda create -y                        \
      -n py39_afni_tiny                \
      python=3.9                       \
      "matplotlib>=2.2.3" numpy scipy  \
      flask flask-cors

This new environment’s name is “py39_afni_tiny”; I called it this because that is basically the minimal set of modules used within AFNI (at present).

Now, if I type conda list env, I will see a list of all my available environments (where ${USER} would actually be replaced by my username):

# conda environments:
#
base                  *  /home/${USER}/miniconda3
py39_afni_tiny           /home/${USER}/miniconda3/envs/py39_afni_tiny

As noted above, to switch to py39_afni_tiny, I would type:

conda activate py39_afni_tiny

To see what modules are installed in your active environment (and their version numbers) you can run:

conda list

... which, in the current “py39_afni_tiny” would be as follows (and you might have slightly different things):

- show list output y/n -

So, in this environment, I could run a program that imports matplotlib, whereas in the “base” environment, I couldn’t.

Make new envs from text file: AFNI minimal Python

This is the command line style to create a new environment (perhaps preferable to command line style, for easier saving and sharing), including both specific and minimal package dependency versions:

  • Make a new text file called environment_ex1.yml:

    name: py3_afni_tiny           # TO MAKE:  conda env create -f env*_ex1.yml
dependencies:
  - python
  - "matplotlib>=2.2.3"       # minimal ver for 'pythonic' APQC HTML
  - numpy
  - scipy
  - "flask>=2.1.2"            # for local server for APQC HTML 
  - "flask-cors>=3.0.10"      # for local server for APQC HTML

    ... and then run:

    conda env create -f environment_ex1.yml

  • (Bonus, because I like IPython, and jupyter-notebooks are common) Make a new text file called environment_ex3.yml:

    name: py3_afni_mid           # TO MAKE:  conda env create -f env*_ex3.yml
dependencies:
  - python
  - "matplotlib>=2.2.3"      # minimal ver for 'pythonic' APQC HTML
  - pillow
  - numpy
  - scipy
  - "flask>=2.1.2"           # for local server for APQC HTML 
  - "flask-cors>=3.0.10"     # for local server for APQC HTML 
  - "ipython>=8.0"           # minimal ver for tab-autocomplete to work
  - jupyter
  - prompt_toolkit=3.0.*     # >=2 for jupyter to autocomplete; >=3 for ipython

    ... and then run:

    conda env create -f environment_ex3.yml

Add to+update conda environments

To add a new package or module NEW_PACK to an existing environment ENV_NAME, one can use the following syntax:

conda install -n ENV_NAME NEW_PACK

... so, for example example, you could add the scipy module to one of the above environments with:

conda install -n py39_afni_tiny scipy

To update a module or package CURR_PACK in a currently active environment, you can use:

conda update CURR_PACK

... for example,:

conda update matplotlib

So, let’s say you want one primary environment on your OS to have all your packages of interest loaded, so you don’t have to hop between environments when using different programs. You could make one that has everything you know you need loaded now, and then in the future you could simply keep adding to it. This might be useful with AFNI, in particular, because there are so few requirements here (modern Python with a very small number of modules).

Specify default environment for the terminal

By default, conda will load the “base” environment in any new terminal. To instead have a different environment ENV_NAME loaded in each new terminal/shell, we can add the line conda activate ENV_NAME in the shell’s ~/.*rc file somewhere after the # >>> conda initialize >>> lines.

Since I am running “bash” shell, I have added the following line in my ~/.bashrc file (by opening that file with a text editor):

conda activate py39_afni_tiny

After sourcing that file or opening a new terminal, conda env list should show that environment loaded, in this and in any new terminals. If that did not work, please check that that the conda version is at least 4.6 (via conda -V).

If you do choose to automatically activate your own env like this, then you might also want to run this in a terminal:

conda config --set auto_activate_base false

so that conda doesn’t pre-load the “base” environment unnecessarily (taking a bit of time).

Disable conda prompt string (opt)

Personally I don’t like having the name of the conda environment always appearing before my prompt, like “(base)” or whatever. To not display that text, you can run:

conda config --set changeps1 False

To make your existing terminal recognize this change, source your shell’s ~/.*rc file, e.g. source ~/.bashrc or source ~/.cshrc. Or open a new terminal.

If in the future you want to re-enable this behavior, then you can always run:

conda config --set changeps1 True

These commands edit a text file called ~/.condarc. You can open it and see what defaults/settings you have made, if you wish.

1.6.3. Set up Conda (quick)

  1. Download and install


    • ... for Linux:

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b

    • ... for Mac (intel chip):

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b

    • ... for Mac (M1 chip):

      curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh -b

  2. Initialize conda in shells

    ~/miniconda3/bin/conda init bash tcsh zsh

  3. Make updates known to terminal


    Open a new terminal, or source your shell’s ~/.*rc file.

  4. Remove annoying prompt string (opt)

    conda config --set changeps1 False

  5. Update conda version (opt)

    conda update -n base -c defaults conda

    ... and to display conda version:

    conda --version

  6. Make new envs from cmd line: AFNI minimal Python

    This is the command line style to create a new environment, including both specific and minimal package dependency versions:

    conda create -y                        \
      -n py39_afni_tiny                \
      python=3.9                       \
      "matplotlib>=2.2.3" numpy scipy  \
      flask flask-cors

    See the next section for a slightly better way.


  7. Make new envs from text file: AFNI minimal Python

    This is the command line style to create a new environment (perhaps preferable to command line style, for easier saving and sharing), including both specific and minimal package dependency versions:

    • Make a new text file called environment_ex1.yml:

      name: py3_afni_tiny           # TO MAKE:  conda env create -f env*_ex1.yml
dependencies:
  - python
  - "matplotlib>=2.2.3"       # minimal ver for 'pythonic' APQC HTML
  - numpy
  - scipy
  - "flask>=2.1.2"            # for local server for APQC HTML 
  - "flask-cors>=3.0.10"      # for local server for APQC HTML

      ... and then run:

      conda env create -f environment_ex1.yml

    • (Bonus, because I like IPython, and jupyter-notebooks are common) Make a new text file called environment_ex3.yml:

      name: py3_afni_mid           # TO MAKE:  conda env create -f env*_ex3.yml
dependencies:
  - python
  - "matplotlib>=2.2.3"      # minimal ver for 'pythonic' APQC HTML
  - pillow
  - numpy
  - scipy
  - "flask>=2.1.2"           # for local server for APQC HTML 
  - "flask-cors>=3.0.10"     # for local server for APQC HTML 
  - "ipython>=8.0"           # minimal ver for tab-autocomplete to work
  - jupyter
  - prompt_toolkit=3.0.*     # >=2 for jupyter to autocomplete; >=3 for ipython

      ... and then run:

      conda env create -f environment_ex3.yml

  8. Load an existing environment

    Copy+paste:

    conda activate ENV_NAME

    For example, from above to setup for AFNI:

    conda activate py39_afni_tiny

  9. Activate an env by default


    To activate some env ENV_NAME by default, put conda activate ENV_NAME in your shell’s ~/.*rc file, after the # <<< conda initialize <<< line.
    For example, to set up for AFNI, put conda activate py39_afni_tiny there.

    NB1: This assumes your conda version (conda -V) is at least 4.6.

    NB2: If you do automatically activate your own env, then also copy+paste the following to not pre-load the “base” env (adding unnecessary time):

    conda config --set auto_activate_base false
    NB3: In general, you don’t want to keep appending different conda activate ... commands in a ~/.*rc file, as each one takes a bit of time.

  10. Add to an existing environment

    Once you have built an environment, if you decide you another package that you might have forgotten, you can do so with:

    conda install -n ENV_NAME PACK_NAME

    For example,

    conda install -n py39_afni_tiny pandas

1.6.4. Short list of conda commands

List available modules (starred/asterisked one is active):

conda env list

Deactivate current module:

conda deactivate

Activate/switch to a specific environment/module:

conda activate ENV_NAME

See module+version list in current env:

conda list

Update a package in the current environment:

conda update PACKAGE

Add a package to some environment:

conda install -n ENV_NAME PACK_NAME

Add a channel to an active environment (with top priority among channels):

conda config --add channels NEW_CHANNEL

Add a channel to an active environment (with bottom priority among channels):

conda config --append channels NEW_CHANNEL

Remove an existing environment (ENV_NAME cannot be active when this command is run):

conda remove --name ENV_NAME --all

Update conda program version:

conda update -n base -c defaults conda

1.6.5. A note on making envs, re. AFNI and more

It is entirely up to you, Dear User, what modules you install and how you organize your environments (and if you even choose to use Conda). At the moment AFNI has very minimal Python requirements. In fact, the AFNI set of recommended modules might simply fit inside those requirements that you have for other software/uses, and you might not need to do anything new.

We certainly don’t anticipate or desire a person to set up one specific environment for running AFNI, then another for running some other software, and then another for another project... While that is possible, it seems annoying and inefficient, and often unnecessary. So, hopefully, you can set up one environment (or a small number of them) and not have to switch too much.

1.6.6. Fancier things with Conda

There are a lot of fancy things that can be done with Conda that we will not describe here. A good starting point is the Managing Environments documentation.

Cloning

One concept with Conda is cloning environments: if I can setup a Conda environment on my laptop with a certain set of modules, each with a certain version number, then I can “clone” it and use that exact recipe to setup a duplicate environment on a different computer. This is a nice concept for reproducibility (as sometimes using different version numbers of modules can affect outputs/results).

More on cloning and building identical conda envs can be read.

Note that in practice, truly duplicating environments exactly is actually pretty tough. Getting very close might be good enough for most purposes, though, in practice.

View Conda Cheatsheet

It’s here: the conda cheatsheet.

Make conda environments, more generally

The environment builder works a bit like a package manager, where it can get a lot of common modules from a default, central repository, but if you want more specialized ones, you might have to add from another place. To add new repositories to pull from, you add a channel to your Conda setup.

Let’s say you want to add the Sphinx module with cloud-theme support (I doubt you will, but just as an example). If you try:

conda create -y                 \
    -n py39_afni_with_sph       \
    python=3.9                  \
    matplotlib numpy scipy      \
    flask flask-cors            \
    sphinx cloud_sptheme

You will likely get the following message:

- show text output y/n -
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - cloud_sptheme

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

This message: 1) tells us our current channels don’t contain this module; 2) shows us our current channels; and 3) helpfully directs us to a webpage to search for a new channel that might have it.

So, searching for “cloud_sptheme” at https://anaconda.org/, one of the top package owners appears to be “conda-forge” (and this is a fairly large platform). So, to add it to my repository list for getting modules, I would run:

conda config --add channels conda-forge

Then, I can retry my conda create .. command above, which should result in success this time.

Thus, if you try to build an environment and get told that some desired module can’t be found, you can search for it amongst available channels, add that channel to your Conda setup, and try again.

1.6.7. Comments on AFNI-specific conda environments

Conda can be used to create environments with a very wide range of dependencies. For the purposes of interacting with AFNI, we tend to just focus on Python modules, since all other dependencies should essentially be sorted out by following the main installation instructions for a given OS.

AFNI has very few Python module dependencies. Most AFNI Python programs in fact have no non-base module dependencies. However, if you are going to use conda/miniconda to set up a Python environment anyways, you might as well install any possible modules, too. Moreover, some programs you might want to use in conjunction with AFNI might have additional dependencies, which you could add to your AFNI-running environment, too.

Every Python program in the AFNI distribution runs in either Python 2.7 or Python 3.* except for the distributed version of Prantik Kundu’s multi-echo FMRI-processing program meica.py, which must be run in Python 2.7 (because of when it was written and its developer has left the field).

Some comments then about possible AFNI environments to make, or dependencies to combine with your other

  • for most projects with AFNI, you could use the py39_afni_tiny environments, described above, on its own or with its dependencies added to a pre-existing environment. NB: there is no strict Python 3.9 requirement for AFNI—it was just used in the example; one should be fine using Python 3.7 or higher. Also, Python 2.7 is technically deprecated now (though most AFNI program run fine in it still, having 3.* makes a lot more sense).

  • If you will be using Prantik’s older meica.py program, then you would want a Python 2.7-based environment available. It could be used as your main environment, or it could just be one you switch in-and-out of for running that specific program. (Being able to easily switch environments is basically the raison-d’etre of conda.) However, for most application purposes (beyond methodological comparison) we would perhaps recommend user’s run the newer tedana group’s version of MEICA (see below).

  • If you want to incorporate the modern TEDANA (Dupre et al., 2021) into your AFNI processing of ME-FMRI data, then here is an example of an environment incorporating both sets of dependencies. You could make a new text file called environment_afni_ted.yml:

    name: py3_afni_ted            # conda env create -f env*afni_ted.yml
channels:
  - conda-forge
  - defaults
dependencies:
  - python
  - "matplotlib>=2.2.3"       # minimal ver for 'pythonic' APQC HTML
  - numpy
  - scipy
  - "flask>=2.1.2"            # for local server for APQC HTML 
  - "flask-cors>=3.0.10"      # for local server for APQC HTML 
  - pip                       # used for TEDANA
  - pip:
    - tedana                  # getting TEDANA

    ... and then run:

    conda env create -f environment_afni_ted.yml

    NB: these reflect current TEDANA installation instructions, which might change over time. We will try to keep up with those.