In this post I describe my strategy for managing source code projects and dependencies with python, Conda, Pip, etc. I also show how to to maintain your own "PyPI" for source code packages.

As a general strategy, I am now trying to host everything on Anaconda Cloud so that all my packages and environments can be installed with Conda.

Overview

Within the Python community, the Python Packaging Authority (PyPA) provides the recommendations of the current working group, and we try to follow their guidelines as laid out in the Python Packaging User Guide.

My general strategy is to maintain Conda environments for each project and each main application, and to rely on these to ensure my projects work and can be easily distributed. In addition, I maintain a common work environment that has everything and the kitchen sink which I can "just use" to get work done. This is hosted on Anaconda Cloud so you can install it and use it with:

conda install anaconda-client
conda env create mforbes/work    # Don't run in a directory with an environment.yml file
conda activate work

Note: do not run this in a folder with an environment.yml file. See Issue #549.

This notebook describes some details about this strategy which consists of the following concepts:

• Jupyter: This section discusses the jupyter environment where I install Jupyter and related tools.

  • I use a script to launch the Jupyter notebook server and/or Jupyter Lab so that this environment is first activated.

  • I use the nb_conda extension so that one can access the conda environments from Jupyter. (This requires ensuring that the ipykernel package is installed in each environment so that they can be found.)

  • I use the Jupytext extension so that you can store notebooks as python files for better version control. This requires a small manual change in your notebooks to enable.

• Conda Environments: I try to package everything into Conda environments. There are several use-cases here:

  • Global environments such as work2 and work3 which have everything including the kitchen sink for python 2 and python 3 respectively. When I just need to get something done, I activate these and work. Don't rely on these for testing though: instead use minimal and dedicated environments for your code.

  • Dedicated and minimal project environments should be used for each project that contain the minimal number of packages required. Your tests should be run in these environments (plus any testing tools) so that you can be sure no undocumented dependencies creep in when you add new features. I usually keep two copies:

    • environment.yml: This contains the packages I need (leaves), usually without version specifications unless there is a known conflict. It can be kept clean as descibed in the section Visualizing Dependencies. To create or update the environment:

      Copy

      conda env update -f environment.yml
      

    • environment.frozen.yml: This contains the explicit versions I am testing against to ensure reproducible computations. It can be formed by:

      Copy

      conda activate <environment name>
      conda env export > environment.frozen.yml
      

• Anaconda Cloud: I try to use my mforbes channel on Anaconda Cloud to host useful environments and packages. Ideally, all packages should be installable with only Conda. Some information about how to build and maintain Conda packages and environments is discussed in section Building a Conda Package.

TL;DR

Conda

• Use conda to manage environments. (Conda can be slow: for faster installs you can try mamba, but it is still experimental.)

• For each project, create an environment which will contain all the packages like NumPy, SciPy etc. needed for that project. (Also specify python 2 or python 3). Store these requirements in a project-specific environment.yml file.

• Create an environment for that project that includes the ipykernel package:

  Copy

  conda env create --file environment.yml
  

  (This assumes that environment.yml specifies the name of the environment, otherwise you need to pass conda the -n <env> flag).

• Install Jupyter (see instructions below) in its own environment and include the nb_conda package. By including the ipykernel package in each environment and the nb_conda package in the jupyter environment, Jupyter will be able to find your environments and use them as kernels.

• Launch Jupyter with a custom kernel for your project using the following function which will activate the jupyter environment first:

Pip

The following recommendations are based on using pip to manage dependencies. We are in the process of migrating to a more conda-centric version.

• Provide a setup.py file for your source repositories that contains all of the dependency information in the install_requires argument to setup().

• Structure your repository with tags and/or named branches so that you can hg update 0.9 etc. to update to the various versions.

• Host an index.html file somewhere that points to the Download links of the various projects.

• Use pip install --extra-index-url <index.html> to allow pip to resolve dependencies against your source code.

Conda

Conda and Anaconda provide quite complete python distributions for scientific computing. Unless I am specifically developing packages for general us, I allow myself to rely on the default Anaconda distribution. This includes the Conda package manager, which provides an alternative to virtualenv for managing environments.

There is a nice discussion of the state of conda here:

• http://penandpants.com/2014/10/09/state-of-conda-oct-2014/

• Conda documentation

Conda has a few important benefits:

• Many binary packages are supported, including some that are not python (and hence cannot be managed by pip).

• Conda including builds of NumPy, SciPy, and Matplotlib that can be difficult to install from source.

• Conda keeps track of your history (try conda list --revisions) which can help if you need to reproduce some history.

• For educational use (or paid commercial use), one can install the Accelerate package.

• Conda is aware of packages installed with pip so these can still be used.

• Conda provides environments, replacing the need for virtualenv.

Cheatsheet

Working with packages:

• conda update --all: Update all packages in an environment.

• conda clean --all: Remove all downloaded packages and unused files. Can free significant amounts of disk space.

Working with environments:

• conda activate <env>/conda deactivate: Activate or deactivate an environment.

• conda env list: Show installed environments (and their location).

• conda env update [ -n name ] ( -f environment.yml | channel/spec ) [ --prune ]: Update packages in an environment from a file or anaconda channel, optionally removing unnecessary files (but see this issue.)

Package Availablity

One issue when dealing with conda is that not all packages are available. There several ways to deal with this:

1. Search for the package on another channel. The following channels seem to be reliable:

   • conda-forge: This is a community driven project that provides an infrastructure for building and maintaining packages cleanly on Windows, Linux, and OS X. It is described in the post - Community powered conda packaging: conda-forge.

   • pipy To add a new channel:

   Copy

   conda config --append channels conda-forge
   

   Then you can look with

   Copy

$ conda config --show add_anaconda_token: True add_pip_as_python_dependency: True allow_softlinks: True always_copy: False always_yes: False auto_update_conda: True binstar_upload: None changeps1: True channel_alias: https://conda.anaconda.org/ channel_priority: True channels:

• defaults

• conda-forge client_cert: client_cert_key: create_default_packages: [] debug: False default_channels:

• https://repo.continuum.io/pkgs/free

• https://repo.continuum.io/pkgs/pro disallow: [] json: False offline: False proxy_servers: {} quiet: False shortcuts: True show_channel_urls: True ssl_verify: True track_features: [] update_dependencies: True use_pip: True verbosity: 0

2. Build your own conda package and upload it to [Anaconda Cloud](https://anaconda.org).  It seems that contributing to [conda-forge] is the preferred way to do this but I have not yet tried.
3. Use `pip`.  The disadvantage here is that packages are then managed by both `conda` and `pip` which can lead to some confusion, but this does work reasonably well and is how I used to do things for many years.  The advantage is that you can specify your dependences simply with the standard `setup.py` file.  I am not sure how to do this with `conda` yet.

[conda-forge]: https://conda-forge.github.io

Getting Started

My strategy is to install a minimal conda installation with python 2.0 and then add various environments as needed. Each project should have its own conda environment so that one can reproducibly manage the dependencies of that project. This ties together as follows:

• The conda base environment and some custom environments (jupyter might be a good candidate) can be maintained at a system level by an admin. On my computers, these reside in:

  Copy

  /data/apps/conda/   # System (mini)conda directory
  |-- envs/           # System-defined environments 
  |   |-- jupyter/    # Jupyter notebooks etc. with nb_conda extension
  |   `-- ...
  |-- bin/            # Executables for `base` environment
  `-- ...
  

• The jupyter environment contains the nb_conda extension which allows one to choose kernels based on known conda environments. With this, you can run jupyter from the jupyter environment and use a specialize kernel for whatever computation you need. (For example, jupyter should be run with Python 3, but you may have kernels that are still Python 2. This approach means you do not need to include jupyter in you project environment.)

• On a system where users cannot write to /data/apps/conda, environments created by the user will be automatically placed in:

  Copy

  ~/.conda/envs/
  

  Thus, users can immediately create their own environments as needed.

• If a user needs to create an environment in ~/.conda/envs/ that shadows an environment in /data/apps/conda/envs, then it seems this can be done by first making the appropriate directory, then using conda to install whatever is needed:

  Copy

  $ conda create -n jupyter
  
  CondaValueError: prefix already exists: /data/apps/conda/envs/jupyter
  
  $ mkdir -p ~/.conda/envs/jupyter
  

$ conda install -n jupyter ...

This is probably a bit of a hack, but seems to work.

Miniconda

I first install Miniconda with Python 2.7.

• I choose to use python 2.7 here because it allows me to install Mercurial which will likely not support Python 3 for some time.

• I add the miniconda bin directory to the end of my path so that the hg command will fall through if I have activated a python 3 environment.

• Other than mercurial and related tools, I keep the default environment clean.

Update Conda

After installing miniconda, I do an update:

Historical note: When I originally wrote this article, I obtained the following outputs at various times:

# packages in environment at /data/apps/anaconda:
#
cffi                      1.9.1                    py27_0    defaults
conda                     4.3.7                    py27_0    defaults
conda-env                 2.6.0                         0    defaults
cryptography              1.7.1                    py27_0    defaults
enum34                    1.1.6                    py27_0    defaults
idna                      2.2                      py27_0    defaults
ipaddress                 1.0.18                   py27_0    defaults
openssl                   1.0.2j                        0    defaults
pip                       9.0.1                    py27_1    defaults
pyasn1                    0.1.9                    py27_0    defaults
pycosat                   0.6.1                    py27_1    defaults
pycparser                 2.17                     py27_0    defaults
pycrypto                  2.6.1                    py27_4    defaults
pyopenssl                 16.2.0                   py27_0    defaults
python                    2.7.13                        0    defaults
readline                  6.2                           2    defaults
requests                  2.12.4                   py27_0    defaults
ruamel_yaml               0.11.14                  py27_1    defaults
setuptools                27.2.0                   py27_0    defaults
six                       1.10.0                   py27_0    defaults
sqlite                    3.13.0                        0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py27_0    defaults
yaml                      0.1.6                         0    defaults
zlib                      1.2.8                         3    defaults

# packages in environment at /data/apps/anaconda:
#
conda                     4.2.7                    py27_0    defaults
conda-env                 2.6.0                         0    defaults
enum34                    1.1.6                    py27_0    defaults
openssl                   1.0.2i                        0    defaults
pip                       8.1.2                    py27_0    defaults
pycosat                   0.6.1                    py27_1    defaults
python                    2.7.12                        1    defaults
pyyaml                    3.12                     py27_0    defaults
readline                  6.2                           2    defaults
requests                  2.11.1                   py27_0    defaults
ruamel_yaml               0.11.14                  py27_0    defaults
setuptools                27.2.0                   py27_0    defaults
sqlite                    3.13.0                        0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py27_0    defaults
yaml                      0.1.6                         0    defaults
zlib                      1.2.8                         3    defaults

# packages in environment at /data/apps/anaconda:
#
conda                     3.12.0                   py27_0  
conda-env                 2.1.4                    py27_0  
openssl                   1.0.1k                        1  
pip                       6.1.1                    py27_0  
pycosat                   0.6.1                    py27_0  
python                    2.7.9                         1  
pyyaml                    3.11                     py27_0  
readline                  6.2                           2  
requests                  2.7.0                    py27_0  
setuptools                15.2                     py27_0  
sqlite                    3.8.4.1                       1  
tk                        8.5.18                        0  
yaml                      0.1.4                         1  
zlib                      1.2.8                         0  

One interesting feature shown here is that conda keeps track of your revisions. You can update the revisions with conda install --revision=0 (the original miniconda installation for example) and you can list them with:

Command Completion

Before doing any more work, add command completion so that you can explore options:

Configure Conda

Conda allows you to configure it with the conda config command. Here I add the conda-forge channel.

Environments

To ensure reproducible computing, I highly recommend working from a clean environment for your projects so that you can enter a clean environment with

conda activate _conda_env

To do this, maintain an environment.yml file such as this:

# environment.yml
name: _conda_env
channels:
  - defaults
  - conda-forge
  - ioam
dependencies:
  - ipykernel        # Use with nb_conda_kernels so jupyter sees this kernel
  - numpy
  - scipy>=0.17.1
  - matplotlib>=1.5
  - uncertainties
  - ipywidgets
  - holoviews
  - pip:
    - mmf_setup>=0.1.11

Then create a new environment as follows:

conda env create -p _conda_env -f environment.yml

Periodically you should check if anything is outdated:

conda search -p _conda_env --outdated

and possibly update everything

conda update -p _conda_env --all

If you then make a _conda_env environment in the project and choose this as the kernel for your notebooks, you can then run in an isolated environment. Don't forget to exclude _cond_env from selective sync with Dropbox etc. as it can get big! For this reason, it can be better to install this globally.

Frozen Envrionments

I manually create environment files like the one above which specify the leave nodes I need. To prune these, see the section Visualizing Dependencies:Conda

I keep a few clean environments for use when testing.

If I want to test against a clean environment, I make a clone:

Jupyter

I install Jupyter into it's own environment and then use the following alias to launch it. This allows me to launch it in any environment without having to install it. To use those environments as their own kernel, make sure you include the ipykernel package in the environment and then the nb_conda package in the jupyter environment.

conda env create --file environment.jupyter.yml

Sometimes I have difficulty getting extensions working. The following may help:

jupyter serverextension enable --sys-prefix jupyter_nbextensions_configurator nb_conda
jupyter nbextension enable --sys-prefix toc2/main init_cell/main
jupyter serverextension disable --sys-prefix nbpresent
jupyter nbextension disable --sys-prefix nbpresent/js/nbpresent.min

Configuration

To see where Jupyter configuration files go, run

I add the following kernel so I can seamlessly match with CoCalc.

# ~/Library/Jupyter/kernels/python2-ubuntu/kernel.json
{
 "display_name": "Python 2 (Ubuntu, plain)",
 "argv": [
  "/data/apps/anaconda/envs/work/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "language": "python"
}

Anaconda Cloud

Once you have useful environments, you can push them to Anaconda Cloud so that you or others can use them:

anaconda login    # If you have not logged in.
anaconda upload environment.work.yml

Once this is done, you can install the environment with one of:

conda env create mforbes/work   
mamba env create mforbes/work   # Faster, but experimental

Clean

The Conda package cache can help speed the creation of new environments, but can take up quite a bit of disk space. One can reclaim this with conda clean:

Basic Packages and Configuration

Here is a list of basic packages that I always use. I install these in two custom environments work2 and work3. I also make a symlink called work which points to the appropriate environment in which I am currently working.

First, however, I install the following in the default conda environment:

Now I add the default path to both the start and end of %PATH with different names in my .bashrc file:

export PATH="/data/apps/anaconda/bin:$PATH:/data/apps/anaconda/bin/."

The reason for the latter addition is:

1. It will be a fallback when activating other environments so I can still use mercurial.

2. By giving it a different name it will not be removed when . deactivate is called.

Here is the process for creating my new work environments including some subtle issues:

• As per issue 280, anaconda's packages for nose and flake8 are broken and do not register the distutils entry point, thus I remove these.

• Many of these tools are for high performance computation.

Activate and Deactivate Scripts

When sourcing the activate and deactivate scripts, files "$CONDA_ROOT/etc/conda/activate.d/*.sh" and "$CONDA_ROOT/etc/conda/deactivate.d/*.sh" will be sourced. This allows one to perform additional configurations such as setting IPYTHONDIR. To find CONDA_ROOT programatically use:

For example, the following two scripts to set and reset (once) IPYTHONDIR:

• /data/apps/anaconda/envs/work/etc/conda/activate.d/set_ipythondir.sh

if [ ! -z \${IPYTHONDIR+x} ]; then
   export OLD_IPYTHONDIR="\$IPYTHONDIR"
fi
export IPYTHONDIR="/data/apps/anaconda/envs/work/.ipython"

• /data/apps/anaconda/envs/work/etc/conda/deactivate.d/unset_ipythondir.sh

if [ -z \${OLD_IPYTHONDIR+x} ]; then
   unset IPYTHONDIR
else
   export IPYTHONDIR="\$OLD_IPYTHONDIR"
   unset OLD_IPYTHONDIR
fi

I make these files with a script like the following which I can run in the appropriate environment to set the .ipython directory in the current folder as the IPYTHONDIR for the environment. Note: It does some checks to make sure that an environment is active.

#!/bin/bash

CONDA_ROOT="$(conda info --root)"
CONDA_ENV="$(conda info -e | grep '*' | cut -d '*' -f2 | sed 's/^ *//g')"
if [ "$CONDA_ROOT" == "$CONDA_ENV" ]; then
  echo "Not in a conda environment.  Activate the environment first. I.e.:"
  echo
  echo "    . activate work"
  echo
  exit 1
else
  echo "Creating activate and deactivate scripts:"
  echo
  echo "   $CONDA_ENV/etc/conda/activate.d/set_ipythondir.sh"
  echo "   $CONDA_ENV/etc/conda/deactivate.d/unset_ipythondir.sh"
  echo
fi

mkdir -p "$CONDA_ENV/etc/conda/activate.d"
mkdir -p "$CONDA_ENV/etc/conda/deactivate.d"
IPYTHONDIR="$(cd .ipython && pwd)"
cat <<EOF > "$CONDA_ENV/etc/conda/activate.d/set_ipythondir.sh"
# This file sets the IPYTHONDIR environmental variable to use the
# settings in $IPYTHONDIR
# It is called when you activate the $CONDA_ENV environment with
#
#   . activate $CONDA_ENV

if [ ! -z \${IPYTHONDIR+x} ]; then
   export OLD_IPYTHONDIR="\$IPYTHONDIR"
fi
echo Setting IPYTHONDIR="$IPYTHONDIR"
export IPYTHONDIR="$IPYTHONDIR"
EOF
cat <<EOF > "$CONDA_ENV/etc/conda/deactivate.d/unset_ipythondir.sh"
# This file resets the IPYTHONDIR environmental to the previous value.
# It is called when you deactivate the $CONDA_ENV environment with
#
#   . deactivate

if [ -z \${OLD_IPYTHONDIR+x} ]; then
   echo Unsetting IPYTHONDIR
   unset IPYTHONDIR
else
   echo Resetting IPYTHONDIR="\$OLD_IPYTHONDIR"
   export IPYTHONDIR="\$OLD_IPYTHONDIR"
   unset OLD_IPYTHONDIR
fi
EOF

. "$CONDA_ENV/etc/conda/activate.d/set_ipythondir.sh"

Development

If you are developing software, the conda-devenv package might help. It allows you to maintain several dependent environment files for example. This might be useful when you want to have separate environment files for testing python 2 and python 3. Useful features include (from the docs):

• Jinja 2 support: gives more flexibility to the environment definition, for example making it simple to conditionally add dependencies based on platform.

• include other environment.devenv.yml files: this allows you to easily work in several dependent projects at the same time, managing a single conda environment with your dependencies.

• Environment variables: you can define a environment: section with environment variables that should be defined when the environment is activated.

Packages and Dependencies

Making sure that you have all of the required packages installed can be a bit of a pain. I have toyed with three different types of strategies outlined below. I currently use method 1. when actively developing, then switch to a version that supports both 2. and 3.

1. Include the source code for projects you need as weak dependencies managed with the myrepos (mr) tool:

   • http://myrepos.branchable.com

   Basically, I include a .mrconfig file which instructs mr how to checkout the required projects:

   Copy

   [_ext/pymmf]
   checkout = hg clone 'https://bitbucket.org/mforbes/pymmf' 'pymmf'
   

   When I run mr checkout, this project will be cloned to _ext/pymmf and I can then symlink the appropriate files to a top-level module ln -s _ext/pymmf/mmf mmf. If you use pymmf in many projects, you can replace _ext/pymmf with a simlink to an appropriate common location (but keep in mind that each project will be using the same revision then.)

   A small set of additional command can give mr freeze which tracks the specific verions. There are some bugs with this, but you can see an example in my mmfhg project, in particular in the mrconfig file.

   This approach is useful if you need to activately develop both the top level project and the subproject, but requires manual intervention to update etc. and is not suitable for general usage since collaborators will need to add myrepos to their toolchain.

2. Write a setup.py file and use this or pip to manage the dependencies.

   This is a better choice for general distribution and collaboration, but still has a few warts. An advantage of this over the next option is that it will then work with PyPI.

3. Write a meta.yaml file and use conda to manage dependencies.

   This is a good choice for general distribution if you want your package to be installable with conda from Anaconda Cloud.

Setuptools and Pip

The Python Packaging User Guide is supposed to provide the definitive guide for managing python packages. Its current recommendations for installing packages is to use pip, so we would like to maintain that strategy. The basic idea here is to include a setup.py file in your top level directory that specifies your dependencies (and example will be provided below). This is designed for and works well will packages published on PyPI but starts failing when you need to depend on projects that are not published (i.e. those hosted on github or bitbucket.)

Setup(.py) and Requirements(.txt)

Some of the issues are discussed in this rant but are centered around the issue of how to specify the location of code not available on PyPI. Both pip and setuptools support installation from version controlled sources. The recommended approach for this (see Setup vs. Requirements is to put these dependencies in a requirements.txt file, but there are problems with this approach:

1. Users must run pip install -r requirements.txt explicitly rather than just pip install -e . or python setup.py develop.

   This is probably not such a big deal as long as the install instructions are clear.

2. If you require a subproject that also has requirements.txt, there is no easy way of recursively parsing the subprojects requirements. (The requirements specified in the subprojects setup.py will get processed.

   The only solution I know to this second issue is to make sure your requirements.txt file is very complete, specifying all possible requirements recursively.

3. Suppose that you use branches or tags in your code to mark versions. Now suppose that one project A that depends on another project B, and that B depends on a specific version of C==0.9. You dutifully include this in the setup.py files:

   Copy

   # A's setup.py
   install_requires = ["B"]
   

   Copy

   # B's setup.py
   install_requires = ["C==0.9"]
   

   However, at this point, neither pip nor setuptools can install A since neither B nor C are installed on PyPI. The recommended solution is to specify all of the dependences in A's requirements.txt file:

   Copy

   -e hg+ssh://hg@bitbucket.org/mforbes/B#egg=B
   -e hg+ssh://hg@bitbucket.org/mforbes/C#egg=C
   -e .
   

   Installing A with pip install -r requirements.txt will now install B and C, but the requirements are now broken. The package C for example will be installed at the latest version, even if this breaks the C==0.9 requirement by B. This appears to be broken even if we provide the explicit versions in the requirements.txt file:

   Copy

   -e hg+ssh://hg@bitbucket.org/mforbes/B#egg=B
   -e hg+ssh://hg@bitbucket.org/mforbes/C@0.8#egg=C-0.8
   -e .
   

   Thus we are left in the awkward position of having to make sure in project A that we pin the exact requirements we need in subproject C even though we do not (directly) depend on it.

The problem described above is consistent with the following roles of setup.py and requirements.txt:

• The install_requires argument to setup() in setup.py specifies what is required and version information should be used to specify what does not work (i.e. avoid this version).

• The requirements.txt specifies exactly one situation that does work.

In case of conflict, the second point here seems to override the first one (C-0.8 above gets installed even though the setup for B specifies C==0.9).

Thus, requirements.txt is completely useless for specifying dependencies in a general way. The only solution here appears to be to carefully construct a requirements.txt file whenever a project or its dependencies change. The following tools may help with this, but I am not happy with this solution:

• pip freeze > requirements.txt

• http://furius.ca/snakefood/

A Better Solution: Host your own Index

Can we get the benefits of the usual dependency resolution while using version controlled software? To do this, it seems that you need to host your packages through an index via one of the following approaches:

• Upload your package to PyPI: This works, but requires a lot of discipline and maintenance to make sure your package is complete, tested, working, and properly described whenever you need to freeze it. The whole release process can be a pain, and submitting a bunch or random tools to PyPI is considered bad practice. (For one thing, there is a global namespace, so you might steal a useful name for a bunch of specialized tools where another package might have been able to better use it for the community.)

• Host your own Package Index.

The latter approach seems at first to require running a web-server somewhere which is off-putting, but actually supports two quite reasonable solutions. The first is to package tarballs of your projects into a single directory with canonical names like C-0.8.tar.gz, and C-0.9.tar.gz and then point to this with the pip --find-links <dir> option. (The tarballs can be produced with python setup.py sdist and will get deposited in the dist directory which you could symlink to the <dir> specified above.)

The second solution is to provide a proper package index somewhere. This allows you to refer to additional locations, and in particular, allows you to refer to something like

<a href="https://bitbucket.org/mforbes/persist/get/0.9.tar.bz2#egg=persist-0.9">persist-0.9</a>.

This uses the packaging feature at bitbucket that will checkout version 0.9 from the repository and deliver it as a tarball. The resulting index file can be hosted somewhere globally and referred to with pip install --extra-index-url <url>.

Package Index API

The Package Index "API" is basically a set of nested directories of the form packagename/version with index.html files that will describe how to download the package with links like those shown above.

This works, but is a PITA since it seems to require:

1. The whole nested directory structure (so you can't just manage a single file).

2. Requires that these index files be server from a proper web server that will dish them up when given a trailing slash https://.../packagename/versions/. I still don't know why they can't just also look for the index files. I have asked on the mailing list.

Visualizing Dependencies

Pip

Try the pipdeptree package. Unfortunately, this lists all dependencies in a conda environment.

pip install pipdeptree

Conda

Here is a strategy based on Eric Dill's render_env.py GIST. We use it to find all of the roots in an environment (packages upon which nothing depends).

Issues

Conda Performance

Conda can be very slow to resolve dependencies. An experimental project mamba attempts to speed this up.

Redundancy between Conda and Pip

The present workflow duplicates information in setup.py for pip and PyPI, and meta.yaml for Conda. This is being discussed as part of the following Conda issue:

• https://github.com/conda/conda-build/issues/884

Building a Conda Package

For a simple example of a Conda package, see the following:

• mforbes/conda-tree

This is a simple python project I forked consisting of a single python file. To turn it into a Conda package, I added the following meta.yaml YAML file, built, and uploaded it as described in Anaconda Cloud: Building Packages:

package:
  name: conda-tree
  version: "0.0.1"

source:
  git_rev: master
  git_url: https://github.com/mforbes/conda-tree.git

requirements:
  host:
    - python

  run:
    - networkx
    - conda

build:
  noarch: python
  script:    # See https://github.com/conda/conda-build/issues/3166
    - mkdir -p "$PREFIX/bin"
    - cp conda-tree.py "$PREFIX/bin/"
    
about:
  home: https://github.com/rvalieris/conda-tree
  license: MIT
  license_file: LICENSE

This follows a discussion about using conda-build to build a package without setup.py (since I wanted to minimally complicate the original package). Once this is built and committed, I can build it an uploaded to [my Conda channel'(https://anaconda.org/mforbes) as follows:

conda install conda-build anaconda-client  # If needed: I have these in (base)
anaconda login                             # If needed: I stay logged in
conda build .

# The name needed below can be found with
conda build . --output

anaconda upload /data/apps/anaconda/conda-bld/noarch/conda-tree-0.0.1-0.tar.bz2

One gotcha: the conda build . command above will attempt to download the source from the appropriate tag/branch on Github (since that is what we specified as a source). Thus, you must make sure to tag and push those tags:

git tag -a v0.0.1
git push v0.0.1

When a Dependency is not Available to Conda

When you are making conda packages, all elements need to be conda-installable. Here is how to push a pip-installable package to your conda channel:

mkdir tmp; cd tmp
conda activate base

conda skeleton pypi husl
# Edit husl/meta.yaml if needed
conda build husl/meta.yaml
anaconda upload --all /data/apps/conda/conda-bld/osx-64/husl-4.0.3-py37_0.tar.bz2

conda skeleton pypi python-hglib
# Edit python-hglib/meta.yaml if needed
conda build python-hglib/meta.yaml
anaconda upload --all /data/apps/conda/conda-bld/osx-64/python-hglib-2.6.1-py37_0.tar.bz2

References

• pip dependencies in conda recipe #548

Testing

By including a section like the following, you can run tests when you issue the conda build command. Some important notes though:

• Conda now moves the working directory, so you need to specifically include the files you will test with the source_files: section.

• If you build from your source directory, be sure to clean out __pycache__ directories (see Gotchas below).

test:
  source_files:
    - persist
  imports:
    - persist
  requires:
    - coverage
    - h5py
    - pytest >=2.8.1
    - pytest-cov >=2.2.0
    - pytest-flake8
    - scipy
  commands:
    - py.test

This will do several things:

1. It will test that import perists works.

2. It will create a test environment with the specified requirements.

3. It will run py.test.

If you do this locally (i.e. src: .) then be sure to remove all __pycache__, *.pyc, and *.pyo files:

find . -name "__pycache__" -exec rm -rf {} \;
find . -name "*.pyc" -delete
find . -name "*.pyo" -delete

Gotchas

• Make sure you have the appropriate channels in your ~/.condarc file. (There is currently no way to specify channels for conda build.)

  Copy

  channels:
    - defaults
    - conda-forge
    - mforbes
    - file://data/apps/conda/conda-bld
  

• The last entry is because sometimes things are not immediately available on my mforbes channel, even though I upload something to anaconda.org. This is where they are built by default. This local channel can be made available by running:

  Copy

  conda index /data/apps/conda/conda-bld
  

• Don't run conda build from a conda environment. Make sure you are in the base environment.

• If you do this locally (i.e. src: .) then be sure to remove all __pycache__, *.pyc, and *.pyo files:

  Copy

  find . -name "__pycache__" -exec rm -rf {} \;
  find . -name "*.pyc" -delete
  find . -name "*.pyo" -delete
  

References

• Building Conda Packages

• Anaconda Cloud: Building Packages

• Anaconda Cloud: Working with Environments

Current Working Environments

I keep track of my working environments through a set of environment.*.yml files which I host here:

• https://bitbucket.org/mforbes/configurations/src/trunk/anaconda/

Complete Install (21 July 2018)

Here is an example of a complete install on Mac OS X.

References

• Python Packaging User Guide: This is the authoritative guide about packaging with "Last Reviewed" dates so you can be sure you are getting the latest information.

• https://caremad.io/2013/07/setup-vs-requirement/

• http://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/

• http://tech.marksblogg.com/better-python-package-management.html: A nice discussion of pip usage.

• https://www.colorado.edu/earthlab/2019/01/03/publishing-your-python-code-pip-and-conda-tips-and-best-practices: Some recent best practice notes.

Old Notes Etc.

TL:DR

• Install miniconda with perhaps a few tools like mercurial, but use conda create -n <env> to create working environments to help maintain isolation for everything else.

• Create the environments from an environment.yml file for reproducibility.

  • Make sure that pip is configured to not install packages in the user directory so that they instead go to the conda environments:

  Copy

  pip config set install.user false
  

This setup requires activating the appropriate conda environment before working. There are three options for making commands available to all environments:

• Added the base miniconda bin/ at the end of your path in a special way so that special way so that you can fall back to these base commands. I do this for mercurial which I install in my base environment.

• Explicitly use pip install --user which will put files in ~/.locals/bin and ~/.locals/lib so that any current python environment can access them. I do this with Nikola for blogging.

• Create symlinks to conda-installed executables in ~/.locals/bin or aliases to the appropriate environment. I use this, for example, with Jupyter.