Python Package Installation - A User's Guide

Modes of installing python packages and associated steps.

The W&M HPC policy has recently changed so as to encourage users to install python packages in their home directories. For python packages required by individual users, it is recommended to try installing the packages in their home directories, whereas for packages required by a group a system-wide installation may be considered. This document is intended to help users decide the mode of installation for their packages and illustrate the essential steps involved. The three modes of installing python packages are as follows:

  1. Pip installation: For packages that have pypi packages and do not conflict with currently installed system python packages
  2. virtualenv installation: For packages with/without pip packages and that conflict with currently installed system python packages.
  3. Source install and private modules: For packages that do not have pip install support and require to be compiled from source. This method can be also be used (with caution) to install packages that need to be used across multiple sub-clusters.

Pip Installation

For packages supported by pip installation method, the following command will install the package in user's home directory:

pip install <packagename> --user

This by default installs the package in a location defined by $PYTHONUSERBASE environment variable, which is defined in the python environment module loaded by the user.

It should be noted that the installation location is different for each python package and each sub-cluster, therefore a package installed using a python module will not be available in other python modules. For example, a package installed by python/2.7.13/intel on Bora will not be available on anaconda3/4.4.0 on Bora or on python/2.7.13 on Vortex. This is true for Storm node types as well.

Virtual Environment

A python virtual environment is recommended if the packages/dependencies required by users' programs are not compatible with the pip packages installed on the system. It creates an isolated environment for users to install packages of choice independent of the system. The python version remains the same.

For example: Say on Vortex, it is recommended for the user to use virtualenv instead of an upgrade of system numpy that may break other packages built on numpy 1.9.2 .

To use python virtual-environment it needs to be created and prepared for usage, activated for use and deactivated after usage. These steps are done as follows:

Creating a virtual-environment:

This is a one time process where the user creates a virtual-environment and install python packages for use.

Generic Python modules:
> module load python/2.7.13
> cd <directory_to_create_virtualenv>
> virtualenv <virtualenvname>

Anaconda modules:
> conda create -n <virtualenvname>     

Activating a virtual-environment

A virtual environment should be activated everytime to use it. This can be done with one of the appropriate commands below:

Generic python modules:
[bora]> source <path_to_virtualenv>/bin/activate.csh # tcsh/csh shell
[bora]> source <path_to_virtualenv>/bin/activate # bash/sh shell

Anaconda modules:
[bora]> conda activate <virtualenvname>

If the command runs successfully the virtual-environment is activated and the prompt changes to reflect this:


Preparing for usage

This is usually a one time process as well, wherein the user must install all the packages required. The installation can be done using pip or from source code.

Deactivating a virtual-environment

 A virtual environment must be deactivated after usage as this resets the $PATH environment variable to the system compatible packages. This can be done with one of the commands below:

(virtualenv_name)[bora]> deactivate                # generic python
(virtualenv_name)[bora]> conda deactivate     # anaconda 

Deleting a virtual environment

For deleting a virtual environment permanently the following command(s) can be used:

> conda remove -n <virtualenvname> --all      # anaconda3 
> rm -rf <path_to_virtualenv>  # generic python

Source Install and Private Modules

This is the preferred method for packages that are not supported by PyPI repos or for packages that require customization during the build process. The user is encouraged to read the package installation documents to decide the best procedure for install. For packages that contain a a flag can be passed to direct installation to a specific directory. It should be noted that packages compiled from source require to be run the same platform for reliable results.

Once the package is installed, the directory must be added to the $PYTHONPATH environment variable to be able to use the package. This can be done by create a private user environment module, and loading the module to use the python package.

The process of creating a private user environment module is detailed below.

Creating a private user environment module: To do this, a module file must be created and placed in the correct location to be loaded. Each of this is described below:

A basic module file for python modules looks as follows (further examples can be found in /usr/local/Modules/modulefiles) :

# The first line above is essential for the file to be considered a module file
# Description: examplemod module
proc ModulesHelp { } {
global version
puts stderr "[module-info name] - An example module file to load examplemod python module"

prereq isa

# Change the module name 'python/2.7.13/intel' below to reflect
# the correct module used to build this python package
if { [module-info mode load] && ![is-loaded python/2.7.13/intel] } { module load python/2.7.13/intel }

set xarch $env(XARCH)
set xchip $env(XCHIP)

set version "2.0b"
module-whatis "examplemod 2.0b"

#change the path below to reflect the correct installation path of your package
set path /sciclone/data10/<user>/<path_to_install_location>

prepend-path PYTHONPATH $path/lib/site-packages
prepend-path LD_LIBRARY_PATH $path/lib
setenv EXAMPLEMOD $path

This file (say examplemod) should be placed in <user_home_directory>/privatemodules directory. The module can then be loaded by loading the use.own module followed by the examplemod module.

The entire process (with examplemod) file would be as follows:

> mkdir privatemodules
> cd privatemodules
> vim examplemod # The filename can be chosen to reflect the package name
> #Loading modules
> module load use.own # Adds visibility to users' private modules
> module load examplemod # Load actual module

The users are encouraged to contact the W&M HPC group for more information or help regarding installation of their packages.