Tools I: Packaging and virtual-environments
OverviewTeaching: 4 min
Exercises: 6 minQuestions
How to use a package manager to install third party tools and librariesObjectives
Use conda to install a reproducible environment
- There are tens of thousands of python packages
- No need to reinvent the square wheel, it’s already out there
- Contributing to existing packages makes it more likely your work will be reused
- Contributing to open-source packages is the best way to learn how to code
Python virtual environments
- Virtual environments isolate your setup from the rest of the system
- It ensures different project do not interfere with each other
- For instance:
- a production environment with tried and true version of your software and tensorflow 1.15
- a development environment with shiny stuff and a migration to tensorflow 2.1
|manage python packages||✅||✅||✅|
|manage non-python packages||✅||❌||❌|
|choose python version||✅||❌||❌|
|manage virtual envs||✅||❌||✅|
Rules for choosing a package manager
- Choose one
- Stick with it
It is also the de facto package manager on Imperial’s HPC cluster systems.
Installing and using an environment
If you haven’t already, see the setup guide for instructions on how to install conda, Visual Studio Code and Git.
Create a new folder to use for this course. Avoid giving it a name that includes spaces. If you’re using an ICT managed PC the folder must be located in your user area on the C: drive i.e.
C:\Users\UserName(Note that files placed here are not persistent so you must remember to take a copy before logging out). Start Visual Studio Code and select “Open folder…” from the welcome screen. Navigate to the folder you just created and press “Select Folder”.
Press “New file” and copy the below text. Save the file as
environment.yml, the location should default to your newly created folder.
name: course dependencies: - python>=3.6 - flake8 - pylint - black - mypy - requests - pip - pip: - -e git+https://github.com/ImperialCollegeLondon/R2T2.git#egg=r2t2
Create a new virtual environment using conda:
Windows users will want to start the app
Anaconda Promptfrom the Start Menu.
Linux and Mac users should use a terminal app of their choice. You may see a warning with instructions. Please follow the instructions.
conda env create -f [path to environment.yml]
You can obtain
[path to environment.yml]by right clicking the file tab near the top of Visual Studio Code and selecting “Copy Path” from the drop-down menu. Right click on the window for your command line interface to paste the path.
We can now activate the environment:
conda activate course
And check python knows about the installed packages. Start a Python interpreter with the command
We expect this to run and not fail. You can see the location of the installed package with:
The file path you see will vary but note that it is within a directory called
coursethat contains the files for the virtual environment you have created. Exit the Python interpreter:
Finally, feel free to remove requests from
environment.yml, then run
conda env update -f [path to environment.yml]
and see whether the package has been updated or removed.
Selecting an environment in Visual Studio Code
If you haven’t already, see the setup guide for instructions on how to install Visual Studio (VS) Code.
On Linux and Mac, one option is to first activate conda, and then start VS Code:
> conda activate name_of_environment > code .
The simplest option for all platforms is to set the interpreter is via the Command Palette:
- For Windows/Linux: Ctrl + Shift + P, and start typing “Python: Select interpreter”
- For macOS: Cmd + Shift + P, and start typing “Python: Select interpreter”
If you already have a Python file open then it’s also possible to set the interpreter using the toolbar at the bottom of the window.
Installing an editable package
Editable packages are packages that you can modify for development and have python immediately recognize your changes.
Look at the last few lines of
environment.yml. It installs r2t2 in editable mode. The package is automatically downloaded from the web and installed next to
environment.ymlin the subfolder
Try and add
Then start python and do
Your greeting should appear: python did indeed take the modified file into account.
Note that r2t2 was setup as a python package with a standard directory structure and a
setup.pyfile. It’s well worth investing 10 minutes into transforming a python script into a package just to make it a shareable development environment.
Choosing the installation directory for R2T2
It would be nice if we could choose the directory where the editable package goes, i.e. rather than have
src/r2t2we might want to install it directly in an
Nominally, pip does allow us to do that with –src.
However, it is not (yet) possible to tell conda to tell to use a given option, as highlighted in this issue. But that’s where the fun begins, because conda is an open-source effort, you could pitch in and try and add a feature or a fix. There is a lot to learn just from lurking around issues of open-source projects, whether it is about the project itself, or even about language design. There is even more to learn from participating.
There are tens of thousands of Python packages
The choice is between reinventing the square wheel or reusing existing work
The state of an environment can be stored in a file
This stored environment is then easy to audit and recreate