How to clone Python working environment on another machine?

Question

I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.

I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.

So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)

Answer 1

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.

---

As for the question itself - you can export your Anaconda environment using:

conda env export > environment.yml

And recreate it using:

conda env create -f environment.yml

Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.

To create a virtual environment in Anaconda you can use:

conda create -n yourenvname python=x.x anaconda

which you activate using:

source activate yourenvname

Answer 2

TLDR
Use Conda Pack
install using conda or pip:

conda-forge:

conda install -c conda-forge conda-pack

PyPI:

pip install conda-pack

Then for:

Backing up:

# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env

# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz

# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env

Restoring:

And to restore it on the other machine(s):

# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env

# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python

# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate

# Run Python from in the environment
(my_env) $ python

# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack

a bit of an explanation :

If you plan on getting an exact copy of your current environment and then move it to another machine with the same platform and OS, without redownloading all packages again from Internet (good for offline machines/behind firewalls). All other previous methods require internet connection. so in case you dont have access to internet, then you can use conda pack.

Conda Pack

Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer.

To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI.

for future updates check the ref

Answer 3

First export environment configuration of your current conda environment using:

conda-env  export -n your_env_name > your_env_name.yml

example:

conda-env  export -n base> base.yml

After running above command their should be yml configuration file in your current directory which contain information of your conda environment

To create new environment using yml configuration file run:

conda-env create -n new_env -f=\path\to\base.yml 

example:

conda-env create -n venv -f=base.yml

In case the above one does not work (due to the various issues of conda itself), it's always worth a try with the following variation:

conda-env create --name new_env --file \path\to\base.yml 

Answer 4

If your program is mostly Python, you could rely solely on virtual environments.

Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments.

In the working virtualenv, create a file with the version of each installed Python library :

pip freeze > requirements.txt

In the new virtualenv, ask pip to install those libraries with the same version :

pip install -r requirements.txt

This makes sure you get the same lib versions on both machines. And since requirements.txt is tracked by your VCS, you can always recreate the environment of an old version of your code.

Of course, if you need a database, a production web server, etc. you end up with a few more steps and you can't rely on virtualenv to ensure both environments match. This is where Docker steps in (See Pieter21's answer).

Answer 5

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization.

It will require some time investment but in the end will provide many benefits.

From the link, where I marked your specific need in bold italic:

Package software into standardized units for development, shipment and deployment

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Available for both Linux and Windows based apps, containerized software will always run the same, regardless of the environment. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.

Answer 6

A wrap up of the existing ways to create an environment based on another one:

• Cloning an environment:

  • From an existing environment:

    $ conda create --name NEW_ENV_NAME --clone ORIG_ENV_NAME

  • From an exported environment file on the same machine:

    $ conda create --name ENV_NAME —-file FILE_NAME.yml

  • From an exported environment file on a different machine:

  $ conda env export > ENV_NAME.yml
  $ conda env create -f ENV_NAME.yml
  

Answer 7

From the very end of this documentation page:

Save packages for future use:

conda list --export > package-list.txt

Reinstall packages from an export file:

conda create -n myenv --file package-list.txt

Answer 8

One-liner:

conda create --clone source_env --name destination_env

Answer 9

to clone a miniconda/anaconda environment for Windows users:

• install miniconda/anaconda with "all users" option. if you install with your current user, the path will be set during the compile of the exe files and you are lost

• using conda-pack is not bad but for me zipping the complete ~/<anaconda/miniconda> path and unzipping it on the target machine was working properly too.

btw. do not forget to copy the dlls libcrypto-xxx.dll and libssl-xxx.dll from the folder ~/Library/bin to the folder ~/DLLs

Answer 10

TLDR: the best method is below:

first, in old server

rsync -avr /home/xxxx/anaconda3/envs/your_env_name your_login_name@new_server_ip:/home/xxxx/miniconda3/envs/

then in new server conda activate your_env_name, and run pip list, you'll get error: /bin/python: bad interpreter: No such file or directory： bad interpreter: No such file or directory

second, in new server

find ~/miniconda3/envs/nn/ -type f  -exec grep --binary-files=without-match -rl "oldprefix" {} \; -exec sed -i 's+newprefix+g' {} +

the below replace all python prefix in envs dir, make commandline command like pip, work as expect.

--------------------------------separating line-----------------------------------

if you dont want to redownload packages in the new server (of same os platform), the official method is use pip install conda-pack . conda-pack archives an entire environment including the package binaries, which is useful in low or no bandwidth situations.

Ab oberversation, by cat $(which pip) you can find the pip command is just a linux file(with execute privilige assisiation to a python.exe path):

#!/home/your_home_name/miniconda3/envs/your_env_name/bin/python
-- coding: utf-8 --
import re
import sys
from pip._internal.cli.main import main
if name == 'main':
    sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0])

see the first line with #! shellbang, all python module command realized by entry_point such as pip, need to replace this python path prefix when move to new server.

AFAIK, what conda pack do is replace all prefix in old envs file form #!/home/your_home_name/miniconda3/envs/your_env_name/bin/python to #!/usr/bin/python, and conda unpack replace it to another path in your new server's conda directory

howerver with conda pack you need to tar.gzip first and rsync to new server, then untargzip.

the best method is below:

first, in old server

rsync -avr /home/xxxx/anaconda3/envs/your_env_name your_login_name@new_server_ip:/home/xxxx/miniconda3/envs/

then in new server conda activate your_env_name, and run pip list, you'll get error: /bin/python: bad interpreter: No such file or directory： bad interpreter: No such file or directory

second, in new server

find ~/miniconda3/envs/nn/ -type f  -exec grep --binary-files=without-match -rl "oldprefix" {} \; -exec sed -i 's+newprefix+g' {} +

the below replace all python prefix in envs dir, make commandline command like pip, work as expect.