6

I'm trying to make a complete local Pypi repository mirror with pip search feature on a server I can only connect an external hard drive to. To be clear, I don't want a simple caching system, the server is connected to other machines in a completely closed network, no access to the internet at all.

What I have done so far is dumping every pypi packages with bandersnatch and I can do pip install with a simple http server in front of it. I also retrieved the pypi legacy source code and got it working without any python packages inside. The problem I encounter now is to link both sides and I'm not even sure this could be done this way.

I also tested pypiserver. It could have done what I wanted, but it's way too slow ending up with pip search throwing timeout (looks like it wasn't built to handle that much packages).

Finally, I gave a look at devpi. Seems to do the job well for what I want to do but I'm looking for a way to import my bandersnatch dump into it easily. It does not look like I can create an index based on a local directory.

Thank you for any response.

axellink
  • 128
  • 1
  • 9
  • Excellently timed question as I'm staring at the same issue: pip clients won't have any Internet access, need a full PyPi mirror, started with Bandersnatch before I noticed it essentially only does caching, saw that devpi ticks all boxes - and now trying to figure out how I can salvage my massive bandersnatch download with superimposing devpi's indexing! – Marakai Jul 29 '16 at 05:36
  • @Marakai I just finished downloading Bandersnatch's massive dump. Are you telling me it was a waste of time, effort and bandwidth ?? – Arjun J Rao Jun 28 '17 at 13:09
  • @ArjunJRao Well, I can only answer that with: does Bandersnatch do everything you want? Then I guess it works for you. As per my comment timestamp we've been using devpi for a year now and it has been brilliant! Only caches what we actually need from remote instead of the entire PyPi archives, we can seamlessly add our own package repository and it has some nice management tools to index and clean up the repositories. – Marakai Jun 28 '17 at 20:53

1 Answers1

5

I might as well provide a proper answer to this on how we got DevPi working quite nicely in our environment:

  1. Install DevPi

DevPi requires Python 3! So make sure you have the Python 3 version of pip installed. Using that:

pip install -U devpi

(likely as root) should do the trick.

  1. Make sure your server firewall is open

DevPi uses port 3141 by default. If you have firewall-cmd installed something like

firewall-cmd --zone=public --add-port=3141/tcp --permanent
firewall-cmd --reload

or equivalent command on your system.

  1. Configure DevPi

DevPi will use PyPi out of the box. We also wanted the ability to "overlay" our own packages that are only provided organisation internally. For local nabCERT packages requires an internal index. The nice thing as this one can itself use PyPi as fallback!

Select the devpi server to work on - which is the server you're on, probably

devpi use  http://localhost:3141

Now create a user that can add and manage the internal packages and login with them

devpi user -c myuser  password=mypassword
devpi login myuser --password mypassword

Now create our internal index to hold local packages, while ensuring it will use PyPi as a "fallback"

devpi index -c myindex bases=/root/pypi volatile=True

  1. Start it up

    devpi-server --host=0.0.0.0 --port=3141 --serverdir=/var/www/pypi

  2. Try and install a package

    pip install -i http://localhost:3141/root/pypi/ simplejson

If something goes wrong check the logs, in our case they were in /var/www/pypi/.xproc/devpi-server/xprocess.log

At this point, if all settings above have been successfully followed, you should be able to open a web browser and point it at the devpi server with

http://localhost:3141/myuser/myindex
  1. Make DevPi start automatically

That varies. We use systemd so I created a file /usr/lib/systemd/system/devpi.service

[Unit]
Requires=network-online.target
After=network-online.target 

[Service]
EnvironmentFile=-/etc/sysconfig/devpi
Type=forking
PIDFile=/var/www/pypi/.xproc/devpi-server/xprocess.PID
Restart=always
ExecStart=/bin/devpi-server --host=0.0.0.0 --port 3141 --serverdir /var/www/pypi --start
ExecStop=/bin/devpi-server --host=0.0.0.0 --port 3141 --serverdir /var/www/pypi --stop
User=root 

[Install]
WantedBy=multi-user.target

Save the file and notify systemd.

systemctl daemon-reload
systemctl enable devpi
  1. Configure a client

To point your clients' pip to use the new DevPi repository create a /etc/pip.conf file with something like this

[global]
trusted-host = <server IP or FQDN>

[install]
index-url = http://<server IP or FQDN>:3141/myuser/myindex/+simple/

[search]
index = http://<server IP or FQDN>:3141/myuser/myindex/
Marakai
  • 1,163
  • 11
  • 26