How to install multiple python namespace packages in editable mode

I created multiple packages according to the PEP420 implicit namespace packaging method. Two of those distribution packages are dende-github-api and dende-gitlab-api. Each of those distributions contains one module which I want to be accessible unter the dende.api namespace. A minimal working example can be found here: https://github.com/dende/example-monorepositry

This is the folder structure of the Repository containing both distributions:

example-monorepository
├── dende-github-api
│   ├── dende
│   │   └── api
│   │       ├── github.py
│   │       └── __init__.py
│   └── setup.py
├── dende-gitlab-api
│   ├── dende
│   │   └── api
│   │       ├── gitlab.py
│   │       └── __init__.py
│   └── setup.py
├── requirements.dev.txt
└── requirements.txt

And the setup.py's look like this (only showing for dende-github-api):

from setuptools import setup

setup(
    name='dende-github-api',
    packages=['dende.api'],
    install_requires=[
        'PyGithub'
    ]
)

I can install both distributions and use them just fine:

$ pip install ./dende-github-api
$ pip install ./dende-gitlab-api
$ python -m dende.api.github
Hi from dende-github-api from /home/c/git/example-monorepositry/venv/lib/python3.8/site-packages/dende/api/github.py
$ python -m dende.api.gitlab
Hi from dende-gitlab-api from /home/c/git/example-monorepositry/venv/lib/python3.8/site-packages/dende/api/gitlab.py

When I install both distributions in editable mode, only one of the installations work:

$ pip install -e ./dende-github-api
$ pip install -e ./dende-gitlab-api
python -m dende.api.gitlab                          
Hi from dende-gitlab-api from /home/c/git/example-monorepositry/venv/lib/python3.8/site-packages/dende/api/gitlab.py
python -m dende.api.github
/home/c/git/example-monorepositry/venv/bin/python: No module named dende.api.github

Somehow the second installation seems to shadow the installation of the older distribution in the overlapping namespace. Is it possible to install overlapping packages like these in editable mode?

Answers

I don't think you can. Editable install works by creating a file <package_name>.egg-link in site-packages/ pointed back to the source directory. You may have only 1 (one) such link so you can install only one package named <package_name> (even if it's a namespace package) in develop/editable mode.

See https://setuptools.readthedocs.io/en/latest/userguide/development_mode.html

Posted on by phd

Both of your modules are in the regular package dende.api, because they both contain an __init__.py. So, they are not really in an implicit namespace package. Only one of the two dende.apis will be available.

It works when installing them in non-editable mode only by accident, because the files are written into the same directory on disk, but not because of the namespace package mechanism.

You can repair that by removing the __init__.pys and thereby making dende.api a namespace package or by renaming the two apis into distinct packages github-api and gitlab-api, which then both are in the namespace package dende.

A more detailed explanation of the namespace package mechanisms for a somehow related problem (why it makes only little sense to have namespace packages inside regular ones) can be found here: https://stackoverflow.com/a/62992832

Posted on by Anonymous