Infinite loop at install time

[lelit]

I'm seeking an advice on how to investigate, and hopefully resolve, a problem I hit today.

The following very simple pyproject.toml, created with poetry init (v1.1.13), has a single dependency against a project of mine:

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["Lele Gaifax <lele@example.com>"]

[tool.poetry.dependencies]
python = "^3.10"
metapensiero-tool-tinject = "^1.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Executing poetry install enters into an infinite loop, and with -vvv I can see:

Using virtualenv: /tmp/e
Updating dependencies
Resolving dependencies...
   1: fact: foo is 0.1.0
   1: derived: foo
   1: fact: foo depends on metapensiero-tool-tinject (^1.0)
   1: selecting foo (0.1.0)
   1: derived: metapensiero-tool-tinject (>=1.0,<2.0)
   1: fact: metapensiero-tool-tinject (1.0) depends on jinja2 (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on jinja2-time (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on questionary (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on ruamel-yaml (*)
   1: selecting metapensiero-tool-tinject (1.0)
   1: derived: ruamel-yaml
   1: derived: questionary
   1: derived: jinja2-time
   1: derived: jinja2
   1: fact: jinja2-time (0.2.0) depends on jinja2 (*)
   1: fact: jinja2-time (0.2.0) depends on arrow (*)
   1: selecting jinja2-time (0.2.0)
   1: derived: arrow
   1: fact: questionary (1.10.0) depends on prompt_toolkit (>=2.0,<4.0)
   1: selecting questionary (1.10.0)
   1: derived: prompt_toolkit (>=2.0,<4.0)
   1: fact: prompt-toolkit (3.0.30) depends on wcwidth (*)
   1: selecting prompt-toolkit (3.0.30)
   1: derived: wcwidth
   1: selecting wcwidth (0.2.5)
   1: fact: jinja2 (3.1.2) depends on MarkupSafe (>=2.0)
   1: selecting jinja2 (3.1.2)
   1: derived: MarkupSafe (>=2.0)
   1: selecting markupsafe (2.1.1)
   1: fact: arrow (1.2.2) depends on python-dateutil (>=2.7.0)
   1: selecting arrow (1.2.2)
   1: derived: python-dateutil (>=2.7.0)
   1: fact: python-dateutil (2.8.2) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.2)
   1: derived: six (>=1.5)
   1: selecting six (1.16.0)
   1: fact: ruamel.yaml (0.17.21) depends on ruamel.yaml.clib (>=0.2.6)
   1: selecting ruamel.yaml (0.17.21)
   1: fact: ruamel.yaml (0.17.21) depends on ruamel.yaml.clib (>=0.2.6)
   1: selecting ruamel.yaml (0.17.21)
   1: fact: ruamel.yaml (0.17.21) depends on ruamel.yaml.clib (>=0.2.6)
   1: selecting ruamel.yaml (0.17.21)
   ...

The same configuration works with poetry 1.2.0b2:

Using virtualenv: /tmp/poetry120b2
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: foo is 0.1.0
   1: derived: foo
   1: fact: foo depends on metapensiero-tool-tinject (^1.0)
   1: selecting foo (0.1.0)
   1: derived: metapensiero-tool-tinject (>=1.0,<2.0)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/metapensiero-tool-tinject/json HTTP/1.1" 200 8155
Source (PyPI): No release information found for metapensiero-tool-tinject-0.0, skipping
Source (PyPI): 1 packages found for metapensiero-tool-tinject >=1.0,<2.0
Source (PyPI): Getting info for metapensiero-tool-tinject (1.0) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/metapensiero-tool-tinject/1.0/json HTTP/1.1" 200 8155
   1: fact: metapensiero-tool-tinject (1.0) depends on jinja2 (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on jinja2-time (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on questionary (*)
   1: fact: metapensiero-tool-tinject (1.0) depends on ruamel-yaml (*)
   1: selecting metapensiero-tool-tinject (1.0)
   1: derived: ruamel-yaml
   1: derived: questionary
   1: derived: jinja2-time
   1: derived: jinja2
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/ruamel-yaml/json HTTP/1.1" 200 330382
Source (PyPI): 236 packages found for ruamel-yaml *
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/questionary/json HTTP/1.1" 200 8296
Source (PyPI): 18 packages found for questionary *
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/jinja2-time/json HTTP/1.1" 200 2918
Source (PyPI): 2 packages found for jinja2-time *
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/jinja2/json HTTP/1.1" 200 15028
Source (PyPI): 44 packages found for jinja2 *
Source (PyPI): Getting info for jinja2-time (0.2.0) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/jinja2-time/0.2.0/json HTTP/1.1" 200 2918
Source (PyPI): No dependencies found, downloading archives
Source (PyPI): Downloading wheel: jinja2_time-0.2.0-py2.py3-none-any.whl
Creating new session for files.pythonhosted.org
[urllib3.connectionpool] Starting new HTTPS connection (1): files.pythonhosted.org:443
[urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/6a/a1/d44fa38306ffa34a7e1af09632b158e13ec89670ce491f8a15af3ebcb4e4/jinja2_time-0.2.0-py2.py3-none-any.whl HTTP/1.1" 200 6360
   1: fact: jinja2-time (0.2.0) depends on jinja2 (*)
   1: fact: jinja2-time (0.2.0) depends on arrow (*)
   1: selecting jinja2-time (0.2.0)
   1: derived: arrow
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/arrow/json HTTP/1.1" 200 16812
Source (PyPI): 62 packages found for arrow *
Source (PyPI): Getting info for questionary (1.10.0) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/questionary/1.10.0/json HTTP/1.1" 200 8296
   1: fact: questionary (1.10.0) depends on prompt_toolkit (>=2.0,<4.0)
   1: selecting questionary (1.10.0)
   1: derived: prompt_toolkit (>=2.0,<4.0)
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/prompt-toolkit/json HTTP/1.1" 200 39862
Source (PyPI): 41 packages found for prompt-toolkit >=2.0,<4.0
Source (PyPI): Getting info for prompt-toolkit (3.0.30) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/prompt-toolkit/3.0.30/json HTTP/1.1" 200 39862
   1: fact: prompt-toolkit (3.0.30) depends on wcwidth (*)
   1: selecting prompt-toolkit (3.0.30)
   1: derived: wcwidth
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/wcwidth/json HTTP/1.1" 200 9186
Source (PyPI): 17 packages found for wcwidth *
Source (PyPI): Getting info for wcwidth (0.2.5) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/wcwidth/0.2.5/json HTTP/1.1" 200 9186
   1: selecting wcwidth (0.2.5)
Source (PyPI): Getting info for jinja2 (3.1.2) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/jinja2/3.1.2/json HTTP/1.1" 200 15028
   1: fact: jinja2 (3.1.2) depends on MarkupSafe (>=2.0)
   1: selecting jinja2 (3.1.2)
   1: derived: MarkupSafe (>=2.0)
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/markupsafe/json HTTP/1.1" 200 52932
Source (PyPI): 4 packages found for markupsafe >=2.0
Source (PyPI): Getting info for markupsafe (2.1.1) from PyPI
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/markupsafe/2.1.1/json HTTP/1.1" 200 52932
Source (PyPI): No dependencies found, downloading archives
Source (PyPI): Downloading sdist: MarkupSafe-2.1.1.tar.gz
[urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/1d/97/2288fe498044284f39ab8950703e88abbac2abbdf65524d576157af70556/MarkupSafe-2.1.1.tar.gz HTTP/1.1" 200 18668
   1: selecting markupsafe (2.1.1)
   1: fact: arrow (1.2.2) depends on python-dateutil (>=2.7.0)
   1: selecting arrow (1.2.2)
   1: derived: python-dateutil (>=2.7.0)
   1: fact: python-dateutil (2.8.2) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.2)
   1: derived: six (>=1.5)
   1: selecting six (1.16.0)
   1: fact: ruamel-yaml (0.17.21) depends on ruamel.yaml.clib (>=0.2.6)
   1: selecting ruamel-yaml (0.17.21)
   1: derived: ruamel.yaml.clib (>=0.2.6)
   1: selecting ruamel-yaml-clib (0.2.6)
   1: Version solving took 26.670 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 11 installs, 0 updates, 0 removals, 1 skipped

  • Installing markupsafe (2.1.1)
  • Installing markupsafe (2.1.1)
  • Installing python-dateutil (2.8.2)
  • Installing python-dateutil (2.8.2)
  • Installing wcwidth (0.2.5)
  • Installing wcwidth (0.2.5)
  • Installing arrow (1.2.2)
  • Installing arrow (1.2.2)
  • Installing jinja2 (3.1.2)
  • Installing jinja2 (3.1.2)
  • Installing prompt-toolkit (3.0.30)
  • Installing prompt-toolkit (3.0.30)
  • Installing ruamel-yaml-clib (0.2.6)
  • Installing ruamel-yaml-clib (0.2.6)
  • Installing jinja2-time (0.2.0)
  • Installing questionary (1.10.0)
  • Installing ruamel-yaml (0.17.21)
  • Installing questionary (1.10.0)
  • Installing jinja2-time (0.2.0)
  • Installing ruamel-yaml (0.17.21)
  • Installing metapensiero-tool-tinject (1.0)
  • Installing metapensiero-tool-tinject (1.0)
  • Installing six (1.16.0): Skipped for the following reason: Already installed

Apparently the problem is related to a new 1.0 release of metapensiero.tool.tinject that I made yesterday, but I really cannot understand what I did wrong to break poetry resolver 😭. The fact that both pip and poetry 1.2.0b2 correctly install it seems to suggest that the package is correct...

I'd appreciate any hint on how to proceed.

Thanks in advance!

[johnthagen]

Could be unrelated but I hit something similar in

• Lockfile generation hangs indefinitely when cache is corrupted #5922

In my case I had to clear Poetry's cache to get Poetry to work properly again.

[lelit]

Thanks for the suggestion but no, unfortunately clearing the cache as suggested didn't change the outcome.

[lelit]

As said, poetry 1.2.0b2 is able to solve the puzzle and install the package, creating the poetry.lock.
By chance, I then tried to install it using poetry 1.1.13, keeping the lock created above, and obtained:

Installing dependencies from lock file

  SolverProblemError

  Because no versions of metapensiero-tool-tinject match >1.0,<2.0
   and metapensiero-tool-tinject (1.0) depends on ruamel-yaml (*), metapensiero-tool-tinject (>=1.0,<2.0) requires ruamel-yaml (*).
  And because no versions of ruamel-yaml match !=0.17.21
   and ruamel-yaml (0.17.21) depends on ruamel.yaml.clib (>=0.2.6), metapensiero-tool-tinject (>=1.0,<2.0) requires ruamel.yaml.clib (>=0.2.6).
  So, because no versions of ruamel.yaml.clib match >=0.2.6
   and foo depends on metapensiero-tool-tinject (^1.0), version solving failed.

  at e1/lib/python3.10/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

I do not understand those statements, in particular the first: why does it say "no versions of metapensiero-tool-tinject match >1.0,<2.0"?

Does this ring any bell?

[lelit]

After several debugging sessions, I think I found the cause.

As said, the package that causes the problem is one of mine, metapensiero.tool.tinject: for the recently released v1 it uses hatchling as build system.

The package declares a dependency on ruamel.yaml. In the PyPI metadata that comes out as canonicalized, ruamel-yaml.

The never ending loop that I get with poetry 1.1 happens because it:

1. fetches info about metapensiero.tools.tinject from PyPI
2. for each dependency, fetches info from PyPI, in particular about ruamel-yaml, for which it creates a Package extracting the .info.name path from the JSON it got from PyPI
3. the Package class canonicalizes that name, using the canonicalize_name()
4. the resulting dependency package is then memoized under the dependency name ruamel-yaml
5. going on, it processes ruamel-yaml dependencies, one of which is ruamel.yaml.clib that in turns depends on ruamel.yaml, and there the loop begins, although I could not identify the exact path: if I understand, there are now two distinct packages that depend on a single one, but the latter is known to the system in two distinct names, ruamel-yaml and ruamel.yaml

The cause of this is in the poetry-core 1.0.8 _canonicalize_regex, that is missing a . (dot), a problem fixed by this commit: I can confirm that simply adding that . to the regex fixes the problem, and I can happily install my package, also with poetry 1.1.14+.

So, should I convert this discussionmonologue to an issue or is the analysis above enough the ask for a backport of the referenced commit?

Thank you for any feedback!

[johnthagen]

I'm not a maintainer, but since you have (very thoroughly) identified the problem, I think an issue would be appropriate and would get the proper attention.