gh-158140: Resolve lazily imported sibling submodules independently

[pablogsal]

Fixes #158140. With lazy import pkg.a followed by lazy import pkg.b, accessing pkg.a currently imports pkg.b first. Resolve ordinary dotted imports through the existing pending-child lookup, without publishing placeholders in package dictionaries. Custom namespace builtins and aliased parents retain their existing import behavior.

The lazy-import, import, importlib, module, sys and C API suites pass in debug and free-threaded builds. Regression tests cover independent siblings, star imports, declaration cleanup, deleted attributes, import hooks and failed-import retries.

• Issue: Accessing one lazily imported submodule (lazy import pkg.a) also imports a later imported pkg.b #158140

[pablogsal]

Sorry fucked up the squash give me sec

[Yhg1s]

Is this ready now @pablogsal?

[Yhg1s]

Here's a somewhat surprising change in behaviour: given these two modules:

A.py:

lazy import xml.missing # CORRECT ONE
import B

xml.missing

B.py:

lazy import xml.missing # WRONG ONE

Before this PR the use of xml.missing in A.py will print a traceback showing the lazy import in A.py as the cause:

% base/python A.py
Traceback (most recent call last):
  File "A.py", line 1, in <module>
    lazy import xml.missing # CORRECT ONE
ImportError: lazy import of 'xml.missing' raised an exception during resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "A.py", line 4, in <module>
    xml.missing
    ^^^
ModuleNotFoundError: No module named 'xml.missing'

With this PR, it reports the lazy import in B.py:

% pr/python A.py
Traceback (most recent call last):
  File "B.py", line 1, in <module>
    lazy import xml.missing # WRONG ONE
ImportError: lazy import of 'xml.missing' raised an exception during resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "A.py", line 4, in <module>
    xml.missing
ModuleNotFoundError: No module named 'xml.missing'

Swapping the order of the lazy imports used to not have an effect on the traceback, but now it does: the last lazy import wins. I'm not sure if that's fixable, but I think if we need a single location the first lazy import would be a better winner.

[Yhg1s]

Swapping the order of the lazy imports used to not have an effect on the traceback, but now it does: the last lazy import wins. I'm not sure if that's fixable, but I think if we need a single location the first lazy import would be a better winner.

Okay I think I see why we can't do that, either. I think this is fixable later (perhaps by merging lazy objects, storing all the locations and selecting an appropriate one when printing the traceback). Still, it's a little annoying that this used to work as users would expect.

[Yhg1s]

I think there's a similar issue with modules defining __import__, although I don't have a simple reproducer: if you have two different modules lazily importing the same pkg.module, with their own __import__ methods shadowing the builtin __import__, module A may end up calling module B's __import__ when it does its reification. I can't see how to avoid that one without giving A and B its own sys.modules entirely.

[Yhg1s]

All things considered, LGTM. I think the semantic changes are acceptable for 3.15 and we can see if we can improve the error reporting later.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @hugovk for commit 4e83bfa 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F158521%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[pablogsal]

I think we can fix the A/B case by keeping the declarations keyed by the importing module’s __name__. When accessing the child, we use the current frame’s globals to pick the right one. Custom-builtins imports can stay out of that registry since they already resolve the full path through their own hook.

I tried this locally. It’s about 70 lines of C. There’s still ambiguity if two namespaces share the same name, or the same filename for anonymous exec, but it fixes this case without keeping globals alive.

What do you think, @Yhg1s?

[Yhg1s]

I think we can fix the A/B case by keeping the declarations keyed by the importing module’s __name__. When accessing the child, we use the current frame’s globals to pick the right one. Custom-builtins imports can stay out of that registry since they already resolve the full path through their own hook.

I tried this locally. It’s about 70 lines of C. There’s still ambiguity if two namespaces share the same name, or the same filename for anonymous exec, but it fixes this case without keeping globals alive.

What do you think, @Yhg1s?

Yes, I was thinking something similar, but I don't think we should try to fix this for 3.15.0. We can pre-emptively file a bug and fix it later.

[encukou]

I merged in the main branch with #157714.

[miss-islington-app]

Thanks @pablogsal for the PR, and @Yhg1s for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

[bedevere-app]

GH-158613 is a backport of this pull request to the 3.15 branch.