Using link to execute relocated 3.11 bin/python3 fails to find libraries

We're using a symbolic link for munkireport-python3 to point to a relocated python framework:

 → ls -l /usr/local/munkireport/munkireport-python3 
lrwxr-xr-x  1 root  wheel  63 24 Feb 10:10 /usr/local/munkireport/munkireport-python3 -> /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3

Executing /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3 works fine:

 → /Library/MunkiReport/Python.framework/Versions/3.11/bin/python3
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/site-packages']
>>> ^D

Calling it with the link fails to find platform dependent libraries:

 → /usr/local/munkireport/munkireport-python3 
Could not find platform dependent libraries <exec_prefix>
Python 3.11.2 (v3.11.2:878ead1ac1, Feb  7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',        # ouch
 '/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/site-packages']
>>> import socket
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/MunkiReport/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 51, in <module>
    import _socket
ModuleNotFoundError: No module named '_socket'
>>> ^D

A relocated 3.10 doesn't seem to have a problem with this:

 → /usr/local/munkireport/munkireport-python3                     
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python310.zip',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10/lib-dynload',
 '/Library/MunkiReport/Python.framework/Versions/3.10/lib/python3.10/site-packages']
>>> ^D

I'm building with these options:

	./make_relocatable_python_framework.py \
		--upgrade-pip \
		--pip-requirements="../$PKGDIR/requirements.txt" \
		--os-version=11 \
		--python-version "$version" \
		--destination "../$FWROOT" \

I suspect that this is the relevant upstream change: python/cpython#29041

Relevant code from getpath.py:

https://github.com/python/cpython/blob/9f3ecd1aa3566947648a053bd9716ed67dd9a718/Modules/getpath.py#L174-L182

https://github.com/python/cpython/blob/9f3ecd1aa3566947648a053bd9716ed67dd9a718/Modules/getpath.py#L598-L622

Possible workaround would be to replace the symlink with a shell script that basically does

/real/path/to/Python.framework/Versions/Current/bin/python3 $additional_args

I've been playing around with PYTHONPATH and PYTHONHOME as documented here which "works", but keep in mind that thse exported environment vars will apply to any other installations of Python on the system (obviously depending on how/where they're exported).
Also, any PYTHONPATH exports are added to the existing path that Python itself generates, so you could end up with a precedence issue as the exported PYTHONPATH may take priority in search order, etc, so it could have unintended consequences for any possible clashing imports.

Without the exports, you get the Could not find platform dependent libraries <exec_prefix> error and can't import standard packages like subprocess, etc.

[appleseed@infiniteloop]:~ # export PYTHONPATH=""                                                                                  
[appleseed@infiniteloop]:~ # export PYTHONHOME=""
[appleseed@infiniteloop]:~ # python3             
Could not find platform dependent libraries <exec_prefix>
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, subprocess
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 104, in <module>
    from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
 '/usr/local/python/site-packages']
>>>

With the exports, the Could not find platform dependent libraries <exec_prefix> error is gone, and can now import standard packages like subprocess, etc.

[appleseed@infiniteloop]:~ # export PYTHONHOME=/Library/ManagedFrameworks/Python.framework/Versions/3.11
[appleseed@infiniteloop]:~ # export PYTHONPATH=/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload                        
[appleseed@infiniteloop]:~ # python3                                                                                                                       
Python 3.11.3 (v3.11.3:f3909b8bc8, Apr  4 2023, 20:12:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from pprint import pprint
>>> import subprocess
>>> pprint(sys.path)
['',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python311.zip',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11',
 '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
 '/usr/local/python/site-packages']
>>> 
KeyboardInterrupt
>>> ^D

This is very early testing, and understandably won't work particularly well if your Python deployments are more complicated than just installing the ManagedFrameworks version, or in the case of munki, so YMMV.

@carlashley

With @carlashley's tip, I was able to work around this by creating a simple shim script that can be invoked (the context here was addressing macadmins/python#48):

#!/bin/zsh

bin="/Library/ManagedFrameworks/Python/Python3.framework/Versions/3.11/bin/python3.11"

PYTHONHOME=/Library/ManagedFrameworks/Python.framework/Versions/3.11
PYTHONPATH=/Library/ManagedFrameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload
exec "${bin}" "$@"

This script can put anywhere (such as /usr/local/bin/managed_python or whatever), and solves the immediate issue:

% /usr/local/bin/managed_python3_fixed
Python 3.11.1 (v3.11.1:a7a450f84a, Dec  6 2022, 15:24:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> exit()

Actually, this works even without the two env variables.

I used Munki's build_python_framework.sh script to build a 3.12.2 Python framework (see munki/munki#1204 for why), and inserted this shell script:

% cat /usr/local/munki/python3.12.fixed
#!/bin/sh
# Wrapper to force arm64 execution of the python3.12 universal binary; the `arch` is probably unnecessary
exec /usr/bin/arch -arm64 /Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/bin/python3 "$@"

Then:

% /usr/local/munki/python3.12.fixed
Python 3.12.2 (v3.12.2:6abddd9f6a, Feb  6 2024, 17:02:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python312.zip', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12/lib-dynload', '/Users/nmcspadden/Documents/GitHub/munki/Python.framework/Versions/3.12/lib/python3.12/site-packages']
>>> import socket
>>>

It seems to correctly utilize the path of the directory it's in, rather than looking in /Library/Frameworks.

Point your symlink to Python.framework/Versions/Current/Resources/Python.app/Contents/MacOS/Python or Python.framework/Versions/<specific version>/Resources/Python.app/Contents/MacOS/Python and the problem goes away, at least for Python 3.12. (Haven't gone back to test with Python 3.11)

Invoke the bundled Python via its resolved real path, not the symlink…

Navigation Menu

Repository navigation

Using link to execute relocated 3.11 bin/python3 fails to find libraries #31

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Using link to execute relocated 3.11 bin/python3 fails to find libraries #31

Description

Activity

MagerValp commented on Feb 24, 2023

MagerValp commented on Feb 24, 2023

gregneagle commented on Mar 20, 2023

carlashley commented on May 31, 2023

nmcspadden commented on Jun 20, 2023

nmcspadden commented on Feb 20, 2024

gregneagle commented on Jun 25, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions