Certain PyObjC frameworks are not properly imported

[maxbelanger]

We've been testing migrating from modulegraph to modulegraph2 for the build tool that produces our embedded Python package, and we stumbled across what seems to be a regression from modulegraph. It seems that in some cases, the modules within certain PyObjC frameworks are not fully discovered. This can be reproduced with the latest version of modulegraph2.

Here's an example with the Quartz framework:

• Install pyobjc-framework-Quartz and modulegraph2.
• In a Python shell, run the following:

import modulegraph2
g = modulegraph2.ModuleGraph()
g.add_module("Quartz")
g.report()

• If you filter the output, there's only a single reference to Quartz:

Package         Quartz                    /opt/homebrew/lib/python3.10/site-packages/Quartz

There are no references to its source modules in the report, or of its extensions (e.g. Quartz/CoreGraphics/_inlines.abi3.so). This issue does not affect modulegraph.

[ronaldoussoren]

This might be related to PyObjC's lazy loader which does some things that can confuse import lib. I'll have to debug to know for sure.

[maxbelanger]

This does seem to correlate with use of ObjCLazyModule, yes.

[ronaldoussoren]

My current plan is to fix this on both ends (if possible):

• Update PyObjC to copy the missing bits to the ObjCLazyModule instances
• Teach modulegraph2 about this pattern

I'm saying this without having tried to reproduce this yet, should have some time tonight.

[ronaldoussoren]

This is what I get:

Package         Quartz                    /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz
Package         Quartz.CoreGraphics       /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics
ExtensionModule Quartz.CoreGraphics._callbacks /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_callbacks.cpython-310-darwin.so
SourceModule    Quartz.CoreGraphics._contextmanager /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_contextmanager.py
ExtensionModule Quartz.CoreGraphics._coregraphics /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_coregraphics.cpython-310-darwin.so
ExtensionModule Quartz.CoreGraphics._doubleindirect /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_doubleindirect.abi3.so
ExtensionModule Quartz.CoreGraphics._inlines /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_inlines.abi3.so
SourceModule    Quartz.CoreGraphics._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_metadata.py
ExtensionModule Quartz.CoreGraphics._sortandmap /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_sortandmap.abi3.so
Package         Quartz.CoreVideo          /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo
ExtensionModule Quartz.CoreVideo._CVPixelBuffer /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo/_CVPixelBuffer.abi3.so
SourceModule    Quartz.CoreVideo._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo/_metadata.py
Package         Quartz.ImageIO            /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageIO
SourceModule    Quartz.ImageIO._metadata  /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageIO/_metadata.py
Package         Quartz.ImageKit           /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit
ExtensionModule Quartz.ImageKit._imagekit /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit/_imagekit.abi3.so
SourceModule    Quartz.ImageKit._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit/_metadata.py
Package         Quartz.PDFKit             /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit
ExtensionModule Quartz.PDFKit._PDFKit     /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit/_PDFKit.abi3.so
SourceModule    Quartz.PDFKit._metadata   /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit/_metadata.py
Package         Quartz.QuartzComposer     /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzComposer
SourceModule    Quartz.QuartzComposer._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzComposer/_metadata.py
Package         Quartz.QuartzCore         /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore
SourceModule    Quartz.QuartzCore._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore/_metadata.py
ExtensionModule Quartz.QuartzCore._quartzcore /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore/_quartzcore.abi3.so
Package         Quartz.QuartzFilters      /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzFilters
SourceModule    Quartz.QuartzFilters._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzFilters/_metadata.py
Package         Quartz.QuickLookUI        /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI
ExtensionModule Quartz.QuickLookUI._QuickLookUI /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI/_QuickLookUI.abi3.so
SourceModule    Quartz.QuickLookUI._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI/_metadata.py

This is with python 3.10 installed from python.org, modulegraph2 is the tip of the tree and pyobjc was installed using pip.

With python3.11 modulegraph2 hits a false assertion, haven't tried to debug that yet. UPDATE: just committed a fix for that, unrelated to this issue.

[ronaldoussoren]

Looks like homebrew is relevant here as well. Did you install pyobjc through homebrew as well?

[maxbelanger]

I did reproduce this using Homebrew, but this was also affecting an internal deployment of PyObjC that does not use Homebrew (something resembling a venv). I think the important factor is that a wheel is in use and on the Python path.

[ronaldoussoren]

I did reproduce this using Homebrew, but this was also affecting an internal deployment of PyObjC that does not use Homebrew (something resembling a venv). I think the important factor is that a wheel is in use and on the Python path.

What do you mean with "a wheel is in use and on the Python path"?

I finally got around to installing homebrew in a VM and cannot reproduce the issue there.

What I did:

• VM running macOS 12
• Install brew from brew.sh
• brew install python3
• Check that python3 now refers to the homebrew installation in /usr/local/bin
• python3 -m pip install -U pip (necessary to be able to install modulegraph2 from the repository)
• python3 -m pip install pyobjc
• git clone https://github.com/ronaldoussoren/modulegraph2
• cd modulegraph2 && pip install -e .
• run the reproducer from the issue

I get the following output:

$ python3 t.py | grep Quartz
Package         Quartz                    /usr/local/lib/python3.11/site-packages/Quartz
Package         Quartz.CoreGraphics       /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics
ExtensionModule Quartz.CoreGraphics._callbacks /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_callbacks.cpython-311-darwin.so
SourceModule    Quartz.CoreGraphics._contextmanager /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_contextmanager.py
ExtensionModule Quartz.CoreGraphics._coregraphics /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_coregraphics.cpython-311-darwin.so
ExtensionModule Quartz.CoreGraphics._doubleindirect /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_doubleindirect.abi3.so
ExtensionModule Quartz.CoreGraphics._inlines /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_inlines.abi3.so
SourceModule    Quartz.CoreGraphics._metadata /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_metadata.py
ExtensionModule Quartz.CoreGraphics._sortandmap /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_sortandmap.abi3.so
Package         Quartz.CoreVideo          /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo
ExtensionModule Quartz.CoreVideo._CVPixelBuffer /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo/_CVPixelBuffer.abi3.so
SourceModule    Quartz.CoreVideo._metadata /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo/_metadata.py
Package         Quartz.ImageIO            /usr/local/lib/python3.11/site-packages/Quartz/ImageIO
SourceModule    Quartz.ImageIO._metadata  /usr/local/lib/python3.11/site-packages/Quartz/ImageIO/_metadata.py
Package         Quartz.ImageKit           /usr/local/lib/python3.11/site-packages/Quartz/ImageKit
ExtensionModule Quartz.ImageKit._imagekit /usr/local/lib/python3.11/site-packages/Quartz/ImageKit/_imagekit.abi3.so
SourceModule    Quartz.ImageKit._metadata /usr/local/lib/python3.11/site-packages/Quartz/ImageKit/_metadata.py
Package         Quartz.PDFKit             /usr/local/lib/python3.11/site-packages/Quartz/PDFKit
ExtensionModule Quartz.PDFKit._PDFKit     /usr/local/lib/python3.11/site-packages/Quartz/PDFKit/_PDFKit.abi3.so
SourceModule    Quartz.PDFKit._metadata   /usr/local/lib/python3.11/site-packages/Quartz/PDFKit/_metadata.py
Package         Quartz.QuartzComposer     /usr/local/lib/python3.11/site-packages/Quartz/QuartzComposer
SourceModule    Quartz.QuartzComposer._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzComposer/_metadata.py
Package         Quartz.QuartzCore         /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore
SourceModule    Quartz.QuartzCore._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore/_metadata.py
ExtensionModule Quartz.QuartzCore._quartzcore /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore/_quartzcore.abi3.so
Package         Quartz.QuartzFilters      /usr/local/lib/python3.11/site-packages/Quartz/QuartzFilters
SourceModule    Quartz.QuartzFilters._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzFilters/_metadata.py
Package         Quartz.QuickLookUI        /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI
ExtensionModule Quartz.QuickLookUI._QuickLookUI /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI/_QuickLookUI.abi3.so
SourceModule    Quartz.QuickLookUI._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI/_metadata.py

This looks correct to me.