Skip to content

windows dll notes

Jump to bottom Edit New page
Matthew Brett edited this page May 8, 2014 · 22 revisions

Python and DLLs on Windows

DLL loading

A DLL can load can be implicit or explicit (see DLL linking.

Implicit loading is also called static loading or load-time linking. An executable (.exe or other .dll) has been compiled against the DLL, using a .lib import library file listing the symbols exported by the DLL. The executable file has an import address table (IAT), with entries for each DLL function that the executable file needs. When the executable is run, Windows finds and loads the DLL file named in the executable, and fills in the executable IAT with the loaded locations of each required function.

Explicit loading is also called dynamic loading or run-time linking. The executable explicitly calls one of the functions LoadLibrary, LoadLibraryEx with the name of the required DLL. Calls to functions inside the loaded DLL need to be made via a function pointer.

Thus, in the implicit case, Windows does the DLL loading based on the contents of the executable. In the explicit case, the executable does the loading.

See DLLs in Visual C++ for an overview.

Which DLL?

DLLs are a problem for general windows builds because it can be tricky making sure your extension gets the right DLL.

This problem is sometimes called DLL hell

Let's say your executable links implicitly to myuseful.dll. How does Windows know where to look for myuseful.dll?

By default, Windows uses the dynamic link library search order.

The default DLL search order for desktop applications on Windows 7 and later is:

  1. Directory containing the loading application
  2. System directory (e.g. C:\Windows\System32)
  3. 16 bit system directory
  4. Windows directory (e.g. C:\Windows)
  5. Current directory
  6. Directories listed in PATH environment variable

There are various exceptions and ways of tuning this order, which are (quotes are from the document above):

  • Previous loading of a DLL with the same module name:

    If a DLL with the same module name is already loaded in memory, the system uses the loaded DLL, no matter which directory it is in. The system does not search for the DLL.

  • Known Windows DLLs are loaded from the system and not searched for

  • The order above is when "safe DLL search mode" is enabled via a entry in the Windows registry. It is "enabled" by default on Windows Vista and later, but disabled by default on XP. If safe search mode is disabled, the current directory gets searched after the directory containing the loading application, rather than after the Windows directory.

  • You can add a single path to the DLL search order by calling the SetDllDirectory(new_path) function. This inserts new_path into the search order before the System directory in the sequence above, and removes the current directory from the search path. See SetDllDirectory API doc. This mode stays in place until specifically disabled by a call SetDllDirectory(NULL).

  • You can affect search order using the LOAD_LIBRARY_SEARCH flags, in systems that support them (updated Vista and above). You can either pass these flags to LoadLibraryEx to affect one call, or use SetDefaultDllDirectories to set the default flags for the process. See LoadLibraryEx API doc for detail. Flags are:

    • LOAD_LIBRARY_SEARCH_APPLICATION_DIR - search directory containing application.
    • LOAD_LIBRARY_SEARCH_SYSTEM32 - search %windows%\system32 directory.
    • LOAD_LIBRARY_SEARCH_USER_DIRS - search any directories added with calls to AddDllDirectory - see below.
    • LOAD_LIBRARY_SEARCH_DEFAULT_DIRS - combination of three flags above.
    • LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR - search for other DLLs in the same directory as the DLL to be loaded. This works when passing an absolute DLL path to LoadLibraryEx.

    In particular, this means you can add multiple paths to the search order with several calls to AddDllDirectory(new_path) etc (see AddDllDirectory API doc.

  • If you call LoadLibraryEx("c:\path\to\my.dll") with the LOAD_WITH_ALTERED_SEARCH_PATH flag, then, for executables resolved on that call, Windows replaces the "Directory containing the loading application" above with the directory containing the DLL - here c:\path\to).

  • You can force Windows to look first for DLLs in the directory containing the loading application, but making a redirection file. Thus, if the application is c:\path\myapp.exe, and you have an (empty) redirection file called c:\path\myapp.exe.local then, any DLLs loaded by myapp.exe will first be looked for in c:\path regardless of any absolute DLL paths given to LoadLibrary or LoadLibraryEx. The search order is otherwise modified as above. Known (named Windows) DLLs can't be redirected in this way. Any such redirection is ignored if the application has a side by side assembly manifest (see below)

Side by side assemblies

You can specify exactly which DLLs get loaded for any executable using `side
by side assemblies <http://msdn.microsoft.com/en-us/library/windows/desktop/dd408052(v=vs.85).aspx>`_.

Some useful links:

Python DLLs

On Windows, Python extension modules have file extensions .pyd as in myextension.pyd.

Imagine you have a package like this:

mypackage/
    myextension.pyd
    runtime.dll
yourpackage/
    yourextension.pyd
    runtime.dll

myextension.pyd depends on my version of runtime.dll, and yourextension.pyd depends on your version of runtime.dll.

This leads to the general Windows "which DLL" problem.

But, when I do: >>> import mypackage.myextension, then myextension.pyd gets loaded, as does my copy of runtime.dll. When I then do >>> import yourpackage.yourextension, Windows sees that we already have a copy of runtime.dll, so doesn't load your copy of runtime.dll. If these two DLLs are not the same, this can cause nasty crashes, and crashes that depend on the order in which the two pyd extensions get loaded.

We also need to make sure that the DLLs we need can be found when the Python extension gets loaded.

When Python loads an extension, it does it using the Windows LoadLibraryEx call, like this:

hDLL = LoadLibraryEx(extension_path, NULL,
                     LOAD_WITH_ALTERED_SEARCH_PATH);

See: http://hg.python.org/cpython/file/3a1db0d2747e/Python/dynload_win.c#l195

LOAD_WITH_ALTERED_SEARCH_PATH causes Windows to look for DLLs first in the directory containing the extension (directory containing extension_path) (see DLL search path)

Specifically, if you do:

>>> import mypackage.myextension

and myextension.pyd is in c:\Python27\Lib\site-packages\mypackage, and myextension.dll loads runtime.dll, then Windows will look for runtime.dll first in c:\Python27\Lib\site-packages\mypackage.

A common situation is that you want to put all needed DLLs in one directory, but there are extensions loading these DLLs are all over the file tree.

For example, default builds of scipy using Mingw-w64 will depend on gcc and gfortran run-time DLLs. There will be extensions needing these DLLs in several places in the scipy package tree. In that case we may want to have a single directory called dlls in the Scipy tree containing these DLLs. We can put this directory on the DLL search path with DLL path tricks used in this ctypes code fragment (thanks to Steve Dower for the fragment, Carl Kleffner for finding it).

Some unsorted links that also seemed useful:

Numpy and Windows DLLs

Add a custom footer
Add a custom sidebar
Clone this wiki locally