27 November 2024
Nuitka Release 2.5
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release focused on Python 3.13 support, but also on improved compatibility, made many performance optimizations, enhanced error reporting, and better debugging support. Windows: Fixed a regression in onefile mode that incorrectly handled program and command line paths containing spaces. Fixed in 2.4.4 already. Windows: Corrected an issue where console output handles were being opened with closed file handles. Fixed in 2.4.2 already. Standalone: Restored the ability to use trailing slashes on the command line to specify the target directory for data files on Windows. Fixed in 2.4.2 already. Compatibility: Fixed a parsing error that occurred with relative imports in Modules: Ensured that extension modules load correctly into packages when using Python 3.12. Fixed in 2.4.4 already. Windows: Improved command line handling for onefile mode to ensure full compatibility with quoting. Fixed in 2.4.4 already. Data Directories: Allowed the use of non-normalized paths on the command line when specifying data directories. Fixed in 2.4.5 already. Python 3.11+: Fixed an issue where importlib_metadata: Improved compatibility with Plugins: Fixed a crash that occurred when using the Scons: Improved error tolerance when reading Scons: Prevented the C standard option from being applied to C++ compilers, resolving an issue with the splash screen on Windows when using Clang. Fixed in 2.4.8 already. macOS: Enhanced handling of DLL self-dependencies to accommodate cases where DLLs use both Compatibility: Fixed a memory leak that occurred when using MSYS2: Excluded the Python 3.10+: Fixed a crash that could occur when a MSYS2: Implemented a workaround for non-normalized paths returned by Python 3.12: Resolved an issue where Nuitka’s constant code was triggering assertions in Python 3.12.7. Fixed in 2.4.10 already. UI: Ensured that the Windows: Prevented encoding issues with CMD files used for accelerated mode on Windows. Standalone: Improved the standard library scan to avoid assuming the presence of specific files, which might have been deleted by the user or a Python distribution. Compatibility: Added Compatibility: Backported the error message change for Compatibility: Fixed an issue where calling built-in functions with keyword-only arguments could result in errors due to incorrect argument passing. Compatibility: Fixed reference leaks that occurred when using Windows: Prioritized relative paths over absolute paths for the result executable when absolute paths are not file system encodable. This helps address issues related to non-ASCII short paths on some Chinese systems. Compatibility: Improved compatibility with C extensions by handling cases where the attribute slot is not properly implemented, preventing potential segfaults. Compatibility: Prevent the leakage of Compatibility: Fixed an issue where matching calls with keyword-only arguments could lead to incorrect optimization and argument passing errors. Compatibility: Corrected the handling of iterators in for loops to avoid assuming the presence of slots, preventing potential issues. macOS: Added support for cyclic DLL dependencies, where DLLs have circular references. Compatibility: Ensured the use of updated expressions during optimization phase for side effects to prevent crashes caused by referencing obsolete information. Python 3.10+: Fixed a crash that could occur in complex cases when re-formulating Python 3.4-3.5: Corrected an issue in Nuitka’s custom Python 3.10+: Ensured that Plugins: Added a data file function that avoids loading packages, preventing potential crashes caused by incompatible dependencies (e.g., Compatibility: Ensured that Nuitka’s package reader closes data files after reading them to prevent resource warnings in certain Python configurations. Standalone: Exposed Accelerated Mode: Enabled the Compatibility: Prevented resource warnings for unclosed files when trace outputs are sent to files via command line options. Compatibility: Enabled the use of Plugins: Fixed an issue in the Python 3.12 (Linux): Enhanced static Standalone: Further improved the standard library scan to avoid assuming the presence of files that might have been manually deleted. UI: Fixed the arcade: Improved standalone configuration for the license-expression: Added a missing data file for the pydantic: Included a missing implicit dependency required for deprecated decorators in the spacy: Added a missing implicit dependency for the trio: Updated standalone support for newer versions of the tensorflow: Updated standalone support for newer versions of the pygame-ce: Added standalone support for the toga: Added standalone support for newer versions of the django: Implemented a workaround for a PySide6: Improved standalone support for polars: Added a missing dependency for the django: Enhanced handling of cases where the win32ctypes: Included missing implicit dependencies for arcade: Added a missing data file for the PySide6: Allowed driverless-selenium: Added standalone support for the tkinterdnd2: Updated standalone support for newer versions of the kivymd: Updated standalone support for newer versions of the gssapi: Added standalone support for the azure.cognitiveservices.speech: Added standalone support for the mne: Added standalone support for the fastapi: Added a missing dependency for the pyav: Updated standalone support for newer versions of the py_mini_racer: Added standalone support for the keras: Improved standalone support for transformers: Updated standalone support for newer versions of the win32com.server.register: Updated standalone support for newer versions of the Python 3.12+: Added support for cv2: Enabled automatic scanning of missing implicit imports for the lttbc: Added standalone support for the win32file: Added a missing dependency for the kivy: Fixed an issue where the paddleocr: Added missing data files for the playwright: Added standalone support for the PySide6: Allowed Python 3.13: Added experimental support for Python 3.13. Warning Python 3.13 support is not yet recommended for production use due to limited testing. On Windows, only MSVC and ClangCL are currently supported due to workarounds needed for incompatible structure layouts. UI: Introduced a new Note The Windows: Added a new UI: Added the Modules: Enabled the generation of type stubs ( Note Nuitka also adds implicit imports to compiled extension modules, ensuring that dependencies are not hidden. Plugins: Changed the data files configuration to a list of items, allowing the use of Onefile: Removed the MSVC requirement for the splash screen in onefile mode. It now works with MinGW64, Clang, and ClangCL. Done for 2.4.8 already. Reports: Added information about the file system encoding used during compilation to help debug encoding issues. Windows: Improved the Distutils: Added the ability to disable Nuitka in Distutils: Added support for commercial file embedding in Linux: Added support for using uninstalled self-compiled Python installations on Linux. Plugins: Enabled the Runtime: Added a new Reports: Added a list of DLLs that are actively excluded because they are located outside of the PyPI package. Plugins: Allowed plugins to override the compilation mode for standard library modules when necessary. Performance: Implemented experimental support for “dual types”, which can significantly speed up integer operations in specific cases (achieving speedups of 12x or more in some very specific loops). This feature is still under development but shows promising potential for future performance gains, esp. when combined with future PGO (Profile Guided Optimization) work revealing likely runtime types more often and more types being covered. Performance: Improved the speed of module variable access. For Python 3.6 to 3.10, this optimization utilizes dictionary version tags but may be less effective when module variables are frequently written to. For Python 3.11+, it relies on dictionary key versions, making it less susceptible to dictionary changes but potentially slightly slower for cache hits compared to Python 3.10. Performance: Accelerated string dictionary lookups for Python 3.11+ by leveraging knowledge about the key and the module dictionary’s likely structure. This also resolves a previous TODO item, where initial 3.11 support was not as fast as our support for 3.10 was in this domain. Performance: Optimized module dictionary updates to occur only when values actually change, improving caching efficiency. Performance: Enhanced exception handling by removing bloat in the abstracted differences between Python 3.12 and earlier versions. This simplifies the generated C code, reduces conversions, and improves efficiency for all Python versions. This affects both C compile time and runtime performance favorably and solves a huge TODO for Python 3.12 performance. Performance: Removed the use of CPython APIs calls for accessing exception context and cause values, which can be slow. Performance: Utilized Nuitka’s own faster methods for creating Performance: Implemented a custom variant of Windows: Aligned with CPython’s change in reference counting implementation on Windows for Python 3.12+, which improves performance with LTO (Link Time Optimization) enabled. Optimization: Expanded static optimization to include unary operations, improving the handling of number operations and preparing for full support of dual types. Optimization: Added static optimization for Performance: Passed the exception state directly into unpacking functions, eliminating redundant exception fetching and improving code efficiency. Performance: Introduced a dedicated helper for unpacking length checks, resulting in faster and more compact code helping scalability as well. Performance: Generated more efficient code for raising built-in exceptions by directly creating them through the base exception’s Performance: Optimized exception creation by avoiding unnecessary tuple allocations for empty exceptions. This hack avoids hitting the memory allocator as much. Performance: Replaced remaining uses of Code Generation: Replaced implicit exception raise nodes with direct exception creation nodes for improved C code generation. Windows: Aligned with CPython’s change in managing object reference counters on Windows for Python 3.12+, improving performance with LTO enabled. Performance: Removed remaining CPython API calls when creating Windows: Avoided scanning for DLLs in the Windows: Updated to a newer MinGW64 version from 13.2 to 14.2 for potentially improved binary code generation with that compiler. Code Size: Reduced the size of constant blobs by avoiding module-level constants for the global values Code Generation: Improved code generation for variables by directly placing Optimization: Statically optimized the Performance: Limited the use of Optimization: Implemented a faster check for non-frame statement sequences by decoupling frames and normal statement sequences and using dedicated accessors. This improves performance during the optimization phase. Prevented the inclusion of Avoided the use of Removed Prevented attempts to include both Python 2 and Python 3 code for the Enhanced handling for the Allowed Avoided Avoided Avoided Updated anti-bloat measures for newer versions of the Avoided Allowed Allowed Avoided test modules when using the Treated the use of the Development Environment: Added experimental support for a devcontainer to the repository, providing an easier way to set up a Linux-based development environment. This feature is still under development and may require further refinement. Issue Reporting: Clarified the issue reporting process on GitHub, emphasizing the importance of testing reproducers against Python first to ensure the issue is related to Nuitka. Issue Reporting: Discouraged the use of UI: Improved the clarity of help message of the option for marking data files as external, emphasizing that files must be included before being used. UI: Added checks to the Qt plugins to ensure that specified plugin families exist, preventing unnoticed errors. UI: Implemented heuristic detection of terminal link support, paving the way for adding links to options and groups in the command line interface. UI: Removed obsolete caching-related options from the help output, as they have been replaced by more general options. Plugins: Improved error messages when retrieving information from packages during compilation. Quality: Implemented a workaround for an Quality: Updated GitHub actions to use Quality: Updated to the latest version of Release Process: Updated the release script tests for Debian and PyPI to use the correct runner names. (Changed in 2.4.1 already. UI: Disabled progress bar locking, as Nuitka currently doesn’t utilize threads. UI: Added heuristic detection of terminal link support and introduced an experimental terminal link as a first step towards a more interactive command line interface. Debugging: Fixed a crash in the “explain reference counts” feature that could occur with unusual Debugging: Included reference counts of tracebacks when dumping reference counts at program end. Debugging: Added assertions and traces to improve debugging of input/output handling. Quality: Added checks for configuration module names in Nuitka package configuration to catch errors caused by using filenames instead of module names. UI: Removed obsolete options controlling cache behavior, directing users to the more general cache options. Scons: Ensured that the Distutils: Added the Dependency Management: Started experimental support for downloading Nuitka dependencies like Added Python 3.13 to the GitHub Actions test matrix. Significantly enhanced construct-based tests for clearer results. The new approach executes code with a boolean flag instead of generating different code, potentially leading to the removal of custom templating. Removed the Fixed an issue where the test runner did not discover and use Python 3.12+, resulting in insufficient test coverage for those versions on GitHub Actions. Ensured that the Set up continuous integration with Azure Pipelines to run Nuitka tests against the factory branch on each commit. Enforced the use of static Improved the robustness of many construct tests, making them less sensitive to unrelated optimization changes. Removed a test that was only applicable to Nuitka Commercial, as it was not useful to always skip it in the standard version. Commercial tests are now also recognized by their names. Added handling for segmentation faults in Prevented resource warnings for unclosed files in a reflected test. WASI: Corrected the signatures of C function getters and setters for compiled types in WASI: Improved code cleanliness by adhering to Code Generation: Fixed a regression in code generation that caused misaligned indentation in some cases. Code Formatting: Changed some code for identical formatting with Caching: Enforced proper indentation in Nuitka cache files stored in JSON format. Code Cleanliness: Replaced checks for Python 3.4 or higher with checks for Python 3, simplifying the code and reflecting the fact that Python 3.3 is no longer supported. Code Cleanliness: Removed remaining Python 3.3 specific code from frame templates. Code Cleanliness: Performed numerous spelling corrections and renamed internal helper functions for consistency and clarity. Plugins: Renamed the Plugins: Moved the This a major release that brings support for Python 3.13, relatively soon after its release. Our plugin system and Nuitka plugin configuration was used a lot for support of many more third-party packages, and numerous other enhancements in the domain of avoiding bloat. This release focuses on improved compatibility, new break through performance optimizations, to build on in the future, enhanced error reporting, and better debugging support.Bug Fixes
.pyi files, which could affect some extension modules with available source code. Fixed in 2.4.3 already.inspect module functions could raise StopIteration when examining compiled functions on the stack. Fixed in 2.4.5 already.importlib_metadata by handling cases where it might be broken, preventing potential compilation crashes. Fixed in 2.4.5 already.no_asserts YAML configuration option. Fixed in 2.4.6 already.ccache log files to prevent crashes on Windows caused by non-ASCII module names or paths. Fixed in 2.4.11 already..so and .dylib extensions for self-references. Fixed in 2.4.8 already.deepcopy on compiled methods. Fixed in 2.4.9 already.bin directory from being considered a system DLL folder when determining DLL inclusion. Fixed in 2.4.9 already.match statement failed to match a class with arguments. Fixed in 2.4.9 already.os.path.normpath in MSYS2 Python environments. Fixed in 2.4.11 already.--include-package option includes both Python modules and extension modules that are sub-modules of the specified package. Fixed in 2.4.11 already.suffix, suffixes, and stem attributes to Nuitka resource readers to improve compatibility with file objects.yield from used at the module level, using dynamic detection instead of hardcoded text per version.list.insert and list.index with 2 or 3 arguments.sys.frozen when using the multiprocessing module and its plugin, resolving a long-standing TODO and potentially breaking compatibility with packages that relied on this behavior.match statements.PyDict_Next implementation that could lead to incorrect results in older Python 3 versions.AttributeError is raised with the correct keyword arguments, avoiding a TypeError that occurred previously.numpy versions).setuptools contained vendor packages in standalone distributions to match the behavior of the setuptools package.django module parameter in accelerated mode to correctly detect used extensions.xmlrpc.server without requiring the pydoc module.anti-bloat configuration where change_function and change_classes ignored “when” clauses, leading to unintended changes.libpython handling for Linux. Static libpython is now used only when the inline copy is available (not in official Debian packages). The inline copy of hacl is used for all Linux static libpython uses with Python 3.12 or higher.--include-raw-dir option, which was not functioning correctly. Only the Nuitka Package configuration was being used previously.Package Support
arcade package. Added in 2.4.3 already.license-expression package in standalone distributions. Added in 2.4.6 already.pydantic package to function correctly in standalone mode. Fixed in 2.4.5 already.spacy package in standalone distributions. Added in 2.4.7 already.trio package. Added in 2.4.8 already.tensorflow package. Added in 2.4.8 already.pygame-ce package. Added in 2.4.8 already.toga package on Windows. Added in 2.4.9 already.django debug feature that attempted to extract column numbers from compiled frames. Added in 2.4.9 already.PySide6 on macOS by allowing the recognition of potentially unusable plugins. Added in 2.4.9 already.polars package in standalone distributions. Added in 2.4.9 already.django settings module parameter is absent in standalone distributions. Added in 2.4.9 already.win32ctypes modules on Windows in standalone distributions. Added in 2.4.9 already.arcade package in standalone distributions. Added in 2.4.9 already.PySide6 extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed. Added in 2.4.11 already.driverless-selenium package. Added in 2.4.11 already.tkinterdnd2 package. Added in 2.4.11 already.kivymd package. Added in 2.4.11 already.gssapi package. Added in 2.4.11 already.azure.cognitiveservices.speech package on macOS.mne package.fastapi package in standalone distributions.pyav package.py_mini_racer package.keras by extending its sub-modules path to include the keras.api sub-package.transformers package.win32com.server.register package.distutils in setuptools for Python 3.12 and later.cv2 package in standalone distributions.lttbc package.win32file package in standalone distributions.kivy clipboard was not working on Linux due to missing dependencies in standalone distributions.paddleocr package in standalone distributions.playwright package with a new plugin.PySide6 extras to be optional on macOS in standalone distributions, preventing complaints about missing DLLs when they are not installed.New Features
--mode selector to replace the options --standalone, --onefile, --module, and --macos-create-app-bundle.app mode creates an app bundle on macOS and a onefile binary on other operating systems to provide the best deployment option for each platform.hide choice for the --windows-console-mode option. This generates a console program that hides the console window as soon as possible, although it may still briefly flash.--python-flag=-B option to disable the use of bytecode cache (.pyc) files during imports. This is mainly relevant for accelerated mode and dynamic imports in non-isolated standalone mode..pyi files) for compiled modules using an inline copy of stubgen. This provides more accurate and informative type hints for compiled code.when conditions for more flexible control. Done in 2.4.6 already.attach mode for --windows-console-mode when forced redirects are used.pyproject.toml builds using the build_with_nuitka setting. This allows falling back to the standard build backend without modifying code or configuration. This setting can also be passed on the command line using --config-setting.distutils packages.matplotlib plugin to react to active Qt and tkinter plugins for backend selection.original_argv0 attribute to the __compiled__ value to provide access to the original start value of sys.argv[0], which might be needed by applications when Nuitka modifies it to an absolute path.Optimization
int and long values, avoiding slower CPython API calls._PyGen_FetchStopIterationValue to avoid CPython API calls in generator handling, further improving performance on generators, coroutines and asyncgen.os.stat and os.lstat calls.new method instead of calling them as functions. This can speed up some things by a lot.PyTuple_Pack with Nuitka’s own helpers to avoid CPython API calls.int values in various parts of the code, including specialization code, helpers, and constants loading.PATH environment variable when they are not intended to be used from the system. This prevents potential crashes related to non-encodable DLL paths and makes those scans faster too.-1, 0, and 1.NameError exceptions into the thread state when raised, making for more compact C code.sys.ps1 and sys.ps2 values to not exist (unless in module mode), potentially enabling more static optimization in packages that detect interactive usage checking them.tqdm locking to no-GIL and Scons builds where threading is actively used.Anti-Bloat
importlib_metadata for the numpy package. Added in 2.4.2 already.dask in the pandera package. Added in 2.4.5 already.numba for newer versions of the shap package. Added in 2.4.6 already.aenum package, avoiding SyntaxError warnings. Added in 2.4.7 already.sympy package. Added in 2.4.7 already.pydoc for the pyqtgraph package. Added in 2.4.7 already.pytest in the time_machine package. Added in 2.4.9 already.pytest in the anyio package.numba in the pandas package.torch package with increased coverage.pygame.tests and cv2 for the pygame package.unittest in the absl.testing package.setuptools in the tufup package.bsdiff4 package.wheel module the same as using the setuptools package.Organizational
--deployment in issue reports, as it hinders the automatic identification of issues, that should be the first thing to remove.isort bug that prevented it from handling UTF-8 comments.clang-format-20.black for code formatting.dict values mistaken for modules.CC environment variable is used consistently for --version and onefile bootstrap builds, as well as the Python build, preventing inconsistencies in compiler usage and outputs.compiled-package-hidden-by-package mnemonic for use in distutils to handle the expected warning when a Python package is replaced with a compiled package and the Python code is yet to be deleted.ordered-set. This feature is not yet ready for general use.Tests
2to3 conversion code from the test suite, as it is being removed from newer Python versions. Tests are now split with version requirements as needed.compare_with_cpython test function defaults to executing the system’s Python interpreter instead of relying on the PYTHON environment variable.libpython for construct-based tests to eliminate DLL call overhead and provide more accurate performance measurements.distutils test cases, providing debug output for easier diagnosis of these failures.Cleanups
WASI to ensure they match the calling conventions. Casts are now performed locally to the compiled types instead of in the function signature. Call entries also have the correct signature used by Python C code.PyCFunction signatures in WASI.clang-format-20 to eliminate differences between the new and old versions.get_module_directory helper function in the Nuitka Package configuration to remove the leading underscore, improving readability.numexpr.cpuinfo workaround to the appropriate location in the Nuitka Package configuration, resolving an old TODO item.Summary