12 April 2024
Nuitka Release 2.2
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release focused on compatibility and significant optimization progress for loops, such as list operations within. The main line of change is to be able to support Python 3.12 in the next release. Standalone: Added support for Standalone: Make ArchLinux: Added platform linker option to be usable with their current Arch Python package. Fixed in 2.1.1 already. Fix, Fix, Standalone: Added support for Standalone: Added support for Standalone: Updated Fix, PySide6 slots failed to be moved between threads. For that we need to make function renames visible in the owning class as well. Fixed in 2.1.3 already. Standalone: Added support for Standalone: Handle “string” import errors of Fix, the Standalone: Added support for Standalone: Added support for newer Standalone: Added support for newer Standalone: Fix, Onefile: Respect Python2: Some cases of Fix, optimized Standalone: Added missing implicit dependencies for Standalone: Added support for Standalone: Added DLLs for Standalone: Added missing metadata of Standalone: Added support for newer Standalone: Fix, older Fix, Standalone: Added data files for Standalone: Added data files for Standalone: Fix, the standalone standard library scan was not fully ignoring git folders, subfolders were still looked at and could cause issues. Fixed in 2.1.5 already. Standalone: Added support for newer Standalone: Add support for newer Scons: Fix, when locating binaries, do not use directories but only files. A directory on Windows: Fix, by default, scan only for Fix, the Standalone: Added support for more parts of Windows: Fix working with UNC paths and re-parse points at compile time. Now Nuitka should work with mapped and even unmapped to drive paths like Windows: Make sure the download path is an external use path in scons as well, otherwise the home directory could be an unusable path for MinGW64, causing it not to find files. Standalone: Added missing dependency of Python3.5+: Add support for using dictionary un-packings in class declarations. That is a rarely used in actual Python code, but was found missing by tests recently. Python3.11: Fix, code objects Anaconda: Fix, must not consider the Anaconda Fix, cannot trust dynamic hard modules as much otherwise, macOS: Fix, on newer macOS the Fix, support for newer Standalone: Added required data files for Standalone: Added required data files for Standalone: Added required data files for Standalone: Fixup Fix, need to make parent module usages more explicit. Otherwise, plugin mechanisms like Fix, the Added experimental support for Python 3.12, this is passing basic tests, but known to crash a lot at run-time still, you are recommended to use pre-releases of Nuitka, as official support is not going to happen before 2.3 release. Standalone: Added support for With preserved source code of decorated functions and we can provide it at run-time to For Nuitka Package Configuration, we now have For the experimental Added new kind of warning of Zig: Added support for For the Use Lower Recognize exact type shapes of loop variables where possible. This enables appends to list to be optimized to their dedicated nodes among other things, with those often being a lot faster than generic code. This speeds up e.g. list Optimization: Have dedicated helper for ArchLinux: Enable static libpython by default, it is usable indeed. Added in 2.1.2 already. Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Avoid using Anti-Bloat: Avoid testing packages for newer Optimization: Avoid recompiling Avoid packages owned by Nuitka plugins in Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Avoid Anti-Bloat: Recognize Faster code generation due to enhancements in how identifiers are cached for module names and the indentation codes. Optimization: Handle Anti-Bloat: Avoid Anti-Bloat: Remove testing module usage when Anti-Bloat: Avoid Anti-Bloat: Avoid UI: Catch conflicts between data files and EXE/DLLs/extension module filenames. Previously, you could overwrite binaries with data files, but that is now rejected as an explicit error. Onefile: Avoid using the program name without suffix inside the dist folder, as that avoids collisions with data file directories of the same name, e.g., if the package and main binary have the same name, they would clash previously, but adding a UI: Don’t force That made it hard for users, who will be surprised to see UI: Catch wrong values for UI: Nicer usage name when invoked with The recommended form of invocation of Nuitka should not have an ugly invocation reference mentioning UI: Reorder options for the plugins group to be more readable. Plugins: Remove obsolete plugins from standard plugin documentation. Removed in 2.1.4 already. UI: The Windows release was coming from the compiling Python and as such wrong, for example, Windows 11 always showed up as Windows 10, and some older versions of Python didn’t know Windows 10, yet, so this could be confusing in issue analysis. UI: Do not warn about static libpython for Python debug mode compilation. It is misleading as often it doesn’t work for that configuration, and it’s only a distraction since debugging Python reference counts is not about performance. Changed in 2.1.4 already. UI: Catch newlines in spec values. They break code C code generation potentially; they also are likely copy&paste mistakes that won’t do what the user expects. Added in 2.1.4 already. Quality: Updated to the latest version of black. Quality: Fix, Debugging: Generate Scons debug script It can serve to quickly re-execute a Scons compilation without re-executing Nuitka again. This is best used where there is no Python level change but only C changes and no expectation of producing a usable result. Because no post-processing is applied, and as a consequence this is not usable to produce binaries that work. In the future, we might expand this to be able to run post-processing still. Debugging: Disabling all freelists is now honored for more code, tuples and empty dictionaries as well. UI: Add macOS version to help output, which is sometimes vital for issue analysis. Reports: Add the OS release to reports as well. Reports: Exclude parent path imports from compilation reports for module usages that are found and end up not being excluded. Watch: Reporting more problems, catching more errors, and adding the ability to create PRs from changes. However, it does not yet do it automatically. Visual Code: Have plugins C files in the include path as well. Tests: Fix, cannot assume Run commercial code signing test only on Windows. Allow for standalone testing file access to the Azure agent folders. For tests on Azure, it’s like the home directory. Make sure optimization tests are named to make it clear that they are tests. Remove useless The JIT mechanism added for The road to Python 3.12 is not fully complete, but the end feels closer now, and the subsequent release hopefully will add the official support for it.Bug Fixes
pypdfium2 package. Fixed in 2.1.1 already.cefpython3 work on Linux. Fixed in 2.1.1 already.ctypes.CDLL optimization used a misspelled argument name for use_last_error, such that keyword argument calls were statically optimized into TypeError at compile-time. Fixed in 2.1.1 already.list.insert was not properly annotating exceptions. Raises by producing the inserted value raised or the index was not annotated and, therefore, could fail to be caught locally. Fixed in 2.1.1 already.selenium package. Fixed in 2.1.2 already.hydra package. Fixed in 2.1.2 already.dotenv workaround for newer version. Fixed in 2.1.3 already.win32com.server.register. Fixed in 2.1.3 already.uvicorn gracefully. Fixed in 2.1.3 already.dill-compat plugin needs also needs to expose the compiled type names as built-ins for the pickle module to find them.gruut package. Fixed in 2.1.3 already.toga to also include toga_winforms metadata. Fixed in 2.1.3 already.tensorflow package. Fixed in 2.1.4 already.matplotlib needs to emit a dependency on the backend to be included. Otherwise it could be missing at run-time in some cases. Fixed in 2.1.4 already.XDG_CACHE_HOME variable on non-Windows platforms. Some users might configure that to not to be ~/.cache, respect that. Fixed in 2.1.4 already.list.insert were not properly handling all index types. Fixed in 2.1.4 already.list.remove failed to handle tuple arguments properly. Removing tuple values from lists could cause errors. Fixed in 2.1.4 already.pyarrow.datasets. Fixed in 2.1.4 already.dask.dataframe module. Fixed in 2.1.4 already.tensorrt_libs package. Fixed in 2.1.4 already.numpy for xarray package. Fixed in 2.1.4 already.scipy. Fixed in 2.1.5 already.gcc could give warning about C code to work with PYTHONPATH which caused build errors on older systems. Fixed in 2.1.5 already.locals representing nodes could not be cloned, and as a result, some code re-formulations failed to compile in try constructs. Fixed in 2.1.5 already.names package. Fixed in 2.1.5 already.randomname package. Fixed in 2.1.5 already.transformers. Fixed in 2.1.5 already.bitsandbytes. Fixed in 2.1.5 already.PATH that was named gcc could be mistaken to be a gcc binary causing errors. Fixed in 2.1.6 already..bin and .exe binaries for Nuitka Package Configuration EXE dependency patterns. This was the intended value, but it had not taken effect yet. Fixed in 2.1.6 already.__compiled__.containing_dir should be an absolute path. For it to be usable after a change of directory is done by the program that is required. Fixed in 2.1.6 already.networkx package. Fixed in 2.1.6 already.\\some-hostname\unc-test as they are common in some VM setups.sspilib that prevented requests-ntlm from working on Windows.co_qualname attribute was not actually the qualified name, but the same as co_name only.lib directory as a system directory, because then those DLLs that are not included.huggingface_hub.utils.tqdm ended up being a module and not the class it’s supposed to be.libc++ and libz DLLs cannot be found anymore, we need to ignore that actively as our code insists on full resolution to catch bugs.zaber_motion was not really working.pyviz_comms.panel package.bokeh package.scipy for Anaconda.no-follow from a parent module cannot affect its child modules, as they can end up being followed to only after them.dill-compat plugin in module mode cannot assume the main module name to be the one from compile time, need to look the actual one up at runtime.New Features
tensorflow.function JITtensorflow JIT so it can do its tracing executions.change_class similar to change_function to replace a full class definition with something else, this can be used to modify classes to become stubs or even unusable.@pyqtSlot decorator, we also should handle the @asyncSlot the same way. Added in 2.1.1 already.plugin category and use it in the Nuitka Package Configuration to inform matplotlib users to select a GUI backend via plugin selection. Added in 2.1.4 already.zig as CC value. Due to it not supporting C11 fully yet, we need to use the C++ workaround and cannot compile for Python 3.11 or higher yet.__compiled__ value, we now have a __compiled__.main that is the name of the compiled module. For modules, Nuitka determines this at run time; in other modes, it is the name of the main module.Optimization
set specific API in contains tests, rather than generic sequence one.value in something tests for known set and list values to use frozenset and tuple respectively.append tests by a significant amount.list.remove, such that it is not using a Python DLL call where that is slow.unittest usage in antlr package.IPython in celery package. Added in 2.1.2 already.setuptools in transformers package for more modules. Added in 2.1.3 already.tensorflow package as well. Added in 2.1.4 already.azure package which is not performance relevant. Added in 2.1.4 already.matploblib backends unless the corresponding plugin is actually active. Added in 2.1.4 already.setuptools in deepspeed package. Added in 2.1.4 already.setuptools in transformers package. Added in 2.1.4 already.scipy usage causing torch or cupy usage. Added in 2.1.4 already.keras testing modules as unittest bloat.no_docstrings issue for torio package.IPython from imgui_bundle package.dask is used.unitest usage in tf_keras package as well.IPython from bokeh package.Organizational
.bin suffix to the binary avoids that entirely.{VERSION} in specs to be resolved to four digits.1.0 become 1.0.0.0 when that is only needed for Windows version information really.--jobs value sooner, negative and non-integer values error exit immediately. Added in 2.1.1 already.python -m nuitka__main__.py instead put the python -m nuitka notion there.isort and black can corrupt outputs, catch that.Tests
setuptools to be installed, some RPM based systems don’t have it.Cleanups
--execute-with-pythonpath option, we don’t use that anymore at all.Summary
tensorflow should be possible to generalize and will be applied to other JITs, like numba and others in the future as well.