Nuitka Home | Nuitka Home

This continues TWN #3 where I explained what is all about.

Good news is, at the time Python2 generators were largely working with the new ways, in the mean time not only did all of the Python 2.7 test suite pass with goto generators, also did the Python 3.4 test suite, i.e. also the yield from is working with it.

The way it was done is to set m_yieldfrom in generators, and then to enter a state, where the code will only be resumed, when that sub-generator that currently it is yielding from, is finished. That makes it very much like normal yield. In fact, code generation is hardly different there.

Since the whole purpose is to get rid of make/get/setcontext, the next stop is coroutines. They have async for, async with and await but at the end of the day, the implementation comes down to yield from really with only a lot of sugar applied.

Right now, I am debugging "goto coroutines". It's hard to tell when it will be finished, and then asyncgen will be waiting still.

This is easily the largest change in a long time, esp. due to the heap storage changes that I already discussed. One this is finished, I expect to turn towards C types with relative ease.

Anthony Shaw took on Tox and Nuitka and created a plugin that allows using Nuitka. I am still wrapping my head around these things. It's only a proof of concept yet. I will give it more coverage in the future.

Follow me on twitter if you like, I will:

Follow @kayhayen

So there have even more hotfixes. One addresses memory leaks found with the yield from while I was adding tests. Usually if I encounter an old issue that has a small fix, that is what I do, push out a hotfix using the git flow model. Also nested namespace packages for Python3, those are the ones without a were not working after the original directory was removed, and that got fixed.

And right now, I have hotfixes for frames close method, which apparently was never updated to work properly for coroutines and asyncgen. That is going to be in the next hotfix.

So the heap storage seems pretty complete now, and goto generators are on the final stretch. As always, things feel right around the corner. But it's unclear how much longer I will have to debug. I am pretty sure the bare work of doing asyncgen is going to be low. Debugging that too then, that is the hard part.

A new release seems justified, but I kind of do not want to make it without that major new code used. Because apparently during the debugging, I tend to find issues that need hotfixes, so I will wait for the goto generator work to finish.

This is working out well so far. I think driving more attention at the things that are going on can only be good. Also to explain will always help. It also kind of motivates me a bit.

Also as part of my communications offensive, I am using my Twitter account more regularily. I used to highlight important fixes, or occasionally releases of some importance there. I will continue to do only important stuff there, but with more regularity.

And I noticed in the past, even when I do not post, followers makes me happy. So here you go:

Follow @kayhayen

This continues TWN #2 where I promised to speak more of it, and this is the main focus of my work on Nuitka right now.

Brief summary, context switches were how this was initially implemented. The main reason being that for C++ there never was going to be a way to save and restore state in the middle of an expression that involves constructors and destructors.

Fast forward some years, and C-ish entered the picture. No objects are used anymore, and Nuitka is purely C11 now, which has convinience of C++, but no objects. Instead goto is used a lot already. So everytime an exception occurs, a goto is done, every time a branch is done, a loop exit or continue, you get it, another goto.

But so far, all Python level variables of a frame live on that C stack still, and the context switch is done with functions that swap stack. That is fast, but the imporant drawback is that it takes more memory. How deep of a stack will we need? And we can use really many, if you imagine a pool of 1000 coroutines, that quickly become impossible to deal with.

So, the new way of doing this basically goes like this:

def g():
 yield 1
 yield 2

This was some far becoming something along this lines:

PyObject *impl_g( NuitkaGenerator *generator )
 YIELD( const_int_1 );
 YIELD( const_int_2 );

 PyErr_SetException( StopIteration );
 return NULL;

The YIELD in there was basically doing the switching of the stacks and for the C code, it looked like a normal function call.

In the new approach, this is done:

PyObject *impl_g( NuitkaGenerator *generator )
 switch( generator->m_resume_point )
 case 1: goto resume_1;
 case 2: goto resume_2;

 generator->m_yielded = const_int_1;
 generator->resume_point = 1
 return NULL;

 generator->m_yielded = const_int_2;
 generator->resume_point = 2
 return NULL;

 PyErr_SetException( StopIteration );
 return NULL;

As you can see, the function has an initial dispatcher. Resume point 0 means we are starting at the top. Then every yield results in a function return with an updated resume point.

I experimented with this actually a long time ago, and experimental code was the result that remained in Nuitka. The problem left to solve was to store the variables that would normally live on the stack, in a heap storage. That is what I am currently working on.

This leads me to "heap storage", which is what I am currently working on and will report on next week. Once that is there, goto generators can work, and will become the norm. Until then, I am refactoring a lot to get accesses to variable go through proper objects that know their storage locations and types.

So there have been 2 more hotfixes. One was to make the enum and __new__ compatibility available that I talked about last week in TWN #2 <./nuitka-this-week-2.html#python3-enumerators> coupled with a new minor things.

And then another one, actually important, where Python3 __annotations__ by default was the empty dictionary, but then could be modified, corrupting the Nuitka internally used one severely.

Right now I have on factory another fix for nested namespace packages in Python3 and that might become another hotfix soon.

As you know, I am following the git flow model, where it's easy to push out small fixes, and just those, on top of the last release. I tend to decide based on importance. However, I feel that with the important fixes in the hotfixes now, it's probably time to make a full release, to be sure everybody gets those.

Finishing heap storage is my top priority right now and I hope to complete the refactorings necessary in the coming week. I will also talk about how it also enables C types work next week.

Until next week then!

As discussed last week in TWN #1 this is a new series that I am using to highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

I made the first release with official 3.7 support, huge milestone in terms of catching up. Generic classes posed a few puzzles, and need more refinements for error handling, but good code works now.

The class creation got a bit more complex, yet again, which will make it even hard to know the exact base classes to be used. But eventually we will manage to overcome this and statically optimize that.

Building the MSI files for Nuitka ran into a 3.7.0 regression of CPython failing to build them, that I reported and seems to be valid bug of theirs.

So they will be missing for some longer time. Actually I wasn't so sure if they are all that useful, or working as expected for the runners, but with the -m nuitka mode of execution, that ought to be a non-issue. so it would be nice to keep them for those who use them for deployment internally.

I have a change here. This is going to be a draft post until I publish it, so I might the link, or mention it on the list, but I do not think I will wait for feedback, where there is not going to be all that much.

So I am shooting this off the web site.

This is an exciting field of work, that I have been busy with this week. I will briefly describe the issue at hand.

So generators in Python are more generally called coroutines in other places, and basically that is code shaking hands, executing resuming in one, handing back a piece of data back and forth.

In Python, the way of doing this is yield and more recently yield from as a convience way for of doing it in a loop in Python3. I still recall the days when that was a statement. Then communication was one way only. Actually when I was still privately doing Nuitka based on then Python 2.5 and was then puzzled for Python 2.6, when I learned in Nuitka about it becoming an expression.

The way this is implemented in Python, is that execution of a frame is simply suspended, and another frame stack bytecode is activated. This switching is of course very fast potentially, the state is already fully preserved on the stack of the virtual machine, which is owned by the frame. For Nuitka, when it still was C++, it wasn't going to be possible to interrupt execution without preserving the stack. So what I did was very similar, and I started to use makecontext/setcontext to implement what I call fibers.

Basically that is C level stack switching, but with a huge issue. Python does not grow stacks, but can need a lot of stack space below. Therefore 1MB or even 2MB per generator was allocated, to be able to make deep function calls if needed.

So using a lot of generators on 32 bits could easily hit a 2GB limit. And now with Python3.5 coroutines people use more and more of them, and hit memory issues.

So, goto generators, now that C is possible, are an entirely new solution. With it, Nuitka will use one stack only. Generator code will become re-entrant, store values between entries on the heap, and continue execution at goto destinations dispatched by a switch according to last exit of the generator.

So I am now making changes to cleanup the way variable declarations and accesses for the C variables are being made. More on that next week though. For now I am very exited about the many cleanups that stem from it. The code generation used to have many special bells and whistles, and they were generalized into one thing now, making for cleaner and easier to understand Nuitka code.

On interesting thing, is that an incompatiblity related to __new__ will go away now.

The automatic staticmethod that we had to hack into it, because the Python core will do it for uncompiled functions only, had to be done while declaring the class. So it was visible and causing issues with at least the Python enum module, which wants to call your __new__ manually. Because why would it not?!

But turns out, for Python3 the staticmethod is not needed anymore. So this is now only done for Python2, where it is needed, and things work smoothly with this kind of code now too. This is currently in my factory testing and will probably become part of a hotfix if it turns out good.

Immediately after the release, some rarely run test, where I compiled all the code on my machine, found 2 older bugs, obscure ones arguably, that I made into a hotfix, also because the test runner was having a regression with 3.7, which prevented some package builds. So that was release.

And then I received a bug report about await where a self test of Nuitka fails and reports an optimization error. Very nice, the new exceptions that automatically dump involved nodes as XML made it immediately clear from the report, what is going on, even without having to reproduce anything. I bundled a 3.7 improvement for error cases in class creation with it. So that was the release.

Finishing goto generators is my top priority, but I am also going over minor issues with the 3.7 test suite, fixing test cases there, and as with e.g. the enum issue, even known issues this now finds.

Until next week.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains substantial new optimization, bug fixes, and already the full support for Python 3.7. Among the fixes, the enhanced coroutine work for compatiblity with uncompiled ones is most important.

  • Fix, was optimizing write backs of attribute in-place assignments falsely.
  • Fix, generator stop future was not properly supported. It is now the default for Python 3.7 which showed some of the flaws.
  • Python3.5: The __qualname__ of coroutines and asyncgen was wrong.
  • Python3.5: Fix, for dictionary unpackings to calls, check the keys if they are string values, and raise an exception if not.
  • Python3.6: Fix, need to check assignment unpacking for too short sequences, we were giving IndexError instead of ValueError for these. Also the error messages need to consider if they should refer to "at least" in their wording.
  • Fix, outline nodes were cloned more than necessary, which would corrupt the code generation if they later got removed, leading to a crash.
  • Python3.5: Compiled coroutines awaiting uncompiled coroutines was not working properly for finishing the uncompiled ones. Also the other way around was raising a RuntimeError when trying to pass an exception to them when they were already finished. This should resolve issues with asyncio module.
  • Fix, side effects of a detected exception raise, when they had an exception detected inside of them, lead to an infinite loop in optimization. They are now optimized in-place, avoiding an extra step later on.
  • Support for Python 3.7 with only some corner cases not supported yet.
  • Delay creation of StopIteration exception in generator code for as long as possible. This gives more compact code for generations, which now pass the return values via compiled generator attribute for Python 3.3 or higher.
  • Python3: More immediate re-formulation of classes with no bases. Avoids noise during optimization.
  • Python2: For class dictionaries that are only assigned from values without side effects, they are not converted to temporary variable usages, allowing the normal SSA based optimization to work on them. This leads to constant values for class dictionaries of simple classes.
  • Explicit cleanup of nodes, variables, and local scopes that become unused, has been added, allowing for breaking of cyclic dependencies that prevented memory release.
  • Adapted 3.5 tests to work with 3.7 coroutine changes.
  • Added CPython 3.7 test suite.
  • Removed remaining code that was there for 3.2 support. All uses of version comparisons with 3.2 have been adapted. For us, Python3 now means 3.3, and we will not work with 3.2 at all. This removed a fair bit of complexity for some things, but not all that much.
  • Have dedicated file for import released helpers, so they are easier to find if necessary. Also do not have code for importing a name in the header file anymore, not performance relevant.
  • Disable Python warnings when running scons. These are particularily given when using a Python debug binary, which is happening when Nuitka is run with --python-debug option and the inline copy of Scons is used.
  • Have a factory function for all conditional statement nodes created. This solved a TODO and handles the creation of statement sequences for the branches as necessary.
  • Split class reformulation into two files, one for Python2 and one for Python3 variant. They share no code really, and are too confusing in a single file, for the huge code bodies.
  • Locals scopes now have a registry, where functions and classes register their locals type, and then it is created from that.
  • Have a dedicated helper function for single argument calls in static code that does not require an array of objects as an argument.
  • There are now requirements-devel.txt and requirements.txt files aimed at usage with scons and by users, but they are not used in installation.

This releases has this important step to add conversion of locals dictionary usages to temporary variables. It is not yet done everywhere it is possible, and the resulting temporary variables are not yet propagated in the all the cases, where it clearly is possible. Upcoming releases ought to achieve that most Python2 classes will become to use a direct dictionary creation.

Adding support for Python 3.7 is of course also a huge step. And also this happened fairly quickly and soon after its release. The generic classes it adds were the only real major new feature. It breaking the internals for exception handling was what was holding back initially, but past that, it was really easy.

Expect more optimization to come in the next releases, aiming at both the ability to predict Python3 metaclasses __prepare__ results, and at more optimization applied to variables after they became temporary variables.

I think I tend to prefer coding over communication too much. I think I need to make more transparent what I am doing. Also things, will be getting exciting continuously for a while now.

I used to status report posts, many years ago, every 3 months or so, and that was nice for me also to get an idea of what changed, but I stopped. What did not happen, was to successfully engage other people to contribute.

This time I am getting more intense. I will aim to do roughly weekly or bi-weekly reports, where I highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.

I will do it this fashion. I will write a post to the mailing list, right about wednesday every week or so. I need to pick a day. I am working from home that day, saving me commute time. I will invest that time into this.

The writing will not be too high quality at times. Bare with me there. Then I will check feedback from the list, if any. Hope is for it to point out the things where I am not correct, missing, or even engage right away.

Topics are going to be random, albeit repeating. I will try and make links to previous issues where applicable. Therefore also the TOC, which makes for link targets in the pages.

When I am speaking of locals dict, I am talking of class scopes (and functions with exec statements). These started to use actual dictionary a while ago, which was a severe setback to optimization.

Right now, so for this week, after a first prototype was making the replacement of local dict assignment and references for Python2, and kind of worked through my buildbots, flawlessly, I immediately noticed that it would require some refactoring to not depend on the locals scopes to be only in one of the trace collections. Thinking of future inlining, maybe part of a locals scope was going to be in multiple functions, that ought to not be affected.

Therefore I made a global registry of locals scopes, and working on those, I checked its variables, if they can be forward propagated, and do this not per module, but after all the modules have been done. This is kind of a setback for the idea of module specific optimization (cachable later on) vs. program optimization, but since that is not yet a thing, it can remain this way for now.

Once I did that, I was interested to see the effect, but to my horror, I noticed, that memory was not released for the locals dict nodes. It was way too involved with cyclic dependencies, which are bad. So that was problematic of course. Compilation to keep nodes in memory for both tracing the usage as a locals dict and temporary variables, wasn't going to help scaling at all.

Solution is finalization

So replaced nodes reference a parent, and then the locals scope references variables, and trace collections referencing variables, which reference locals scopes, and accesses referencing traces, and so on. The garbage collector can handle some of this, but seems I was getting past that.

For a solution, I started to add a finalize method, which released the links for locals scopes, when they are fully propagated, on the next run.

Adding a finalize to all nodes, ought to make sure, memory is released soon, and might even find bugs, as nodes become unusable after they are supposedly unused. Obviously, there will currently be cases, where nodes becomes unused, but they are not finalized yet. Also, often this is more manual, because part of the node is to be released, but one child is re-used. That is messy.

The result was a bit disappointing. Yes, memory usage of mercurial compilation went back again, but mostly to what it had been. Some classes are now having their locals dict forward propagated, but the effect is not always a single dictionary making yet. Right now, function definitions, are not forward at all propagated. This is a task I want to take on before next release though, but maybe not, there is other things too. But I am assuming that will make most class dictionaries created without using any variables at all anymore, which should make it really lean.

Then, asking about type hints, I got the usual question about Nuitka going to use it. And my stance is unchanged. They are just hints, not reliable. Need to behave the same if users do it wrong. Suggested to create decorated which make type hints enforced. But I expect nobody takes this on though. I need to make it a Github issue of Nuitka, although technically it is pure CPython work and ought to be done independently. Right now Nuitka is not yet there anyway yet, to take full advantage.

Then, for Python 3.7, I have long gotten the 3.6 test suite to pass. I raised 2 bugs with CPython, one of which lead to update of a failing test. Nuitka had with large delay, caught of with what del __annotations__ was doing in a class. Only with the recent work for proper locals dict code generation, we could enforce a name to be local, and have proper code generation, that allows for it to be unset.

This was of course a bit of work. But the optimization behind was always kind of necessary to get right. But now, that I got this, think of my amazement when for 3.7 they reverted to the old behavior, where annotiatons then corrupt the module annotations

The other bug is a reference counting bug, where Nuitka tests were failing with CPython 3.7, and turns out, there is a bug in the dictionary implementation of 3.7, but it only corrupts counts reported, not actual objects, so it's harmless, but means for 3.7.0 the reference count tests are disabled.

Working through the 3.7 suite, I am cherry picking commits, that e.g. allow the repr of compiled functions to contain <compiled_function ...> and the like. Nothing huge yet. There is now a subscript of type, and foremost the async syntax became way more liberal, so it is more complex for Nuitka to make out if it is a coroutine due to something happening inside a generator declared inside of it. Also cr_origin was added to coroutines, but that is mostly it.

A bigger thing was that I debugged coroutines and their interaction with uncompiled and compiled coroutines awaiting one another, and turns out, there was a lot to improve.

The next release will be much better compatible with asyncio module and its futures, esp with exceptions to cancel tasks passed along. That required to clone a lot of CPython generator code, due to how ugly they mess with bytecode instruction pointers in yield from on an uncompiled coroutine, as they don't work with send method unlike everything else has to.

For PyLint, the 2.0.0 release found new things, but unfortunately for 2.0.1 there is a lot of regressions that I had to report. I fixed the versions of first PyLint, and now also Astroid, so Travis cannot suddenly start to fail due to a PyLint release finding new warnings.

Currently, if you make a PR on Github, a PyLint update will break it. And also the cron job on Travis that checks master.

As somebody pointed out, I am now using <> to check for Nuitka dependencies. But since 1.9.2 is still needed for Python2, that kind of is bound to give alarms for now.

I have a habit of doing off tasks, when I am with my notebook in some place, and don't know what to work on. So I have some 2 hours recently like this, and used it to look at TODO and resolve them.

I did a bunch of cleanups for static code helpers. There was one in my mind about calling a function with a single argument. That fast call required a local array with one element to put the arg into. That makes using code ugly.

So the enum module of Python3 hates compiled classes and their staticmethod around __new__. Since it manually unwraps __new__ and then calls it itself, it then finds that a staticmethod object cannot be called. It's purpose is to sit in the class dictionary to give a descriptor that removes the self arg from the call.

I am contemplating submitting an upstream patch for CPython here. The hard coded check for PyFunction on the __new__ value is hard to emulate.

So I am putting the staticmethod into the dictionary passed already. But the undecorated function should be there for full compatibility.

If I were to make compiled function type that is both a staticmethod alike and a function, maybe I can work around it. But it's ugly and a burden. But it would need no change. And maybe there is more core wanting to call __new__ manually

I intend to make a release, probably this weekend. It might not contain full 3.7 compatibility yet, although I am aiming at that.

Then I want to turn to "goto generators", a scalability improvement of generators and coroutines that I will talk about next week then.

Until next week.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is massive in terms of fixes, but also adds a lot of refinement to code generation, and more importantly adds experimental support for Python 3.7, while enhancing support for Pyt5 in standalone mode by a lot.

  • Standalone: Added missing dependencies for PyQt5.Qt module.

  • Plugins: Added support for PyQt5.Qt module and its qml plugins.

  • Plugins: The sensible plugin list for PyQt now includes that platforms plugins on Windows too, as they are kind of mandatory.

  • Python3: Fix, for uninstalled Python versions wheels that linked against the Python3 library as opposed to Python3X, it was not found.

  • Standalone: Prefer DLLs used by main program binary over ones used by wheels.

  • Standalone: For DLLs added by Nuitka plugins, add the package directory to the search path for dependencies where they might live.

  • Fix, the vars built-in didn't annotate its exception exit.

  • Python3: Fix, the bytes and complex built-ins needs to be treated as a slot too.

  • Fix, consider if del variable must be assigned, in which case no exception exit should be created. This prevented Tkinter compilation.

  • Python3.6: Added support for the following language construct:

    d = {"metaclass" : M}
    class C(**d):
  • Python3.5: Added support for cyclic imports. Now a from import with a name can really cause an import to happen, not just a module attribute lookup.

  • Fix, hasattr was never raising exceptions.

  • Fix, bytearray constant values were considered to be non-iterable.

  • Python3.6: Fix, now it is possible to del __annotations__ in a class and behave compatible. Previously in this case we were falling back to the module variable for annotations used after that which is wrong.

  • Fix, some built-in type conversions are allowed to return derived types, but Nuitka assumed the excact type, this affected bytes, int, long, unicode.

  • Standalone: Fix, the _socket module was insisted on to be found, but can be compiled in.

  • Added experimental support for Python 3.7, more work will be needed though for full support. Basic tests are working, but there are are at least more coroutine changes to follow.
  • Added support for building extension modules against statically linked Python. This aims at supporting manylinux containers, which are supposed to be used for creating widely usable binary wheels for Linux. Programs won't work with statically linked Python though.
  • Added options to allow ignoring the Windows cache for DLL dependencies or force an update.
  • Allow passing options from distutils to Nuitka compilation via setup options.
  • Added option to disable the DLL dependency cache on Windows as it may become wrong after installing new software.
  • Added experimental ability to provide extra options for Nuitka to setuptools.
  • Python3: Remove frame preservation and restoration of exceptions. This is not needed, but leaked over from Python2 code.
  • Apply value tracing to local dict variables too, enhancing the optimization for class bodies and function with exec statements by a lot.
  • Better optimization for "must not have value", wasn't considering merge traces of uninitialized values, for which this is also the case.
  • Use 10% less memory at compile time due to specialized base classes for statements with a single child only allowing __slots__ usage by not having multiple inheritance for those.
  • More immediately optimize branches with known truth values, so that merges are avoided and do not prevent trace based optimization before the pass after the next one. In some cases, optimization based on traces could fail to be done if there was no next pass caused by other things.
  • Much faster handling for functions with a lot of eval and exec calls.
  • Static optimization of type with known type shapes, the value is predicted at compile time.
  • Optimize containers for all compile time constants into constant nodes. This also enables further compile time checks using them, e.g. with isinstance or in checks.
  • Standalone: Using threads when determining DLL dependencies. This will speed up the un-cached case on Windows by a fair bit.
  • Also remove unused assignments for mutable constant values.
  • Python3: Also optimize calls to bytes built-in, this was so far not done.
  • Statically optimize iteration over constant values that are not iterable into errors.
  • Removed Fortran, Java, LaTex, PDF, etc. stuff from the inline copies of Scons for faster startup and leaner code. Also updated to 3.0.1 which is no important difference over 3.0.0 for Nuitka however.
  • Make sure to always release temporary objects before checking for error exits. When done the other way around, more C code than necessary will be created, releasing them in both normal case and error case after the check.
  • Also remove unused assignments in case the value is a mutable constant.
  • Don't store "version" numbers of variable traces for code generation, instead directly use the references to the value traces instead, avoiding later lookups.
  • Added dedicated module for complex built-in nodes.
  • Moved C helpers for integer and complex types to dedicated files, solving the TODOs around them.
  • Removed some Python 3.2 only codes.
  • For better bug reports, the --version output now contains also the Python version information and the binary path being used.
  • Started using specialized exceptions for some types of errors, which will output the involved data for better debugging without having to reproduce anything. This does e.g. output XML dumps of problematic nodes.
  • When encountering a problem (compiler crash) in optimization, output the source code line that is causing the issue.
  • Added support for Fedora 28 RPM builds.
  • Remove more instances of mentions of 3.2 as supported or usable.
  • Renovated the graphing code and made it more useful.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release has improvements in all areas. Many bug fixes are accompanied with optimization changes towards value tracing.

  • Fix, the new setuptools runners were not used by pip breaking the use of Nuitka from PyPI.

  • Fix, imports of six.moves could crash the compiler for built-in names. Fixed in already.

  • Windows: Make the nuitka-run not a symlink as these work really bad on that platform, instead make it a full copy just like we did for nuitka3-run already. Fixed in already.

  • Python3.5: In module mode, types.coroutine was monkey patched into an endless recursion if including more than one module, e.g. for a package. Fixed in already.

  • Python3.5: Dictionary unpackings with both star arguments and non star arguments could leak memory. Fixed in already.

    c = {a : 1, **d}
  • Fix, distutils usage was not working for Python2 anymore, due to using super for what are old style classes on that version.

  • Fix, some method calls to C function members could leak references.

    class C:
     for_call = functools.partial
     def m():
     self.for_call() # This leaked a reference to the descriptor.
  • Python3.5: The bases classes should be treated as an unpacking too.

    class C(*D): # Allowed syntax that was not supported.
  • Windows: Added back batch files to run Nuitka from the command line. Fixed in already.

  • Added option --include-package to force inclusion of a whole package with the submodules in a compilation result.
  • Added options --include-module to force inclusion of a single module in a compilation result.
  • The `multiprocessing plug-in got adapted to Python 3.4 changes and will now also work in accelerated mode on Windows.
  • It is now possible to specify the Qt plugin directories with e.g. --enable-plugin=qt_plugins=imageformats and have only those included. This should avoid dependency creep for shared libraries.
  • Plugins can now make the decision about recursing to a module or not.
  • Plugins now can get their own options passed.
  • The re-raising of exceptions has gotten its own special node type. This aims at more readability (XML output) and avoiding the overhead of checking potential attributes during optimization.
  • Changed built-in int, long, and float to using a slot mechanism that also analyses the type shape and detects and warns about errors at compile time.
  • Changed the variable tracing to value tracing. This meant to cleanup all the places that were using it to find the variable.
  • Enable must have / must not value value optimization for all kinds of variables including module and closure variables. This often avoids error exits and leads to smaller and faster generated code.
  • Added burn test with local install of pip distribution to virtualenv before making any PyPI upload. It seems pip got its specific error sources too.
  • Avoid calling 2to3 and prefer <python> -m lib2to3 instead, as it seems at least Debian Testing stopped to provide the binary by default. For Python 2.6 and 3.2 we continue to rely on it, as the don't support that mode of operation.
  • The PyLint checks have been made more robust and even more Python3 portable.
  • Added PyLint to Travis builds, so PRs are automatically checked too.
  • Added test for distutils usage with Nuitka that should prevent regressions for this new feature and to document how it can be used.
  • Make coverage taking work on Windows and provide the full information needed, the rendering stage is not there working yet though.
  • Expanded the trick assignment test cases to cover more slots to find bugs introduced with more aggressive optimization of closure variables.
  • New test to cover multiprocessing usage.
  • Generating more code tests out of doctests for increased coverage of Nuitka.
  • Stop using --python-version in tests where they still remained.
  • Split the forms of int and long into two different nodes, they share nothing except the name. Create the constants for the zero arg variant more immediately.
  • Split the output comparison part into a dedicated testing module so it can be re-used, e.g. when doing distutils tests.
  • Removed dead code from variable closure taking.
  • Have a dedicated module for the metaclass of nodes in the tree, so it is easier to find, and doesn't clutter the node base classes module as much.
  • Have a dedicated node for reraise statements instead of checking for all the arguments to be non-present.
  • There is now a pull request template for Github when used.
  • Deprecating the --python-version argument which should be replaced by using -m nuitka with the correct Python version. Outputs have been updated to recommend this one instead.
  • Make automatic import sorting and autoformat tools properly executable on Windows without them changing new lines.
  • The documentation was updated to prefer the call method with -m nuitka and manually providing the Python binary to use.

This release continued the distutils integration adding first tests, but more features and documentation will be needed.

Also, for the locals dictionary work, the variable tracing was made generic, but not yet put to use. If we use this to also trace dictionary keys, we can expect a lot of improvements for class code again.

The locals dictionary tracing will be the focus before resuming the work on C types, where the ultimate performance boost lies. However, currently, not the full compatibility has been achieved even with currently using dictionaries for classes, and we would like to be able to statically optimize those better anyway.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes with a lot of improvements across the board. A lot of focus has been givevn to the packaging side of Nuitka, but also there is a lot of compatibility work.

  • Windows: When using Scons for Python3 and Scons for Python2 on the same build directory, a warning would be given about the need to migrate. Make the Scons cache directory use the Python ABI version as a key too, to avoid these issues. Fixed in already.
  • Windows: Fixup for Python3 and Scons no more generating the MinGW64 import library for Python anymore properly. Was only working if cached from a previous install of Nuitka. Fixed in already.
  • Plugins: Made the data files plugin mandatory and added support for the scrapy package needs.
  • Fix, added implicit dependencies for pkg_resources.external package. Fixed in already.
  • Fix, an import of x.y where this was not a package didn't cause the package x to be included.
  • Standalone: Added support for six.moves and requests.packages meta imports, these cause hidden implicit imports, that are now properly handled.
  • Standalone: Patch the __file__ value for technical bytecode modules loaded during Python library initialization in a more compatible way.
  • Standalone: Extension modules when loaded might actually raise legit errors, e.g. ImportError of another module, don't make those into SystemError anymore.
  • Python3.2: The __package__ of sub-packages was wrong, which could cause issues when doing relative imports in that sub-package.
  • Python3: Contractions in a finally clause could crash the compiler.
  • Fix, unused closure variables could lead to a crash in they were passed to a nested function.
  • Linux: Standalone dependency analysis could enter an endless recursion in case of cyclic dependencies.
  • Python3.6: Async generation expressions need to return a None value too.
  • Python3.4: Fix, __spec__ is a package attribute and not a built-in value.
  • It is now possible to run Nuitka with some_python_you_choose -m nuitka ... and therefore know exactly which Python installation is going to be used. It does of course need Nuitka installed for this to work. This mechanism is going to replace the --python-version mechanism in the future.
  • There are dedicated runners for Python3, simply use nuitka3 or nuitka3-run to execute Nuitka if your code is Python3 code.
  • Added warning for implicit exception raises due to mismatch in unpacking length. These are statically detected, but so far were not warned about.
  • Added cache for depends.exe results. This speeds up standalone mode again as some of these calls were really slow.
  • The import tracer is more robust against recursion and works with Python3 now.
  • Added an option to assume yes for downloading questions. The currently only enables the download of depends.exe and is intended for CI servers.
  • There is now a report file for scons, which records the values used to run things, this could be useful for debugging.
  • Nuitka now registers with distutils and can be used with bdist_wheel directly, but this lacks documentation and tests. Many improvements in the distutils build.
  • Forward propagate compile time constants even if they are only potential usages. This is actually the case where this makes the most sense, as it might remove its use entirely from the branches that do not use it.
  • Avoid extra copy of finally code. The cloning operation takes time and memory, and this shaved of 0.3% of Nuitka memory usage, as these can also become dangling.
  • Class dictionaries are now proper dictionarties in optimization, using some dedicated code for name lookups that are transformed to dedicated locals dictionary or mapping (Python3) accesses. This currently does not fully optimize, but will in coming releases, and saves about 25% of memory compared to the old code.
  • Treating module attributes __package__, __loader__, __file__, and __spec__ with dedicated nodes, that allow or forbid optimization dependent on usage.
  • Python3.6: Async generator expressions were not working fully, become more compatible.
  • Fix, using super inside a contraction could crash the compiler.
  • Fix, also accept __new__ as properly decorated in case it's a classmethod too.
  • Fix, removed obsolete --nofreeze-stdlib which only complicated using the --recurse-stdlib which should be used instead.
  • The nuitka Python package is now installed into the public namespace and used from there. There are distinct copies to be installed for both Python2 and Python3 on platforms where it is supported.
  • Using twine for upload to PyPI now as recommended on their site.
  • Running pylint on Windows became practical again.
  • Added RPM packages for Fedora 26 and 27, these used to fail due to packaging issues.
  • Added RPM packages for openSUSE Leap 42.2, 42.3 and 15.0 which were simply missing.
  • Added RPM packages for SLE 15.
  • Added support for PyLint 1.8 and its new warnings.
  • The RPM packages no longer contain nuitka-run3, it will be replaced by the new nuitka3-run which is in all packages.
  • The runners used for installation are now easy install created, but patched to avoid overhead at run time.
  • Added repository for Ubuntu Artful (17.10) for download, removed support for Ubuntu Yakkety, Vivid and Zesty (no more supported by them).
  • Removed support for Debian Wheezy and Ubuntu Precise (they are too old for modern packaging used).
  • There is now a issue template for Github when used.
  • Windows: Standalone tests were referencing an old path to depends.exe that wasn't populated on new installs.
  • Refinements for CPython test suites to become more stable in results. Some tests occasionally fail to clean up, or might do indetermistic outputs, or are not relevant at all.
  • The tests don't use the runners, but more often do -m nuitka to become executable without having to find the proper runner. This improves usage during the RPM builds and generally.
  • Travis: Do not test development versions of CPython, even for stable release, they break too often.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release has a focus on compatibility work and contains bug fixes and work to enhance the usability of Nuitka by integrating with distutils. The major improvement is that contractions no longer use pseudo functions to achieve their own local scope, but that there is now a dedicated structure for that representing an in-lined function.

  • Python3.6: Fix, async for was not yet implemented for async generators.
  • Fix, functions with keyword arguments where the value was determined to be a static raise could crash the compiler.
  • Detect using MinGW64 32 bits C compiler being used with 64 bits Python with better error message.
  • Fix, when extracting side effects of a static raise, extract them more recursively to catch expressions that themselves have no code generation being used. This fixes at least static raises in keyword arguments of a function call.
  • Compatibility: Added support for proper operation of `pkgutil.get_data by implementing get_data in our meta path based loader.
  • Compatibility: Added __spec__ module attribute was previously missing, present on Python3.4 and higher.
  • Compatibility: Made __loader__ module attribute set when the module is loading already.
  • Standalone: Resolve the @rpath and @loader_path from otool on MacOS manually to actual paths, which adds support for libraries compiled with that.
  • Fix, nested functions calling super could crash the compiler.
  • Fix, could not use --recurse-directory with arguments that had a trailing slash.
  • Fix, using --recurse-directory on packages that are not in the search crashed the compiler.
  • Compatibility: Python2 set and dict contractions were using extra frames like Python3 does, but those are not needed.
  • Standalone: Fix, the way PYTHONHOME was set on Windows had no effect, which allowed the compiled binary to access the original installation still.
  • Standalone: Added some newly discovered missing hidden dependencies of extension modules.
  • Compatiblity: The name mangling of private names (e.g. __var) in classes was applied to variable names, and function declarations, but not to classes yet.
  • Python3.6: Fix, added support for list contractions with await expressions in async generators.
  • Python3.6: Fix, async for was not working in async generators yet.
  • Fix, for module tracebacks, we output the module name <module name> instead of merely <module>, but if the module was in a package, that was not indicated. Now it is <module>.
  • Windows: The cache directory could be unicode which then failed to pass as an argument to scons. We now encode such names as UTF-8 and decode in Scons afterwards, solving the problem in a generic way.
  • Standalone: Need to recursively resolve shared libraries with ldd, otherwise not all could be included.
  • Standalone: Make sure sys.path has no references to CPython compile time paths, or else things may work on the compiling machine, but not on another.
  • Standalone: Added various missing dependencies.
  • Standalone: Wasn't considering the DLLs directory for standard library extensions for freezing, which would leave out these.
  • Compatibility: For __future__ imports the __import__ function was called more than once.
  • Contractions are now all properly inlined and allow for optimization as if they were fully local. This should give better code in some cases.
  • Classes are now all building their locals dictionary inline to the using scope, allowing for more compact code.
  • The dictionary API was not used in module template code, although it helps to generate more compact code.
  • Experimental support for building platform dependent wheel distribution.

    python --command-packages=nuitka.distutils clean -a bdist_nuitka

    Use with caution, this is incomplete work.

  • Experimental support for running tests against compiled installation with nose and py.test.

  • When specifiying what to recurse to, now patterns can be used, e.g. like this --recurse-not-to=*.tests which will skip all tests in submodules from compilation.

  • By setting NUITKA_PACKAGE_packagename=/some/path the __path__ of packages can be extended automatically in order to allow and load uncompiled sources from another location. This can be e.g. a tests sub-package or other plug-ins.

  • By default when creating a module, now also a module.pyi file is created that contains all imported modules. This should be deployed alongside the extension module, so that standalone mode creation can benefit from knowing the dependencies of compiled code.

  • Added option --plugin-list that was mentioned in the help output, but still missing so far.

  • The import tracing of the hints module has achieved experimental status and can be used to test compatibility with regards to import behavior.

  • Rename tree and codegen Helper modules to unique names, making them easier to work with.
  • Share the code that decides to not warn for standard library paths with more warnings.
  • Use the bool enum definition of Python2 which is more elegant than ours.
  • Move quality tools, autoformat, isort, etc. to the namespace.
  • Move output comparison tool to the namespace.
  • Made frame code generation capable of using nested frames, allowing the real inline of classes and contraction bodies, instead of "direct" calls to pseudo functions being used.
  • Proper base classes for functions that are entry points, and functions that are merely a local expression using return statements.
  • The search mode with pattern, was not working anymore.
  • Resume hash values now consider the Python version too.
  • Added test that covers using test runners like nose and py.test with Nuitka compiled extension modules.
  • Added support for Scons 3.0 and running Scons with Python3.5 or higher. The option to specifiy the Python to use for scons has been renamed to reflect that it may also be a Python3 now. Only for Python3.2 to Python3.4 we now need another Python installation.
  • Made recursion the default for --recurse-directory with packages. Before you also had to tell it to recurse into that package or else it would only include the top level package, but nothing below.
  • Updated the man pages, correct mentions of its C++ to C and don't use now deprecated options.
  • Updated the help output which still said that standalone mode implies recursion into standard library, which is no longer true and even not recommended.
  • Added option to disable the output of .pyi file when creating an extension module.
  • Removed Ubuntu Wily package download, no longer supported by Ubuntu.

This release was done to get the fixes and new features out for testing. There is work started that should make generators use an explicit extra stack via pointer, and restore instruction state via goto dispatchers at function entry, but that is not complete.

This feature, dubbed "goto generators" will remove the need for fibers (which is itself a lot of code), reduce the memory footprint at run time for anything that uses a lot of generators, or coroutines.

Integrating with distutils is also a new thing, and once completed will make use of Nuitka for existing projects automatic and trivial to do. There is a lot missing for that goal, but we will get there.

Also, documenting how to run tests against compiled code, if that test code lives inside of that package, will make a huge difference, as that will make it easier for people to torture Nuitka with their own test cases.

And then of course, nested frames now mean that every function could be inlined, which was previously not possible due to collisions of frames. This will pave the route for better optimization in those cases in future releases.

The experimental features will require more work, but should make it easier to use Nuitka for existing projects. Future releases will make integrating Nuitka dead simple, or that is the hope.

And last but not least, now that Scons works with Python3, chances are that Nuitka will more often work out the of the box. The older Python3 versions that still retain the issue are not very widespread.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes a lot of bug fixes and improvements.

  • Fix, need to add recursed modules immediately to the working set, or else they might first be processed in second pass, where global names that are locally assigned, are optimized to the built-in names although that should not happen. Fixed in already.
  • Fix, the accelerated call of methods could crash for some special types. This had been a regress of 0.5.25, but only happens with custom extension types. Fixed in already.
  • Python3.5: For async def functions parameter variables could fail to properly work with in-place assignments to them. Fixed in already.
  • Compatability: Decorators that overload type checks didn't pass the checks for compiled types. Now isinstance and as a result inspect module work fine for them.
  • Compatiblity: Fix, imports from __init__ were crashing the compiler. You are not supposed to do them, because they duplicate the package code, but they work.
  • Compatiblity: Fix, the super built-in on module level was crashing the compiler.
  • Standalone: For Linux, BSD and MacOS extension modules and shared libraries using their own $ORIGIN to find loaded DLLs resulted in those not being included in the distribution.
  • Standalone: Added more missing implicit dependencies.
  • Standalone: Fix, implicit imports now also can be optional, as e.g. _tkinter if not installed. Only include those if available.
  • The --recompile-c-only was only working with C compiler as a backend, but not in the C++ compatibility fallback, where files get renamed. This prevented that edit and test debug approach with at least MSVC.
  • Plugins: The PyLint plug-in didn't consider the symbolic name import-error but only the code F0401.
  • Implicit exception raises in conditional expressions would crash the compiler.
  • Added support for Visual Studio 2017. Issue#368.
  • Added option --python2-for-scons to specify the Python2 execute to use for calling Scons. This should allow using AnaConda Python for that task.
  • References to known unassigned variables are now statically optimized to exception raises and warned about if the according option is enabled.
  • Unhashable keys in dictionaries are now statically optimized to exception raises and warned about if the according option is enabled.
  • Enable forward propagation for classes too, resulting in some classes to create only static dictionaries. Currently this never happens for Python3, but it will, once we can statically optimize __prepare__ too.
  • Enable inlining of class dictionary creations if they are mere return statements of the created dictionary. Currently this never happens for Python3, see above for why.
  • Python2: Selecting the metaclass is now visible in the tree and can be statically optimized.
  • For executables, we now also use a freelist for traceback objects, which also makes exception cases slightly faster.
  • Generator expressions no longer require the use of a function call with a .0 argument value to carry the iterator value, instead their creation is directly inlined.
  • Remove "pass through" frames for Python2 list contractions, they are no longer needed. Minimal gain for generated code, but more lightweight at compile time.
  • When compiling Windows x64 with MinGW64 a link library needs to be created for linking against the Python DLL. This one is now cached and re-used if already done.
  • Use common code for NameError and UnboundLocalError exception code raises. In some cases it was creating the full string at compile time, in others at run time. Since the later is more efficient in terms of code size, we now use that everywhere, saving a bit of binary size.
  • Make sure to release unused functions from a module. This saves memory and can be decided after a full pass.
  • Avoid using OrderedDict in a couple of places, where they are not needed, but can be replaced with a later sorting, e.g. temporary variables by name, to achieve deterministic output. This saves memory at compile time.
  • Add specialized return nodes for the most frequent constant values, which are None, True, and False. Also a general one, for constant value return, which avoids the constant references. This saves quite a bit of memory and makes traversal of the tree a lot faster, due to not having any child nodes for the new forms of return statements.
  • Previously the empty dictionary constant reference was specialized to save memory. Now we also specialize empty set, list, and tuple constants to the same end. Also the hack to make is not say that {} is {} was made more general, mutable constant references and now known to never alias.
  • The source references can be marked internal, which means that they should never be visible to the user, but that was tracked as a flag to each of the many source references attached to each node in the tree. Making a special class for internal references avoids storing this in the object, but instead it's now a class property.
  • The nodes for named variable reference, assignment, and deletion got split into separate nodes, one to be used before the actual variable can be determined during tree building, and one for use later on. This makes their API clearer and saves a tiny bit of memory at compile time.
  • Also eliminated target variable references, which were pseudo children of assignments and deletion nodes for variable names, that didn't really do much, but consume processing time and memory.
  • Added optimization for calls to staticmethod and classmethod built-in methods along with type shapes.
  • Added optimization for open built-in on Python3, also adding the type shape file for the result.
  • Added optimization for bytearray built-in and constant values. These mutable constants can now be compile time computed as well.
  • Added optimization for frozenset built-in and constant values. These mutable constants can now be compile time computed as well.
  • Added optimization for divmod built-in.
  • Treat all built-in constant types, e.g. type itself as a constant. So far we did this only for constant values types, but of course this applies to all types, giving slightly more compact code for their uses.
  • Detect static raises if iterating over non-iterables and warn about them if the option is enabled.
  • Split of locals node into different types, one which needs the updated value, and one which just makes a copy. Properly track if a functions needs an updated locals dict, and if it doesn't, don't use that. This gives more efficient code for Python2 classes, and exec using functions in Python2.
  • Build all constant values without use of the pickle module which has a lot more overhead than marshal, instead use that for too large long values, non-UTF8 unicode values, nan float, etc.
  • Detect the linker arch for all Linux platforms using objdump instead of only a hand few hard coded ones.
  • The use of INCREASE_REFCOUNT got fully eliminated.
  • Use functions not vulenerable for buffer overflow. This is generally good and avoids warnings given on OpenBSD during linking.
  • Variable closure for classes is different from all functions, don't handle the difference in the base class, but for class nodes only.
  • Make sure mayBeNone doesn't return None which means normally "unclear", but False instead, since it's always clear for those cases.
  • Comparison nodes were using the general comparison node as a base class, but now a proper base class was added instead, allowing for cleaner code.
  • Valgrind test runners got changed to using proper tool namespace for their code and share it.
  • Made construct case generation code common testing code for re-use in the speedcenter web site. The code also has minor beauty bugs which will then become fixable.
  • Use appdirs package to determine place to store the downloaded copy of depends.exe.
  • The code still mentioned C++ in a lot of places, in comments or identifiers, which might be confusing readers of the code.
  • Code objects now carry all information necessary for their creation, and no longer need to access their parent to determine flag values. That parent is subject to change in the future.
  • Our import sorting wrapper automatically detects imports that could be local and makes them so, removing a few existing ones and preventing further ones on the future.
  • Cleanups and annotations to become Python3 PyLint clean as well. This found e.g. that source code references only had __cmp__ and need rich comparison to be fully portable.
  • The test runner for construct tests got cleaned up and the constructs now avoid using xrange so as to not need conversion for Python3 execution as much.
  • The main test runner got cleaned up and uses common code making it more versatile and robust.
  • Do not run test in debugger if CPython also segfaulted executing the test, then it's not a Nuitka issue, so we can ignore that.
  • Improve the way the Python to test with is found in the main test runner, prefer the running interpreter, then PATH and registry on Windows, this will find the interesting version more often.
  • Added support for "" to ignore the inline copies of code, they are not under our control.
  • The test runner for Valgrind got merged with the usage for constructs and uses common code now.
  • Construct generation is now common code, intended for sharing it with the Speedcenter web site generation.
  • Rebased Python 3.6 test suite to 3.6.1 as that is the Python generally used now.
  • Added inline copy of appdirs package from PyPI.
  • Added credits for RedBaron and isort.
  • The --experimental flag is now creating a list of indications and more than one can be used that way.
  • The PyLint runner can also work with Python3 pylint.
  • The Nuitka Speedcenter got more fine tuning and produces more tags to more easily identify trends in results. This needs to become more visible though.
  • The MSI files are also built on AppVeyor, where their building will not depend on me booting Windows. Getting these artifacts as downloads will be the next step.