Pickling — Serialisation of running tasklets

One of the most impressive features of Stackless-Python, is the ability to pickle tasklets. This allows you to take a tasklet mid-execution, serialise it to a chunk of data and then unserialise that data at a later point, creating a new tasklet from it that resumes where the last left off.

What makes this particularly impressive is the fact that the Python® pickle structure is platform independent. Code can for instance initially be run on a x86 Windows machine, then interrupted, pickled and sent over the network to be resumed on an ARM Linux machine.

Example - pickling a tasklet:

>>> def func():
...    busy_count = 0
...    while 1:
...        busy_count += 1
...        if busy_count % 10 == 0:
...            print(busy_count)
...
>>> stackless.tasklet(func)()
<stackless.tasklet object at 0x01BD16B0>
>>> t1 = stackless.run(100)
10
20
>>> s = pickle.dumps(t1)
>>> t1.kill()
>>> t2 = pickle.loads(s)
>>> t2.insert()
>>> stackless.run(100)
30
40
50

In the above example, a tasklet is created that increments the counter busy_count and outputs the value when it is a multiple of 10.

Run the tasklet for a while:

>>> t1 = stackless.run(100)
10
20

The tasklet has been interrupted at some point in its execution. If it were to be resumed, we would expect its output to be the values following those previously displayed.

Serialise the tasklet:

>>> s = pickle.dumps(t1)

As any other object is pickled, so are tasklets. In this case, the serialised representation of the tasklet is a string, stored in s.

Destroy the tasklet:

>>> t1.kill()

We want to show that the old code cannot be resumed, and in order to do so, we destroy the tasklet it was running within.

Unserialise the stored representation:

>>> t2 = pickle.loads(s)

As any other object is unpickled, so are tasklets. We take the string and by unpickling it, get a new tasklet object back.

Schedule the new tasklet:

>>> t2.insert()

Now the newly recreated tasklet is inserted into the scheduler, so that when the scheduler is next run, the tasklet is resumed.

Run the scheduler:

>>> stackless.run(100)
30
40
50
<stackless.tasklet object at 0x01BD1D30>

When the scheduler is run, the values displayed are indeed the ones that follow those displayed by the original tasklet. The value returned by stackless.run() is not stored in a variable this time, so the interpreter displays the recreated tasklet. You can see that it has a different address than t1, which was displayed earlier.

Note

It should be possible to pickle any tasklets that you might want to. However, not all tasklets can be unpickled. One of the cases in which this is true, is where not all the functions called by the code within the tasklet are Python® functions. The Stackless-Python pickling mechanism has no ability to deal with C functions that may have been called.

Note

It is not possible to unpickle running tasklets which were pickled by a different minor version of Stackless-Python. A running tasklet contains frame objects and frame objects contain code objects. And code objects are usually incompatible between different minor versions of standard Python®.

Note

If you pickle a tasklet, its Context won’t be pickled, because Context objects can’t be pickled. See PEP 567 for an explanation.

It is sometimes possible enable pickling of Context objects in an application specific way (see for instance: copyreg.pickle() or pickle.Pickler.dispatch_table or pickle.Pickler.persistent_id). Such an application can set the pickle flag PICKLEFLAGS_PICKLE_CONTEXT to include the context in the pickled state of a tasklet.

Another option is to subclass tasklet and overload the methods tasklet.__reduce_ex__() and tasklet.__setstate__() to pickle the values of particular ContextVar objects together with the tasklet.

Pickling other objects

In order to be able to pickle tasklets Stackless-Python needs to be able to pickle several other objects, which can’t be pickled by standard Python®. If the module stackless gets imported for the first time, Stackless-Python uses copyreg.pickle() to register “reduction” functions for the following types: AsyncGeneratorType, CodeType, CoroutineType, FunctionType, GeneratorType, ModuleType, TracebackType, Cell Objects, C-types PyAsyncGenASend and PyAsyncGenAThrow (see PEP 525) as well as all kinds of Dictionary view objects.

Frames

Stackless-Python can pickle frames, but only as part of a tasklet, a traceback-object, a generator, a coroutine or an asynchronous generator. Stackless-Python does not register a “reduction” function for FrameType. This way Stackless-Python stays compatible with application code that registers its own “reduction” function for FrameType.

Asynchronous Generators

New in version 3.7.

At C-level asynchronous generators have an attribute ag_finalizer and a flag, if ag_finalizer has been initialised. The value of ag_finalizer is a callable Python®-object, which has been set by sys.set_asyncgen_hooks(). You can use stackless.pickle_flags() to control how Stackless-Python pickles and unpickles an asynchronous generator.

Pickling

By default (no flags set) Stackless-Python does not pickle ag_finalizer but a marker, if a ag_finalizer has been set. If PICKLEFLAGS_PRESERVE_AG_FINALIZER has been set, Stackless-Python pickles ag_finalizer by value. Otherwise, if PICKLEFLAGS_RESET_AG_FINALIZER has been set, Stackless-Python pickles ag_finalizer as uninitialised.

Unpickling

By default Stackless-Python initialises the generator upon unpickling using the firstiter and finalizer values set by sys.set_asyncgen_hooks(), if ag_finalizer of the original asynchronous generator was initialised. If PICKLEFLAGS_PRESERVE_AG_FINALIZER has been set and if ag_finalizer has been pickled by value, Stackless-Python unpickles ag_finalizer by value. Otherwise, if PICKLEFLAGS_RESET_AG_FINALIZER has been set, Stackless-Python unpickles ag_finalizer as uninitialised.