Pickling — Serialisation of running tasklets¶
One of the most impressive features of Stackless-Python, is the ability to pickle tasklets. This allows you to take a tasklet mid-execution, serialise it to a chunk of data and then unserialise that data at a later point, creating a new tasklet from it that resumes where the last left off.
What makes this
particularly impressive is the fact that the Python® pickle
structure
is platform independent. Code can for instance initially be run on a x86
Windows machine, then interrupted, pickled and sent over the network to be
resumed on an ARM Linux machine.
Example - pickling a tasklet:
>>> def func():
... busy_count = 0
... while 1:
... busy_count += 1
... if busy_count % 10 == 0:
... print(busy_count)
...
>>> stackless.tasklet(func)()
<stackless.tasklet object at 0x01BD16B0>
>>> t1 = stackless.run(100)
10
20
>>> s = pickle.dumps(t1)
>>> t1.kill()
>>> t2 = pickle.loads(s)
>>> t2.insert()
>>> stackless.run(100)
30
40
50
In the above example, a tasklet is created that increments the counter
busy_count and outputs the value when it is a multiple of 10
.
Run the tasklet for a while:
>>> t1 = stackless.run(100)
10
20
The tasklet has been interrupted at some point in its execution. If it were to be resumed, we would expect its output to be the values following those previously displayed.
Serialise the tasklet:
>>> s = pickle.dumps(t1)
As any other object is pickled, so are tasklets. In this case, the serialised representation of the tasklet is a string, stored in s.
Destroy the tasklet:
>>> t1.kill()
We want to show that the old code cannot be resumed, and in order to do so, we destroy the tasklet it was running within.
Unserialise the stored representation:
>>> t2 = pickle.loads(s)
As any other object is unpickled, so are tasklets. We take the string and by unpickling it, get a new tasklet object back.
Schedule the new tasklet:
>>> t2.insert()
Now the newly recreated tasklet is inserted into the scheduler, so that when the scheduler is next run, the tasklet is resumed.
Run the scheduler:
>>> stackless.run(100)
30
40
50
<stackless.tasklet object at 0x01BD1D30>
When the scheduler is run, the values displayed are indeed the ones that
follow those displayed by the original tasklet. The value returned by
stackless.run()
is not stored in a variable this time, so the
interpreter displays the recreated tasklet. You can see that it has a
different address than t1, which was displayed earlier.
Note
It should be possible to pickle any tasklets that you might want to. However, not all tasklets can be unpickled. One of the cases in which this is true, is where not all the functions called by the code within the tasklet are Python® functions. The Stackless-Python pickling mechanism has no ability to deal with C functions that may have been called.
Note
It is not possible to unpickle running tasklets which were pickled by a different minor version of Stackless-Python. A running tasklet contains frame objects and frame objects contain code objects. And code objects are usually incompatible between different minor versions of standard Python®.
Note
If you pickle a tasklet, its Context
won’t be pickled,
because Context
objects can’t be pickled. See
PEP 567 for an explanation.
It is sometimes possible enable pickling of Context
objects
in an application specific way (see for instance: copyreg.pickle()
or
pickle.Pickler.dispatch_table
or pickle.Pickler.persistent_id
).
Such an application can set the pickle flag
PICKLEFLAGS_PICKLE_CONTEXT
to include the
context in the pickled state of a tasklet.
Another option is to subclass tasklet
and overload the methods
tasklet.__reduce_ex__()
and tasklet.__setstate__()
to
pickle the values of particular ContextVar
objects together
with the tasklet.
Pickling other objects¶
In order to be able to pickle tasklets Stackless-Python needs to be able to pickle
several other objects, which can’t be pickled by standard Python®. If the module
stackless
gets imported for the first time, Stackless-Python
uses copyreg.pickle()
to register “reduction” functions for the following
types:
AsyncGeneratorType
,
CodeType
,
CoroutineType
,
FunctionType
,
GeneratorType
,
ModuleType
,
TracebackType
,
Cell Objects,
C-types PyAsyncGenASend and PyAsyncGenAThrow (see PEP 525) as well as
all kinds of Dictionary view objects.
Frames¶
Stackless-Python can pickle frames, but only as part of a
tasklet, a traceback-object, a generator, a coroutine or an asynchronous
generator. Stackless-Python does not register a “reduction” function for
FrameType
. This way Stackless-Python stays compatible with application
code that registers its own “reduction” function for FrameType
.
Asynchronous Generators¶
New in version 3.7.
At C-level asynchronous generators have an attribute ag_finalizer
and a flag,
if ag_finalizer has been initialised. The value of ag_finalizer
is a callable
Python®-object, which has been set by sys.set_asyncgen_hooks()
.
You can use stackless.pickle_flags()
to control how Stackless-Python pickles and
unpickles an asynchronous generator.
Pickling¶
By default (no flags set) Stackless-Python does not pickle ag_finalizer
but a
marker, if a ag_finalizer
has been set.
If PICKLEFLAGS_PRESERVE_AG_FINALIZER
has been set,
Stackless-Python pickles ag_finalizer
by value.
Otherwise, if PICKLEFLAGS_RESET_AG_FINALIZER
has
been set, Stackless-Python pickles ag_finalizer
as uninitialised.
Unpickling¶
By default Stackless-Python initialises the generator upon unpickling using the
firstiter
and finalizer
values set by sys.set_asyncgen_hooks()
,
if ag_finalizer
of the original asynchronous generator was initialised.
If PICKLEFLAGS_PRESERVE_AG_FINALIZER
has been set and if
ag_finalizer
has been pickled by value, Stackless-Python unpickles
ag_finalizer
by value.
Otherwise, if PICKLEFLAGS_RESET_AG_FINALIZER
has
been set, Stackless-Python unpickles ag_finalizer
as uninitialised.