When performing large calculations it’s often useful to split the
work over multiple processes or threads. The
tskit API can
be used without issues across multiple processes, and the Python
multiprocessing module often provides a very effective way to
work with many replicate simulations in parallel.
When we wish to work with a single very large dataset, however, threads can
offer better resource usage because of the shared memory space. The Python
threading library gives a very simple interface to lightweight CPU
threads and allows us to perform several CPU intensive tasks in parallel. The
tskit API is designed to allow multiple threads to work in parallel when
CPU intensive tasks are being undertaken.
In the CPython implementation the
Global Interpreter Lock ensures that
only one thread executes Python bytecode at one time. This means that
Python code does not parallelise well across threads, but avoids a large
number of nasty pitfalls associated with multiple threads updating
data structures in parallel. Native C extensions like
release the GIL while expensive tasks are being performed, therefore
allowing these calculations to proceed in parallel.
This tutorial previously used code with an old interface, and hence has been removed.
We must recreate an example of parallel processing, giving examples of both
threads and processes (but see
this stackoverflow post
for why it may be difficult to get
multiprocessing working in this notebook).
A reasonable example might be to calculate many pairwise statistics between sample sets
We should also show how, for large tree sequences that it is better to pass the filenames to each subprocess, and load the tree sequence, rather than transferring the entire tree sequence (via pickle) to the subprocesses.