parallel_map
- sherpa.utils.parallel.parallel_map(function: Callback[I_contra, O_co], sequence: Sequence[I_contra], numcores: int | None = None) list[O_co] [source] [edit on github]
Run a function on a sequence of inputs in parallel.
A parallelized version of the native Python map function that utilizes the Python multiprocessing module to divide and conquer sequence. If
function
uses random numbers thenparallel_map_rng
should be used instead.- Parameters:
function (function) – This function accepts a single argument (an element of
sequence
) and returns a value.sequence (array_like) – The data to be passed to
function
.numcores (int or None, optional) – The number of calls to
function
to run in parallel. When set toNone
, all the available CPUs on the machine - as set either by the ‘numcores’ setting of the ‘parallel’ section of Sherpa’s preferences or by multiprocessing.cpu_count - are used.
- Returns:
ans – The return values from the calls, in the same order as the
sequence
array.- Return type:
array
See also
Notes
A tuple or dictionary should be used to pass multiple values to the function.
The input list is split into
numcores
chunks, and then each chunk is run in parallel. There is no guarantee to the ordering of the tasks.Examples
In the following examples a simple set of computations are used; in reality the function is expected to be run on computations that take a significant amount of time to run.
Run the computation (summing up each element of the input array) on a separate core and return the results (unless the machine only has a single core or the parallel.numcores setting is set to 1).
>>> import numpy as np >>> args = [np.arange(5), np.arange(3), np.arange(7)] >>> parallel_map(np.sum, args) [10, 3, 21]
Use two jobs to evaluate the results: one job will sum up two arrays while the other will only sum one array since there are 3 jobs to run.
>>> parallel_map(np.sum, args, numcores=2) [10, 3, 21]
An example of sending in multiple arguments to a function (
comp
) via a dictionary (although in this case there is only one task to execute):>>> parallel_map(comp, [{'idx1': 23, 'idx2': 47}])
Here the
tcomp
function accepts a single parameter which it can deconstruct to extract the two values it needs:>>> parallel_map(tcomp, [(23, 47), (2, 20), (5, 10)])