parallel_map

sherpa.utils.parallel_map(function, sequence, numcores=None)[source] [edit on github]

Run a function on a sequence of inputs in parallel.

A parallelized version of the native Python map function that utilizes the Python multiprocessing module to divide and conquer sequence.

Parameters
  • function (function) – This function accepts a single argument (an element of sequence) and returns a value.

  • sequence (array_like) – The data to be passed to function.

  • numcores (int or None, optional) – The number of calls to function to run in parallel. When set to None, all the available CPUs on the machine - as set either by the ‘numcores’ setting of the ‘parallel’ section of Sherpa’s preferences or by multiprocessing.cpu_count - are used.

Returns

ans – The return values from the calls, in the same order as the sequence array.

Return type

array

Notes

A tuple or dictionary should be used to pass multiple values to the function.

The input list is split into numcores chunks, and then each chunk is run in parallel. There is no guarantee to the ordering of the tasks.

Examples

In the following examples a simple set of computations are used; in reality the function is expected to be run on computations that take a significant amount of time to run.

Run the computation (summing up each element of the input array) on a separate core and return the results (unless the machine only has a single core or the parallel.numcores setting is set to 1).

>>> args = [np.arange(5), np.arange(3), np.arange(7)]
>>> parallel_map(np.sum, args)
[10, 3, 21]

Use two jobs to evaluate the results: one job will sum up two arrays while the other will only sum one array since there are 3 jobs to run.

>>> parallel_map(np.sum, args, numcores=2)
[10, 3, 21]

An example of sending in multiple arguments to a function (comp) via a dictionary (although in this case there is only one task to execute):

>>> parallel_map(comp, [{'idx1': 23, 'idx2': 47}])

Here the tcomp function accepts a single parameter which it can deconstruct to extract the two values it needs:

>>> parallel_map(tcomp, [(23, 47), (2, 20), (5, 10)])