charred.parallel

Parallelism helpers

default-executor-service

(default-executor-service)

Default executor service that is created via 'newCachedThreadPool with a custom thread factory that creates daemon threads. This is an executor service that is suitable for blocking operations as it creates new threads as needed.

in-fork-join-task?

(in-fork-join-task?)

indexed-map-reduce

(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)(indexed-map-reduce num-iters indexed-map-fn reduce-fn)(indexed-map-reduce num-iters indexed-map-fn)

Execute indexed-map-fn over n-groups subranges of (range num-iters). Then call reduce-fn passing in entire in order result value sequence.

  • num-iters - Indexes are the values of (range num-iters).
  • indexed-map-fn - Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced using reduce-fn.
  • reduce-fn - Function from sequence of values to result.
  • max-batch-size - Safepoint batch size, defaults to 64000. For more information, see java safepoint documentation.

Implementation:

This function uses the ForkJoinPool/commonPool to parallelize iteration over (range num-iters) indexes via splitting the index space up into (>= n-groups ForkJoinPool/getCommonPoolParallelism) tasks. In order to respect safepoints, n-groups may be only be allowed to iterate over up to max-batch-size indexes.

If the current thread is already in the common pool, this function executes in the current thread.

queue-supplier

(queue-supplier src-fn & [options])

Given a supplier or clojure fn, create a new thread that will read that fn and place the results into a queue of a fixed size. Returns new suplier. Iteration stops when the src-fn returns nil.

Options:

  • :queue-depth - Queue depth. Defaults to 16.
  • :log-fn - When set a message is logged when the iteration is finished. If no error was encounted log-fn receives the message. If an error is encountered it receives the exception followed by the message. log-fn must be able to take either 1 or 2 arguments.
  • :executor-service - Which executor service to use to run the thread. Defaults to a default one created via default-executor-service.
  • :close-fn - Function to call to close upstream iteration. When not provided src-fn is checked and if it implements AutoCloseable then it's close method is called.