charred.parallel
Parallelism helpers
default-executor-service
(default-executor-service)
Default executor service that is created via 'newCachedThreadPool with a custom thread factory that creates daemon threads. This is an executor service that is suitable for blocking operations as it creates new threads as needed.
indexed-map-reduce
(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)
(indexed-map-reduce num-iters indexed-map-fn reduce-fn)
(indexed-map-reduce num-iters indexed-map-fn)
Execute indexed-map-fn
over n-groups
subranges of (range num-iters)
.
Then call reduce-fn
passing in entire in order result value sequence.
num-iters
- Indexes are the values of(range num-iters)
.indexed-map-fn
- Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced usingreduce-fn
.reduce-fn
- Function from sequence of values to result.max-batch-size
- Safepoint batch size, defaults to 64000. For more information, see java safepoint documentation.
Implementation:
This function uses the ForkJoinPool/commonPool
to parallelize iteration over
(range num-iters) indexes via splitting the index space up into
(>= n-groups ForkJoinPool/getCommonPoolParallelism)
tasks. In order to respect
safepoints, n-groups may be only be allowed to iterate over up to max-batch-size
indexes.
If the current thread is already in the common pool, this function executes in the current thread.
queue-supplier
(queue-supplier src-fn & [options])
Given a supplier or clojure fn, create a new thread that will read that fn and place the results into a queue of a fixed size. Returns new suplier. Iteration stops when the src-fn returns nil.
Options:
:queue-depth
- Queue depth. Defaults to 16.:log-fn
- When set a message is logged when the iteration is finished. If no error was encounted log-fn receives the message. If an error is encountered it receives the exception followed by the message. log-fn must be able to take either 1 or 2 arguments.:executor-service
- Which executor service to use to run the thread. Defaults to a default one created via default-executor-service.:close-fn
- Function to call to close upstream iteration. When not provided src-fn is checked and if it implements AutoCloseable then it's close method is called.