tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
->consumer
(->consumer consumer)
Convert a generic object into a consumer. Works for any java.util.consumer and a Clojure IFn. Returns an implementation of java.util.Consumer
as-spliterator
(as-spliterator item)
Convert a stream or a spliterator into a spliterator.
batch-iter
(batch-iter batch-size item)
Given an iterator, batch it up into an implementation of java.util.List
that
contains batches of the data. Return a sequence of batches.
common-pool-parallelism
(common-pool-parallelism)
Integer number of threads used in the ForkJoinPool's common pool
consume!
(consume! consumer item)
Consume (terminate) a sequence or stream. If the stream is parallel then the consumer had better be threadsafe. Returns the consumer.
convertible-to-iterator?
(convertible-to-iterator? item)
True when this is an iterable or a stream.
cpu-pool-map-reduce
(cpu-pool-map-reduce map-fn reduce-fn fork-join-pool)
Execute map-fn in the separate threads of ForkJoinPool's common pool. Map-fn takes a single long which is it's task index.
doiter
macro
(doiter varname iterable & body)
Execute body for every item in the iterable. Expecting side effects, returns nil.
indexed-consume!
(indexed-consume! consumer item)
Consume (terminate) an iterable. Consumer in this case is expected to be an clojure IFn and it will receive two arguments, a long and the value.
indexed-doiter
macro
(indexed-doiter idxvarname varname iterable & body)
Execute body for every item in the iterable. Expecting side effects, returns nil.
indexed-map-reduce
(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)
(indexed-map-reduce num-iters indexed-map-fn reduce-fn)
(indexed-map-reduce num-iters indexed-map-fn)
Execute indexed-map-fn
over n-groups
subranges of (range num-iters)
.
Then call reduce-fn
passing in entire in order result value sequence.
num-iters
- Indexes are the values of(range num-iters)
.indexed-map-fn
- Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced usingreduce-fn
.reduce-fn
- Function from sequence of values to result.max-batch-size
- Safepoint batch size, defaults to 64000. For more information, see java safepoint documentation.
Implementation:
This function uses the ForkJoinPool/commonPool
to parallelize iteration over
(range num-iters) indexes via splitting the index space up into
(>= n-groups ForkJoinPool/getCommonPoolParallelism)
tasks. In order to respect
safepoints, n-groups may be only be allowed to iterate over up to max-batch-size
indexes.
If the current thread is already in the common pool, this function executes in the current thread.
launch-parallel-for
(launch-parallel-for num-iters indexed-map-fn reduce-fn)
(launch-parallel-for num-iters indexed-map-fn)
Legacy name. See indexed-map-reduce
parallel-for
macro
(parallel-for idx-var num-iters & body)
Like clojure.core.matrix.macros c-for except this expects index that run from 0 -> num-iters. Idx is named idx-var and body will be called for each idx in parallel. Uses forkjoinpool's common pool for parallelism. Assumed to be side effecting; returns nil.
pmap
(pmap map-fn & sequences)
pmap using the commonPool. This is useful for interacting with other primitives, namely indexed-map-reduce which are also based on this pool.
rest-iter
(rest-iter item)
Mutable update the iterator calling 'next' and return iterator.
spliterator-map-reduce
(spliterator-map-reduce data map-fn reduce-fn)
Given a spliterator, a function from spliterator to scalar and a reduction function do an efficient parallelized reduction.
upmap
(upmap map-fn & sequences)
Unordered pmap using the commonPool. This is useful for interacting with other primitives, namely indexed-map-reduce which are also based on this pool.