tech.v3.parallel.for

Serial and parallel iteration strategies across iterators and index spaces.

->consumer

(->consumer consumer)

Convert a generic object into a consumer. Works for any java.util.consumer and a Clojure IFn. Returns an implementation of java.util.Consumer

->iterator

(->iterator item)

Convert a stream or an iterable into an iterator.

as-spliterator

(as-spliterator item)

Convert a stream or a spliterator into a spliterator.

batch-iter

(batch-iter batch-size item)

Given an iterator, batch it up into an implementation of java.util.List that contains batches of the data. Return a sequence of batches.

common-pool-parallelism

(common-pool-parallelism)

Integer number of threads used in the ForkJoinPool's common pool

consume!

(consume! consumer item)

Consume (terminate) a sequence or stream. If the stream is parallel then the consumer had better be threadsafe. Returns the consumer.

convertible-to-iterator?

(convertible-to-iterator? item)

True when this is an iterable or a stream.

cpu-pool-map-reduce

(cpu-pool-map-reduce map-fn reduce-fn fork-join-pool)

Execute map-fn in the separate threads of ForkJoinPool's common pool. Map-fn takes a single long which is it's task index.

default-max-batch-size

Default batch size to allow reasonable safe-point access

doiter

macro

(doiter varname iterable & body)

Execute body for every item in the iterable. Expecting side effects, returns nil.

in-fork-join-task?

(in-fork-join-task?)

indexed-consume!

(indexed-consume! consumer item)

Consume (terminate) an iterable. Consumer in this case is expected to be an clojure IFn and it will receive two arguments, a long and the value.

indexed-doiter

macro

(indexed-doiter idxvarname varname iterable & body)

Execute body for every item in the iterable. Expecting side effects, returns nil.

indexed-map-reduce

(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)(indexed-map-reduce num-iters indexed-map-fn reduce-fn)(indexed-map-reduce num-iters indexed-map-fn)

Execute indexed-map-fn over n-groups subranges of (range num-iters). Then call reduce-fn passing in entire in order result value sequence.

  • num-iters - Indexes are the values of (range num-iters).
  • indexed-map-fn - Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced using reduce-fn.
  • reduce-fn - Function from sequence of values to result.
  • max-batch-size - Safepoint batch size, defaults to 64000. For more information, see java safepoint documentation.

Implementation:

This function uses the ForkJoinPool/commonPool to parallelize iteration over (range num-iters) indexes via splitting the index space up into (>= n-groups ForkJoinPool/getCommonPoolParallelism) tasks. In order to respect safepoints, n-groups may be only be allowed to iterate over up to max-batch-size indexes.

If the current thread is already in the common pool, this function executes in the current thread.

launch-parallel-for

(launch-parallel-for num-iters indexed-map-fn reduce-fn)(launch-parallel-for num-iters indexed-map-fn)

Legacy name. See indexed-map-reduce

parallel-for

macro

(parallel-for idx-var num-iters & body)

Like clojure.core.matrix.macros c-for except this expects index that run from 0 -> num-iters. Idx is named idx-var and body will be called for each idx in parallel. Uses forkjoinpool's common pool for parallelism. Assumed to be side effecting; returns nil.

pmap

(pmap map-fn & sequences)

pmap using the commonPool. This is useful for interacting with other primitives, namely indexed-map-reduce which are also based on this pool.

rest-iter

(rest-iter item)

Mutable update the iterator calling 'next' and return iterator.

spliterator-map-reduce

(spliterator-map-reduce data map-fn reduce-fn)

Given a spliterator, a function from spliterator to scalar and a reduction function do an efficient parallelized reduction.

unchunk

(unchunk s)

Given a possibly chunked sequence, return an unchunked sequence.

upmap

(upmap map-fn & sequences)

Unordered pmap using the commonPool. This is useful for interacting with other primitives, namely indexed-map-reduce which are also based on this pool.