tech.v3.parallel.queue-iter

Read an iterator from in a separate thread into a queue returning a new iterator. This allows a traversal of potentially blocking information to happen in a separate thread with a fixed queue size.

default-executor-service

(default-executor-service)

Default executor service that is created via 'newCachedThreadPool with a custom thread factory that creates daemon threads. This is an executor service that is suitable for blocking operations as it creates new threads as needed.

iter-fn

(iter-fn item)

Create a non threadsafe clojure fn that will iterate the iterator.

queue-fn

(queue-fn src-fn & [options])

Given a clojure fn, create a new thread that will read that fn and place the results into a queue of a fixed size. Returns new fn. Iteration stops when the src-fn returns nil.

Options:

  • :queue-depth - Queue depth. Defaults to 16.
  • :log-level - When set a message is logged when the iteration is finished.
  • :executor-service - Which executor service to use to run the thread. Defaults to a default one created via default-executor-service.
  • :close-fn - Function to call to close upstream iteration.

queue-iter

(queue-iter iter & [options])

Given an object with a conversion to iterator, create a new thread that will read that iterator and place the results into a queue of a fixed size. Returns new iterator.

Options:

  • :queue-depth - Queue depth. Defaults to 16.
  • :log-level - When set a message is logged when the iteration is finished.
  • :executor-service - Which executor service to use to run the thread. Defaults to a default one created via default-executor-service.