tech.v3.datatype

Base namespace for container creation and elementwise access of data

->array

(->array datatype options item)(->array datatype item)(->array item)

Perform a NaN-aware conversion into an array. Default nan-strategy is :keep. Nan strategies can be: :keep :remove :exception

->array-buffer

(->array-buffer datatype options item)(->array-buffer datatype item)(->array-buffer item)

Perform a NaN-aware conversion into an array buffer. Default nan-strategy is :keep.

Options:

  • :nan-strategy - Nan strategies can be: :keep :remove :exception. :remove can result in a shorter array buffer than the input data. Default is :keep.

->buffer

(->buffer item)

Convert this item to a buffer implementation or throw an exception.

->byte-array

(->byte-array data)

Efficiently convert nearly anything into a byte array.

->char-array

(->char-array data)

Efficiently convert nearly anything into a char array.

->double-array

(->double-array options data)(->double-array data)

Nan-aware conversion to double array. See documentation for ->array-buffer. Returns a double array. options -

  • nan-strategy - :keep (default) :remove :exception

->float-array

(->float-array options data)(->float-array data)

Nan-aware conversion to double array. See documentation for ->array-buffer. Returns a double array. options -

  • nan-strategy - :keep (default) :remove :exception

->int-array

(->int-array data)

Efficiently convert nearly anything into a int array.

->iterable

(->iterable item)

Ensure this object is iterable. If item is iterable, return the item. If the item is convertible to a buffer object , convert to it and return the object. Else if the item is a scalar constant, return an infinite sequence of items.

->long-array

(->long-array data)

Efficiently convert nearly anything into a long array.

->native-buffer

(->native-buffer data)

->reader

(->reader item)(->reader item read-datatype)

If this object has a read-only or read-write conversion to a buffer object return the buffer object. Else throw an exception.

->short-array

(->short-array data)

Efficiently convert nearly anything into a short array.

->vector

(->vector item)

Copy a thing into a persistent vector.

->writer

(->writer item)

If this item is convertible to a PrimtiveIO writing the perform the conversion and return the interface. Else throw an excepion.

alloc-uninitialized

(alloc-uninitialized dtype ec)(alloc-uninitialized dtype ec options)

Allocate an uninitialized buffer. See options for malloc.

alloc-zeros

(alloc-zeros dtype ec)(alloc-zeros dtype ec options)

Allocate a buffer of zeros. See options for malloc.

arg-type

(arg-type arg)

Return the type of a thing. Types could be: :scalar :iterable :reader :tensor (reader with more than 1 dimension)

as-array-buffer

(as-array-buffer item)

If this item is convertible to a tech.v3.datatype.array_buffer.ArrayBuffer then convert it and return the typed buffer

as-array-buffer-data

(as-array-buffer-data item)

as-buffer

(as-buffer item)

If this item is or has a conversion to an implementation of the Buffer interface then return the buffer interface for this object. Else return nil.

as-concrete-buffer

(as-concrete-buffer item)

If this item is representable as a base buffer type, either native or array, return that buffer.

as-native-buffer

(as-native-buffer item)

If this item is convertible to a tech.v3.datatype.native_buffer.NativeBuffer return it. Return nil otherwise.

as-native-buffer-data

(as-native-buffer-data item)

as-nd-buffer-descriptor

(as-nd-buffer-descriptor src-item)

If this item is convertible to a buffer descriptor, convert it. Else return nil.

Buffer descriptors are a ND description of data. For example, a native 3x3 tensor has a buffer description like thus:

{:ptr 140330005614816
 :datatype :float64
 :endianness :little-endian
 :shape [3 3]
 :strides [24 8]}

This design allows zero-copy transfer between neanderthal, numpy, tvm, etc.

as-reader

(as-reader item)(as-reader item read-datatype)

If this object has a read-only or read-write conversion to a primitive io object return the buffer object.

as-roaring-bitmap

(as-roaring-bitmap item)

as-writer

(as-writer item)

If this item is convertible to a PrimtiveIO writing the perform the conversion and return the interface.

cast

(cast value datatype)

Perform a checked cast of a value to specific datatype.

clone

(clone item)

Clone an object. Can clone anything convertible to a reader.

clone-native

(clone-native data options)(clone-native data)

Extremely fast clone assuming src is a native buffer. Exception otherwise. See options for malloc.

coalesce!

(coalesce! buffer raw-data)

Coalesce data from a raw sequence of things into a contiguous buffer. Returns the buffer; uses the copy-raw->item! pathway.

coalesce-blocks!

(coalesce-blocks! dst src-seq)

Copy a sequence of blocks of countable things into a larger countable thing.

concat-buffers

(concat-buffers datatype buffers)(concat-buffers buffers)

Concatenate a list of buffers into a single unified buffer. This shares the data and it works when the number of buffers is relatively small. The failure case for this is a large number of small buffers where it will be faster to simply copy all the data into a new buffer.

const-reader

(const-reader item n-elems)

Create a new reader that only returns the item for the provided indexes.

copy!

(copy! src dst options)(copy! src dst)

Mutably copy values from a src container into a destination container. Returns the destination container.

copy-raw->item!

(copy-raw->item! raw-data buffer options)(copy-raw->item! raw-data buffer)

Copy raw data into a buffer. Data may be a sequence of numbers or a sequence of containers. Data will be coalesced into the buffer. Returns a tuple of: buffer final-offset.

datatype

(datatype item)

Return this object's actual datatype. This is not the same as the DEPRECATED get-datatype. That function maps to elemwise-datatype. This maps to the parameterization of the object, so for instance a list of ints might be:

  {:container-type :list :elemwise-datatype :int32}

Defaults to this object's elemwise-datatype.

ecount

(ecount item)

Return the long number of elements in the object.

elemwise-cast

(elemwise-cast item new-dtype)

Create a new thing by doing a checked cast from the old values to the new values upon at read time. Return value will report new-dtype as it's elemwise-datatype.

elemwise-datatype

(elemwise-datatype item)

Return the datatype one would expect when iterating through a container of this type. For scalars, return your elemental datatype.

emap

(emap map-fn res-dtype x)(emap map-fn res-dtype x y)(emap map-fn res-dtype x y & args)

Elemwise map:

  1. If input are all scalars, results in a scalar.
  2. If any inputs are iterables, results in an iterable.
  3. Either a reader or a tensor is returned. All input shapes have to match.

res-dtype is nil it is deduced from unifying the argument datatypes meaning map-fn, unless it is typed, is assumed to keep data in the same numeric space as the input.

ensure-native

(ensure-native input outbuf)(ensure-native input)

If input is already a native buffer and has the same datatype as output, return input. Else copy input into output and return output.

ensure-reader

(ensure-reader item n-const-elems)(ensure-reader item)

Ensure item is randomly addressable. This may copy the data into a randomly accessible container.

ensure-serializeable

(ensure-serializeable item)

Ensure this is an efficiently serializeable datatype object. For nippy support across buffers and tensors require tech.v3.datatype.nippy.

get-datatype

(get-datatype item)

Legacy method, returns elemwise-datatype

get-value

(get-value item idx)

Get a value from an object via conversion to a reader.

indexed-buffer

(indexed-buffer indexes item)

Create a new Buffer implementation that indexes into a previous Buffer implementation via the provided indexes.

iterable?

(iterable? item)

Return true if this is either a instance of Iterable or an instance of java.util.streams.Stream.

make-container

(make-container container-type datatype options elem-seq-or-count)(make-container container-type datatype elem-seq-or-count)(make-container datatype elem-seq-or-count)(make-container elem-seq-or-count)

Make a container of a given datatype. Options are container specific and in general unused. Values will be copied into given container using the most efficient pathway possible.

make-list

(make-list datatype n-elems-or-data)(make-list datatype)

Make an instance of a tech.v3.datatype.Buffer. These have typed add* methods, implement tech.v3.datatype.Buffer, and a guaranteed in-place transformation to a concrete buffer (defaults to an array buffer).

If an integer is passed in an empty list that has preallocated storage is returned. If a sequence of data is passed in then a non-empty list that contains just those elements is returned.

Example:

user> (require '[tech.v3.datatype :as dt])
nil
user> (dt/make-list :float32 10)
#list<float32>[0]
[]
user> (dt/make-list :float32 (range 10))
#list<float32>[10]
[0.000, 1.000, 2.000, 3.000, 4.000, 5.000, 6.000, 7.000, 8.000, 9.000]

make-reader

macro

(make-reader datatype n-elems read-op)(make-reader reader-datatype advertised-datatype n-elems read-op)

Make a reader. Datatype must be a compile time visible object. read-op has 'idx' in scope which is the index to read from. Returns a reader of the appropriate type for the passed in datatype. Results are unchecked casted to the appropriate datatype. It is up to you to ensure this is the result you want or throw an exception.

This function creates a compile-time object that can allow maximum performance. For most use cases this is probably overkill and 'emap' is a more ideal pathway that matches Clojure constructs a bit closer.

reader-datatype must be a compile time constant but advertised datatype need not be.

user> (dtype/make-reader :float32 5 idx) 0.0 1.0 2.0 3.0 4.0 user> (dtype/make-reader :boolean 5 idx) true true true true true user> (dtype/make-reader :boolean 5 (== idx 0)) true false false false false user> (dtype/make-reader :float32 5 (* idx 2)) 0.0 2.0 4.0 6.0 8.0

make-reader-fn

(make-reader-fn datatype advertised-datatype n-elems read-fn)

malloc

(malloc n-bytes opts)(malloc n-bytes)

Malloc memory. If a desired buffer type is needed follow up with set-native-datatype.

Options:

  • :resource-type - defaults to :gc - maps to :track-type in tech.v3.resource but can also be set to nil in which case the data is not tracked this library will not clean it up.
  • :uninitialized? - do not initialize to zero. Use for perf in very very rare cases.
  • :endianness - Either :little-endian or :big-endian - defaults to platform.
  • :log-level - one of #{:debug :trace :info :warn :error :fatal} or nil if no logging is desired. When enabled allocations and frees will be logged in the same manner as tech.jna.

map-factory

(map-factory key-seq)

Create an IFn taking exactly n-keys arguments to rapidly create a map. This moves the key-checking to the factory creation and simply creates a new map as fast as possible when requrested

prealloc-list

(prealloc-list datatype n-elems)

Make an list with preallocated storage. This function exists to cause a compilation error if older versions of dtype-next are included and is the equivalent (in latest versions of dtype-next) to (make-list datatype n-elems).

reader-like?

(reader-like? argtype)

Returns true if this argument type is convertible to a reader.

reader?

(reader? item)

True if this item is convertible to a read-only or read-write Buffer interface.

reverse

(reverse item)

Reverse an sequence, range or reader.

  • If range, returns a new range.
  • If sequence, uses clojure.core/reverse
  • If reader, returns a new reader that performs an in-place reverse.

set-constant!

(set-constant! item offset length value)(set-constant! item offset value)(set-constant! item value)

Set a contiguous region of a buffer to a constant value

set-datatype

(set-datatype item new-dtype)

Legacy method. Performs elemwise-cast.

set-native-datatype

(set-native-datatype item datatype)

Set the datatype of a native buffer. n-elems will be recalculated.

set-value!

(set-value! item idx value)

Set a value on an object via conversion to a writer. set-value can also take a tuple in which case a select operation will be done on the tensor and value applied to the result. See also set-constant!.

Example:

tech.v3.tensor.integration-test> (def test-tens (dtt/->tensor (->> (range 27)
                                     (partition 3)
                                     (partition 3))
                                :datatype :float64))

#'tech.v3.tensor.integration-test/test-tens
tech.v3.tensor.integration-test> test-tens
#tech.v3.tensor<float64>[3 3 3]
[[[0.000 1.000 2.000]
  [3.000 4.000 5.000]
  [6.000 7.000 8.000]]
 [[9.000 10.00 11.00]
  [12.00 13.00 14.00]
  [15.00 16.00 17.00]]
 [[18.00 19.00 20.00]
  [21.00 22.00 23.00]
  [24.00 25.00 26.00]]]
tech.v3.tensor.integration-test> (dtype/set-value! (dtype/clone test-tens) [:all :all (range 2)] 0)
#tech.v3.tensor<float64>[3 3 3]
[[[0.000 0.000 2.000]
  [0.000 0.000 5.000]
  [0.000 0.000 8.000]]
 [[0.000 0.000 11.00]
  [0.000 0.000 14.00]
  [0.000 0.000 17.00]]
 [[0.000 0.000 20.00]
  [0.000 0.000 23.00]
  [0.000 0.000 26.00]]]
tech.v3.tensor.integration-test> (def sv-tens (dtt/reshape (double-array [1 2 3]) [3 1]))
#'tech.v3.tensor.integration-test/sv-tens
tech.v3.tensor.integration-test> (dtype/set-value! (dtype/clone test-tens) [:all :all (range 2)] sv-tens)
#tech.v3.tensor<float64>[3 3 3]
[[[1.000 1.000 2.000]
  [2.000 2.000 5.000]
  [3.000 3.000 8.000]]
 [[1.000 1.000 11.00]
  [2.000 2.000 14.00]
  [3.000 3.000 17.00]]
 [[1.000 1.000 20.00]
  [2.000 2.000 23.00]
  [3.000 3.000 26.00]]]

shape

(shape item)

Return a persistent vector of the shape of the object. Dimensions are returned from least-rapidly-changing to most-rapidly-changing.

sub-buffer

(sub-buffer item offset length)(sub-buffer item offset)

Create a sub-buffer (view) of the given buffer.

unchecked-cast

(unchecked-cast value datatype)

Perform an unchecked cast of a value to specific datatype.

vectorized-dispatch-1

(vectorized-dispatch-1 scalar-fn iterable-fn reader-fn options arg1)(vectorized-dispatch-1 scalar-fn iterable-fn reader-fn arg1)(vectorized-dispatch-1 scalar-fn reader-fn arg1)

Perform a vectorized dispatch meaning return a different object depending on the argument type of arg1. If arg1 is scalar, use scalar-fn. If arg1 is an iterable, use iterable-fn. Finally if arg1 is a reader, use reader-fn. The space the operation is to be performed in can be loosely set with :operation-space in the options. The actual space the operation will be performed in (and the return type of the system) is a combination of :operation-space if it exists and the elemwise datatype of arg1. See tech.v3.datatype.casting/widest-datatype for more information.

vectorized-dispatch-2

(vectorized-dispatch-2 scalar-fn iterable-fn reader-fn options arg1 arg2)(vectorized-dispatch-2 scalar-fn reader-fn arg1 arg2)

Perform a vectorized dispatch meaning return a different object depending on the argument types of arg1 and arg2. If arg1 and arg2 are both scalar, use scalar-fn. If neither argument is a reader, use iterable-fn. Finally if both arguments are readers, use reader-fn. The space the operation is to be performed in can be loosely set with :operation-space in the options. The actual space the operation will be performed in (and the return type of the system) is a combination of :operation-space if it exists and the elemwise datatypes of arg1 and arg2. See tech.v3.datatype.casting/widest-datatype for more information.

writer?

(writer? item)

True if this item is convertible to a Buffer interface that supports writing.