tech.v3.tensor

ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.

This system relies heavily on the tech.v3.tensor.dimensions namespace to provide the optimized indexing operator from ND space to buffer space and back.

There is an ABI in the form of nd-buffer-descriptors that is a map containing:

:ptr - long value
:elemwise-datatype - primitive datatype of the buffer.
:shape - buffer of :int64 dimensions.
:strides - buffer of :int64 byte stride counts.

Optionally more keys and the source agrees not to release the source data until this map goes out of scope.

->jvm

(->jvm tens & args)

Conversion to storage that structurally represented the tensor so for exampe a 2 2 tensor will be of the form [1 2]. Base storage is either jvm-array or persistent-vector.

10.000 - Note that this function now takes an option map as opposed to a variable number of option arguments.

->tensor

(->tensor data & args)

Convert some data into a tensor via copying the data. The datatype and container type can be specified. The datatype defaults to the datatype of the input data and container type defaults to jvm-heap.

10.000 - Note that this function now takes an option map as opposed to a variable number of option arguments.

Options:

:datatype - Data of the storage. Defaults to the datatype of the passed-in data.
:container-type - Specify the container type of the new tensor. Defaults to :jvm-heap.
:resource-type - One of tech.v3.resource/track :track-type options. If allocating native tensors, nil corresponds to :gc:.

as-tensor

(as-tensor tens)

Attempts an in-place conversion of this object to a tech.v3.datatype.NDBuffer interface. For a guaranteed conversion, use ensure-tensor.

broadcast

(broadcast tens new-shape)

Broadcast an element into a new (larger) shape. The new shape's dimension must be even multiples of the old shape's dimensions. Elements are repeated.

See reduce-axis for the opposite operation.

clone

(clone tens & args)

Clone a tensor via copying the tensor into a new container. Datatype defaults to the datatype of the tensor and container-type defaults to :java-heap.

10.000 - Note that this function now takes an option map as opposed to a variable number of option arguments.

Options:

:datatype - Specify a new datatype to copy data into.
:container-type - Specify the container type of the new tensor. Defaults to :jvm-heap.
- :resource-type - One of tech.v3.resource/track :track-type options. If allocating native tensors, nil corresponds to gc:.

columns

(columns tens)

Return the columns of the tensor in a randomly-addressable structure.

compute-tensor

(compute-tensor shape per-pixel-op datatype)(compute-tensor output-shape per-pixel-op)

Create a new tensor which calls into op for every operation. Op will receive n-dimensional long arguments and the result will be :unchecked-casted to whatever datatype the tensor is reporting.

Example:

user> (require '[tech.v3.tensor :as dtt])
nil
user> (dtt/compute-tensor [2 2] (fn [& args] (vec args)) :object)
#tech.v3.tensor<object>[2 2]
[[[0 0] [0 1]]
 [[1 0] [1 1]]]
user> (dtt/compute-tensor [2 2 2] (fn [& args] (vec args)) :object)
#tech.v3.tensor<object>[2 2 2]
[[[[0 0 0] [0 0 1]]
  [[0 1 0] [0 1 1]]]
 [[[1 0 0] [1 0 1]]
  [[1 1 0] [1 1 1]]]]

const-tensor

(const-tensor value shape)

Construct a tensor from a value and a shape. Data is represented efficiently via a const-reader.

construct-tensor

(construct-tensor buffer dimensions & args)

Construct an implementation of tech.v3.datatype.NDBuffer from a buffer and a dimensions object. See tech.v3.tensor.dimensions/dimensions.

dimensions-dense?

(dimensions-dense? tens)

Returns true of the dimensions of a tensor are dense, meaning no gaps due to striding.

dims-suitable-for-desc?

(dims-suitable-for-desc? tens)

Are the dimensions of this object suitable for use in a buffer description? breaks due to striding.

ensure-native

(ensure-native tens options)(ensure-native tens)

Ensure this tensor is native backed and packed. Tens is cloned into a native tensor with the same datatype and :resource-type :auto by default.

Options are the same as clone with the exception of :resource-type.

:resource-type - Defaults to :auto - used as tech.v3.resource/track track-type.

ensure-nd-buffer-descriptor

(ensure-nd-buffer-descriptor tens)

Get a buffer descriptor from the tensor. This may copy the data. If you want to ensure sharing, use the protocol ->nd-buffer-descriptor function.

ensure-tensor

(ensure-tensor tens)

Create an implementation of tech.v3.datatype.NDBuffer from an object. If possible, represent the data in-place.

map-axis

(map-axis tens map-fn axis)(map-axis tens map-fn)

Map a function from vector->vector replacing the values along an axis.

tens - input tensor to use.
map-fn - maps from vector->vector. Must return a vector of the same count as the input.
axis - Defaults to -1 meaning the last axis. So the default would map across the rows of a matrix.

Example:

user> t
#tech.v3.tensor<float64>[4 3]
[[0.000 1.000 2.000]
 [3.000 4.000 5.000]
 [6.000 7.000 8.000]
 [9.000 10.00 11.00]]
user> (require '[tech.v3.datatype.functional :as dfn])
nil
user> (defn center-1d [d] (dfn// d (dfn/mean d)))
#'user/center-1d
user> (dtt/map-axis t center-1d -1)
#tech.v3.tensor<float64>[4 3]
[[ 0.000 1.000 2.000]
 [0.7500 1.000 1.250]
 [0.8571 1.000 1.143]
 [0.9000 1.000 1.100]]
user> (dtt/map-axis t center-1d -2)
#tech.v3.tensor<float64>[4 3]
[[ 0.000 0.1818 0.3077]
 [0.6667 0.7273 0.7692]
 [ 1.333  1.273  1.231]
 [ 2.000  1.818  1.692]]

mget

(mget tens x)(mget tens x y)(mget tens x y z)(mget tens x y z & args)

Get an item from an ND object. If fewer dimensions are specified than exist then the return value is a new tensor as a select operation is performed.

mset!

(mset! tens value)(mset! tens x value)(mset! tens x y value)(mset! tens x y z value)(mset! tens x y z w & args)

Set value(s) on an ND object. If fewer indexes are provided than dimension then a tensor assignment is done and value is expected to be the same shape as the subrect of the tensor as indexed by the provided dimensions. Returns t.

native-tensor

(native-tensor shape datatype options)(native-tensor shape datatype)(native-tensor shape)

Create a new native-backed tensor with a :resource-type :auto default resource type.

Options are the same as new-tensor with some additions:

:resource-type - Defaults to :auto - used as tech.v3.resource/track track-type.
:uninitialized? - Defaults to false - do not 0-initialize the memory.

nd-buffer->buffer-reader

(nd-buffer->buffer-reader b)

nd-buffer-descriptor->tensor

(nd-buffer-descriptor->tensor desc)

Given a buffer descriptor, produce a tensor

nd-copy!

(nd-copy! src dst)

similar to tech.v3.datatype/copy! except this copy is ND aware and parallelizes over the outermost dimension. This useful for compute tensors. If you have tensors such as images, see tensor-copy!.

new-tensor

(new-tensor shape & args)

Create a new tensor with a given shape.

10.000 - Note that this function now takes an option map as opposed to a variable number of option arguments.

Options:

:datatype - Data of the storage. Defaults to :float64.
:container-type - Specify the container type of the new tensor. Defaults to :jvm-heap.
:resource-type - One of tech.v3.resource/track :track-type options. If allocating native tensors, nil corresponds to :gc:.

reduce-axis

(reduce-axis tens reduce-fn axis res-dtype)(reduce-axis tens reduce-fn axis)(reduce-axis tens reduce-fn)

Reduce a tensor along an axis using reduce-fn on the elemwise entries.

tensor - input tensor to use.
reduce-fn - lazily applied reduction applied to each input. Inputs are 1-dimensional vectors. Use clone to force the operation.
axis - Defaults to -1 meaning the last axis. So the default would reduce across the rows of a matrix.
res-dtype - result datatype, defaults to the datatype of the incoming tensor.

Example:

user> (def t (dtt/->tensor (partition 3 (range 12)) {:datatype :float64}))
#'user/t
user> t
#tech.v3.tensor<float64>[4 3]
[[0.000 1.000 2.000]
 [3.000 4.000 5.000]
 [6.000 7.000 8.000]
 [9.000 10.00 11.00]]
user> (dtt/reduce-axis t dfn/sum 0)
#tech.v3.tensor<float64>[3]
[18.00 22.00 26.00]
user> (dtt/reduce-axis t dfn/sum 1)
user> (def t (dtt/new-tensor [2 3 5]))
#'user/t
user> (dtype/shape (dtt/reduce-axis t dfn/sum 0))
Syntax error compiling at (*cider-repl cnuernber/dtype-next:localhost:37199(clj)*:83:7).
No such namespace: dtype
user> (require '[tech.v3.datatype :as dt])
nil
user> (dt/shape (dtt/reduce-axis t dfn/sum 0))
[3 5]
user> (dt/shape (dtt/reduce-axis t dfn/sum 1))
[2 5]
user> (dt/shape (dtt/reduce-axis t dfn/sum 2))
[2 3]

For the opposite - adding dimensions via repetition - see broadcast.

reshape

(reshape tens new-shape)

Reshape this item into a new shape. For this to work, the tensor namespace must be required. Always returns a tensor.

rotate

(rotate tens offset-vec)

Rotate dimensions. Offset-vec must have same count as the rank of t. Elements of that dimension are rotated by the amount specified in the offset vector with 0 indicating no rotation.

rows

(rows tens)

Return the rows of the tensor in a randomly-addressable structure.

select

(select tens & args)

Perform a subrect projection or selection Shape arguments may be readers, ranges, integers, or the keywords :all :lla. :all means take the entire dimension, :lla means reverse the dimension. Arguments are applied left to right and any missing arguments are assumed to be :all.

Example:

  user> (dtt/select (dtt/->tensor [1 2 3]) [0 2])
  #tech.v3.tensor<object>[2]
  [1 3]

  user> (def tensor (dtt/->tensor
              [[1 2 3]
               [4 5 6]]))
  #'user/tensor
  user> (dtt/select tensor [1] [0 2])
  #tech.v3.tensor<object>[2]
  [4 6]
  user> (dtt/select tensor [0 1] [1 2])
  #tech.v3.tensor<object>[2 2]
  [[2 3]
   [5 6]]

simple-dimensions?

(simple-dimensions? tens)

Are the dimensions of this object simple meaning read in order with no breaks due to striding.

slice

(slice tens n-dims)

Slice off Y leftmost dimensions returning a reader of objects. If all dimensions are sliced of then the reader reads actual elements, else it reads subrect tensors.

slice-right

(slice-right tens n-dims)

Slice off Y rightmost dimensions returning a reader of objects. If all dimensions are sliced of then the reader reads actual elements, else it reads subrect tensors.

tensor->buffer

(tensor->buffer tens)

Get the buffer from a tensor.

tensor->dimensions

(tensor->dimensions tens)

Get the dimensions object from a tensor.

tensor-copy!

(tensor-copy! src dst options)(tensor-copy! src dst)

Specialized copy with optimized pathways for when tensors have regions of contiguous data. As an example consider a sub-image of a larger image. Each row can be copied contiguously into a new image but there are gaps between them.

tensor?

(tensor? item)

Returns true if this implements the tech.v3.datatype.NDBuffer interface.

transpose

(transpose tens reorder-indexes)

In-place transpose an n-d object into a new shape. Returns a tensor.

reorder-indexes are the relative indexes of the old indexes

Example:

user> (def tensor (dtt/->tensor (partition 2 (partition 3 (flatten (repeat 6 [:r :g :b]))))))
#'user/tensor
user> tensor
#tech.v3.tensor<object>[3 2 3]
[[[:r :g :b]
  [:r :g :b]]
 [[:r :g :b]
  [:r :g :b]]
 [[:r :g :b]
  [:r :g :b]]]
user> (dtt/transpose tensor [2 0 1])
#tech.v3.tensor<object>[3 3 2]
[[[:r :r]
  [:r :r]
  [:r :r]]
 [[:g :g]
  [:g :g]
  [:g :g]]
 [[:b :b]
  [:b :b]
  [:b :b]]]
user> (dtt/transpose tensor [1 2 0])
#tech.v3.tensor<object>[2 3 3]
[[[:r :r :r]
  [:g :g :g]
  [:b :b :b]]
 [[:r :r :r]
  [:g :g :g]
  [:b :b :b]]]

typed-compute-tensor

macro

(typed-compute-tensor datatype advertised-datatype rank shape op-code-args op-code)(typed-compute-tensor advertised-datatype rank shape op-code-args op-code)(typed-compute-tensor advertised-datatype shape op-code-args op-code)

Fastest possible inline compute tensor. The code to generate the next element is output inline into the tensor definition.

For the 4 argument version to work, shape must be compile time introspectable object with count so for instance [a b c] will work but item-shape will throw an exception.

:datatype - One of #{:int64 :float64} or :object is assumed. This indicates the tensor interface definition and read operations that will be implemented. See 'java/tech/v3/datatype/Long|Double|ObjectTensorReader.java.
:advertised-datatype - Datatype you will tell the world.
:rank - compile time introspectable rank. Indicates which ndReadX overloads will be implemented.
:shape - Shape of the output tensor.
:op-code-args - Op code arguments. Expected to be a vector of argument names such as [y x c]. Let destructuring is *NOT* supported beyond 3 variables at this time.!!!.
:op-code - Code which executes the read operation.

Results in an implementation of NDBuffer which efficiently performs a 1,2 or 3 dimension ND read operation.