DType Next 10.124
High Performance Clojure Primitives.
Topics
- dtype-next Buffered Image Support
- dtype-next Cheatsheet
- Why dtype-next?
- dtype-next Tensor Dimensions and Bytecode Generation
- dtype-next Overview
Namespaces
tech.v3.datatype
Base namespace for container creation and elementwise access of data
Public variables and functions:
- ->array
- ->array-buffer
- ->buffer
- ->byte-array
- ->char-array
- ->double-array
- ->float-array
- ->int-array
- ->iterable
- ->long-array
- ->native-buffer
- ->reader
- ->short-array
- ->vector
- ->writer
- alloc-uninitialized
- alloc-zeros
- arg-type
- as-array-buffer
- as-array-buffer-data
- as-buffer
- as-concrete-buffer
- as-native-buffer
- as-native-buffer-data
- as-nd-buffer-descriptor
- as-reader
- as-roaring-bitmap
- as-writer
- cast
- clone
- clone-native
- coalesce!
- coalesce-blocks!
- concat-buffers
- const-reader
- copy!
- copy-raw->item!
- datatype
- ecount
- elemwise-cast
- elemwise-datatype
- emap
- ensure-native
- ensure-reader
- ensure-serializeable
- export-symbols
- get-datatype
- get-value
- indexed-buffer
- iterable?
- make-container
- make-list
- make-reader
- make-reader-fn
- malloc
- prealloc-list
- reader-like?
- reader?
- reverse
- set-constant!
- set-datatype
- set-native-datatype
- set-value!
- shape
- sub-buffer
- unchecked-cast
- vectorized-dispatch-1
- vectorized-dispatch-2
- writer?
tech.v3.datatype.argops
Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.
Public variables and functions:
- ->binary-operator
- ->binary-predicate
- ->comparator
- ->double-comparator
- ->long-comparator
- ->unary-operator
- ->unary-predicate
- arg-min-n
- argfilter
- arggroup
- arggroup-by
- arglast-every
- argmax
- argmin
- argpartition
- argpartition-by
- argshuffle
- argsort
- binary-argfilter
- binary-search
- binary-search-impl
- find-base-comparator
- impl-index-of
- index-comparator
- index-of
- last-index-of
- tech-numerics-kwd?
tech.v3.datatype.bitmap
Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.
Public variables and functions:
tech.v3.datatype.convolve
Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.
Public variables and functions:
tech.v3.datatype.datetime
Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.
Public variables and functions:
- between
- datetime->epoch
- datetime->milliseconds
- datetime-datatype?
- days-since-epoch->local-date
- duration
- duration->milliseconds
- duration->nanoseconds
- duration-datatype?
- epoch->datetime
- epoch-days->epoch-months
- epoch-months->epoch-days
- epoch-months->local-date
- instant
- instant->local-date-time
- instant->microseconds-since-epoch
- instant->milliseconds-since-epoch
- instant->seconds-since-epoch
- instant->zoned-date-time
- local-date
- local-date->days-since-epoch
- local-date->epoch-months
- local-date->instant
- local-date->local-date-time
- local-date->milliseconds-since-epoch
- local-date->zoned-date-time
- local-date-time
- local-date-time->instant
- local-date-time->local-date
- local-date-time->local-time
- local-date-time->milliseconds-since-epoch
- local-date-time->zoned-date-time
- local-time
- local-time->instant
- local-time->local-date-time
- local-time->milliseconds
- local-time->seconds
- long-temporal-field
- microseconds-since-epoch->instant
- millisecond-descriptive-statistics
- milliseconds->datetime
- milliseconds->duration
- milliseconds->local-time
- milliseconds-in-day
- milliseconds-in-hour
- milliseconds-in-minute
- milliseconds-in-second
- milliseconds-in-week
- milliseconds-since-epoch->instant
- milliseconds-since-epoch->local-date
- milliseconds-since-epoch->local-date-time
- milliseconds-since-epoch->local-time
- milliseconds-since-epoch->zoned-date-time
- minus-temporal-amount
- nanoseconds->duration
- nanoseconds-in-day
- nanoseconds-in-hour
- nanoseconds-in-millisecond
- nanoseconds-in-minute
- nanoseconds-in-second
- nanoseconds-in-week
- plus-temporal-amount
- seconds->local-time
- seconds-in-day
- seconds-in-hour
- seconds-in-minute
- seconds-since-epoch->instant
- system-zone-id
- system-zone-offset
- utc-zone-id
- utc-zone-offset
- variable-rolling-window-ranges
- zoned-date-time
- zoned-date-time->instant
- zoned-date-time->milliseconds-since-epoch
tech.v3.datatype.errors
Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.
Public variables and functions:
tech.v3.datatype.ffi
Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.
Public variables and functions:
- c->string
- clear-string->c-stats!
- define-foreign-interface
- define-library
- define-library!
- define-library-functions
- define-library-interface
- enable-string->c-stats!
- find-library
- find-symbol
- foreign-interface-instance->c
- if-class
- instantiate-class
- instantiate-foreign-interface
- instantiate-library
- jdk-ffi?
- jdk-mmodel?
- jna-ffi?
- library-loadable?
- library-singleton
- make-ptr
- PLibrarySingleton
- pointer->address
- PToPointer
- ptr->struct
- set-ffi-impl!
- string->c
- string->c-data-histogram
- struct-member-ptr
- utf16-platform-encoding
- utf32-platform-encoding
- windows-platform?
tech.v3.datatype.ffi.clang
General utilites to work with clang in order to define things like structs or enumerations.
Public variables and functions:
tech.v3.datatype.ffi.graalvm
Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.
Public variables and functions:
tech.v3.datatype.functional
Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to '+' checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the '+' function itself.
Public variables and functions:
- *
- +
- -
- /
- <
- <=
- >
- >=
- abs
- acos
- and
- asin
- atan
- atan2
- bit-and
- bit-and-not
- bit-clear
- bit-flip
- bit-not
- bit-or
- bit-set
- bit-shift-left
- bit-shift-right
- bit-test
- bit-xor
- bool-reader->indexes
- cbrt
- ceil
- cos
- cosh
- cummax
- cummin
- cumprod
- cumsum
- descriptive-statistics
- distance
- distance-squared
- dot-product
- eq
- equals
- even?
- exp
- expm1
- fill-range
- finite?
- fixed-rolling-window
- floor
- get-significand
- hypot
- identity
- ieee-remainder
- infinite?
- kendalls-correlation
- kurtosis
- linear-regressor
- log
- log10
- log1p
- logistic
- magnitude
- magnitude-squared
- mathematical-integer?
- max
- mean
- mean-fast
- median
- min
- mode
- nan?
- neg?
- next-down
- next-up
- normalize
- not
- not-eq
- odd?
- or
- pearsons-correlation
- percentiles
- pos?
- pow
- quartile-1
- quartile-3
- quartile-outlier-fn
- quartiles
- quot
- reduce-*
- reduce-+
- reduce-max
- reduce-min
- rem
- rint
- round
- shift
- signum
- sin
- sinh
- skew
- spearmans-correlation
- sq
- sqrt
- standard-deviation
- sum
- sum-fast
- tan
- tanh
- to-degrees
- to-radians
- ulp
- unsigned-bit-shift-right
- variance
- zero?
tech.v3.datatype.gradient
Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.
Public variables and functions:
tech.v3.datatype.mmap-writer
Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.
Public variables and functions:
tech.v3.datatype.native-buffer
Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.
Public variables and functions:
- accessor-maps
- alloc-uninitialized
- alloc-zeros
- as-native-buffer
- clone-native
- construct-binary-buffer
- data->jvm
- ensure-native
- free
- jvm->data
- malloc
- native-buffer->byte-array
- native-buffer->map
- native-buffer->string
- native-buffer-byte-len
- raw-read
- raw-write
- read-byte
- read-double
- read-float
- read-int
- read-long
- read-short
- set-endianness
- set-gc-obj
- set-native-datatype
- set-parent
- swap-value
- unsafe
- wrap-address
- write-int
- write-long
tech.v3.datatype.nippy
Nippy bindings for datatype base types and tensor types
Public variables and functions:
tech.v3.datatype.packing
Implements the 'packed/unpacked' concept in datatype. This allows us to take objects like LocalDate's and store them in int32 storage which dramatically decreases their size.
Public variables and functions:
tech.v3.datatype.reductions
High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.
tech.v3.datatype.sampling
Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.
Public variables and functions:
tech.v3.datatype.statistics
Nan-aware, high quality and reasonably efficient summation and descriptive statistics.
Public variables and functions:
- all-descriptive-stats-names
- define-descriptive-stats
- descriptive-statistics
- kendalls-correlation
- kurtosis
- max
- mean
- median
- min
- mode
- moment-2
- moment-3
- moment-4
- pearsons-correlation
- percentiles
- quartile-1
- quartile-3
- quartile-outlier-fn
- quartiles
- skew
- spearmans-correlation
- standard-deviation
- sum
- variance
tech.v3.datatype.struct
Structs are datatypes composed of primitive datatypes or other structs.
Similar to records except they do not support string or object columns,
only numeric values. They have memset-0 initialization, memcpy copy
semantics. For correct equals, hashing, convert struct into a normal
persistent map via into
.
Public variables and functions:
tech.v3.datatype.wavelet
Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.
tech.v3.libs.buffered-image
Bindings to buffered images for the datatype system
tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
Public variables and functions:
- ->consumer
- ->iterator
- as-spliterator
- batch-iter
- common-pool-parallelism
- consume!
- convertible-to-iterator?
- cpu-pool-map-reduce
- default-max-batch-size
- doiter
- in-fork-join-task?
- indexed-consume!
- indexed-doiter
- indexed-map-reduce
- launch-parallel-for
- parallel-for
- pmap
- rest-iter
- spliterator-map-reduce
- unchunk
- upmap
tech.v3.parallel.queue-iter
Read an iterator from in a separate thread into a queue returning a new iterator. This allows a traversal of potentially blocking information to happen in a separate thread with a fixed queue size.
Public variables and functions:
tech.v3.tensor
ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.
Public variables and functions:
- ->jvm
- ->tensor
- as-tensor
- broadcast
- clone
- columns
- compute-tensor
- const-tensor
- construct-tensor
- dimensions-dense?
- dims-suitable-for-desc?
- ensure-native
- ensure-nd-buffer-descriptor
- ensure-tensor
- map-axis
- mget
- mset!
- native-tensor
- nd-buffer->buffer-reader
- nd-buffer-descriptor->tensor
- nd-copy!
- new-tensor
- reduce-axis
- reshape
- rotate
- rows
- select
- simple-dimensions?
- slice
- slice-right
- tensor->buffer
- tensor->dimensions
- tensor-copy!
- tensor?
- transpose
- typed-compute-tensor
tech.v3.tensor.color-gradients
Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.
Public variables and functions:
tech.v3.tensor.dimensions
Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.
Public variables and functions:
- ->2d-shape
- ->batch-shape
- ->global->local
- ->least-rapidly-changing-dimension
- ->local->global
- ->most-rapidly-changing-dimension
- access-increasing?
- broadcast
- buffer-ecount
- create-dimension-transforms
- dense?
- dimensions
- dimensions->column-stride
- direct?
- ecount
- get-elem-dims-local->global
- in-place-reshape
- indirect?
- local-address->local-shape
- matrix-column-stride
- matrix-element-stride
- native?
- rotate
- select
- shape
- slice
- slice-right
- strides
- trans-2d-shape
- transpose