DType Next 10.112

High Performance Clojure Primitives.

Topics

Namespaces

tech.v3.datatype

Base namespace for container creation and elementwise access of data

tech.v3.datatype.argops

Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.

tech.v3.datatype.bitmap

Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.

tech.v3.datatype.convolve

Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.

Public variables and functions:

tech.v3.datatype.datetime

Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.

Public variables and functions:

tech.v3.datatype.errors

Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.

tech.v3.datatype.ffi

Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.

tech.v3.datatype.ffi.clang

General utilites to work with clang in order to define things like structs or enumerations.

Public variables and functions:

tech.v3.datatype.ffi.graalvm

Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.

Public variables and functions:

tech.v3.datatype.ffi.size-t

tech.v3.datatype.functional

Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to '+' checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the '+' function itself.

tech.v3.datatype.gradient

Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.

Public variables and functions:

tech.v3.datatype.list

Public variables and functions:

tech.v3.datatype.mmap

Public variables and functions:

tech.v3.datatype.mmap-writer

Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.

Public variables and functions:

tech.v3.datatype.native-buffer

Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.

tech.v3.datatype.nippy

Nippy bindings for datatype base types and tensor types

Public variables and functions:

tech.v3.datatype.packing

Implements the 'packed/unpacked' concept in datatype. This allows us to take objects like LocalDate's and store them in int32 storage which dramatically decreases their size.

tech.v3.datatype.reductions

High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.

tech.v3.datatype.rolling

tech.v3.datatype.sampling

Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.

Public variables and functions:

tech.v3.datatype.statistics

Nan-aware, high quality and reasonably efficient summation and descriptive statistics.

tech.v3.datatype.struct

Structs are datatypes composed of primitive datatypes or other structs. Similar to records except they do not support string or object columns, only numeric values. They have memset-0 initialization, memcpy copy semantics. For correct equals, hashing, convert struct into a normal persistent map via into.

tech.v3.datatype.wavelet

Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.

Public variables and functions:

tech.v3.libs.buffered-image

Bindings to buffered images for the datatype system

tech.v3.parallel.for

Serial and parallel iteration strategies across iterators and index spaces.

tech.v3.parallel.queue-iter

Read an iterator from in a separate thread into a queue returning a new iterator. This allows a traversal of potentially blocking information to happen in a separate thread with a fixed queue size.

Public variables and functions:

tech.v3.tensor

ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.

tech.v3.tensor.color-gradients

Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.

tech.v3.tensor.dimensions

Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.