avclj

libavcodec (FFMPEG) bindings for Clojure.

user> (require '[avclj :as avclj])
nil
user> (require '[avclj.av-codec-ids :as codec-ids])
nil
user> (require '[tech.v3.tensor :as dtt])
nil
user> (avclj/initialize!)
:ok
user> (defn img-tensor
  [shape ^long offset]
  (dtt/compute-tensor shape
                      (fn [^long y ^long x ^long c]
                        (let [ymod (-> (quot (+ y offset) 32)
                                       (mod 2))
                              xmod (-> (quot (+ x offset) 32)
                                       (mod 2))]
                          (if (and (== 0 xmod)
                                   (== 0 ymod))
                            255
                            0)))
                      :uint8))
#'user/img-tensor
user> (let [encoder-name codec-ids/AV_CODEC_ID_H264
            output-fname "test/data/test-video.mp4"]
        (with-open [encoder (avclj/make-video-encoder 256 256 output-fname
                                                      {:encoder-name encoder-name})]
          (dotimes [iter 125]
            (avclj/encode-frame! encoder (img-tensor [256 256 3] iter)))))
nil
  • To use this with buffered images, make sure the pixel formats match and be sure to require tech.v3.libs.buffered-image.
  • If you have a system that is producing java.nio.ByteBuffers then require tech.v3.datatype.nio-buffer.

For manipulating h264 encoder properties, use the :codec-private-options and the various possibilities from the ffmpeg libx264 page.

decoder-names

(decoder-names)

List all decoder names

encoder-names

(encoder-names)

List all of the encoder names

find-decoder

(find-decoder decoder-name)

Find an encoder by name. Name may either be the ffmpeg encoder name such as “libx264” or it may be a codec id in avclj.av-codec-ids such as avclj.av-codec-ids/AV_CODEC_ID_H264.

find-encoder

(find-encoder encoder-name)

Find an encoder by name. Name may either be the ffmpeg encoder name such as “libx264” or it may be a codec id in avclj.av-codec-ids such as avclj.av-codec-ids/AV_CODEC_ID_H264.

frame-buffer-shape

(frame-buffer-shape height width pixel-fmt)

Return a vector of buffer shapes. Corresponds to the require input format of encode-frame!. Note that for planar pixel formats you will have to pass in multiple buffers.

initialize!

(initialize!)

Initialize the library. Dynamically must find libavcodec and libswscale. Attempts to load x264 to enable h264 encode.

list-codecs

(list-codecs)

List all available encoder/decoders

list-pix-formats

(list-pix-formats)

List all available pixel format names.

make-video-decoder

(make-video-decoder fname & [{:keys [output-pixfmt output-height output-width], :or {output-pixfmt "AV_PIX_FMT_BGR24"}}])

Make an auto-closeable video decoder - you can use this decoder with with-open. Do not assume that you can keep a reference to the frame data; it is only good until the next decode-frame! call. The decoder will do simple scaling and pixel format translations using highly optimized ffmpeg code.

The decoder returns a frame of data which in ffmpeg terms means a sequence of buffers. For the RGB-based pixel formats the first data-frame will be a tensor reshaped to the appropriate size.

Decoders have metadata on them which indicate the size of the image, the pixel format, and the time base of the decoder. Each frame has metadata on it which indicates again the width/height and more important :pts which when transformed by the time-base information yields the millisecond offset of the frame.

:Options

  • :output-pixfmt - a valid ffmpeg pixfmt string. Defaults to "AV_PIX_FMT_BGR24" as this corresponds to buffered-image :byte-bgr format. For list of valid pixfmt strings see (keys avclj.av-pixfmt/pixfmt-name-value-map). I recommend sticking to either "AV_PIX_FMT_RGB24" or "AV_PIX_FMT_BGR24".
  • :output-height, :output-width - If none are specified, use video width/height. If one is specified use aspect-ratio-preserving scaling to find the other. If both are specified use these precisely - which may distort the image.

Example:

user> (require '[avclj :as avclj])
nil
user> (require '[tech.v3.tensor :as dtt])
nil
user> (require '[tech.v3.libs.buffered-image :as bufimg])
nil
user> (avclj/initialize!)
:ok
user> (def decoder (avclj/make-video-decoder "test/data/test-video.mp4"))
#'user/decoder
user> (meta decoder)
{:time-base {:num 1, :den 15360},
 :width 256,
 :height 256,
 :pix-fmt ["AV_PIX_FMT_BGR24"]}
user> (def frame-data (avclj/decode-frame! decoder))
#'user/frame-data
user> (first frame-data)
#tech.v3.tensor<uint8>[256 256 3]
[[[250 253 251]
  [250 253 251]
  [250 253 251]
  ...
  [  0   0   0]
  [  0   0   0]
  [  0   0   0]]]
user> (meta frame-data)
{:pts 0, :width 256, :height 256, :linesize 768}
user> (def dest-img (bufimg/new-image ((meta decoder) :height) ((meta decoder) :width) :byte-bgr))
#'user/dest-img
user> (require '[tech.v3.datatype :as dtype])
nil
user> (dtype/copy! (first frame-data) dest-img)
#object[java.awt.image.BufferedImage 0x1888c431 "BufferedImage@1888c431: type = 5 ColorModel: #pixelBits = 24 numComponents = 3 color space = java.awt.color.ICC_ColorSpace@7fb1c295 transparency = 1 has alpha = false isAlphaPre = false ByteInterleavedRaster: width = 256 height = 256 #numDataElements 3 dataOff[0] = 2"]
user> (bufimg/save! dest-img "test.png")
true
user> (.close decoder)
nil

make-video-encoder

(make-video-encoder height width out-fname {:keys [fps-numerator fps-denominator input-pixfmt encoder-pixfmt encoder-name file-format bit-rate codec-options codec-private-options], :or {fps-numerator 60, fps-denominator 1, input-pixfmt "AV_PIX_FMT_BGR24", encoder-pixfmt "AV_PIX_FMT_YUV420P", encoder-name "mpeg4"}})(make-video-encoder height width output-fname)

Make a video encoder.

  • height - divisible by 2
  • width - divisible by 2
  • out-fname - Output filepath. Must be a c-addressable file path, not a url or an input stream. The file format will be divined from the file extension.

Selected Options:

  • :input-pixfmt - One of the pixel formats. Defaults to “AV_PIX_FMT_BGR24”
  • :encoder-pixfmt - One of the pixel formats. Defaults to “AV_PIX_FMT_YUV420P”. Changing this will probably cause opening the codec to fail with an invalid argument. To see the valid encoder pixel formats, use find-encoder and analyze the :pix-fmts member.
  • :encoder-name - Name (or integer codec id) of the encoder to use.
  • :fps-numerator - :int32 defaults to 60.
  • :fps-denominator - :int32 defaults to 1.
  • :codec-options - Map of string option name to string option key.
  • :codec-private-options - Codec-private options you can set. For libx264 an example is {“preset” “slow”}.
  • :bit-rate - If specified, the system will set a constant bit-rate for the video. In this case with h264 encoding it will switch from crf encoding to abr encoding with a 40 frame rate control look-ahead.

PVideoDecoder

protocol

members

decode-frame!

(decode-frame! decoder)

Decode a frame returning a persistent vector of buffers, one for each data plane in the frame. When this function returns nil there are no more frames to decode. Users can expect the frames will be decoded in-place so the data is only valid until the next decode-frame! call

PVideoEncoder

protocol

members

encode-frame!

(encode-frame! enc frame)

Handle a frame of data. A frame is a persistent vector of buffers, one for each data plane. If you are passing in a nio buffer, ensure to require ’tech.v3.datatype.nio-buffer` for zero-copy support. If frame is not a persistent vector it is assumed to a single buffer and is wrapped in a persistent vector