Documentation · Neural Kit reference

Neural Kit reference.

Every class, function, constant and message of the kit, header by header, from NeuralDefs.h to the back-end API.

Types, flags, messages and errors

neural/NeuralDefs.h

The constants every other header uses. <NeuralKit.h> includes all of the kit's headers at once.

Header source ↗
enum

neural_data_type

How one element of a tensor is stored.

NameMeaning
B_NEURAL_NO_TYPE0: no type
B_NEURAL_FLOAT3232-bit float
B_NEURAL_FLOAT1616-bit float
B_NEURAL_INT8signed 8-bit integer, usually quantized
B_NEURAL_UINT8unsigned 8-bit integer, usually quantized
B_NEURAL_INT16signed 16-bit integer
B_NEURAL_INT32signed 32-bit integer (shapes, indices)
B_NEURAL_INT64signed 64-bit integer
B_NEURAL_BOOLboolean
define

B_NEURAL_MAX_RANK

The most dimensions a tensor can have.

#define B_NEURAL_MAX_RANK 8
enum

neural_device_kind

What kind of processor a device is. neural_device_kind_name() returns "CPU", "GPU" or "NPU".

NameMeaning
B_NEURAL_DEVICE_CPUthe processor itself
B_NEURAL_DEVICE_GPUa graphics processor
B_NEURAL_DEVICE_NPUa neural processing unit
enum

Device flags

Returned by BNeuralDevice::Flags().

NameMeaning
B_NEURAL_DEVICE_QUANTIZEDruns networks of 8-bit quantized numbers
B_NEURAL_DEVICE_FLOATruns networks of floating-point numbers
B_NEURAL_DEVICE_REMOVABLEthe device is not always there (USB, hot plug)
B_NEURAL_DEVICE_REFERENCEthe kit's own CPU device, which runs every operator the kit knows
enum

Session and device-choice flags

Passed to BNeuralRoster::GetDefaultDevice(), BNeuralSession::SetTo() and BSpeechRecognizer::SetTo().

NameMeaning
B_NEURAL_PREFER_SPEED0, the default: the fastest kind of device (NPU, then GPU, then CPU)
B_NEURAL_PREFER_LOW_POWERthe device that uses the least power (NPU, then CPU, then GPU)
B_NEURAL_NO_FALLBACKfail with B_NEURAL_NOT_ACCELERATED rather than run operators the device cannot run on the CPU
B_NEURAL_CPU_ONLYuse the CPU only, whatever else the machine has
B_NEURAL_PROFILEkeep per-operator times; see BNeuralSession::GetOperatorInfo()
enum

Messages

Message codes the kit sends. The fields are listed with each one.

NameMeaning
B_NEURAL_DEVICE_ADDED ('NRda')to BNeuralRoster::StartWatching() targets: "device" (the identifier), "name", "kind"
B_NEURAL_DEVICE_REMOVED ('NRdr')the same fields, when a device goes away
B_NEURAL_RUN_COMPLETED ('NRrc')from BNeuralSession::RunAsync() unless the caller gives a message of its own. Fields added to either: "neural:status" (int32), "neural:token" (int32), "neural:session" (pointer; the session may be gone by the time the message is read), "neural:duration" (int64, µs)
B_SPEECH_RESULT ('NRsr')from a BSpeechRecognizer: "speech:text", "speech:final" (bool), "speech:start" and "speech:end" (int64, µs from the start of the recording), "speech:token", "speech:recognizer" (pointer). A result that is not final replaces the previous one.
B_SPEECH_FINISHED ('NRsf')after the last result of a recording: "speech:status", "speech:token", "speech:recognizer", "speech:audio" (int64, µs of audio recognized), "speech:duration" (int64, µs of work)
enum

Errors

From B_NEURAL_ERROR_BASE (B_GENERAL_ERROR_BASE + 0xb000). The kit also returns the usual B_NO_INIT, B_BAD_VALUE, B_NO_MEMORY, B_BUSY and file errors.

NameMeaning
B_NEURAL_BAD_MODELthe file is not a model, or is damaged
B_NEURAL_UNKNOWN_FORMATno loader knows the model's format or version
B_NEURAL_UNSUPPORTED_OPERATORthe model uses an operator the kit does not implement; BNeuralModel::ErrorDetails() names it
B_NEURAL_TYPE_MISMATCHa tensor's type is not the one the model expects
B_NEURAL_SHAPE_MISMATCHa tensor's shape is not the one the model expects
B_NEURAL_NO_DEVICEno device of the kind asked for
B_NEURAL_DEVICE_LOSTthe device went away or stopped answering, and the CPU could not take over
B_NEURAL_NOT_ACCELERATEDB_NEURAL_NO_FALLBACK was given and the device cannot run the whole model
functions

Functions

size_t neural_data_type_size(neural_data_type type);
Size of one element in bytes; 0 for an unknown type.
const char* neural_data_type_name(neural_data_type type);
"float32", "uint8" and so on.
const char* neural_device_kind_name(neural_device_kind kind);
"CPU", "GPU" or "NPU".
const char* neural_error_string(status_t error);
Describes the kit's own errors in words, and all others through strerror().

BTensorInfo and BTensor

neural/Tensor.h

What goes in and comes out of a network. Quantized tensors map stored integers to real values as real = scale × (stored − zero point). Pictures are laid out batch, height, width, channels (NHWC).

Header source ↗
class

BTensorInfo

The description of a tensor without its numbers: element type, shape, an optional name and, for quantized tensors, scale and zero point. Models describe their inputs and outputs with these.

BTensorInfo();
BTensorInfo(neural_data_type type, int32 rank, const int32* dimensions, const char* name = NULL);
An empty description, or one of the given type and shape.
status_t SetTo(neural_data_type type, int32 rank, const int32* dimensions, const char* name = NULL);
void Unset();
bool IsValid() const;
Sets or clears the description.
bool operator==(const BTensorInfo& other) const;
Same type, shape and quantization. The name is not compared.
const char* Name() const;
void SetName(const char* name);
The tensor's name in the model, such as "input".
neural_data_type Type() const;
int32 Rank() const;
int32 DimensionAt(int32 index) const;
Element type, number of dimensions, and the size of one dimension (0 when the index is out of range).
int64 CountElements() const;
size_t ByteSize() const;
Number of elements, and the bytes they take.
bool IsQuantized() const;
float Scale() const;
int32 ZeroPoint() const;
void SetQuantization(float scale, int32 zeroPoint);
Quantization parameters. IsQuantized() is true when scale and zero point mean something.
BString ToString() const;
A line for logs: input uint8 [1, 224, 224, 3] scale 0.0078 zero 128.
status_t Archive(BMessage* into) const;
status_t Unarchive(const BMessage* from);
Stores the description in a message and reads it back.
class

BTensor : public BArchivable

A block of numbers with a shape. A tensor owns its memory unless it was given memory to use. The inputs and outputs of a BNeuralSession belong to the session. Tensors can be archived: put one in a BMessage with its numbers to send it to another team or keep it in a file.

BTensor();
BTensor(const BTensorInfo& info);
BTensor(const BTensorInfo& info, void* data, size_t size);
BTensor(BMessage* archive);
An empty tensor; one with zeroed memory of its own; one that uses the caller's memory; one from an archive.
status_t SetTo(const BTensorInfo& info);
Makes the tensor one of that description, with zeroed memory of its own.
status_t SetTo(const BTensorInfo& info, void* data, size_t size);
Uses the caller's memory, which must stay valid while the tensor uses it and be at least info.ByteSize() long.
status_t InitCheck() const;
void Unset();
Whether the tensor is usable; frees or drops its memory.
const BTensorInfo& Info() const;
neural_data_type Type() const;
int32 Rank() const;
int32 DimensionAt(int32 index) const;
int64 CountElements() const;
The description, and shortcuts into it.
void* Data();
const void* Data() const;
size_t ByteSize() const;
The raw storage.
status_t CopyFrom(const void* data, size_t size);
status_t CopyTo(void* data, size_t size) const;
status_t CopyFrom(const BTensor& other);
Copies raw bytes in or out, or the numbers of a tensor of the same type and shape.
float ValueAt(int64 index) const;
status_t SetValueAt(int64 index, float value);
status_t GetValues(float* values, int64 count, int64 first = 0) const;
status_t SetValues(const float* values, int64 count, int64 first = 0);
Elements as real values, whatever the storage: quantized values are converted with scale and zero point, and values that do not fit are clamped. Indices count elements in storage order (the last dimension varies fastest). The same code handles a float model and its quantized sibling.
int64 IndexOfMaximum() const;
Index of the largest element; −1 for an empty tensor. For a classifier, the best class.
virtual status_t Archive(BMessage* into, bool deep = true) const;
static BArchivable* Instantiate(BMessage* archive);
BArchivable support: shape and numbers.

BNeuralModel and BNeuralModelBuilder

neural/NeuralModel.h

A network: its operators, its weights and the description of its inputs and outputs.

Header source ↗
class

BNeuralModel

A network read from a file. The kit reads TensorFlow Lite files (.tflite, schema version 3) itself, with a reader that checks every offset. A model cannot be changed once it is made, and copies share the network. Files are mapped, not read, so the weights of a model used by several teams are in memory once. A model does nothing by itself: a BNeuralSession runs it.

BNeuralModel(const entry_ref* ref);
BNeuralModel(const char* path);
Reads a model file.
status_t SetTo(const entry_ref* ref);
status_t SetTo(const char* path);
status_t SetTo(BDataIO* source);
status_t SetTo(const void* buffer, size_t size);
Reads a model from a file, a stream (read to the end) or a buffer (copied).
status_t InitCheck() const;
const char* ErrorDetails() const;
Whether the model loaded, and why not, for developers: "operator 126 is not supported". Empty when there is nothing to say.
const char* Name() const;
const char* Description() const;
const char* Format() const;
The name the model gives itself (or its file name), its description, and "tflite", or "built" for a model from a BNeuralModelBuilder.
int32 CountInputs() const;
status_t GetInputInfo(int32 index, BTensorInfo* info) const;
int32 FindInput(const char* name) const;
The inputs. FindInput() returns an index, or a negative error.
int32 CountOutputs() const;
status_t GetOutputInfo(int32 index, BTensorInfo* info) const;
int32 FindOutput(const char* name) const;
The outputs.
int32 CountOperators() const;
neural_operator_code OperatorAt(int32 index) const;
The operators, in an order in which they can run.
status_t GetInfo(BMessage* info) const;
Everything else: "name", "description", "format", "version" (int32), "operators" (int32), "weights:bytes" (int64), and "operator" (one string per kind of operator used).
static bool IsModel(const entry_ref* ref);
Whether a file looks like a model the kit can read, without reading all of it.
class

BNeuralModelBuilder

Builds a model one operator at a time, for programs that compute their network and for tests. The Add…() methods return tensor numbers. A negative number is an error, which Build() also returns.

BNeuralModelBuilder(const char* name = NULL);
An empty network.
int32 AddInput(const BTensorInfo& info);
A tensor the caller fills in before each run.
int32 AddConstant(const BTensorInfo& info, const void* data, size_t size);
A tensor of fixed numbers (weights). The data is copied.
status_t SetChannelQuantization(int32 tensor, const float* scales, int32 count, int32 axis);
Per-channel scales for the weights of a quantized convolution.
int32 AddOperator(neural_operator_code code, const int32* inputs, int32 inputCount, const BTensorInfo& output, const void* params = NULL, size_t paramsSize = 0);
Adds an operator and returns its output. params must be the structure NeuralOperators.h names for the code.
int32 AddOperator(neural_operator_code code, const int32* inputs, int32 inputCount, const BTensorInfo* outputs, int32 outputCount, const void* params = NULL, size_t paramsSize = 0);
For operators with several outputs; returns the first.
status_t MarkOutput(int32 tensor);
Makes a tensor an output of the model.
status_t Build(BNeuralModel* model);
Checks types, shapes and order, and makes the model. The builder is empty afterwards.

BNeuralDevice and BNeuralRoster

neural/NeuralRoster.h

The processors of this machine that can run networks.

Header source ↗
class

BNeuralDevice

A value that names one processor: an NPU, a graphics processor, or the CPU. A device stays valid when it is unplugged, but is no longer IsAvailable(). The identifier is stable across restarts and is what a program should keep in its settings: "cpu", "rk3588_npu/0", "vulkan/0", "amdxdna/0".

bool IsValid() const;
bool IsAvailable() const;
Whether the value names a device, and whether that device is there now.
const char* ID() const;
The stable identifier.
const char* Name() const;
const char* Vendor() const;
"Rockchip RK3588 NPU" and its vendor.
neural_device_kind Kind() const;
uint32 Flags() const;
CPU, GPU or NPU; and B_NEURAL_DEVICE_QUANTIZED, B_NEURAL_DEVICE_FLOAT, B_NEURAL_DEVICE_REMOVABLE and B_NEURAL_DEVICE_REFERENCE.
const char* Backend() const;
The name of the back-end add-on that drives the device.
status_t GetInfo(BMessage* info) const;
Whatever the back end knows: "driver", "version", "cores" (int32), "memory" (int64), "path" and more. None of the fields is guaranteed.
int32 CountSupportedOperators(const BNeuralModel& model) const;
How many of the model's operators the device would run itself (the rest run on the CPU), or a negative error. Loads nothing onto the device.
class

BNeuralRoster

The devices of the machine. There is one roster per team, BNeuralRoster::Default(). It loads the back-end add-ons the first time it is asked something; a back end whose hardware is absent reports no devices and costs nothing more.

static BNeuralRoster* Default();
The team's roster.
int32 CountDevices();
status_t GetDeviceAt(int32 index, BNeuralDevice* device);
status_t FindDevice(const char* id, BNeuralDevice* device);
Lists the devices, or finds one by its identifier.
status_t GetDefaultDevice(BNeuralDevice* device, uint32 flags = 0, const BNeuralModel* model = NULL);
The device to use when a program has no reason to choose: an NPU, else a GPU, else the CPU. B_NEURAL_PREFER_LOW_POWER and B_NEURAL_CPU_ONLY change the order. Given a model, devices that would run none of it are skipped, and with B_NEURAL_NO_FALLBACK so are those that cannot run all of it. The user's preferred device comes first when it qualifies.
status_t Rescan();
Asks the back ends to look for devices again.
status_t StartWatching(const BMessenger& target);
status_t StopWatching(const BMessenger& target);
Sends B_NEURAL_DEVICE_ADDED and B_NEURAL_DEVICE_REMOVED to the target. The roster watches /dev/npu.

BNeuralSession

neural/NeuralSession.h

A model loaded onto a device and ready to run.

Header source ↗
class

BNeuralSession

Setting a session up is the slow part: the device's back end translates the network and uploads its weights. Running it is fast and can be repeated as often as needed. When a session is set up, the kit asks the back end which operators the device can run, splits the network into stretches for the device and stretches for the CPU, and has the back end make a program for each of its stretches. A run the device fails is done by the CPU; a device that goes away, or fails three runs in a row, loses its stretches to the CPU for good. A session therefore always works and gives the same answers. The input and output tensors belong to the session and keep their memory between runs. A session runs one thing at a time; several sessions, including sessions of the same model, can run at once and share a device.

BNeuralSession(const BNeuralModel& model, uint32 flags = 0);
BNeuralSession(const BNeuralModel& model, const BNeuralDevice& device, uint32 flags = 0);
Sets the session up on the default device or on a chosen one.
status_t SetTo(const BNeuralModel& model, uint32 flags = 0);
status_t SetTo(const BNeuralModel& model, const BNeuralDevice& device, uint32 flags = 0);
void Unset();
status_t InitCheck() const;
Sets up or ends the session. Without a device, BNeuralRoster::GetDefaultDevice() chooses one, passing the same flags.
const BNeuralModel& Model() const;
const BNeuralDevice& Device() const;
What runs, and where.
int32 CountInputs() const;
BTensor* InputAt(int32 index);
BTensor* FindInput(const char* name);
The input tensors, to fill in before a run.
int32 CountOutputs() const;
const BTensor* OutputAt(int32 index) const;
const BTensor* FindOutput(const char* name) const;
The output tensors, filled in by a run.
status_t Run();
Runs the network on the inputs and returns when the outputs are filled in.
status_t RunAsync(const BMessenger& target, const BMessage* message = NULL, int32* _token = NULL);
Starts a run and returns at once. When it is over, message (or B_NEURAL_RUN_COMPLETED) goes to the target with the result in "neural:status". Leave the inputs alone and do not read the outputs until then. B_BUSY while a run is in progress.
status_t WaitForRun(bigtime_t timeout = B_INFINITE_TIMEOUT);
bool IsRunning() const;
Waits for a run started by RunAsync() and returns its result. Deleting a session also waits; a run cannot be cancelled yet.
status_t GetRunInfo(neural_run_info* info) const;
How the model was split and how long runs take.
status_t GetOperatorInfo(int32 index, neural_operator_info* info) const;
Where one operator runs and, with B_NEURAL_PROFILE, how long it took.
struct

neural_run_info

Filled in by BNeuralSession::GetRunInfo().

NameMeaning
int64 runsruns since the session was set up
bigtime_t last_durationtime the last run took, in µs
bigtime_t last_device_durationof that, the time spent in the device's parts of the model
bigtime_t average_durationmean over all runs
int32 device_operatorsoperators the device runs itself
int32 cpu_operatorsoperators the CPU runs
int32 device_partitionsstretches of operators handed to the device at once
int32 device_failuresdevice runs that failed and were done by the CPU instead
struct

neural_operator_info

Filled in by BNeuralSession::GetOperatorInfo().

NameMeaning
neural_operator_code codewhat the operator does
bool on_devicetrue: runs on the session's device; false: on the CPU
int32 partitionthe device partition the operator is part of, or −1
bigtime_t last_durationwith B_NEURAL_PROFILE: the operator's own time on the CPU, or its whole partition's time on the device

BNeuralUtils: pictures, labels and results

neural/NeuralUtils.h

Conveniences that tie tensors to the Interface Kit and to the files models come with.

Header source ↗
class

BNeuralUtils

Static functions only.

static status_t SetTensorFromBitmap(BTensor* tensor, const BBitmap* bitmap, const BRect* source = NULL, const neural_image_options* options = NULL, neural_image_mapping* _mapping = NULL);
Fills an image tensor ([1, height, width, 3 or 1] or [height, width, channels]; uint8, int8 or float32) from all or part of a bitmap of any size, scaling with bilinear filtering. Takes B_RGB32, B_RGBA32, B_RGB24, B_RGB16, B_RGB15, B_GRAY8 and B_CMAP8. _mapping says where the picture went, to map results back.
static status_t GetBitmapFromTensor(const BTensor& tensor, BBitmap** _bitmap, const neural_image_options* options = NULL);
The reverse: a B_RGB32 bitmap of an image tensor; one channel becomes gray. The caller owns the bitmap.
static status_t GetLabels(const entry_ref* ref, BStringList* labels);
static status_t GetLabels(const char* path, BStringList* labels);
Reads a label file: one class name per line, as shipped with classification and detection models.
static bool IsDetector(const BNeuralSession& session);
Whether the model's outputs are those of DETECTION_POSTPROCESS (boxes, classes, scores, count).
static int32 GetDetections(const BNeuralSession& session, neural_detection* detections, int32 maxCount);
What a detector found in its last run, best first. Returns how many were written, or B_BAD_TYPE when the model is not a detector.
static int32 GetTopValues(const BTensor& tensor, int32 count, int64* indices, float* values = NULL);
The count largest elements of a tensor, best first, with their real values if wanted. The top classes of a classifier.
struct

neural_image_options

How SetTensorFromBitmap() prepares a picture.

NameMeaning
neural_image_fit fitB_NEURAL_FIT_STRETCH (fill, ignoring proportions), B_NEURAL_FIT_CROP (cover, cutting off the overflow evenly) or B_NEURAL_FIT_LETTERBOX (fit inside, the rest gray)
bool bgrstore blue, green, red instead of red, green, blue
float mean[3], deviation[3]float tensors get (pixel − mean) / deviation per channel, pixels being 0–255; the default 127.5 for both gives −1 to 1. Quantized 8-bit tensors get the pixel values as they are (int8: minus 128).
struct

neural_image_mapping

Where the picture went, to map what a model says about its input back to the bitmap.

NameMeaning
BRect sourcethe part of the bitmap used, in bitmap coordinates
BRect destinationwhere it was put, in tensor pixels
int32 tensor_width, tensor_heightthe size of the tensor's picture
BRect ToBitmap(float left, float top, float right, float bottom) constturns a box in fractions of the tensor (0–1, as detection models give them) into bitmap coordinates
struct

neural_detection

One thing a detection model found.

NameMeaning
float left, top, right, bottomwhere, in fractions of the model's input picture (0–1)
int32 class_indexthe class. Label files of detection models start with a line for "nothing", so the label is at class_index + 1.
float scorehow sure, 0–1

BSpeechRecognizer

neural/SpeechRecognizer.h

Speech to text on the machine itself, with a Whisper model the kit runs. Nothing leaves the machine.

Header source ↗
class

BSpeechRecognizer

Audio goes in as it is recorded, in any raw format of the Media Kit; text comes out as messages, and the caller never waits for the model. The recognizer has its own thread, which does the work. Its methods may be called from any thread, and deleting it is safe at any time: it waits for the step the model is in (at most one encoder run). Whisper works on 30-second windows and marks where phrases end; a phrase is final when the model has seen its end.

BSpeechRecognizer(const char* model, uint32 flags = 0);
Loads a model at once; see SetTo().
status_t SetTo(const char* model = NULL, uint32 flags = 0);
status_t SetTo(const char* model, const BNeuralDevice& device, uint32 flags = 0);
Loads a model: a name ("whisper-tiny.en", a directory under neural/models of the data directories), the path of a model directory, or NULL for the first one installed. The device and flags work as for BNeuralSession.
void Unset();
status_t InitCheck() const;
const char* ErrorDetails() const;
Unloads; reports whether SetTo() worked, and why not.
static status_t FindModel(const char* name, BPath* _directory);
static status_t GetModels(BStringList* names);
The installed models: directories with a speech.conf under neural/models. B_ENTRY_NOT_FOUND when none is installed.
const char* ModelName() const;
const BNeuralDevice& Device() const;
"tiny.en", and the device the encoder (the larger network) runs on.
status_t GetLanguages(BStringList* codes) const;
status_t SetLanguage(const char* code);
const char* Language() const;
ISO 639-1 codes. The language is "en" to begin with; a change takes effect with the next Start(). B_NAME_NOT_FOUND when the model does not know the language.
void SetTimestamps(bool enabled);
bool Timestamps() const;
Whether the model is asked for phrase times (the default). Without them a window is one phrase.
status_t SetTarget(const BMessenger& target, const BMessage* message = NULL);
Where B_SPEECH_RESULT and B_SPEECH_FINISHED go. With a message of the caller's, the fields are added to it, and "speech:finished" (bool) tells the two apart.
void SetInterimInterval(bigtime_t interval);
bigtime_t InterimInterval() const;
How often, at most, the audio that is not final yet is read for a result that is not final. 0, the default, means never. A reading costs as much as a whole window.
status_t Start(const media_raw_audio_format* format = NULL, int32* _token = NULL);
Begins a recording in the given format (NULL: 16 kHz, mono, float). A recording in progress is dropped. The token comes back in the messages of this recording.
status_t Feed(const void* buffer, size_t size);
More audio; returns at once. B_BUSY when ten minutes of audio are waiting for the model.
status_t Finish();
The recording is over. What is left is recognized, then B_SPEECH_FINISHED is sent. Returns at once.
void Cancel();
Drops the recording; no more results of it are sent.
status_t WaitUntilFinished(bigtime_t timeout = B_INFINITE_TIMEOUT);
bool IsRecognizing() const;
For callers that do want to wait.
status_t Transcribe(const float* samples, size_t count, BString* _text);
status_t Transcribe(const media_raw_audio_format& format, const void* buffer, size_t size, BString* _text);
Recognizes a complete recording and returns when done: 16 kHz mono float samples, or any raw format. No messages are sent. B_BUSY while a recording is in progress.
int32 CountSegments() const;
status_t GetSegment(int32 index, speech_segment* _segment) const;
The final phrases so far: start, end (µs) and text.
status_t GetInfo(speech_recognizer_info* info) const;
Audio recognized, time spent (front end, encoder, decoder), windows and tokens.

Operators

neural/NeuralOperators.h

The operators a network is made of, whatever file it came from. They follow TensorFlow Lite's, whose quantized arithmetic is specified to the bit; the CPU kernels reproduce it, which makes the CPU device the reference every back end is checked against. Each operator has a fixed list of inputs and, where noted, a parameter structure. An input index of −1 means "not given" (a convolution without bias). neural_operator_name() and neural_operator_params_size() describe a code.

Header source ↗
enum

neural_operator_code

Inputs, then the parameter structure.

NameMeaning
B_NEURAL_OP_CONV_2D, B_NEURAL_OP_DEPTHWISE_CONV_2Dinput, weights, bias; neural_conv_params
B_NEURAL_OP_FULLY_CONNECTEDinput, weights, bias; neural_fully_connected_params
B_NEURAL_OP_TRANSPOSE_CONVoutput shape, weights, input, bias; neural_conv_params
B_NEURAL_OP_AVERAGE_POOL_2D, B_NEURAL_OP_MAX_POOL_2Dinput; neural_pool_params
B_NEURAL_OP_ADD, _SUB, _MUL, _DIV, _MAXIMUM, _MINIMUMa, b (broadcast); neural_binary_params
B_NEURAL_OP_RELU, _RELU6, _LOGISTIC, _TANH, _HARD_SWISHinput; no parameters
B_NEURAL_OP_LEAKY_RELUinput; neural_leaky_relu_params
B_NEURAL_OP_PRELUinput, slope
B_NEURAL_OP_SOFTMAXinput; neural_softmax_params
B_NEURAL_OP_RESHAPEinput, shape (optional); neural_reshape_params
B_NEURAL_OP_SQUEEZEinput; neural_squeeze_params
B_NEURAL_OP_CONCATENATIONany number of inputs; neural_concat_params
B_NEURAL_OP_PADinput, paddings [rank, 2], constant value (optional)
B_NEURAL_OP_MEANinput, axes; neural_reduce_params
B_NEURAL_OP_RESIZE_BILINEAR, _RESIZE_NEARESTinput, size [height, width]; neural_resize_params
B_NEURAL_OP_TRANSPOSEinput, permutation
B_NEURAL_OP_STRIDED_SLICEinput, begin, end, strides; neural_strided_slice_params
B_NEURAL_OP_QUANTIZE, _DEQUANTIZEinput; converts between types and quantizations
B_NEURAL_OP_DETECTION_POSTPROCESSbox encodings [1, anchors, 4], class scores [1, anchors, classes], anchors (constant). Outputs boxes [1, n, 4] (top, left, bottom, right), classes, scores, count; neural_detection_params
B_NEURAL_OP_SQUARED_DIFFERENCEa, b (broadcast): (a − b)²
B_NEURAL_OP_RSQRTinput: 1 / √x
B_NEURAL_OP_GELUinput; neural_gelu_params (exact, or the tanh approximation)
B_NEURAL_OP_BATCH_MATMULa, b: […, m, k] × […, k, n], leading dimensions broadcast; neural_batch_matmul_params
B_NEURAL_OP_GATHERinput, indices (int32 or int64): slices along one dimension (embeddings); an index out of range fails the run; neural_gather_params
enum

neural_activation and neural_padding

What is done to an operator's result before it is stored, and how windows meet the edges.

NameMeaning
B_NEURAL_ACTIVATION_NONE, _RELU, _RELU_N1_TO_1, _RELU6, _TANHfused activations; _RELU_N1_TO_1 clamps to −1…1
B_NEURAL_PADDING_SAMEas much padding as it takes to keep size / stride outputs
B_NEURAL_PADDING_VALIDno padding: only positions where the whole window fits
struct

Parameter structures

One per operator family.

NameMeaning
neural_conv_paramspadding, stride_width, stride_height, dilation_width, dilation_height, depth_multiplier (depthwise only), activation
neural_fully_connected_paramsactivation, keep_dimensions
neural_pool_paramspadding, stride_width, stride_height, filter_width, filter_height, activation
neural_binary_paramsactivation
neural_leaky_relu_paramsalpha
neural_softmax_paramsbeta
neural_reshape_paramsrank, shape[B_NEURAL_MAX_RANK], used when no shape input is given
neural_squeeze_paramscount (0: every dimension of size 1), dimensions[]
neural_concat_paramsaxis, activation
neural_reduce_paramskeep_dimensions
neural_resize_paramsalign_corners, half_pixel_centers
neural_strided_slice_paramsbegin_mask, end_mask, ellipsis_mask, new_axis_mask, shrink_axis_mask
neural_gelu_paramsapproximate
neural_batch_matmul_paramsadjoint_a, adjoint_b
neural_gather_paramsaxis, batch_dimensions (only 0)
neural_detection_paramsmax_detections, max_classes_per_detection, detections_per_class, num_classes, score_threshold, iou_threshold, y_scale, x_scale, h_scale, w_scale, use_regular_nms

The back-end API

add-ons/neural/NeuralBackend.h

For people adding hardware. A back end is an add-on in add-ons/neural that makes one kind of hardware usable: it finds its devices (usually through the kernel driver's nodes under /dev/npu), says which operators a device can run, and turns stretches of such operators into programs. The kit does the rest. See Writing a back end and the example back end.

Header source ↗
class

BNeuralBackend

One per add-on, made by instantiate_neural_backend(), living as long as the team's roster. It must be thread safe: programs of different sessions run at the same time.

virtual const char* Name() const = 0;
Short and unique, used in device identifiers: "rk3588_npu".
virtual int32 CountDevices() = 0;
Looks for devices, once after loading and again on BNeuralRoster::Rescan(). A back end with no devices here returns 0.
virtual status_t GetDeviceInfo(int32 device, neural_device_info* info, BMessage* details) = 0;
Describes a device; details (may be NULL) gets what BNeuralDevice::GetInfo() returns.
virtual status_t GetSupportedOperators(int32 device, const BNeuralGraph& graph, bool* supported) = 0;
Sets supported[i] for every operator the device can compute correctly. Asked before anything is loaded, also just to choose a device: look at codes, types, shapes and parameters only.
virtual status_t CreateProgram(int32 device, const BNeuralGraph& graph, const neural_partition& partition, uint32 flags, BNeuralProgram** _program) = 0;
Makes the program for a stretch of supported operators: converts and uploads weights. An error means "run this stretch on the CPU".
class

BNeuralProgram

A stretch of a network made ready for a device.

virtual status_t Run(void* const* tensors) = 0;
Runs once. tensors has one entry per tensor of the graph; those of the partition's inputs and outputs point to memory in the graph's type and NHWC layout. On an error the kit computes the stretch on the CPU for this run and tries the device again next time (three failures in a row end that). Return B_NEURAL_DEVICE_LOST when the device is gone for good.
virtual bigtime_t LastDeviceTime() const;
The device's own time for the last run, without copies; 0 if unknown.
class

BNeuralGraph

The network as back ends see it, operators in an order in which they can run.

int32 CountTensors() const;
const neural_graph_tensor* TensorAt(int32 index) const;
Type, shape, quantization (including per-channel scales) and, for constants, the numbers.
int32 CountOperators() const;
const neural_graph_operator* OperatorAt(int32 index) const;
Code, input and output tensor indices, and the parameter structure.
int32 CountInputs() const;
int32 InputAt(int32 index) const;
int32 CountOutputs() const;
int32 OutputAt(int32 index) const;
The network's own inputs and outputs, as tensor indices.
struct

neural_partition and neural_device_info

What CreateProgram() is given, and what GetDeviceInfo() fills in.

NameMeaning
neural_partitionoperators/operator_count: operators that follow one another; inputs: tensors they need from outside (constants are not listed); outputs: tensors they make that are needed outside
neural_device_infoid (unique in the back end, stable: "0"), name, vendor, kind, flags, priority (100 for the device that is the reason the machine has an NPU, 50 for a general-purpose GPU)
BNeuralBackend* instantiate_neural_backend(image_id image)the one function an add-on exports; return NULL when the back end cannot work on this machine