Documentation · Neural Kit
Neural networks, on every processor.
The Neural Kit runs models on NPUs, graphics processors and the CPU through one API, splitting each network between the device and the CPU so it always gives the same answers. It also turns speech into text on the machine itself.
Overview
The Neural Kit runs neural networks on the processors a machine has for them: neural processing units (NPUs), graphics processors and the CPU. A program hands the kit a model file and gets answers. Which processor computed them is the kit’s business, and the answers are the same on all of them: whatever a device cannot do, the CPU does.
It is a kit like Haiku’s others: a library, libneural.so; public headers, <NeuralKit.h>; back-end add-ons for each kind of hardware in add-ons/neural; and kernel drivers that publish the processors under /dev/npu.
#include <NeuralKit.h>
BNeuralModel model("/boot/home/models/mobilenet_v1_1.0_224_quant.tflite");
BNeuralSession session(model); // loads it onto the best device
if (session.InitCheck() != B_OK)
return session.InitCheck();
BNeuralUtils::SetTensorFromBitmap(session.InputAt(0), bitmap);
if (session.Run() == B_OK)
int64 best = session.OutputAt(0)->IndexOfMaximum();Link with -lneural -lbe. The kit ships in air/OS releases from 9 November 2026 on (the first was the ARM64 release hrev60207+2552), together with the neural command and the Lookout demo.
What runs where
| Machine | Device | Back end | Runs |
|---|---|---|---|
| Every machine | cpu | built into the kit | every operator the kit knows, float and quantized; the reference the others are checked against |
| ROCK 5 ITX and other RK3588 boards | rk3588_npu/0 | rk3588_npu.neural, driver /dev/npu/rk3588/0 | 8-bit quantized convolutions; the rest of the network on the CPU |
| Machines with a Vulkan driver (NVK on NVIDIA, RADV on AMD) | vulkan/0 | vulkan.neural, the graphics driver | float32 networks, in compute shaders |
| Framework 13 with Ryzen AI 300 (AMD XDNA NPUs) | amdxdna/0 | amdxdna.neural, driver /dev/npu/amdxdna/0 | compiled NPU designs (“design bundles”) for parts of a network |
See Hardware status for what was measured on each.
How it fits together
- A model (
BNeuralModel) is a network read into the kit’s own form: tensors, operators and weights. The file is mapped, not read, so weights are paged in on demand and shared between teams. Models cannot be changed, and copies share the network. - A session (
BNeuralSession) is a model made ready on one device. The kit asks the device’s back end which operators it can run, cuts the network into stretches of those and stretches of the rest, has the back end make a program for each of its stretches, and prepares the CPU interpreter for everything else. A run the device fails is done by the CPU, and a device that goes away, or fails three runs in a row, loses its stretches to the CPU for good. A session always works and gives the same answers: bit for bit for quantized networks on the CPU, and within measured tolerances on devices. - The roster (
BNeuralRoster) knows the devices. It loads the back ends the first time it is asked something. A back end whose hardware is not there reports no devices and costs nothing more. - Back ends are add-ons, found like translators: in
add-ons/neuralof the user’s non-packaged directory, the user’s packages, the system’s non-packaged directory and the system package, in that order. The first file of a name wins, so a back end under development shadows the packaged one. - Kernel drivers only give access to a processor: memory it can reach, power, a queue of jobs. They know nothing of networks. Listing
/dev/npuloads them, so a machine that never runs a network never loads them.
Tensors
A BTensorInfo describes a tensor: element type (B_NEURAL_FLOAT32, B_NEURAL_UINT8, B_NEURAL_INT8 and others), shape (up to eight dimensions; pictures are batch, height, width, channels) and, for quantized tensors, the scale and zero point that map stored integers to real values: real = scale × (stored − zero point).
A BTensor is a description plus the numbers. It owns its memory or uses the caller’s. Data() is the raw storage; ValueAt(), GetValues(), SetValueAt() and SetValues() read and write real values whatever the storage, so the same code handles a float model and its quantized sibling. A tensor is a BArchivable: archived into a BMessage with its numbers, it can be sent to another team or kept in a file or attribute.
Models
BNeuralModel model;
status_t status = model.SetTo(&ref); // or a path, a BDataIO, a buffer
if (status != B_OK)
fprintf(stderr, "%s: %s\n", neural_error_string(status), model.ErrorDetails());
for (int32 i = 0; i < model.CountInputs(); i++) {
BTensorInfo info;
model.GetInputInfo(i, &info);
puts(info.ToString()); // input uint8 [1, 224, 224, 3] scale 0.0078125 zero 128
}The kit reads TensorFlow Lite files (.tflite, schema version 3) itself, with a reader that checks every offset, since a model file may come from anywhere. BNeuralModel::IsModel() tells whether a file is a model without reading all of it, and GetInfo() returns the name, description, format, kinds of operator and bytes of weights.
Shapes are fixed when a model is made: there are no dynamic shapes and no control flow. A model keeps nothing between runs: one that works in steps, like the decoder of a speech model, gets what its earlier steps computed as inputs. A model with an operator the kit does not know fails to load with B_NEURAL_UNSUPPORTED_OPERATOR, and ErrorDetails() names the operator.
BNeuralModelBuilder makes a model in memory, operator by operator, for programs that compute their network and for tests. See the builder example.
Devices and choosing one
A BNeuralDevice names a device. Its ID() ("cpu", "rk3588_npu/0", "vulkan/0") is stable across restarts: keep it in your settings. CountSupportedOperators(model) says how much of a model a device would run without loading anything.
When a program does not choose, BNeuralRoster::GetDefaultDevice() does: an NPU, else a graphics processor, else the CPU, among the devices that run at least part of the model. Flags change the choice:
| Flag | Effect |
|---|---|
B_NEURAL_PREFER_SPEED (0) | the fastest kind of device: NPU, GPU, CPU (the default) |
B_NEURAL_PREFER_LOW_POWER | NPU, else CPU, else GPU |
B_NEURAL_NO_FALLBACK | only devices that run the whole model; a session fails with B_NEURAL_NOT_ACCELERATED rather than use the CPU for part of it |
B_NEURAL_CPU_ONLY | the CPU, whatever else there is |
B_NEURAL_PROFILE | sessions keep per-operator times |
The user’s choice comes first when it can do the job. It is set in ~/config/settings/neural (driver settings syntax) or in the environment:
preferred_device rk3588_npu/0 # or NEURAL_DEVICE=rk3588_npu/0
disable_backend vulkan # or NEURAL_DISABLE_BACKENDS="vulkan" ("all": the CPU only)NEURAL_THREADS=n sets how many threads the CPU device uses (default: one per CPU, at most 16).
BNeuralRoster::StartWatching() sends B_NEURAL_DEVICE_ADDED and B_NEURAL_DEVICE_REMOVED as devices come and go. A session whose device went away keeps working on the CPU; set it up again to move it back. See the devices example.
Sessions and runs
BNeuralSession session;
status_t status = session.SetTo(model); // the default device
// or: session.SetTo(model, device, B_NEURAL_NO_FALLBACK);
BTensor* input = session.InputAt(0); // owned by the session
// fill input->Data(), or use BNeuralUtils
status = session.Run(); // returns when done
const BTensor* output = session.OutputAt(0);Setting a session up is the slow part; running is fast and can be repeated as often as needed. Inputs and outputs belong to the session and keep their memory between runs. A session runs one thing at a time (B_BUSY otherwise); several sessions, including sessions of the same model, run at once and share a device.
A window must not block on a network. RunAsync() returns at once and sends a message when the run is over:
session.RunAsync(BMessenger(this)); // B_NEURAL_RUN_COMPLETED when done
void MyWindow::MessageReceived(BMessage* message)
{
if (message->what == B_NEURAL_RUN_COMPLETED
&& message->GetInt32("neural:status", B_ERROR) == B_OK) {
// session.OutputAt() is ready; "neural:duration" says how long it took
}
}A message of your own can replace B_NEURAL_RUN_COMPLETED; the neural: fields are added to it, including a token and a pointer to the session. Leave the inputs alone and do not read the outputs until the message arrives. Deleting a session waits for its run to end: a run cannot be cancelled yet. GetRunInfo() tells how the model was split and how long runs take; GetOperatorInfo() tells where each operator runs and, with B_NEURAL_PROFILE, how long it took. See the asynchronous example.
Pictures, labels and results
BNeuralUtils ties tensors to the Interface Kit. SetTensorFromBitmap() scales a bitmap of any size and of the usual colour spaces into an image tensor (uint8, int8 or float; one or three channels), stretched, cropped or letterboxed, and records a neural_image_mapping to map results back. GetLabels() reads the label files models come with, GetTopValues() picks a classifier’s best classes, and IsDetector() and GetDetections() read what a detection model found:
neural_image_mapping mapping;
BNeuralUtils::SetTensorFromBitmap(session.InputAt(0), bitmap, NULL, NULL, &mapping);
session.Run();
neural_detection found[32];
int32 count = BNeuralUtils::GetDetections(session, found, 32);
for (int32 i = 0; i < count; i++) {
BRect box = mapping.ToBitmap(found[i].left, found[i].top,
found[i].right, found[i].bottom);
// labels.StringAt(found[i].class_index + 1), found[i].score
}See classifying and detecting pictures.
Speech recognition
BSpeechRecognizer turns speech into text on the machine itself, with a Whisper model the kit runs. Nothing leaves the machine. Audio goes in as it is recorded, in any raw format of the Media Kit; text comes out as messages, and nothing the caller does waits for the model.
BSpeechRecognizer recognizer;
status_t status = recognizer.SetTo(); // the first model installed
recognizer.SetTarget(BMessenger(this));
recognizer.SetInterimInterval(2000000); // also read unfinished audio
recognizer.Start(&format); // a media_raw_audio_format
// for every buffer the recording gives, from whatever thread has it:
recognizer.Feed(buffer, size);
// when the recording is over:
recognizer.Finish();
void MyWindow::MessageReceived(BMessage* message)
{
switch (message->what) {
case B_SPEECH_RESULT:
{
const char* text = message->GetString("speech:text", "");
if (message->GetBool("speech:final", false))
fTranscript << text << " "; // a phrase that stands
else
fPending = text; // replaces the one before
break;
}
case B_SPEECH_FINISHED:
// "speech:status": the recording is done with
break;
}
}For a complete recording, Transcribe() returns the text directly, and CountSegments() and GetSegment() give each phrase with its start and end time.
How it works. Whisper takes 30 seconds of audio at a time. The recognizer’s thread makes a log-mel spectrogram of each window, runs the encoder once, and runs the decoder once per token, choosing the likeliest token each time. The model writes timestamps between phrases: a phrase is final once the model has seen its end, and the next window starts where the last whole phrase ended. A recording shorter than a window therefore gives its final phrases only at Finish(); with SetInterimInterval() you also get non-final readings while it is spoken. Audio of any rate, sample format and up to eight channels is brought to 16 kHz mono as it is fed.
Models. A speech model is a directory with a speech.conf, the encoder and decoder networks, the vocabulary and the mel filters, under neural/models of a data directory (for example ~/config/non-packaged/data/neural/models/whisper-tiny.en). No model ships in the images yet. tools/airos/neural/fetch-models.sh <directory> speech in the haiku repository downloads OpenAI’s Whisper tiny.en checkpoint by checksum and converts it on a Linux host (Python with numpy, flatbuffers and ai-edge-litert); speech-multilingual fetches Whisper tiny, which knows 99 languages.
Limits. Decoding is greedy, with no beam search and no second tries, so a hard stretch of audio can give repeated or wrong text. Nothing detects the end of speech: the caller says when a recording is over. All of the work runs on the CPU. Each recording costs about 425 MB of memory while it is recognized, the 150 MB of mapped model files included. On a ROCK 5 ITX a 30-second window full of speech takes 2.2 to 2.3 s, and 11 seconds of speech take 1.5 to 1.6 s.
Summit’s webkitSpeechRecognition is built on this class. See the transcription example and Summit’s speech recognition.
Operators
The kit’s operators follow TensorFlow Lite’s, whose quantized arithmetic is specified to the bit. The CPU kernels reproduce it, which makes the CPU device the reference every back end is checked against.
| Group | Operators |
|---|---|
| Convolution | CONV_2D, DEPTHWISE_CONV_2D, TRANSPOSE_CONV, FULLY_CONNECTED |
| Pooling | AVERAGE_POOL_2D, MAX_POOL_2D, MEAN |
| Arithmetic | ADD, SUB, MUL, DIV, MAXIMUM, MINIMUM (broadcasting) |
| Activation | RELU, RELU6, LEAKY_RELU, PRELU, LOGISTIC, TANH, HARD_SWISH, SOFTMAX |
| Shape | RESHAPE, SQUEEZE, CONCATENATION, PAD, TRANSPOSE, STRIDED_SLICE, RESIZE_BILINEAR, RESIZE_NEAREST |
| Types | QUANTIZE, DEQUANTIZE |
| Transformers | BATCH_MATMUL, GELU, GATHER, SQUARED_DIFFERENCE, RSQRT (float32) |
| Detection | DETECTION_POSTPROCESS (boxes, classes, scores and count from a single-shot detector) |
That covers the MobileNet family (V1 and V2, classification and SSD detection), EfficientNet-Lite, DeepLab V3, BlazeFace and the encoder and decoder of Whisper. Weights stored as 8-bit integers with floating-point activations (“dynamic range” quantization) are not supported yet. Parameters are in the reference.
Errors
Besides the usual ones (B_NO_INIT, B_BAD_VALUE, B_NO_MEMORY, B_BUSY, file errors), the kit has its own. neural_error_string() describes any of them.
| Error | When |
|---|---|
B_NEURAL_BAD_MODEL | the file is not a model, or is damaged |
B_NEURAL_UNKNOWN_FORMAT | no loader knows the format or version |
B_NEURAL_UNSUPPORTED_OPERATOR | the model has an operator the kit has no kernel for |
B_NEURAL_TYPE_MISMATCH, B_NEURAL_SHAPE_MISMATCH | a tensor does not fit |
B_NEURAL_NO_DEVICE | no such device |
B_NEURAL_DEVICE_LOST | the device went away and the CPU could not take over (B_NEURAL_NO_FALLBACK) |
B_NEURAL_NOT_ACCELERATED | B_NEURAL_NO_FALLBACK was given and the device cannot run the whole model |
The neural command
neural devices the devices, and what their back ends say about them
neural info model.tflite inputs, outputs, operators, and what each device runs of it
neural run model.tflite -i cat.png -l labels.txt
the best classes or the detections for a picture;
-d <device>, --cpu, --no-fallback, -n <runs>,
-p (where each operator ran and how long), -o <file>
neural compare model.tflite -d rk3588_npu/0
a device against the CPU on the same input;
--tolerance <steps>, --float-tolerance <fraction>
neural bench model.tflite times the model on every device
neural transcribe speech.wav what is said in a recording, phrase by phrase;
--model <name or directory>, -d <device>, --cpu,
--language <code>, --no-timestamps, --interim <seconds>Without -i, the input is noise that is the same on every run.
Writing a back end
A back end makes a kind of hardware available to every program that uses the kit. It is one class in an add-on named <name>.neural, implementing BNeuralBackend:
CountDevices()looks for hardware, usually by opening the driver’s nodes under/dev/npu;GetDeviceInfo()describes each device.GetSupportedOperators()must be exact and quick: it sees the network as aBNeuralGraph(types, shapes, quantization, parameters) and says yes only to what the device computes correctly. The kit asks it before anything is loaded, and also just to choose a device.CreateProgram()gets a partition: operators that follow one another, the tensors they need from outside and the tensors they make that are needed outside. It converts and uploads the weights and returns aBNeuralProgram. An error means “run this stretch on the CPU”.BNeuralProgram::Run()gets the memory of the partition’s inputs and outputs in the graph’s types and NHWC layout, computes, and returns when the outputs are written. On an error the kit computes the stretch on the CPU for that run and tries the device again next time; three failures in a row, orB_NEURAL_DEVICE_LOST, hand the stretch to the CPU for good.- Programs of different sessions run at the same time, so a back end must be thread safe.
Check a back end with neural compare <model> -d <device>, which runs the device and the CPU on the same input and shows the differences, and with single-operator models made by BNeuralModelBuilder. The example back end is a complete add-on to start from.
Kernel drivers live in add-ons/kernel/drivers/npu and publish /dev/npu/<driver>/<n>. They give access to memory the processor can reach, power and a job queue, and should touch no hardware until the device is opened. The RK3588 driver also isolates clients: each open has its own IOMMU address space, and a job that reaches outside its buffers fails with B_BAD_ADDRESS.
Hardware status
Measured on 9 and 10 November 2026. “Agrees” means neural compare against the CPU device on the same input.
| Device | Model | On the device | Against the CPU | CPU | Device |
|---|---|---|---|---|---|
| RK3588 NPU (ROCK 5 ITX) | MobileNet V1 quantized | 28 of 31 operators | equal | 21.8 ms | 8.5 ms |
| MobileNet V2 quantized | 63 of 65 | within 3 quantized steps; same top five | 19.0 ms | 10.0 ms | |
| SSD MobileNet V1 quantized | 47 of 64 | same detections; scores within 0.008 | 34.3 ms | 14.7 ms | |
| Vulkan, GTX 1070 (NVK) | MobileNet V2 float | 66 of 66 | relative difference 2e-6 | 29.5 ms | 3.9 ms |
| Vulkan, Radeon 840M (RADV) | MobileNet V2 float | 66 of 66 | relative difference 2e-6 | 15.1 ms | 8.9 ms |
| AMD XDNA NPU (Ryzen AI 5 340) | 1×1 int8 convolution (one design bundle) | all | equal in all 65536 outputs | 0.35 ms | 0.11 ms |
- CPU. 726 generated single-operator models and 9 real models agree with TensorFlow Lite’s reference kernels on every tensor (integers equal, floats within 1e-4). 3737 deliberately damaged model files are refused or run without a crash, also under the address and undefined-behaviour sanitizers. x86-64 uses SSE2 only so far.
- RK3588 NPU. Quantized
CONV_2DandDEPTHWISE_CONV_2D(uint8 or int8, kernels up to 7, strides 1 and 2), with a fusedADD; one of the three cores, at 198 MHz. - Vulkan. float32 only: convolutions, fully connected, pooling, arithmetic, activations, softmax, reshapes, concatenation,
MEAN,PAD,RESIZE_BILINEAR. Not yet tried on Turnip, V3DV or PowerVR. - AMD XDNA. The NPU runs designs compiled on a Linux host with mlir-aie and Peano, found as design bundles by a key computed from a stretch of a network. Only one bundle exists. There is no AMD IOMMU driver, so the device can only be opened by root; for everyone else the CPU runs the model.
- Speech. CPU only; the same text and times as Whisper’s reference on three test recordings, on ARM64 and x86-64.
The full measurements, test methods and open points are in the Neural Kit design notes ↗.
Not yet
- Model formats other than TensorFlow Lite (ONNX would be a second loader).
- Cancelling a run, and sharing device memory between sessions and the program’s tensors (tensors are copied across).
- Speech on a GPU or NPU, voice activity detection, quantized speech models, and recognition as a media node.
- The RK3588’s other two NPU cores; designs for real networks on AMD XDNA.
- The Dragon Q8B’s Hexagon NPU, which has no open compiler or runtime (its Adreno GPU would work through the Vulkan back end, untried).
- Models in the system images: fetch them with
fetch-models.shfor now.
