Documentation · Neural Kit examples

Neural Kit examples.

Complete programs, each compiled against air/OS and run on a ROCK 5 ITX with its NPU, with the output they printed. Copy one and change it.

Before you start

Each example is a complete program. They were compiled against air/OS’s headers and libraries and run on a ROCK 5 ITX (RK3588, air/OS ARM64 release hrev60207+2552) on 10 November 2026; the output shown is what they printed there. That board’s image has no JPEG translator, so the pictures are PNG.

The public test models come from TensorFlow’s model zoo. The script in the haiku repository downloads them by checksum into a data directory, where Lookout also finds them:

# in a checkout of airos-works/haiku
# MobileNet V1 and V2, SSD MobileNet V1 and their labels:
tools/airos/neural/fetch-models.sh ~/config/non-packaged/data
# Whisper tiny.en, converted on a Linux host:
tools/airos/neural/fetch-models.sh ~/config/non-packaged/data speech

fetch-models.sh ↗ lists the files, their sources and licences.

Classify a picture

classify.cpp

The smallest useful program: load a model, set up a session on the best device, put a picture into the input tensor, run it, and print the five best classes with their labels. On the ROCK 5 the kit puts 28 of the 31 operators on the NPU by itself.

// classify.cpp: the five best classes of a picture.
//
//   classify mobilenet_v1_1.0_224_quant.tflite labels.txt cat.jpg
//
// Build: g++ classify.cpp -o classify -lneural -lbe -ltranslation

#include <stdio.h>

#include <Application.h>
#include <Bitmap.h>
#include <NeuralKit.h>
#include <StringList.h>
#include <TranslationUtils.h>


int
main(int argc, char** argv)
{
	if (argc != 4) {
		fprintf(stderr, "usage: %s <model> <labels> <picture>\n", argv[0]);
		return 2;
	}

	// The Translation Kit makes bitmaps, and bitmaps need an application.
	BApplication app("application/x-vnd.example-classify");

	BNeuralModel model(argv[1]);
	if (model.InitCheck() != B_OK) {
		fprintf(stderr, "%s: %s %s\n", argv[1],
			neural_error_string(model.InitCheck()), model.ErrorDetails());
		return 1;
	}

	// The slow part: the kit picks a device (an NPU, else a GPU, else the
	// CPU) and the device's back end uploads the weights.
	BNeuralSession session(model);
	if (session.InitCheck() != B_OK) {
		fprintf(stderr, "no session: %s\n",
			neural_error_string(session.InitCheck()));
		return 1;
	}

	BBitmap* picture = BTranslationUtils::GetBitmap(argv[3]);
	if (picture == NULL) {
		fprintf(stderr, "%s: not a picture\n", argv[3]);
		return 1;
	}

	// Scales the picture into the input tensor, whatever its size and
	// element type (uint8, int8 or float).
	neural_image_options options;
	options.fit = B_NEURAL_FIT_STRETCH;
	status_t status = BNeuralUtils::SetTensorFromBitmap(session.InputAt(0),
		picture, NULL, &options);
	delete picture;
	if (status == B_OK)
		status = session.Run();
	if (status != B_OK) {
		fprintf(stderr, "run: %s\n", neural_error_string(status));
		return 1;
	}

	BStringList labels;
	BNeuralUtils::GetLabels(argv[2], &labels);

	int64 indices[5];
	float scores[5];
	int32 count = BNeuralUtils::GetTopValues(*session.OutputAt(0), 5,
		indices, scores);
	for (int32 i = 0; i < count; i++) {
		printf("%5.1f%%  %s\n", scores[i] * 100,
			labels.StringAt(indices[i]).String());
	}

	neural_run_info info;
	session.GetRunInfo(&info);
	printf("on %s (%s): %d operators on the device, %d on the CPU, "
		"%.1f ms\n", session.Device().Name(), session.Device().ID(),
		(int)info.device_operators, (int)info.cpu_operators,
		info.last_duration / 1000.0);
	return 0;
}

Output on the ROCK 5 ITX

$ classify mobilenet_v1_1.0_224_quant.tflite labels_mobilenet_quant_v1_224.txt grace_hopper.png
 83.2%  military uniform
  7.0%  Windsor tie
  2.3%  bow tie
  1.2%  bulletproof vest
  1.2%  mortarboard
on Rockchip RK3588 NPU (rk3588_npu/0): 28 operators on the device, 3 on the CPU, 8.9 ms

Find objects in a picture

detect.cpp

A detection model returns boxes as fractions of its input. The neural_image_mapping filled in by SetTensorFromBitmap() takes them back to the picture’s pixels. Detection label files begin with a line for “nothing”, so the label of class n is at n + 1.

// detect.cpp: what a detection model finds in a picture, and where.
//
//   detect ssd_mobilenet_v1_quant.tflite ssd_mobilenet_v1_quant.txt street.jpg
//
// Build: g++ detect.cpp -o detect -lneural -lbe -ltranslation

#include <stdio.h>

#include <Application.h>
#include <Bitmap.h>
#include <NeuralKit.h>
#include <StringList.h>
#include <TranslationUtils.h>


int
main(int argc, char** argv)
{
	if (argc != 4) {
		fprintf(stderr, "usage: %s <model> <labels> <picture>\n", argv[0]);
		return 2;
	}

	BApplication app("application/x-vnd.example-detect");

	BNeuralModel model(argv[1]);
	BNeuralSession session(model);
	if (session.InitCheck() != B_OK) {
		fprintf(stderr, "%s: %s %s\n", argv[1],
			neural_error_string(session.InitCheck()), model.ErrorDetails());
		return 1;
	}
	if (!BNeuralUtils::IsDetector(session)) {
		fprintf(stderr, "%s is not a detection model\n", argv[1]);
		return 1;
	}

	BBitmap* picture = BTranslationUtils::GetBitmap(argv[3]);
	if (picture == NULL) {
		fprintf(stderr, "%s: not a picture\n", argv[3]);
		return 1;
	}

	// The mapping remembers where the picture went in the tensor, to take
	// the model's boxes (fractions of its input) back to the picture.
	neural_image_mapping mapping;
	status_t status = BNeuralUtils::SetTensorFromBitmap(session.InputAt(0),
		picture, NULL, NULL, &mapping);
	delete picture;
	if (status == B_OK)
		status = session.Run();
	if (status != B_OK) {
		fprintf(stderr, "run: %s\n", neural_error_string(status));
		return 1;
	}

	BStringList labels;
	BNeuralUtils::GetLabels(argv[2], &labels);

	neural_detection found[32];
	int32 count = BNeuralUtils::GetDetections(session, found, 32);
	for (int32 i = 0; i < count; i++) {
		if (found[i].score < 0.4f)
			continue;

		BRect box = mapping.ToBitmap(found[i].left, found[i].top,
			found[i].right, found[i].bottom);
		// Detection label files begin with a line for "nothing".
		printf("%3.0f%%  %-16s at %.0f, %.0f, %.0f × %.0f pixels\n",
			found[i].score * 100,
			labels.StringAt(found[i].class_index + 1).String(),
			box.left, box.top, box.Width(), box.Height());
	}
	return 0;
}

Output on the ROCK 5 ITX

$ detect ssd_mobilenet_v1_quant.tflite labelmap.txt grace_hopper.png
 72%  person           at 4, 31, 506 × 567 pixels
 63%  tie              at 225, 429, 67 × 114 pixels
 48%  person           at 8, 9, 121 × 404 pixels
 44%  tie              at 233, 437, 54 × 77 pixels

List devices and choose one

devices.cpp

What the roster knows: each device, how much of a model it would run (asked without loading anything), what its back end reports, and which device the kit would choose under each preference. A float model would show the NPU running none of it, and the kit choosing the CPU.

// devices.cpp: the machine's devices, how much of a model each would run,
// and the device the kit would choose.
//
//   devices mobilenet_v2_1.0_224.tflite
//
// Build: g++ devices.cpp -o devices -lneural -lbe

#include <stdio.h>

#include <Message.h>
#include <NeuralKit.h>


static void
print_choice(const char* what, const BNeuralModel& model, uint32 flags)
{
	BNeuralDevice device;
	status_t status = BNeuralRoster::Default()->GetDefaultDevice(&device,
		flags, &model);
	if (status == B_OK)
		printf("%-28s %s\n", what, device.ID());
	else
		printf("%-28s none (%s)\n", what, neural_error_string(status));
}


int
main(int argc, char** argv)
{
	if (argc != 2) {
		fprintf(stderr, "usage: %s <model>\n", argv[0]);
		return 2;
	}

	BNeuralModel model(argv[1]);
	if (model.InitCheck() != B_OK) {
		fprintf(stderr, "%s: %s %s\n", argv[1],
			neural_error_string(model.InitCheck()), model.ErrorDetails());
		return 1;
	}

	// Asking the roster anything loads the back-end add-ons; one whose
	// hardware is not in this machine reports no devices.
	BNeuralRoster* roster = BNeuralRoster::Default();
	for (int32 i = 0; i < roster->CountDevices(); i++) {
		BNeuralDevice device;
		if (roster->GetDeviceAt(i, &device) != B_OK)
			continue;

		uint32 flags = device.Flags();
		const char* types = (flags & B_NEURAL_DEVICE_QUANTIZED) == 0 ? "float"
			: (flags & B_NEURAL_DEVICE_FLOAT) == 0 ? "quantized"
			: "quantized and float";
		printf("%-14s %s %s: %s networks, back end %s\n", device.ID(),
			neural_device_kind_name(device.Kind()), device.Name(), types,
			device.Backend());

		// Asks the back end only; nothing is loaded onto the device.
		printf("               runs %d of the model's %d operators\n",
			(int)device.CountSupportedOperators(model),
			(int)model.CountOperators());

		BMessage info;
		if (device.GetInfo(&info) == B_OK)
			info.PrintToStream();
	}

	print_choice("fastest:", model, B_NEURAL_PREFER_SPEED);
	print_choice("least power:", model, B_NEURAL_PREFER_LOW_POWER);
	print_choice("whole model on the device:", model, B_NEURAL_NO_FALLBACK);
	return 0;
}

Output on the ROCK 5 ITX

$ devices mobilenet_v1_1.0_224_quant.tflite
cpu            CPU CPU: quantized and float networks, back end cpu
               runs 31 of the model's 31 operators
BMessage(0x0) {
        cores = int32(0x8 or 8)
        threads = int32(0x8 or 8)
        driver = string("Neural Kit", 11 bytes)
}
rk3588_npu/0   NPU Rockchip RK3588 NPU: quantized networks, back end rk3588_npu
               runs 28 of the model's 31 operators
BMessage(0x0) {
        driver = string("rk3588_npu", 11 bytes)
        version = string("0x46495245", 11 bytes)
        cores = int32(0x1 or 1)
        clock = int32(0xbcd3d80 or 198000000)
        path = string("/dev/npu/rk3588/0", 18 bytes)
}
fastest:                     rk3588_npu/0
least power:                 rk3588_npu/0
whole model on the device:   cpu

Run without blocking

async.cpp

The pattern Lookout uses to keep up with a camera. A looper owns the session and starts runs with RunAsync(); frames that arrive during a run replace each other, and the newest is taken when the run is over. Three frames went in while the first was being processed, so the second was dropped: two results.

// async.cpp: runs without blocking. A looper owns the session and starts
// each run with RunAsync(); frames that arrive while a run is going on
// replace each other, and the newest is taken when the run is over. This is
// how Lookout keeps up with a camera.
//
//   async mobilenet_v1_1.0_224_quant.tflite a.jpg b.jpg c.jpg
//
// Build: g++ async.cpp -o async -lneural -lbe -ltranslation

#include <stdio.h>

#include <Application.h>
#include <Bitmap.h>
#include <Looper.h>
#include <NeuralKit.h>
#include <TranslationUtils.h>


enum {
	kMsgFrame = 'fram',		// "bitmap" (pointer): a new picture to look at
	kMsgRunDone = 'done',	// our own completion message
	kMsgNoMore = 'nomo'		// quit the application when all is done
};


class Classifier : public BLooper {
public:
	Classifier(const BNeuralModel& model)
		:
		BLooper("classifier"),
		fSession(model),
		fRunning(false),
		fDone(false),
		fNext(NULL)
	{
	}

	~Classifier()
	{
		// The session's destructor waits for a run that is going on.
		delete fNext;
	}

	status_t InitCheck() const { return fSession.InitCheck(); }

	virtual void MessageReceived(BMessage* message)
	{
		switch (message->what) {
			case kMsgFrame:
			{
				BBitmap* bitmap;
				if (message->FindPointer("bitmap", (void**)&bitmap) != B_OK)
					break;
				// Only the newest frame waits; older ones are dropped.
				delete fNext;
				fNext = bitmap;
				if (!fRunning)
					_StartNext();
				_QuitWhenDone();
				break;
			}

			case kMsgRunDone:
			{
				fRunning = false;
				status_t status = message->GetInt32("neural:status", B_ERROR);
				if (status == B_OK) {
					const BTensor* output = fSession.OutputAt(0);
					int64 best = output->IndexOfMaximum();
					printf("class %d (%.0f%%) in %.1f ms on %s\n", (int)best,
						output->ValueAt(best) * 100,
						message->GetInt64("neural:duration", 0) / 1000.0,
						fSession.Device().ID());
				} else
					printf("run failed: %s\n", neural_error_string(status));

				if (fNext != NULL)
					_StartNext();
				_QuitWhenDone();
				break;
			}

			case kMsgNoMore:
				fDone = true;
				_QuitWhenDone();
				break;

			default:
				BLooper::MessageReceived(message);
		}
	}

private:
	void _QuitWhenDone()
	{
		if (fDone && !fRunning && fNext == NULL)
			be_app->PostMessage(B_QUIT_REQUESTED);
	}

	void _StartNext()
	{
		BBitmap* bitmap = fNext;
		fNext = NULL;
		status_t status = BNeuralUtils::SetTensorFromBitmap(
			fSession.InputAt(0), bitmap);
		delete bitmap;

		// Returns at once; kMsgRunDone comes back with "neural:status",
		// "neural:duration" and the other neural: fields added. Inputs
		// and outputs are not to be touched until then.
		BMessage done(kMsgRunDone);
		if (status == B_OK)
			status = fSession.RunAsync(BMessenger(this), &done);
		if (status == B_OK)
			fRunning = true;
		else
			printf("cannot run: %s\n", neural_error_string(status));
	}

	BNeuralSession	fSession;
	bool			fRunning;
	bool			fDone;
	BBitmap*		fNext;
};


int
main(int argc, char** argv)
{
	if (argc < 3) {
		fprintf(stderr, "usage: %s <model> <picture>...\n", argv[0]);
		return 2;
	}

	BApplication app("application/x-vnd.example-async");

	BNeuralModel model(argv[1]);
	Classifier* classifier = new Classifier(model);
	if (classifier->InitCheck() != B_OK) {
		fprintf(stderr, "%s: %s\n", argv[1],
			neural_error_string(classifier->InitCheck()));
		delete classifier;
		return 1;
	}
	classifier->Run();

	for (int i = 2; i < argc; i++) {
		BBitmap* bitmap = BTranslationUtils::GetBitmap(argv[i]);
		if (bitmap == NULL)
			continue;
		BMessage frame(kMsgFrame);
		frame.AddPointer("bitmap", bitmap);
		classifier->PostMessage(&frame);
	}
	classifier->PostMessage(kMsgNoMore);

	app.Run();

	classifier->Lock();
	classifier->Quit();
	return 0;
}

Output on the ROCK 5 ITX

$ async mobilenet_v1_1.0_224_quant.tflite grace_hopper.png grace_hopper.bmp grace_hopper.png
class 653 (83%) in 8.9 ms on rk3588_npu/0
class 653 (83%) in 8.5 ms on rk3588_npu/0

Build a network in code

builder.cpp

A network made with BNeuralModelBuilder, run on the CPU as the reference and on the device the kit chooses, then compared. This is also how a back end is tested, one operator at a time. Binary operators take neural_binary_params, and an activation can be fused into them. On the ROCK 5 the kit chose the CPU, because the NPU runs only quantized networks.

// builder.cpp: a network made in code, y = min(max(x * 2 + 1, 0), 6), run on
// the CPU (the reference) and on the device the kit chooses, and compared.
// This is also how a back end is tested: one operator at a time against
// the CPU.
//
// Build: g++ builder.cpp -o builder -lneural -lbe

#include <math.h>
#include <stdio.h>

#include <NeuralKit.h>


static status_t
build(BNeuralModel* model)
{
	BNeuralModelBuilder builder("scale and clamp");

	int32 shape[] = { 1, 8 };
	int32 one[] = { 1 };
	int32 x = builder.AddInput(BTensorInfo(B_NEURAL_FLOAT32, 2, shape, "x"));

	float two = 2.0f;
	int32 factor = builder.AddConstant(
		BTensorInfo(B_NEURAL_FLOAT32, 1, one, "two"), &two, sizeof(two));
	float oneValue = 1.0f;
	int32 offset = builder.AddConstant(
		BTensorInfo(B_NEURAL_FLOAT32, 1, one, "one"), &oneValue,
		sizeof(oneValue));

	// Binary operators broadcast and take neural_binary_params; the
	// clamp to 0 .. 6 is fused into the ADD as its activation.
	neural_binary_params none = { B_NEURAL_ACTIVATION_NONE };
	int32 inputs[2] = { x, factor };
	int32 scaled = builder.AddOperator(B_NEURAL_OP_MUL, inputs, 2,
		BTensorInfo(B_NEURAL_FLOAT32, 2, shape, "scaled"), &none,
		sizeof(none));

	neural_binary_params relu6 = { B_NEURAL_ACTIVATION_RELU6 };
	inputs[0] = scaled;
	inputs[1] = offset;
	int32 y = builder.AddOperator(B_NEURAL_OP_ADD, inputs, 2,
		BTensorInfo(B_NEURAL_FLOAT32, 2, shape, "y"), &relu6, sizeof(relu6));

	builder.MarkOutput(y);
	// Checks types, shapes and order; an error of any Add...() above comes
	// back here.
	return builder.Build(model);
}


static status_t
run(BNeuralSession& session, const float* input, float* output)
{
	session.InputAt(0)->SetValues(input, 8);
	status_t status = session.Run();
	if (status == B_OK)
		status = session.OutputAt(0)->GetValues(output, 8);
	return status;
}


int
main()
{
	BNeuralModel model;
	status_t status = build(&model);
	if (status != B_OK) {
		fprintf(stderr, "build: %s\n", neural_error_string(status));
		return 1;
	}

	BNeuralSession reference(model, B_NEURAL_CPU_ONLY);
	BNeuralSession session(model);
	if (reference.InitCheck() != B_OK || session.InitCheck() != B_OK) {
		fprintf(stderr, "no session\n");
		return 1;
	}

	float input[8] = { -3, -1, -0.5f, 0, 0.5f, 1, 2.5f, 4 };
	float expected[8];
	float actual[8];
	if (run(reference, input, expected) != B_OK
		|| run(session, input, actual) != B_OK) {
		fprintf(stderr, "run failed\n");
		return 1;
	}

	float worst = 0;
	for (int i = 0; i < 8; i++) {
		printf("%5.1f -> %4.1f\n", input[i], actual[i]);
		worst = fmaxf(worst, fabsf(actual[i] - expected[i]));
	}
	printf("%s against the CPU: largest difference %g\n",
		session.Device().ID(), worst);
	return 0;
}

Output on the ROCK 5 ITX

$ builder
 -3.0 ->  0.0
 -1.0 ->  0.0
 -0.5 ->  0.0
  0.0 ->  1.0
  0.5 ->  2.0
  1.0 ->  3.0
  2.5 ->  6.0
  4.0 ->  6.0
cpu against the CPU: largest difference 0

Transcribe a recording

transcribe.cpp

Speech to text, fed as a live recording would be: a file decoded by the Media Kit, handed to BSpeechRecognizer buffer by buffer in whatever format the decoder gives, with the phrases arriving as messages to a looper. Feeding from a microphone works the same way, from the thread that records.

// transcribe.cpp: speech to text, fed as a recording would be. The audio
// file is decoded with the Media Kit and handed to the recognizer buffer by
// buffer; the text comes back as messages to a looper. Feeding from a
// microphone works the same way, from the thread that records.
//
//   transcribe speech.wav [model]
//
// The model is a name ("whisper-tiny.en") or a model directory; without
// one, the first speech model installed in neural/models of a data
// directory, for example ~/config/non-packaged/data/neural/models/
// whisper-tiny.en (see fetch-models.sh). Build: g++ transcribe.cpp -o transcribe -lneural -lmedia -lbe

#include <stdio.h>
#include <stdlib.h>

#include <Entry.h>
#include <Looper.h>
#include <MediaFile.h>
#include <MediaTrack.h>
#include <NeuralKit.h>


class Printer : public BLooper {
public:
	Printer()
		:
		BLooper("printer"),
		fFinished(create_sem(0, "finished")),
		fStatus(B_OK)
	{
	}

	~Printer()
	{
		delete_sem(fFinished);
	}

	virtual void MessageReceived(BMessage* message)
	{
		switch (message->what) {
			case B_SPEECH_RESULT:
				// A result that is not final is the recognizer's reading,
				// so far, of everything after the last final phrase: it
				// replaces the one before. Here only final ones are shown.
				if (message->GetBool("speech:final", false)) {
					printf("[%6.2f – %6.2f]  %s\n",
						message->GetInt64("speech:start", 0) / 1e6,
						message->GetInt64("speech:end", 0) / 1e6,
						message->GetString("speech:text", ""));
				}
				break;

			case B_SPEECH_FINISHED:
				// After the last result of the recording.
				fStatus = message->GetInt32("speech:status", B_ERROR);
				printf("%.1f s of audio in %.1f s\n",
					message->GetInt64("speech:audio", 0) / 1e6,
					message->GetInt64("speech:duration", 0) / 1e6);
				release_sem(fFinished);
				break;

			default:
				BLooper::MessageReceived(message);
		}
	}

	status_t WaitForEnd()
	{
		acquire_sem(fFinished);
		return fStatus;
	}

private:
	sem_id		fFinished;
	status_t	fStatus;
};


static status_t
feed_file(BSpeechRecognizer& recognizer, const char* path)
{
	entry_ref ref;
	status_t status = get_ref_for_path(path, &ref);
	if (status != B_OK)
		return status;

	BMediaFile file(&ref);
	if ((status = file.InitCheck()) != B_OK)
		return status;

	for (int32 i = 0; i < file.CountTracks(); i++) {
		BMediaTrack* track = file.TrackAt(i);
		media_format format;
		format.Clear();
		format.type = B_MEDIA_RAW_AUDIO;
		format.u.raw_audio = media_raw_audio_format::wildcard;
		if (track->DecodedFormat(&format) != B_OK
			|| format.type != B_MEDIA_RAW_AUDIO) {
			file.ReleaseTrack(track);
			continue;
		}

		// Any raw format will do: the recognizer makes 16 kHz mono of it.
		const media_raw_audio_format& raw = format.u.raw_audio;
		size_t frameSize = (raw.format & media_raw_audio_format
			::B_AUDIO_SIZE_MASK) * raw.channel_count;
		char* buffer = NULL;
		if (frameSize > 0 && raw.buffer_size >= frameSize)
			buffer = (char*)malloc(raw.buffer_size);
		status = buffer != NULL ? recognizer.Start(&raw) : B_NO_MEMORY;

		int64 frames;
		while (status == B_OK && track->ReadFrames(buffer, &frames) == B_OK
			&& frames > 0) {
			// Returns at once; B_BUSY only when ten minutes of audio are
			// waiting for the model.
			while ((status = recognizer.Feed(buffer, frames * frameSize))
					== B_BUSY)
				snooze(50000);
		}
		free(buffer);
		file.ReleaseTrack(track);
		return status;
	}
	return B_MEDIA_NO_HANDLER;
}


int
main(int argc, char** argv)
{
	if (argc != 2 && argc != 3) {
		fprintf(stderr, "usage: %s <audio file> [model]\n", argv[0]);
		return 2;
	}

	// NULL: the first speech model installed. The device is chosen as for
	// a BNeuralSession.
	BSpeechRecognizer recognizer;
	status_t status = recognizer.SetTo(argc == 3 ? argv[2] : NULL);
	if (status != B_OK) {
		fprintf(stderr, "no recognizer: %s (%s)\n",
			neural_error_string(status), recognizer.ErrorDetails());
		return 1;
	}
	printf("model %s on %s\n", recognizer.ModelName(),
		recognizer.Device().ID());

	Printer* printer = new Printer();
	printer->Run();
	recognizer.SetTarget(BMessenger(printer));

	status = feed_file(recognizer, argv[1]);
	if (status == B_OK) {
		// The rest is recognized, then B_SPEECH_FINISHED is sent.
		recognizer.Finish();
		status = printer->WaitForEnd();
	} else
		recognizer.Cancel();

	printer->Lock();
	printer->Quit();

	if (status != B_OK) {
		fprintf(stderr, "%s: %s\n", argv[1], neural_error_string(status));
		return 1;
	}
	return 0;
}

Output on the ROCK 5 ITX

$ transcribe jfk.wav whisper-tiny.en
model tiny.en on cpu
[  0.00 –   8.00]  And so my fellow Americans ask not what your country can do for you
[  8.00 –  11.00]  ask what you can do for your country.
11.0 s of audio in 1.5 s

A back end for new hardware

backend.cpp

A complete back-end add-on. It claims float32 RELU operators and runs stretches of them in its own program, standing in for a device. Swap the body of Run() for a job submitted to your driver and the rest of the kit (loading, partitioning, CPU fallback, comparison) comes for free. Installed in ~/config/non-packaged/add-ons/neural, it appeared in neural devices and ran its two operators of a test network correctly.

// backend.cpp: the smallest back end. It claims the float32 RELU operators
// of a network and runs stretches of them in its own "program", which here
// is plain C++ standing in for a device. A real back end opens its driver
// under /dev/npu in CountDevices(), converts and uploads weights in
// CreateProgram() and submits jobs in Run().
//
// Build: g++ -shared -fPIC backend.cpp -o example.neural -lneural -lbe
// Install: ~/config/non-packaged/add-ons/neural/example.neural, then
//   neural devices        lists "example/0"
//   neural compare model.tflite -d example/0

#include <map>
#include <new>
#include <set>
#include <string.h>
#include <vector>

#include <Message.h>
#include <add-ons/neural/NeuralBackend.h>


static bool
is_float_relu(const BNeuralGraph& graph, const neural_graph_operator* op)
{
	if (op->code != B_NEURAL_OP_RELU || op->input_count != 1
		|| op->output_count != 1) {
		return false;
	}
	const neural_graph_tensor* in = graph.TensorAt(op->inputs[0]);
	const neural_graph_tensor* out = graph.TensorAt(op->outputs[0]);
	return in->type == B_NEURAL_FLOAT32 && out->type == B_NEURAL_FLOAT32;
}


static size_t
element_count(const neural_graph_tensor* tensor)
{
	size_t count = 1;
	for (int32 i = 0; i < tensor->rank; i++)
		count *= tensor->dimensions[i];
	return count;
}


class ExampleProgram : public BNeuralProgram {
public:
	ExampleProgram(const BNeuralGraph& graph,
		const neural_partition& partition)
		:
		fGraph(graph),
		fOperators(partition.operators,
			partition.operators + partition.operator_count)
	{
		// The kit promises memory only for the tensors that cross the
		// stretch's edges; what is made and used inside is the back end's
		// own (a device keeps it in its memory).
		std::set<int32> edges(partition.inputs,
			partition.inputs + partition.input_count);
		edges.insert(partition.outputs,
			partition.outputs + partition.output_count);
		for (int32 index : fOperators) {
			int32 output = graph.OperatorAt(index)->outputs[0];
			if (edges.count(output) == 0) {
				fScratch[output].resize(
					element_count(graph.TensorAt(output)));
			}
		}
	}

	// tensors[] has one entry per tensor of the graph; the partition's
	// inputs are filled in, its outputs are to be written.
	virtual status_t Run(void* const* tensors)
	{
		for (int32 index : fOperators) {
			const neural_graph_operator* op = fGraph.OperatorAt(index);
			const float* in = _Memory(tensors, op->inputs[0]);
			float* out = _Memory(tensors, op->outputs[0]);
			if (in == NULL || out == NULL)
				return B_BAD_VALUE;
				// the kit runs this stretch on the CPU this time

			size_t count = element_count(fGraph.TensorAt(op->outputs[0]));
			for (size_t i = 0; i < count; i++)
				out[i] = in[i] > 0 ? in[i] : 0;
		}
		return B_OK;
	}

private:
	float* _Memory(void* const* tensors, int32 index)
	{
		auto scratch = fScratch.find(index);
		if (scratch != fScratch.end())
			return scratch->second.data();
		return (float*)tensors[index];
	}

	const BNeuralGraph&					fGraph;
	std::vector<int32>					fOperators;
	std::map<int32, std::vector<float>>	fScratch;
};


class ExampleBackend : public BNeuralBackend {
public:
	ExampleBackend(image_id image)
		:
		BNeuralBackend(image)
	{
	}

	virtual const char* Name() const
	{
		return "example";
	}

	virtual int32 CountDevices()
	{
		// Look for the hardware here; 0 costs nothing more.
		return 1;
	}

	virtual status_t GetDeviceInfo(int32 device, neural_device_info* info,
		BMessage* details)
	{
		if (device != 0)
			return B_BAD_INDEX;
		memset(info, 0, sizeof(*info));
		strlcpy(info->id, "0", sizeof(info->id));
		strlcpy(info->name, "Example accelerator", sizeof(info->name));
		strlcpy(info->vendor, "Example", sizeof(info->vendor));
		info->kind = B_NEURAL_DEVICE_NPU;
		info->flags = B_NEURAL_DEVICE_FLOAT;
		info->priority = 10;
		if (details != NULL)
			details->AddString("driver", "none");
		return B_OK;
	}

	// Asked before anything is loaded, also just to choose a device: look
	// at codes, types, shapes and parameters only, and say yes only to what
	// the device computes correctly.
	virtual status_t GetSupportedOperators(int32 device,
		const BNeuralGraph& graph, bool* supported)
	{
		for (int32 i = 0; i < graph.CountOperators(); i++)
			supported[i] = is_float_relu(graph, graph.OperatorAt(i));
		return B_OK;
	}

	// One call per stretch of supported operators: convert and upload
	// weights here. An error means "run this stretch on the CPU". The
	// graph outlives the program; the partition is copied.
	virtual status_t CreateProgram(int32 device, const BNeuralGraph& graph,
		const neural_partition& partition, uint32 flags,
		BNeuralProgram** _program)
	{
		ExampleProgram* program
			= new(std::nothrow) ExampleProgram(graph, partition);
		if (program == NULL)
			return B_NO_MEMORY;
		*_program = program;
		return B_OK;
	}
};


extern "C" _EXPORT BNeuralBackend*
instantiate_neural_backend(image_id image)
{
	return new(std::nothrow) ExampleBackend(image);
}

Output on the ROCK 5 ITX

$ neural devices
example/0            NPU Example Example accelerator (float)
    driver           none
...
A test network relu(relu(x)) + x, run on example/0:
supported 2 of 3
-2 -1 0 2 4 6
device ops 2 cpu ops 1 partitions 1 failures 0

Speech in a web page

Summit

Summit’s webkitSpeechRecognition is a BSpeechRecognizer fed from the microphone. A page uses the standard Web Speech API; nothing leaves the machine. The interface exists only where the system has the recognizer, so 'webkitSpeechRecognition' in window is false on stock Haiku.

const recognition = new webkitSpeechRecognition();
recognition.lang = 'en-US';
recognition.interimResults = true;      // SetInterimInterval()
recognition.onresult = event => {
	const result = event.results[event.results.length - 1];
	console.log(result[0].transcript, result.isFinal);   // "speech:final"
};
recognition.start();                    // Start(), then Feed() from the microphone

The first start() asks for the microphone. With a model installed, it was checked on a Framework 13 with a Logitech C920’s microphone. Summit’s speech recognition notes ↗ cover events, errors and what the machine needs.

Lookout: the kit in an application

Demos menu

Lookout, in the Demos menu, shows what a detection or classification model finds in a picture or in a camera’s video, on the device of your choice. It is the kit used the way an application should use it:

  • An engine looper owns the model and the session, so setting a session up (seconds on a GPU) never holds up the window.
  • Runs use RunAsync() with the engine as target; camera frames replace each other in a slot while a run is going on.
  • It watches the roster: when a device is added or removed, the session is set up again, and the user’s choice is kept by device ID.
  • After each run, GetRunInfo() shows how many operators ran on the device and on the CPU, and whether the device failed runs.
  • Pictures are fitted with SetTensorFromBitmap(), and detection boxes taken back with the mapping.

On a Framework 13 it ran SSD MobileNet V1 on the CPU at the camera’s 30 frames per second, 13 to 15 ms a run, and MobileNet V2 on the Radeon GPU through Vulkan at 15 to 17 ms. Engine.cpp ↗ is the place to start reading; the whole app ↗.

More code to read