Hubris Reference

We’re trying to inch closer to robustness than our previous systems could, through a combination of decisions.

More memory safety. The bulk of both the Hubris kernel and our applications are written in safe Rust, with careful sprinklings of unsafe Rust where required. “Unsafe” Rust is still a much safer language than C or assembler and helps us avoid thinking about a bunch of potential bugs.

Fault isolation. Tasks, including drivers, can crash independently. An application might choose to have a driver crash ripple out into clients, but could also choose to notify clients and have them retry requests — whichever is appropriate. Memory protection is vital for ensuring this; without it, once some invariant in the system is observed to be broken, you have to assume they’re all in jeopardy.

Holistic deployment. It’s critical to ship the code you test, but once a program has been factored into a bunch of separately compiled pieces, there’s a temptation to update each of these pieces independently. This leads to a combinatorial explosion in the configurations you’d have to test to be thorough. To avoid that, engineering processes pick up a lot of overhead about conscious forward- and backward-compatible API design, etc. We’ve chosen to bypass this and assume that all the software that runs on a given processor was built — and tested! — together.

There are a class of “ideal attractors” in engineering, concepts like “everything is an object,” “homoiconicity,” “purely functional,” “pure capability system,” etc. Engineers fall into orbit around these ideas quite easily. Systems that follow these principles often get useful properties out of the deal.

However, going too far in any of these directions is also a great way to find a deep reservoir of unsolved problems, which is part of why these are popular directions in academia.

In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features. For instance:

We are doing our best to avoid writing code.

That might seem like an odd statement coming from a group that has written an operating system from whole-cloth, and it is.

Adding code to a system like this adds attack surface, new corner cases that must be tested, things the programmer has to reason about, and — most mundanely — more code we have to understand and maintain.

We’re working hard to avoid adding features to the lower layers of the system, even when it takes a little more work at higher levels to compensate. For instance, the original Hubris proposal included a MINIX-3-inspired asynchronous messaging primitive for use by the supervisor task; we’re conscious of the massive impact this would have on the system, and have been going out of our way to avoid implementing it.

Now, that being said: we are doing our best to ensure that the code we do write is correct.

In many ways, Rust makes this part of the job easy, but “cowboy coding” is as feasible in Rust as in other languages, given a sufficiently motivated cowboy. Culturally, we try to avoid being “clever” in pursuit of a few saved bytes or cycles, and instead solve problems in ways that are more likely to be correct. We also prize correctness by construction where possible, meaning, designing pieces of the system such that illegal or undesirable states simply can’t be represented, and defining operations that compose in predictable and useful ways that can be discovered by applying local reasoning.

This is not the only way to design an IPC system. The main alternatives involve some sort of asynchrony or queueing, where a task can issue several messages and then do other work while waiting for responses. This can be useful, so it’s worth asking why Hubris is fully synchronous.

It turns out that synchronous IPC has significant advantages for systems like Hubris.

It’s fast. Synchronous IPC can be implemented using a single message copy from sender to recipient. The message need only be read once. This reduces the number of clock cycles required for a message transfer. (The operating system that originally made this point was L4, and Hubris’s IPC mechanism is directly inspired by L4’s.)

There are no queues to size. Asynchronous systems usually queue messages in kernel memory, which means there are now kernel-managed queues that must have their resources accounted for. In extreme cases, this can allow a single task to exhaust the kernel’s memory with impunity by expanding queues to the breaking point — which isn’t great for reliability. Many systems that try to avoid that problem do so by allowing the user to impose size limits on queues, which then become another thing users need to tune carefully for best performance. With synchronous rendezvous messaging, we avoid having queues in the first place.

It limits the power of any single task. A Hubris task can either be doing local work, or sending one message. It can’t, for instance, spam every other task in the system with messages simultaneously, or set up hundreds of messages to a single task. This avoids “fault amplification” scenarios where a bug in one task cascades into a much larger problem, as in a denial-of-service attack.

It lets message recipients reason about what their callers are doing. When your task is processing a message, it knows that the sending task is parked waiting for a reply.^[1] It also knows that, if it goes to receive a second message before replying, that message won’t be from the same task. This allows message recipients to coordinate when senders run and don’t run, which is useful for implementing mutual exclusion, as we’ll discuss later.

Finally, synchronous IPC makes the system much easier to think about. Each task operates as a synchronous state machine, with IPC operations appearing as atomic and predictably ordered. Larger systems composed out of tasks can be easily halted and inspected to see who’s waiting on who. Lots of potential race conditions are eliminated. We’ve found this point to be hugely important during our early experience with the system.

From the caller’s perspective, send deposits any returned data into the incoming buffer given to send. But it also returns two integers:

The response code, like the operation code, is largely application-defined, but it’s intended to be used to distinguish success from error. Specifically, 0 meaning success, and non-zero meaning error.

The length gives the number of bytes in the incoming buffer that have been written by the kernel and are now valid. This happens no matter what the response code is, so an operation can return detailed data even in the case of an error.

This scheme is specifically designed to allow IPC wrapper functions to translate the results of send into a Rust Result<T, E> type, where T is built from the data returned on success, and E is built from the combination of the non-zero response code and any additional data returned. Most wrappers do this.

When a message transfer happens, the kernel diligently copies the message data from one place to another. This operation is uninterruptible, so it may delay processing of interrupts or timers. To limit this, we impose a maximum length on messages, currently 256 bytes.

Sending and receiving small structs by copy is a good start, but what if you want something more complex? For example, how do you send 1kiB of data to a serial port, or read it back, when messages are limited to 256 bytes?

The answer is the same as in Rust: when you call the operation, you loan it some of your memory.

Any send operation can include leases, which are small descriptors that tell the kernel to allow the recipient of the message to access parts of the sender’s memory space. Each lease can be read-only, write-only, or read-write. While the sender is waiting for a reply, the recipient has exclusive control of the leased memory it is borrowing. If the caller resumes (generally after reply, but also possible in some corner cases involving task supervision) the leases are reliably and atomically revoked.

This means it’s safe to lend out any memory that the caller can safely access, including memory from the caller’s stack.

This property also means that lending data can be expressed, in Rust, as simple & or &mut borrows, and checked for correctness at compile time.

Each send can include up to 255 leases (currently — this number may be reduced because what on earth do you need that many leases for).

On the recipient side, leases are referred to by index (0 through 255), and an IPC operation would typically declare that it needs certain arguments passed as leases in a certain order. For instance, a simple serial write operation might expect a single readable lease giving the data to send, while a more nuanced I2C operation might take a sequence of readable and writable leases.

Let’s sketch a concrete IPC interface, to get a feeling for how the various options on send fit together. Imagine a task that implements a very simple streaming data access protocol consisting of two functions (written as in Rust):

These are basically POSIX read and write, only expressed in Rust style.

A concrete mapping of these operations to IPCs might go as follows.

Read. Operation code 0.

Write. Operation code 1.

A very simple IPC stub for the read operation might be written as follows.

(write would be nearly identical, but with the operation code changed.)

Hubris does not require that you reply before calling recv again. You could instead start an operation, do some bookkeeping to keep track of that sender, and then recv the next, with the intent of replying later. This allows you to implement a pipelined server that overlaps requests.

Hubris also doesn’t require that you reply in the same order as recv. For example, in a pipelined server, you might want to promptly reply with an error to a bogus request while still processing others. Or, in a fully asynchronous server (such as a network stack for something like UDP), you might reply whenever operations finish, regardless of their order.

Hubris doesn’t actually require that you reply, ever. The caller will wait patiently. This means if you want to halt a task, sending a message to someone who will never reply is a reasonable technique. Or, a server could halt malfunctioning callers by never replying (see next section).

What is required for reply to succeed is that the sender must actually be blocked in a send to your task. If you reply to a random task ID that has never messaged you, the reply will not go through. If the sending task has been forceably restarted by some supervising entity, the reply will not go through. Similarly, if an application implements IPC timeouts by forceably unblocking senders that have waited too long (something you can choose to do), the reply to the timed-out sender won’t go through.

Because the latter two cases (sender timed out, sender rebooted) are expected to be possible in an otherwise functioning application, and because it isn’t clear in general how a server should handle a behavior error in one of its clients, the reply operation does not return an error to the server, even if it doesn’t go through. The server moves on.

Hubris assumes that you mistrust tasks sending you messages, and provides enough information to detect the following error cases:

Any of these suggest that the sender is confused or malfunctioning. You have a few options for dealing with these cases:

recv can operate in two modes, called open receive and closed receive. The general case that we’ve been discussing so far is the open receive case.

In a closed receive, the task selects a single other task to receive messages from. Any other sender will be blocked.

This can be used to implement mutual exclusion. If a client sends a lock request to a server, for instance, the server could then perform a closed receive and only accept messages from that client. Other clients could send lock requests, but they’d queue up, until the first client either sends a “release” message, or dies (see below).

Notifications are used by the kernel to route hardware interrupts to tasks: a task can request that an interrupt appear as one of its notification bits. (More on that in the chapter on Interrupts, below.)

Notifications are also used to signal tasks that the deadline they loaded into their timer has elapsed. (More on that in the chapter on Timers.)

Finally, notifications can be valuable between tasks in an application, as a way for one task to notify another of an event without blocking. In particular, notifications are the only safe way for a high-priority server shared by many clients to signal a single client — if it used send instead, that client could decide not to reply, starving all the other clients.

In developing firmware on Hubris we’ve found a particular pattern to be useful, called “pingback.” In this pattern, a high-priority shared server (such as a network stack) has obtained data that it needs to give to one of its clients — but it can’t just send the data for the reason described above. One option is to have all clients forever blocked in send to the server until data arrived, but that keeps the clients from ever doing anything else! Instead, the server and clients can agree on a protocol where

This pattern is useful, in part, because it’s very tolerant of a defective client task. If the server posts the notification and the client never responds, it’s no skin off the server’s back — it’s still free to continue serving other clients.

Syscalls are invoked using the SVC instruction. The 8-bit immediate in the instruction is ignored, because reading it from user text is potentially sketchy.

Syscalls provide for up to 7 arguments and 8 return values in registers. Syscalls never use arguments from the stack, to make it easier to reason about possible memory management faults during syscall entry (i.e. now there aren’t any).

Arguments to syscalls are passed in r4 through r10, with the syscall index in r11.

Return values from syscalls are returned in r4 through r11.

Syscalls are invoked using the ECALL instruction. The rest is TBD.

Sends a message.

The error-free path:

Receives a pending message or notification.

The error-free path:

If the provided notification mask is not zero, the receive operation may be interrupted by a notification message from the kernel instead. This happens if any of the notification bits specified in the mask (by 1 bits) have been set on the calling task. When RECV returns, you can distinguish these notification messages because they have the kernel’s virtual task ID 0xFFFF as the message sender.

Replies to a received message.

If all goes well, this copies a slice of data from your task’s memory into the caller’s memory and resumes the caller.

Configures your task’s timer.

Copies data from memory borrowed from a caller (a “borrow”).

Copies data into memory borrowed from a caller (a “borrow”).

Collects information about one entry in a sender’s lease table.

Delivers a Panic fault to the calling task, recording an optional message.

This is roughly equivalent to the Rust panic! operation and is used in its implementation.

Reads the contents of the task’s timer: both the current time, and any configured deadline.

Given a task ID that may have the wrong generation, produces a corrected task ID with the target task’s current generation.

This is intended for two use cases:

Accumulates a set of notification bits into another task’s notification word using bitwise OR. This enables a simple inter-task asynchronous communication mechanism. See Notifications: the other IPC mechanism for more information on the mechanism.

Like REPLY, this resumes a task that is blocked waiting for a reply from the invoking task. Unlike REPLY, this does not set the task runnable, and instead marks it as faulted by a recognizable code.

Returns the current status of interrupts mapped to the calling task.

Reads out status information about a task, by index. This is intended to be used for task management purposes by the supervisor — because tasks can restart whenever, the supervisor generally doesn’t want to concern itself with getting generation numbers right.

Reinitializes a task, chosen by index, and optionally starts it running.

This is valid in any task state, and can be used to interrupt otherwise uninterruptible operations like the SEND syscall.

A successful call to reinit_task has the following effects:

Forces a task into a Faulted state. Specifically, this will set the task’s fault to FaultInfo::Injected(caller), where caller is the TaskId of the task that called fault_task (i.e. you). This means that a fault caused by fault_task is both easily distinguished from any other fault, and traceable.

fault_task immediately prevents the targeted task from running, and any other tasks that were blocked in IPC with the targeted task are interrupted and given a dead code to indicate that the IPC will never complete.

A dump region is an area of memory for a specified task that can be read by the supervisor for purpose of creating a memory dump for debugging. For a specified task and region index, get_task_dump_region will return the details of the dump region, if any. This entry point is only present if the kernel’s dump feature is enabled.

For a given task and task dump region, this will read the specified region and return its contents. The region should be entirely contained by a region that has been returned by a call to get_task_dump_region but is otherwise unconstrained. This entry point is only present if the kernel’s dump feature is enabled.

Scans forward from a given task index searching for a faulted task. If a faulted task is found, returns its index. Otherwise, returns zero, the index of the supervisor task, which is by definition not faulted.

To simplify supervisor implementations, the given task index may equal the number of tasks in the system. In this case, there are no tasks to search, and you always get zero back.

We do not permit greater indices than that, because that would strongly suggest a bug.

Servers normally spend most of their time hanging out in RECV. This ensures that they’re ready to handle incoming messages.

In the simplest case, after doing any initialization required on startup, a server will:

This simple version covers most servers on Hubris, believe it or not. All the complexity, and application-specific logic, is hidden in the "do it" step.

The vast majority of servers need to send messages to other servers to do their jobs. Most servers will turn a single incoming client message into a sequence of messages to other servers to perform useful work.

When designing a collection of servers in an application, remember that it’s only safe to send messages to higher priority servers (called the "uphill send rule"). Sending messages to lower priority servers can cause starvation and deadlock.

Servers are tasks. Tasks are relatively expensive — they require separate code and data storage and stack space. When designing an API consider whether it should be a server task — or just a crate.

You may want a server if any of these are true:

Signs that you may just want a crate:

Here is a full implementation of a server for a very simple IPC protocol: it maintains a 32-bit integer, and can add or subtract values and return the result.

This implementation uses syscalls directly and no abstractions, to show you exactly what’s happening under the hood. In practice, we rarely write servers this way — the next section shows a higher-level equivalent.

This server supports two messages, add (0) and sub (1). Both messages expect a four-byte payload, which is a u32 in little-endian byte order. On success, the messages update the server state the_integer and return the new value, as another four-byte little-endian integer.

The userlib::hl module provides wrappers for common patterns in server implementation. Here’s the same server from the last section, rewritten using the hl conveniences.

It’s polite to provide a wrapper crate that turns your server’s IPC API into a Rust API. We write these by hand at the moment, since we don’t have any sort of IDL. The general pattern is:

The wrapper crate should not depend on the server implementation crate. This may require moving types around.

One of the decisions wrapper crates must make is how to handle server death — that is, what if the server crashes while the client is talking to it, or between messages? There are three common ways to respond.

Here is a wrapper crate for the server presented earlier in this chapter, expressed entirely using low-level Hubris API, under the assumption that we just want to retry on server restart:

Now, notice that the server can generate error codes, such as BadArg if the buffers are the wrong size, but the client doesn’t have any representation for them. This is deliberate. In the case of the integer server protocol, all potential errors returned from IPCs represent programming errors in the client:

In the first two cases the server will return a non-zero response code; in the last case, it will succeed, but the response_len will show that our response was truncated. Either case represents a mismatch between the wrapper crate and the server, and the normal thing to do in such situations on Hubris is to panic!.

The server loop described above handles a single request at a time. Things become more complex if the server wants to be able to handle multiple requests concurrently. In that case, the reply step is delayed until the work actually completes, so the server may RECV another message before replying to the first.

For each incoming request, the server needs to record at least the caller’s Task ID, so that it can respond. In practice, the server will also need to record some details about each request, and some information about the state of processing. While it’s nice to pretend that we can resize buffers forever, that’s simply not the environment we work in. Eventually, the server’s internal storage for this sort of thing will fill up. At this point, the server should finish at least one outstanding request before doing another RECV.

Typically, a pipelined server will keep information about outstanding requests in a table. The maximum size of that table is dictated by the number of potential clients. If the server has specific knowledge of this number in the application, it can use that to size the table — or it be conservative and set the size of the table to hubris_num_tasks::NUM_TASKS, the number of tasks in the system. Such a table should never overflow.

The supervisor is a task like any other. It is compiled with the application, runs in the processor’s unprivileged mode, and is subject to memory protection.

Two things make the supervisor different from other tasks:

The kernel can spot the supervisor because the supervisor always has task index 0, and is listed first in the app.toml. The kernel treats task index 0 differently:

The design of Hubris assumes that the supervisor is responsible for taking action on task crashes. It may also do other things, but, that’s the basics.

When any task crashes, the kernel will post a notification to the supervisor task (as chosen by the supervisor.notification key in the app.toml). Since notifications don’t carry data payloads, this tells the supervisor that something has crashed, but not what or why. The supervisor can use kernel IPC messages to figure out the rest.

Currently, the supervisor needs to scan the set of tasks using the read_task_state kernel IPC until it finds faults. (If the supervisor sometimes lets tasks stay in faulted states, then it will need to keep track of that and look for new faults here.) It can then record that fault information somewhere (maybe a log) and use the reinit_task call to fix the problem.

The basic supervisor main loop reads, then, reads as follows:

(This is almost verbatim from the reference implementation.)

A supervisor may expose an IPC interface that can be used by other tasks to report information. (Because the supervisor is the highest priority task, any task can SEND to it, but it is not allowed to SEND anywhere but the kernel.)

Why would you want to do this? Some examples might include:

If the supervisor wishes to expose an IPC interface, its main loop changes as follows:

Since tasks are relatively expensive in terms of resources (primarily RAM and Flash), it’s important to have the right number of tasks, rather than a separate task for everything (or just one task).

Drivers should not always be servers. Hubris is not religious about this, and it’s useful to have some flexibility here.

We’ve found the following distinction to be useful:

By convention, a driver crate for the encoder peripheral on the xyz SoC is called drv-xyz-encoder, while the crate wrapping it in a server is called drv-xyz-encoder-server.

If, in a given application, there’s only one user for a given driver — say, the board has a SPI controller with only one device wired to it — then it doesn’t necessarily make sense to have a task for the SPI controller. Instead, the task responsible for managing the device could link the SPI driver crate in directly.

There’s also the question of mutual exclusion. On an I2C bus, for instance, we can only talk to one device at any given time — and we may need to issue several transactions to a single device without risk of interruption. This means that a single device driver needs exclusive access to the I2C bus, for a combination of inherent hardware reasons (I2C is not pipelined) and software requirements.

If we allocated a separate server per I2C device, only one of those servers would be doing useful work at any given time — the rest would be waiting their turn.

In this case it might make more sense to assign the task to the bus and have it call into driver crates for each device as needed. This ensures that we’re only spending enough stack space for one device at a time, and helps the device drivers share common code. It also puts the drivers for the devices and the bus controller in the same fault domain, so that a crash in one affects the other — in I2C, a heavily stateful protocol with poor error recovery, this is almost certainly what you want, since a crash in a device during a transaction will likely require global recovery actions on the bus controller.

A typical driver server has to multiplex hardware events and client requests, which requires both configuration in the app.toml and code in the server itself. Here is an example server written against the userlib::hl library. (For more details on userlib::hl and server implementations in general, see the chapter on servers.)

Some details are omitted — this is pseudocode.

A server called drv-xyz-encoder-server should, by convention, provide clients with a corresponding API wrapper crate called drv-xyz-encoder-api. This will normally use the userlib::hl module under the hood to generate IPC.

An example API might look like: