CBOB SPI Protocol

Stock driver

The stock driver’s protocol is contained in cbob_spi.c and chumby.c. All communications consists of a special spi message format, which allows the chumby to send to the cbob a command and optionally some command-specific data, and receive some data in return. Every message consists of 4 spi transactions of 16-bit words, which are performed together in the chumby’s cbob_spi_message function, and separately as an interrupt driven state machine in the cbob’s chumby.c. Here’s a brief description of them:

1. Header/command: chumby sends cbob 3 words: 0xCB07, a command word, and an output count
2. Chumby Data: chumby sends cbob data related to the command. At least one byte must be sent
3. Length: cbob sends chumby a word containing the length of the reply, and a dummy word (it would appear that the hand-written SPI drivers for the cbob can’t transmit a single word)
4. Cbob Data: cbob sends chumby data related to the command. At least one byte must be sent.

Overall, except for the dummy words, this should be perfectly workable and reasonably efficient. The current implementation falls short however.

Poor performance

A full SPI message takes just over 6ms to complete, at1.5ms per transaction. Worse yet, it busy waits the entire time. If you have trouble getting motors to start simultaneously, or reading lots of sensors quickly or otherwise have high CPU usage, this is almost certainly why.

The cbob’s UARTs fortunately aren’t polled, however whenever data comes two SPI messages take place. This means 12ms of cycles burned, not even counting context switches. The create’s stream mode will bring the system to its knees. We poll create sensors at 10hz right now (cbcv1 was actually better than this), and the cbob work queue takes up 40% of our CPU.

De-sync

Attempting to eliminate these busy waits exposes an even bigger issue, de-syncing. Both the chumby and the cbc assume a perfect stream of bytes back and forth to one another. If either misses a single word, the cbob will stop responding to messages correctly, and the chumby crashes due to the corrupted data triggering buffer overflows. With the overflows fixed, a de-sync simply causes all sensor readings to report garbage and motor or servo commands to silently fail. I’m not entirely sure why the cbob gets de-synced, but my theory is that other interrupts are occasionally preventing the cbob’s SPI transfer complete interrupt from being handled before the chumby begins to send or receive the next transaction, causing it to drop data.

The reason the desync is such a big issue is because the chumby has little means to fix it, as the cbob has no reset lines. The cbob does send a brief ack pulse after every complete SPI transaction, but it is sent to a pin on the chumby’s processor that lacks an irq, and thus is nearly impossible to detect even while busywaiting. Even if this pulse could be received, it would only allow desync to be detected, not corrected except by retrying. The system does, however, have a tendency to by chance get itself in sync again, as eventually the chumby will start a header on the byte the cbob expects it to. Still, the busy-waiting is definitely a lesser evil than the chance of a de-sync.

What now?

Unfortunately, I don’t have a JTAG. That means that cbob reprogramming is out of reach for me, as I could potentially brick it. Though the cbob has a bootloader, it is actually a part of the firmware and is overwritten on each download, and thus can’t be used to unbrick a cbob. So for now, I’m going to try to make the chumby end of things do the absolute best it can. Here’s what I’ve fixed so far:

• you can safely call select() on the cbob’s uart ttys
• bounds checking is performed so that a desync doesn’t crash the chumby
• Transactions are performed safely every 1.2 msec (down from 1.5 msec), using a hardware timer interrupt while the main process sleeps.
• If a full transaction delay passes between two messages, the second message skips its initial delay and completes in 3.6 msec.

This solution is pretty much the best that can be done without reprogramming the cbob. Though each transaction still takes 4.8 msec or 3.6 msec, the process properly sleeps during this time, meaning CPU usage is kept to a minimum. Still, the SPI link and the cbob processor are both fast enough that this should me measured in usec, not msec, with a better interrupt driven protocol, and have no risk of desync. If I do get a JTAG, I’d change the protocol slightly to fix these issues within the bounds of the current cbc hardware:

• The cbob would send ack pulses on the chumby’s SS1 signal, which does have interrupt support. All transactions would be performed in this interrupt, with the calling process sleeping for the entire duration.
• The chumby would start/reset communications with the cbob using its “bend” signal. After receiving the start/reset signal, the cbob would send an ack pulse, indicating it is ready to receive the first transaction in the message, and triggering the chumby’s interrupt handler which would then send it. A separate timeout thread on the chumby would detect if a transaction locks up and send a new start signal to reattempt the message from the beginning.
• The cbob would signal data arriving on a uart by holding the ack line high instead of pulsing it. The chumby would detect this as a special case in its interrupt handler, immediately clock out uart data outside of the message format. The cbob would only do this if it was not in the process of receiving an SPI message. If a cbob does raise its ack line, it will ignore any subsequent start pulses until after the uart data is clocked out.

This would allow very low cpu usage on the chumby, faster interrupt-driven transactions, and complete immunity to desyncs.