R – How to handle large data transfers on very memory constrained, embedded systems


I have a microcontroller that must download a large file from a PC serial port (115200 baud) and write it to serial flash memory over SPI (~2 MHz). The flash writes must be in 256 byte blocks preceded by a write command and page address. The total RAM available on the system is 1 kB with an 80 byte stack size.

This is currently working by filling a 256 byte buffer from the UART and then ping-ponging to a another 256 byte buffer being filled by an interrupt on the RX buffer ready signal while the flash is written to with busy writes. The buffer swapping is repeated until the operation is complete.

I would prefer to setup TX/RX interrupt handlers for both the SPI and UART ports that operate on seperate circular buffers. So, instead of polling for new bytes and waiting for operations to complete I can simply fill the TX buffers and enable the interrupt or check the buffers for incoming data. This would give a lot more clock cycles for real work instead of waiting on peripherals.

After implementing the IRQ's with 128 byte circular buffers, I poll the UART RX buffer for data and immediately place it in the SPI TX buffer to do the file transfer. The problem I am having with this approach is that I don't have sufficient RAM for the buffers and the PC receive buffer is filling up faster than I get the data over to the flash transmit buffer. Obviously, transmission speed is not the problem (115.2 kHz in and 2 MHz out), but there is a write cycle wait after each 256-byte page is transmitted.

It appears the frequent SPI interrupts were blocking some of the UART interrupts and causing bytes to be missed. The solution I chose was to use a ring buffer for the UART receive interrupt and feed the data into a 256 byte page buffer that is sent to the serial flash by polling for byte transfers and write completion. A 128 ring buffer is big enough to prevent overflows during the SPI write.

Best Solution

I'd do something like a scatter gather on a PC. Create a linked list of a struct like this:

typedef struct data_buffer {
    char flags;
    char[128] data;

Have one of the bits in the flag mean "ReadyToFlash" and one for "Flashing". You should be able to tune the number of buffers in your linked list to keep the flash from catching the UART as it writes or vice versa.

If the flash gets to a buffer block that isn't "ReadyToFlash" it would stall and you'd need to have your UART IRQ start it back up. If the UART gets to a block that is "ReadyToFlash" or "Flashing" it is filling too fast and you probably need another buffer, if you have dynamic memory you could do this tuning at runtime and add a buffer to the list on the fly, otherwise you'll just need to do some empirical testing.

Related Question