First I thought there was something wrong with my macOS version of the
avutil shared library, as the error was occurring on macOS but not in Linux. After some debugging I discovered that it didn't occur every single time the code was run, indicating that it's some kind of buffer overflow that was occurring instead.
When increasing the size of the output buffer by one, ie.,
malloc(outputBufferSize + 1), the error stopped happening. Then my colleague Lennart discovered that the LZO code in libavutil requires the buffers to be padded by a certain number of bytes. I added the
AV_LZO_OUTPUT_PADDING constants in the header file to the buffer sizes, and so far everything finally seems to work without any problems.