RecordIO: robust record-based streaming I/O
Summary:
RecordIO provides an interface to write and read streams of variable-length
records that is resilient in the face of failure and data corruption. If
the stream is corrupted in any way, you will lose records, but the stream
will resynchronize.
We have one implementation of RecordIO reader/writer that reads from / writes
to regular files (using mmap(), even for writes, so we can preserve state
even in case of process death -- the buffer cache will flush things to disk
eventually) and we expose enough of the guts (in a reasonably clean way)
so you can build your own on top of other backends.
Test Plan: test added
Reviewed By: mmcurtiss@fb.com
FB internal diff:
D790275