RecordIO: robust record-based streaming I/O
authorTudor Bosman <tudorb@fb.com>
Thu, 25 Apr 2013 03:29:08 +0000 (20:29 -0700)
committerSara Golemon <sgolemon@fb.com>
Mon, 20 May 2013 18:01:26 +0000 (11:01 -0700)
commit6cee46136f0825424e442831c2d844ac94e23f7b
tree501636965ae190bab363d43259e8bdc082171f31
parentd1028d196481191a6e3adf1b081f736b89c52cfa
RecordIO: robust record-based streaming I/O

Summary:
RecordIO provides an interface to write and read streams of variable-length
records that is resilient in the face of failure and data corruption.  If
the stream is corrupted in any way, you will lose records, but the stream
will resynchronize.

We have one implementation of RecordIO reader/writer that reads from / writes
to regular files (using mmap(), even for writes, so we can preserve state
even in case of process death -- the buffer cache will flush things to disk
eventually) and we expose enough of the guts (in a reasonably clean way)
so you can build your own on top of other backends.

Test Plan: test added

Reviewed By: mmcurtiss@fb.com

FB internal diff: D790275
folly/io/RecordIO-inl.h [new file with mode: 0644]
folly/io/RecordIO.cpp [new file with mode: 0644]
folly/io/RecordIO.h [new file with mode: 0644]
folly/io/test/RecordIOTest.cpp [new file with mode: 0644]