2 * Copyright 2016 Facebook, Inc.
4 * Licensed under the Apache License, Version 2.0 (the "License");
5 * you may not use this file except in compliance with the License.
6 * You may obtain a copy of the License at
8 * http://www.apache.org/licenses/LICENSE-2.0
10 * Unless required by applicable law or agreed to in writing, software
11 * distributed under the License is distributed on an "AS IS" BASIS,
12 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 * See the License for the specific language governing permissions and
14 * limitations under the License.
19 #include <glog/logging.h>
27 #include <type_traits>
29 #include <boost/iterator/iterator_facade.hpp>
31 #include <folly/FBString.h>
32 #include <folly/Range.h>
33 #include <folly/FBVector.h>
34 #include <folly/portability/SysUio.h>
36 // Ignore shadowing warnings within this file, so includers can use -Wshadow.
37 #pragma GCC diagnostic push
38 #pragma GCC diagnostic ignored "-Wshadow"
43 * An IOBuf is a pointer to a buffer of data.
45 * IOBuf objects are intended to be used primarily for networking code, and are
46 * modelled somewhat after FreeBSD's mbuf data structure, and Linux's sk_buff
49 * IOBuf objects facilitate zero-copy network programming, by allowing multiple
50 * IOBuf objects to point to the same underlying buffer of data, using a
51 * reference count to track when the buffer is no longer needed and can be
58 * The IOBuf itself is a small object containing a pointer to the buffer and
59 * information about which segment of the buffer contains valid data.
61 * The data layout looks like this:
69 * +------------+--------------------+-----------+
70 * | headroom | data | tailroom |
71 * +------------+--------------------+-----------+
73 * buffer() data() tail() bufferEnd()
75 * The length() method returns the length of the valid data; capacity()
76 * returns the entire capacity of the buffer (from buffer() to bufferEnd()).
77 * The headroom() and tailroom() methods return the amount of unused capacity
78 * available before and after the data.
84 * The buffer itself is reference counted, and multiple IOBuf objects may point
85 * to the same buffer. Each IOBuf may point to a different section of valid
86 * data within the underlying buffer. For example, if multiple protocol
87 * requests are read from the network into a single buffer, a separate IOBuf
88 * may be created for each request, all sharing the same underlying buffer.
90 * In other words, when multiple IOBufs share the same underlying buffer, the
91 * data() and tail() methods on each IOBuf may point to a different segment of
92 * the data. However, the buffer() and bufferEnd() methods will point to the
93 * same location for all IOBufs sharing the same underlying buffer.
95 * +-----------+ +---------+
96 * | IOBuf 1 | | IOBuf 2 |
97 * +-----------+ +---------+
99 * data | tail |/ data | tail
101 * +-------------------------------------+
103 * +-------------------------------------+
105 * If you only read data from an IOBuf, you don't need to worry about other
106 * IOBuf objects possibly sharing the same underlying buffer. However, if you
107 * ever write to the buffer you need to first ensure that no other IOBufs point
108 * to the same buffer. The unshare() method may be used to ensure that you
109 * have an unshared buffer.
115 * IOBuf objects also contain pointers to next and previous IOBuf objects.
116 * This can be used to represent a single logical piece of data that its stored
117 * in non-contiguous chunks in separate buffers.
119 * A single IOBuf object can only belong to one chain at a time.
121 * IOBuf chains are always circular. The "prev" pointer in the head of the
122 * chain points to the tail of the chain. However, it is up to the user to
123 * decide which IOBuf is the head. Internally the IOBuf code does not care
124 * which element is the head.
126 * The lifetime of all IOBufs in the chain are linked: when one element in the
127 * chain is deleted, all other chained elements are also deleted. Conceptually
128 * it is simplest to treat this as if the head of the chain owns all other
129 * IOBufs in the chain. When you delete the head of the chain, it will delete
130 * the other elements as well. For this reason, prependChain() and
131 * appendChain() take ownership of of the new elements being added to this
134 * When the coalesce() method is used to coalesce an entire IOBuf chain into a
135 * single IOBuf, all other IOBufs in the chain are eliminated and automatically
136 * deleted. The unshare() method may coalesce the chain; if it does it will
137 * similarly delete all IOBufs eliminated from the chain.
139 * As discussed in the following section, it is up to the user to maintain a
140 * lock around the entire IOBuf chain if multiple threads need to access the
141 * chain. IOBuf does not provide any internal locking.
147 * When used in multithread programs, a single IOBuf object should only be used
148 * in a single thread at a time. If a caller uses a single IOBuf across
149 * multiple threads the caller is responsible for using an external lock to
150 * synchronize access to the IOBuf.
152 * Two separate IOBuf objects may be accessed concurrently in separate threads
153 * without locking, even if they point to the same underlying buffer. The
154 * buffer reference count is always accessed atomically, and no other
155 * operations should affect other IOBufs that point to the same data segment.
156 * The caller is responsible for using unshare() to ensure that the data buffer
157 * is not shared by other IOBufs before writing to it, and this ensures that
158 * the data itself is not modified in one thread while also being accessed from
161 * For IOBuf chains, no two IOBufs in the same chain should be accessed
162 * simultaneously in separate threads. The caller must maintain a lock around
163 * the entire chain if the chain, or individual IOBufs in the chain, may be
164 * accessed by multiple threads.
167 * IOBuf Object Allocation
168 * -----------------------
170 * IOBuf objects themselves exist separately from the data buffer they point
171 * to. Therefore one must also consider how to allocate and manage the IOBuf
174 * It is more common to allocate IOBuf objects on the heap, using the create(),
175 * takeOwnership(), or wrapBuffer() factory functions. The clone()/cloneOne()
176 * functions also return new heap-allocated IOBufs. The createCombined()
177 * function allocates the IOBuf object and data storage space together, in a
178 * single memory allocation. This can improve performance, particularly if you
179 * know that the data buffer and the IOBuf itself will have similar lifetimes.
181 * That said, it is also possible to allocate IOBufs on the stack or inline
182 * inside another object as well. This is useful for cases where the IOBuf is
183 * short-lived, or when the overhead of allocating the IOBuf on the heap is
186 * However, note that stack-allocated IOBufs may only be used as the head of a
187 * chain (or standalone as the only IOBuf in a chain). All non-head members of
188 * an IOBuf chain must be heap allocated. (All functions to add nodes to a
189 * chain require a std::unique_ptr<IOBuf>, which enforces this requrement.)
191 * Copying IOBufs is only meaningful for the head of a chain. The entire chain
192 * is cloned; the IOBufs will become shared, and the old and new IOBufs will
193 * refer to the same underlying memory.
198 * The IOBuf class manages sharing of the underlying buffer that it points to,
199 * maintaining a reference count if multiple IOBufs are pointing at the same
202 * However, it is the callers responsibility to manage sharing and ownership of
203 * IOBuf objects themselves. The IOBuf structure does not provide room for an
204 * intrusive refcount on the IOBuf object itself, only the underlying data
205 * buffer is reference counted. If users want to share the same IOBuf object
206 * between multiple parts of the code, they are responsible for managing this
207 * sharing on their own. (For example, by using a shared_ptr. Alternatively,
208 * users always have the option of using clone() to create a second IOBuf that
209 * points to the same underlying buffer.)
212 // Is T a unique_ptr<> to a standard-layout type?
213 template <class T, class Enable=void> struct IsUniquePtrToSL
214 : public std::false_type { };
215 template <class T, class D>
216 struct IsUniquePtrToSL<
217 std::unique_ptr<T, D>,
218 typename std::enable_if<std::is_standard_layout<T>::value>::type>
219 : public std::true_type { };
220 } // namespace detail
226 enum CreateOp { CREATE };
227 enum WrapBufferOp { WRAP_BUFFER };
228 enum TakeOwnershipOp { TAKE_OWNERSHIP };
229 enum CopyBufferOp { COPY_BUFFER };
231 typedef ByteRange value_type;
232 typedef Iterator iterator;
233 typedef Iterator const_iterator;
235 typedef void (*FreeFunction)(void* buf, void* userData);
238 * Allocate a new IOBuf object with the requested capacity.
240 * Returns a new IOBuf object that must be (eventually) deleted by the
241 * caller. The returned IOBuf may actually have slightly more capacity than
244 * The data pointer will initially point to the start of the newly allocated
245 * buffer, and will have a data length of 0.
247 * Throws std::bad_alloc on error.
249 static std::unique_ptr<IOBuf> create(uint64_t capacity);
250 IOBuf(CreateOp, uint64_t capacity);
253 * Create a new IOBuf, using a single memory allocation to allocate space
254 * for both the IOBuf object and the data storage space.
256 * This saves one memory allocation. However, it can be wasteful if you
257 * later need to grow the buffer using reserve(). If the buffer needs to be
258 * reallocated, the space originally allocated will not be freed() until the
259 * IOBuf object itself is also freed. (It can also be slightly wasteful in
260 * some cases where you clone this IOBuf and then free the original IOBuf.)
262 static std::unique_ptr<IOBuf> createCombined(uint64_t capacity);
265 * Create a new IOBuf, using separate memory allocations for the IOBuf object
266 * for the IOBuf and the data storage space.
268 * This requires two memory allocations, but saves space in the long run
269 * if you know that you will need to reallocate the data buffer later.
271 static std::unique_ptr<IOBuf> createSeparate(uint64_t capacity);
274 * Allocate a new IOBuf chain with the requested total capacity, allocating
275 * no more than maxBufCapacity to each buffer.
277 static std::unique_ptr<IOBuf> createChain(
278 size_t totalCapacity, uint64_t maxBufCapacity);
281 * Create a new IOBuf pointing to an existing data buffer.
283 * The new IOBuffer will assume ownership of the buffer, and free it by
284 * calling the specified FreeFunction when the last IOBuf pointing to this
285 * buffer is destroyed. The function will be called with a pointer to the
286 * buffer as the first argument, and the supplied userData value as the
287 * second argument. The free function must never throw exceptions.
289 * If no FreeFunction is specified, the buffer will be freed using free()
290 * which will result in undefined behavior if the memory was allocated
293 * The IOBuf data pointer will initially point to the start of the buffer,
295 * In the first version of this function, the length of data is unspecified
296 * and is initialized to the capacity of the buffer
298 * In the second version, the user specifies the valid length of data
301 * On error, std::bad_alloc will be thrown. If freeOnError is true (the
302 * default) the buffer will be freed before throwing the error.
304 static std::unique_ptr<IOBuf> takeOwnership(void* buf, uint64_t capacity,
305 FreeFunction freeFn = nullptr,
306 void* userData = nullptr,
307 bool freeOnError = true) {
308 return takeOwnership(buf, capacity, capacity, freeFn,
309 userData, freeOnError);
311 IOBuf(TakeOwnershipOp op, void* buf, uint64_t capacity,
312 FreeFunction freeFn = nullptr, void* userData = nullptr,
313 bool freeOnError = true)
314 : IOBuf(op, buf, capacity, capacity, freeFn, userData, freeOnError) {}
316 static std::unique_ptr<IOBuf> takeOwnership(void* buf, uint64_t capacity,
318 FreeFunction freeFn = nullptr,
319 void* userData = nullptr,
320 bool freeOnError = true);
321 IOBuf(TakeOwnershipOp, void* buf, uint64_t capacity, uint64_t length,
322 FreeFunction freeFn = nullptr, void* userData = nullptr,
323 bool freeOnError = true);
326 * Create a new IOBuf pointing to an existing data buffer made up of
327 * count objects of a given standard-layout type.
329 * This is dangerous -- it is essentially equivalent to doing
330 * reinterpret_cast<unsigned char*> on your data -- but it's often useful
331 * for serialization / deserialization.
333 * The new IOBuffer will assume ownership of the buffer, and free it
334 * appropriately (by calling the UniquePtr's custom deleter, or by calling
335 * delete or delete[] appropriately if there is no custom deleter)
336 * when the buffer is destroyed. The custom deleter, if any, must never
339 * The IOBuf data pointer will initially point to the start of the buffer,
340 * and the length will be the full capacity of the buffer (count *
343 * On error, std::bad_alloc will be thrown, and the buffer will be freed
344 * before throwing the error.
346 template <class UniquePtr>
347 static typename std::enable_if<detail::IsUniquePtrToSL<UniquePtr>::value,
348 std::unique_ptr<IOBuf>>::type
349 takeOwnership(UniquePtr&& buf, size_t count=1);
352 * Create a new IOBuf object that points to an existing user-owned buffer.
354 * This should only be used when the caller knows the lifetime of the IOBuf
355 * object ahead of time and can ensure that all IOBuf objects that will point
356 * to this buffer will be destroyed before the buffer itself is destroyed.
358 * This buffer will not be freed automatically when the last IOBuf
359 * referencing it is destroyed. It is the caller's responsibility to free
360 * the buffer after the last IOBuf has been destroyed.
362 * The IOBuf data pointer will initially point to the start of the buffer,
363 * and the length will be the full capacity of the buffer.
365 * An IOBuf created using wrapBuffer() will always be reported as shared.
366 * unshare() may be used to create a writable copy of the buffer.
368 * On error, std::bad_alloc will be thrown.
370 static std::unique_ptr<IOBuf> wrapBuffer(const void* buf, uint64_t capacity);
371 static std::unique_ptr<IOBuf> wrapBuffer(ByteRange br) {
372 return wrapBuffer(br.data(), br.size());
376 * Similar to wrapBuffer(), but returns IOBuf by value rather than
377 * heap-allocating it.
379 static IOBuf wrapBufferAsValue(const void* buf, uint64_t capacity);
380 static IOBuf wrapBufferAsValue(ByteRange br) {
381 return wrapBufferAsValue(br.data(), br.size());
384 IOBuf(WrapBufferOp op, const void* buf, uint64_t capacity);
385 IOBuf(WrapBufferOp op, ByteRange br);
388 * Convenience function to create a new IOBuf object that copies data from a
389 * user-supplied buffer, optionally allocating a given amount of
390 * headroom and tailroom.
392 static std::unique_ptr<IOBuf> copyBuffer(const void* buf, uint64_t size,
394 uint64_t minTailroom=0);
395 static std::unique_ptr<IOBuf> copyBuffer(ByteRange br,
397 uint64_t minTailroom=0) {
398 return copyBuffer(br.data(), br.size(), headroom, minTailroom);
400 IOBuf(CopyBufferOp op, const void* buf, uint64_t size,
401 uint64_t headroom=0, uint64_t minTailroom=0);
402 IOBuf(CopyBufferOp op, ByteRange br,
403 uint64_t headroom=0, uint64_t minTailroom=0);
406 * Convenience function to create a new IOBuf object that copies data from a
407 * user-supplied string, optionally allocating a given amount of
408 * headroom and tailroom.
410 * Beware when attempting to invoke this function with a constant string
411 * literal and a headroom argument: you will likely end up invoking the
412 * version of copyBuffer() above. IOBuf::copyBuffer("hello", 3) will treat
413 * the first argument as a const void*, and will invoke the version of
414 * copyBuffer() above, with the size argument of 3.
416 static std::unique_ptr<IOBuf> copyBuffer(const std::string& buf,
418 uint64_t minTailroom=0);
419 IOBuf(CopyBufferOp op, const std::string& buf,
420 uint64_t headroom=0, uint64_t minTailroom=0)
421 : IOBuf(op, buf.data(), buf.size(), headroom, minTailroom) {}
424 * A version of copyBuffer() that returns a null pointer if the input string
427 static std::unique_ptr<IOBuf> maybeCopyBuffer(const std::string& buf,
429 uint64_t minTailroom=0);
432 * Convenience function to free a chain of IOBufs held by a unique_ptr.
434 static void destroy(std::unique_ptr<IOBuf>&& data) {
435 auto destroyer = std::move(data);
439 * Destroy this IOBuf.
441 * Deleting an IOBuf will automatically destroy all IOBufs in the chain.
442 * (See the comments above regarding the ownership model of IOBuf chains.
443 * All subsequent IOBufs in the chain are considered to be owned by the head
444 * of the chain. Users should only explicitly delete the head of a chain.)
446 * When each individual IOBuf is destroyed, it will release its reference
447 * count on the underlying buffer. If it was the last user of the buffer,
448 * the buffer will be freed.
453 * Check whether the chain is empty (i.e., whether the IOBufs in the
454 * chain have a total data length of zero).
456 * This method is semantically equivalent to
457 * i->computeChainDataLength()==0
458 * but may run faster because it can short-circuit as soon as it
459 * encounters a buffer with length()!=0
464 * Get the pointer to the start of the data.
466 const uint8_t* data() const {
471 * Get a writable pointer to the start of the data.
473 * The caller is responsible for calling unshare() first to ensure that it is
474 * actually safe to write to the buffer.
476 uint8_t* writableData() {
481 * Get the pointer to the end of the data.
483 const uint8_t* tail() const {
484 return data_ + length_;
488 * Get a writable pointer to the end of the data.
490 * The caller is responsible for calling unshare() first to ensure that it is
491 * actually safe to write to the buffer.
493 uint8_t* writableTail() {
494 return data_ + length_;
498 * Get the data length.
500 uint64_t length() const {
505 * Get the amount of head room.
507 * Returns the number of bytes in the buffer before the start of the data.
509 uint64_t headroom() const {
510 return data_ - buffer();
514 * Get the amount of tail room.
516 * Returns the number of bytes in the buffer after the end of the data.
518 uint64_t tailroom() const {
519 return bufferEnd() - tail();
523 * Get the pointer to the start of the buffer.
525 * Note that this is the pointer to the very beginning of the usable buffer,
526 * not the start of valid data within the buffer. Use the data() method to
527 * get a pointer to the start of the data within the buffer.
529 const uint8_t* buffer() const {
534 * Get a writable pointer to the start of the buffer.
536 * The caller is responsible for calling unshare() first to ensure that it is
537 * actually safe to write to the buffer.
539 uint8_t* writableBuffer() {
544 * Get the pointer to the end of the buffer.
546 * Note that this is the pointer to the very end of the usable buffer,
547 * not the end of valid data within the buffer. Use the tail() method to
548 * get a pointer to the end of the data within the buffer.
550 const uint8_t* bufferEnd() const {
551 return buf_ + capacity_;
555 * Get the total size of the buffer.
557 * This returns the total usable length of the buffer. Use the length()
558 * method to get the length of the actual valid data in this IOBuf.
560 uint64_t capacity() const {
565 * Get a pointer to the next IOBuf in this chain.
570 const IOBuf* next() const {
575 * Get a pointer to the previous IOBuf in this chain.
580 const IOBuf* prev() const {
585 * Shift the data forwards in the buffer.
587 * This shifts the data pointer forwards in the buffer to increase the
588 * headroom. This is commonly used to increase the headroom in a newly
591 * The caller is responsible for ensuring that there is sufficient
592 * tailroom in the buffer before calling advance().
594 * If there is a non-zero data length, advance() will use memmove() to shift
595 * the data forwards in the buffer. In this case, the caller is responsible
596 * for making sure the buffer is unshared, so it will not affect other IOBufs
597 * that may be sharing the same underlying buffer.
599 void advance(uint64_t amount) {
600 // In debug builds, assert if there is a problem.
601 assert(amount <= tailroom());
604 memmove(data_ + amount, data_, length_);
610 * Shift the data backwards in the buffer.
612 * The caller is responsible for ensuring that there is sufficient headroom
613 * in the buffer before calling retreat().
615 * If there is a non-zero data length, retreat() will use memmove() to shift
616 * the data backwards in the buffer. In this case, the caller is responsible
617 * for making sure the buffer is unshared, so it will not affect other IOBufs
618 * that may be sharing the same underlying buffer.
620 void retreat(uint64_t amount) {
621 // In debug builds, assert if there is a problem.
622 assert(amount <= headroom());
625 memmove(data_ - amount, data_, length_);
631 * Adjust the data pointer to include more valid data at the beginning.
633 * This moves the data pointer backwards to include more of the available
634 * buffer. The caller is responsible for ensuring that there is sufficient
635 * headroom for the new data. The caller is also responsible for populating
636 * this section with valid data.
638 * This does not modify any actual data in the buffer.
640 void prepend(uint64_t amount) {
641 DCHECK_LE(amount, headroom());
647 * Adjust the tail pointer to include more valid data at the end.
649 * This moves the tail pointer forwards to include more of the available
650 * buffer. The caller is responsible for ensuring that there is sufficient
651 * tailroom for the new data. The caller is also responsible for populating
652 * this section with valid data.
654 * This does not modify any actual data in the buffer.
656 void append(uint64_t amount) {
657 DCHECK_LE(amount, tailroom());
662 * Adjust the data pointer forwards to include less valid data.
664 * This moves the data pointer forwards so that the first amount bytes are no
665 * longer considered valid data. The caller is responsible for ensuring that
666 * amount is less than or equal to the actual data length.
668 * This does not modify any actual data in the buffer.
670 void trimStart(uint64_t amount) {
671 DCHECK_LE(amount, length_);
677 * Adjust the tail pointer backwards to include less valid data.
679 * This moves the tail pointer backwards so that the last amount bytes are no
680 * longer considered valid data. The caller is responsible for ensuring that
681 * amount is less than or equal to the actual data length.
683 * This does not modify any actual data in the buffer.
685 void trimEnd(uint64_t amount) {
686 DCHECK_LE(amount, length_);
693 * Postcondition: headroom() == 0, length() == 0, tailroom() == capacity()
696 data_ = writableBuffer();
701 * Ensure that this buffer has at least minHeadroom headroom bytes and at
702 * least minTailroom tailroom bytes. The buffer must be writable
703 * (you must call unshare() before this, if necessary).
705 * Postcondition: headroom() >= minHeadroom, tailroom() >= minTailroom,
706 * the data (between data() and data() + length()) is preserved.
708 void reserve(uint64_t minHeadroom, uint64_t minTailroom) {
709 // Maybe we don't need to do anything.
710 if (headroom() >= minHeadroom && tailroom() >= minTailroom) {
713 // If the buffer is empty but we have enough total room (head + tail),
714 // move the data_ pointer around.
716 headroom() + tailroom() >= minHeadroom + minTailroom) {
717 data_ = writableBuffer() + minHeadroom;
720 // Bah, we have to do actual work.
721 reserveSlow(minHeadroom, minTailroom);
725 * Return true if this IOBuf is part of a chain of multiple IOBufs, or false
726 * if this is the only IOBuf in its chain.
728 bool isChained() const {
729 assert((next_ == this) == (prev_ == this));
730 return next_ != this;
734 * Get the number of IOBufs in this chain.
736 * Beware that this method has to walk the entire chain.
737 * Use isChained() if you just want to check if this IOBuf is part of a chain
740 size_t countChainElements() const;
743 * Get the length of all the data in this IOBuf chain.
745 * Beware that this method has to walk the entire chain.
747 uint64_t computeChainDataLength() const;
750 * Insert another IOBuf chain immediately before this IOBuf.
752 * For example, if there are two IOBuf chains (A, B, C) and (D, E, F),
753 * and B->prependChain(D) is called, the (D, E, F) chain will be subsumed
754 * and become part of the chain starting at A, which will now look like
757 * Note that since IOBuf chains are circular, head->prependChain(other) can
758 * be used to append the other chain at the very end of the chain pointed to
759 * by head. For example, if there are two IOBuf chains (A, B, C) and
760 * (D, E, F), and A->prependChain(D) is called, the chain starting at A will
761 * now consist of (A, B, C, D, E, F)
763 * The elements in the specified IOBuf chain will become part of this chain,
764 * and will be owned by the head of this chain. When this chain is
765 * destroyed, all elements in the supplied chain will also be destroyed.
767 * For this reason, appendChain() only accepts an rvalue-reference to a
768 * unique_ptr(), to make it clear that it is taking ownership of the supplied
769 * chain. If you have a raw pointer, you can pass in a new temporary
770 * unique_ptr around the raw pointer. If you have an existing,
771 * non-temporary unique_ptr, you must call std::move(ptr) to make it clear
772 * that you are destroying the original pointer.
774 void prependChain(std::unique_ptr<IOBuf>&& iobuf);
777 * Append another IOBuf chain immediately after this IOBuf.
779 * For example, if there are two IOBuf chains (A, B, C) and (D, E, F),
780 * and B->appendChain(D) is called, the (D, E, F) chain will be subsumed
781 * and become part of the chain starting at A, which will now look like
784 * The elements in the specified IOBuf chain will become part of this chain,
785 * and will be owned by the head of this chain. When this chain is
786 * destroyed, all elements in the supplied chain will also be destroyed.
788 * For this reason, appendChain() only accepts an rvalue-reference to a
789 * unique_ptr(), to make it clear that it is taking ownership of the supplied
790 * chain. If you have a raw pointer, you can pass in a new temporary
791 * unique_ptr around the raw pointer. If you have an existing,
792 * non-temporary unique_ptr, you must call std::move(ptr) to make it clear
793 * that you are destroying the original pointer.
795 void appendChain(std::unique_ptr<IOBuf>&& iobuf) {
796 // Just use prependChain() on the next element in our chain
797 next_->prependChain(std::move(iobuf));
801 * Remove this IOBuf from its current chain.
803 * Since ownership of all elements an IOBuf chain is normally maintained by
804 * the head of the chain, unlink() transfers ownership of this IOBuf from the
805 * chain and gives it to the caller. A new unique_ptr to the IOBuf is
806 * returned to the caller. The caller must store the returned unique_ptr (or
807 * call release() on it) to take ownership, otherwise the IOBuf will be
808 * immediately destroyed.
810 * Since unlink transfers ownership of the IOBuf to the caller, be careful
811 * not to call unlink() on the head of a chain if you already maintain
812 * ownership on the head of the chain via other means. The pop() method
813 * is a better choice for that situation.
815 std::unique_ptr<IOBuf> unlink() {
816 next_->prev_ = prev_;
817 prev_->next_ = next_;
820 return std::unique_ptr<IOBuf>(this);
824 * Remove this IOBuf from its current chain and return a unique_ptr to
825 * the IOBuf that formerly followed it in the chain.
827 std::unique_ptr<IOBuf> pop() {
829 next_->prev_ = prev_;
830 prev_->next_ = next_;
833 return std::unique_ptr<IOBuf>((next == this) ? nullptr : next);
837 * Remove a subchain from this chain.
839 * Remove the subchain starting at head and ending at tail from this chain.
841 * Returns a unique_ptr pointing to head. (In other words, ownership of the
842 * head of the subchain is transferred to the caller.) If the caller ignores
843 * the return value and lets the unique_ptr be destroyed, the subchain will
844 * be immediately destroyed.
846 * The subchain referenced by the specified head and tail must be part of the
847 * same chain as the current IOBuf, but must not contain the current IOBuf.
848 * However, the specified head and tail may be equal to each other (i.e.,
849 * they may be a subchain of length 1).
851 std::unique_ptr<IOBuf> separateChain(IOBuf* head, IOBuf* tail) {
852 assert(head != this);
853 assert(tail != this);
855 head->prev_->next_ = tail->next_;
856 tail->next_->prev_ = head->prev_;
861 return std::unique_ptr<IOBuf>(head);
865 * Return true if at least one of the IOBufs in this chain are shared,
866 * or false if all of the IOBufs point to unique buffers.
868 * Use isSharedOne() to only check this IOBuf rather than the entire chain.
870 bool isShared() const {
871 const IOBuf* current = this;
873 if (current->isSharedOne()) {
876 current = current->next_;
877 if (current == this) {
884 * Return true if all IOBufs in this chain are managed by the usual
885 * refcounting mechanism (and so the lifetime of the underlying memory
886 * can be extended by clone()).
888 bool isManaged() const {
889 const IOBuf* current = this;
891 if (!current->isManagedOne()) {
894 current = current->next_;
895 if (current == this) {
902 * Return true if this IOBuf is managed by the usual refcounting mechanism
903 * (and so the lifetime of the underlying memory can be extended by
906 bool isManagedOne() const {
911 * Return true if other IOBufs are also pointing to the buffer used by this
912 * IOBuf, and false otherwise.
914 * If this IOBuf points at a buffer owned by another (non-IOBuf) part of the
915 * code (i.e., if the IOBuf was created using wrapBuffer(), or was cloned
916 * from such an IOBuf), it is always considered shared.
918 * This only checks the current IOBuf, and not other IOBufs in the chain.
920 bool isSharedOne() const {
921 // If this is a user-owned buffer, it is always considered shared
922 if (UNLIKELY(!sharedInfo())) {
926 if (UNLIKELY(sharedInfo()->externallyShared)) {
930 if (LIKELY(!(flags() & kFlagMaybeShared))) {
934 // kFlagMaybeShared is set, so we need to check the reference count.
935 // (Checking the reference count requires an atomic operation, which is why
936 // we prefer to only check kFlagMaybeShared if possible.)
937 bool shared = sharedInfo()->refcount.load(std::memory_order_acquire) > 1;
939 // we're the last one left
940 clearFlags(kFlagMaybeShared);
946 * Ensure that this IOBuf has a unique buffer that is not shared by other
949 * unshare() operates on an entire chain of IOBuf objects. If the chain is
950 * shared, it may also coalesce the chain when making it unique. If the
951 * chain is coalesced, subsequent IOBuf objects in the current chain will be
952 * automatically deleted.
954 * Note that buffers owned by other (non-IOBuf) users are automatically
957 * Throws std::bad_alloc on error. On error the IOBuf chain will be
960 * Currently unshare may also throw std::overflow_error if it tries to
961 * coalesce. (TODO: In the future it would be nice if unshare() were smart
962 * enough not to coalesce the entire buffer if the data is too large.
963 * However, in practice this seems unlikely to become an issue.)
974 * Ensure that this IOBuf has a unique buffer that is not shared by other
977 * unshareOne() operates on a single IOBuf object. This IOBuf will have a
978 * unique buffer after unshareOne() returns, but other IOBufs in the chain
979 * may still be shared after unshareOne() returns.
981 * Throws std::bad_alloc on error. On error the IOBuf will be unmodified.
990 * Mark the underlying buffers in this chain as shared with external memory
991 * management mechanism. This will make isShared() always returns true.
993 * This function is not thread-safe, and only safe to call immediately after
994 * creating an IOBuf, before it has been shared with other threads.
996 void markExternallyShared();
999 * Mark the underlying buffer that this IOBuf refers to as shared with
1000 * external memory management mechanism. This will make isSharedOne() always
1003 * This function is not thread-safe, and only safe to call immediately after
1004 * creating an IOBuf, before it has been shared with other threads.
1006 void markExternallySharedOne() {
1007 SharedInfo* info = sharedInfo();
1009 info->externallyShared = true;
1014 * Ensure that the memory that IOBufs in this chain refer to will continue to
1015 * be allocated for as long as the IOBufs of the chain (or any clone()s
1016 * created from this point onwards) is alive.
1018 * This only has an effect for user-owned buffers (created with the
1019 * WRAP_BUFFER constructor or wrapBuffer factory function), in which case
1020 * those buffers are unshared.
1022 void makeManaged() {
1024 makeManagedChained();
1031 * Ensure that the memory that this IOBuf refers to will continue to be
1032 * allocated for as long as this IOBuf (or any clone()s created from this
1033 * point onwards) is alive.
1035 * This only has an effect for user-owned buffers (created with the
1036 * WRAP_BUFFER constructor or wrapBuffer factory function), in which case
1037 * those buffers are unshared.
1039 void makeManagedOne() {
1040 if (!isManagedOne()) {
1041 // We can call the internal function directly; unmanaged implies shared.
1047 * Coalesce this IOBuf chain into a single buffer.
1049 * This method moves all of the data in this IOBuf chain into a single
1050 * contiguous buffer, if it is not already in one buffer. After coalesce()
1051 * returns, this IOBuf will be a chain of length one. Other IOBufs in the
1052 * chain will be automatically deleted.
1054 * After coalescing, the IOBuf will have at least as much headroom as the
1055 * first IOBuf in the chain, and at least as much tailroom as the last IOBuf
1058 * Throws std::bad_alloc on error. On error the IOBuf chain will be
1061 * Returns ByteRange that points to the data IOBuf stores.
1063 ByteRange coalesce() {
1067 return ByteRange(data_, length_);
1071 * Ensure that this chain has at least maxLength bytes available as a
1072 * contiguous memory range.
1074 * This method coalesces whole buffers in the chain into this buffer as
1075 * necessary until this buffer's length() is at least maxLength.
1077 * After coalescing, the IOBuf will have at least as much headroom as the
1078 * first IOBuf in the chain, and at least as much tailroom as the last IOBuf
1079 * that was coalesced.
1081 * Throws std::bad_alloc or std::overflow_error on error. On error the IOBuf
1082 * chain will be unmodified. Throws std::overflow_error if maxLength is
1083 * longer than the total chain length.
1085 * Upon return, either enough of the chain was coalesced into a contiguous
1086 * region, or the entire chain was coalesced. That is,
1087 * length() >= maxLength || !isChained() is true.
1089 void gather(uint64_t maxLength) {
1090 if (!isChained() || length_ >= maxLength) {
1093 coalesceSlow(maxLength);
1097 * Return a new IOBuf chain sharing the same data as this chain.
1099 * The new IOBuf chain will normally point to the same underlying data
1100 * buffers as the original chain. (The one exception to this is if some of
1101 * the IOBufs in this chain contain small internal data buffers which cannot
1104 std::unique_ptr<IOBuf> clone() const;
1107 * Similar to clone(). But returns IOBuf by value rather than heap-allocating
1110 IOBuf cloneAsValue() const;
1113 * Return a new IOBuf with the same data as this IOBuf.
1115 * The new IOBuf returned will not be part of a chain (even if this IOBuf is
1116 * part of a larger chain).
1118 std::unique_ptr<IOBuf> cloneOne() const;
1121 * Similar to cloneOne(). But returns IOBuf by value rather than
1122 * heap-allocating it.
1124 IOBuf cloneOneAsValue() const;
1127 * Similar to Clone(). But use other as the head node. Other nodes in the
1128 * chain (if any) will be allocted on heap.
1130 void cloneInto(IOBuf& other) const {
1131 other = cloneAsValue();
1135 * Similar to CloneOne(). But to fill an existing IOBuf instead of a new
1138 void cloneOneInto(IOBuf& other) const {
1139 other = cloneOneAsValue();
1143 * Return an iovector suitable for e.g. writev()
1145 * auto iov = buf->getIov();
1146 * auto xfer = writev(fd, iov.data(), iov.size());
1148 * Naturally, the returned iovector is invalid if you modify the buffer
1151 folly::fbvector<struct iovec> getIov() const;
1154 * Update an existing iovec array with the IOBuf data.
1156 * New iovecs will be appended to the existing vector; anything already
1157 * present in the vector will be left unchanged.
1159 * Naturally, the returned iovec data will be invalid if you modify the
1162 void appendToIov(folly::fbvector<struct iovec>* iov) const;
1165 * Fill an iovec array with the IOBuf data.
1167 * Returns the number of iovec filled. If there are more buffer than
1168 * iovec, returns 0. This version is suitable to use with stack iovec
1171 * Naturally, the filled iovec data will be invalid if you modify the
1174 size_t fillIov(struct iovec* iov, size_t len) const;
1177 * Overridden operator new and delete.
1178 * These perform specialized memory management to help support
1179 * createCombined(), which allocates IOBuf objects together with the buffer
1182 void* operator new(size_t size);
1183 void* operator new(size_t size, void* ptr);
1184 void operator delete(void* ptr);
1187 * Destructively convert this IOBuf to a fbstring efficiently.
1188 * We rely on fbstring's AcquireMallocatedString constructor to
1191 fbstring moveToFbString();
1194 * Iteration support: a chain of IOBufs may be iterated through using
1195 * STL-style iterators over const ByteRanges. Iterators are only invalidated
1196 * if the IOBuf that they currently point to is removed.
1198 Iterator cbegin() const;
1199 Iterator cend() const;
1200 Iterator begin() const;
1201 Iterator end() const;
1204 * Allocate a new null buffer.
1206 * This can be used to allocate an empty IOBuf on the stack. It will have no
1207 * space allocated for it. This is generally useful only to later use move
1208 * assignment to fill out the IOBuf.
1213 * Move constructor and assignment operator.
1215 * In general, you should only ever move the head of an IOBuf chain.
1216 * Internal nodes in an IOBuf chain are owned by the head of the chain, and
1217 * should not be moved from. (Technically, nothing prevents you from moving
1218 * a non-head node, but the moved-to node will replace the moved-from node in
1219 * the chain. This has implications for ownership, since non-head nodes are
1220 * owned by the chain head. You are then responsible for relinquishing
1221 * ownership of the moved-to node, and manually deleting the moved-from
1224 * With the move assignment operator, the destination of the move should be
1225 * the head of an IOBuf chain or a solitary IOBuf not part of a chain. If
1226 * the move destination is part of a chain, all other IOBufs in the chain
1229 IOBuf(IOBuf&& other) noexcept;
1230 IOBuf& operator=(IOBuf&& other) noexcept;
1232 IOBuf(const IOBuf& other);
1233 IOBuf& operator=(const IOBuf& other);
1236 enum FlagsEnum : uintptr_t {
1237 // Adding any more flags would not work on 32-bit architectures,
1238 // as these flags are stashed in the least significant 2 bits of a
1239 // max-align-aligned pointer.
1240 kFlagFreeSharedInfo = 0x1,
1241 kFlagMaybeShared = 0x2,
1242 kFlagMask = kFlagFreeSharedInfo | kFlagMaybeShared
1247 SharedInfo(FreeFunction fn, void* arg);
1249 // A pointer to a function to call to free the buffer when the refcount
1250 // hits 0. If this is null, free() will be used instead.
1251 FreeFunction freeFn;
1253 std::atomic<uint32_t> refcount;
1254 bool externallyShared{false};
1256 // Helper structs for use by operator new and delete
1259 struct HeapFullStorage;
1262 * Create a new IOBuf pointing to an external buffer.
1264 * The caller is responsible for holding a reference count for this new
1265 * IOBuf. The IOBuf constructor does not automatically increment the
1268 struct InternalConstructor {}; // avoid conflicts
1269 IOBuf(InternalConstructor, uintptr_t flagsAndSharedInfo,
1270 uint8_t* buf, uint64_t capacity,
1271 uint8_t* data, uint64_t length);
1273 void unshareOneSlow();
1274 void unshareChained();
1275 void makeManagedChained();
1276 void coalesceSlow();
1277 void coalesceSlow(size_t maxLength);
1278 // newLength must be the entire length of the buffers between this and
1279 // end (no truncation)
1280 void coalesceAndReallocate(
1284 size_t newTailroom);
1285 void coalesceAndReallocate(size_t newLength, IOBuf* end) {
1286 coalesceAndReallocate(headroom(), newLength, end, end->prev_->tailroom());
1288 void decrementRefcount();
1289 void reserveSlow(uint64_t minHeadroom, uint64_t minTailroom);
1290 void freeExtBuffer();
1292 static size_t goodExtBufferSize(uint64_t minCapacity);
1293 static void initExtBuffer(uint8_t* buf, size_t mallocSize,
1294 SharedInfo** infoReturn,
1295 uint64_t* capacityReturn);
1296 static void allocExtBuffer(uint64_t minCapacity,
1297 uint8_t** bufReturn,
1298 SharedInfo** infoReturn,
1299 uint64_t* capacityReturn);
1300 static void releaseStorage(HeapStorage* storage, uint16_t freeFlags);
1301 static void freeInternalBuf(void* buf, void* userData);
1308 * Links to the next and the previous IOBuf in this chain.
1310 * The chain is circularly linked (the last element in the chain points back
1311 * at the head), and next_ and prev_ can never be null. If this IOBuf is the
1312 * only element in the chain, next_ and prev_ will both point to this.
1318 * A pointer to the start of the data referenced by this IOBuf, and the
1319 * length of the data.
1321 * This may refer to any subsection of the actual buffer capacity.
1323 uint8_t* data_{nullptr};
1324 uint8_t* buf_{nullptr};
1325 uint64_t length_{0};
1326 uint64_t capacity_{0};
1328 // Pack flags in least significant 2 bits, sharedInfo in the rest
1329 mutable uintptr_t flagsAndSharedInfo_{0};
1331 static inline uintptr_t packFlagsAndSharedInfo(uintptr_t flags,
1333 uintptr_t uinfo = reinterpret_cast<uintptr_t>(info);
1334 DCHECK_EQ(flags & ~kFlagMask, 0);
1335 DCHECK_EQ(uinfo & kFlagMask, 0);
1336 return flags | uinfo;
1339 inline SharedInfo* sharedInfo() const {
1340 return reinterpret_cast<SharedInfo*>(flagsAndSharedInfo_ & ~kFlagMask);
1343 inline void setSharedInfo(SharedInfo* info) {
1344 uintptr_t uinfo = reinterpret_cast<uintptr_t>(info);
1345 DCHECK_EQ(uinfo & kFlagMask, 0);
1346 flagsAndSharedInfo_ = (flagsAndSharedInfo_ & kFlagMask) | uinfo;
1349 inline uintptr_t flags() const {
1350 return flagsAndSharedInfo_ & kFlagMask;
1353 // flags_ are changed from const methods
1354 inline void setFlags(uintptr_t flags) const {
1355 DCHECK_EQ(flags & ~kFlagMask, 0);
1356 flagsAndSharedInfo_ |= flags;
1359 inline void clearFlags(uintptr_t flags) const {
1360 DCHECK_EQ(flags & ~kFlagMask, 0);
1361 flagsAndSharedInfo_ &= ~flags;
1364 inline void setFlagsAndSharedInfo(uintptr_t flags, SharedInfo* info) {
1365 flagsAndSharedInfo_ = packFlagsAndSharedInfo(flags, info);
1368 struct DeleterBase {
1369 virtual ~DeleterBase() { }
1370 virtual void dispose(void* p) = 0;
1373 template <class UniquePtr>
1374 struct UniquePtrDeleter : public DeleterBase {
1375 typedef typename UniquePtr::pointer Pointer;
1376 typedef typename UniquePtr::deleter_type Deleter;
1378 explicit UniquePtrDeleter(Deleter deleter) : deleter_(std::move(deleter)){ }
1379 void dispose(void* p) {
1381 deleter_(static_cast<Pointer>(p));
1392 static void freeUniquePtrBuffer(void* ptr, void* userData) {
1393 static_cast<DeleterBase*>(userData)->dispose(ptr);
1398 * Hasher for IOBuf objects. Hashes the entire chain using SpookyHashV2.
1401 size_t operator()(const IOBuf& buf) const;
1402 size_t operator()(const std::unique_ptr<IOBuf>& buf) const {
1403 return buf ? (*this)(*buf) : 0;
1408 * Equality predicate for IOBuf objects. Compares data in the entire chain.
1411 bool operator()(const IOBuf& a, const IOBuf& b) const;
1412 bool operator()(const std::unique_ptr<IOBuf>& a,
1413 const std::unique_ptr<IOBuf>& b) const {
1416 } else if (!a || !b) {
1419 return (*this)(*a, *b);
1424 template <class UniquePtr>
1425 typename std::enable_if<detail::IsUniquePtrToSL<UniquePtr>::value,
1426 std::unique_ptr<IOBuf>>::type
1427 IOBuf::takeOwnership(UniquePtr&& buf, size_t count) {
1428 size_t size = count * sizeof(typename UniquePtr::element_type);
1429 auto deleter = new UniquePtrDeleter<UniquePtr>(buf.get_deleter());
1430 return takeOwnership(buf.release(),
1432 &IOBuf::freeUniquePtrBuffer,
1436 inline std::unique_ptr<IOBuf> IOBuf::copyBuffer(
1437 const void* data, uint64_t size, uint64_t headroom,
1438 uint64_t minTailroom) {
1439 uint64_t capacity = headroom + size + minTailroom;
1440 std::unique_ptr<IOBuf> buf = create(capacity);
1441 buf->advance(headroom);
1442 memcpy(buf->writableData(), data, size);
1447 inline std::unique_ptr<IOBuf> IOBuf::copyBuffer(const std::string& buf,
1449 uint64_t minTailroom) {
1450 return copyBuffer(buf.data(), buf.size(), headroom, minTailroom);
1453 inline std::unique_ptr<IOBuf> IOBuf::maybeCopyBuffer(const std::string& buf,
1455 uint64_t minTailroom) {
1459 return copyBuffer(buf.data(), buf.size(), headroom, minTailroom);
1462 class IOBuf::Iterator : public boost::iterator_facade<
1463 IOBuf::Iterator, // Derived
1464 const ByteRange, // Value
1465 boost::forward_traversal_tag // Category or traversal
1467 friend class boost::iterator_core_access;
1469 // Note that IOBufs are stored as a circular list without a guard node,
1470 // so pos == end is ambiguous (it may mean "begin" or "end"). To solve
1471 // the ambiguity (at the cost of one extra comparison in the "increment"
1472 // code path), we define end iterators as having pos_ == end_ == nullptr
1473 // and we only allow forward iteration.
1474 explicit Iterator(const IOBuf* pos, const IOBuf* end)
1477 // Sadly, we must return by const reference, not by value.
1485 val_ = ByteRange(pos_->data(), pos_->tail());
1488 void adjustForEnd() {
1490 pos_ = end_ = nullptr;
1497 const ByteRange& dereference() const {
1501 bool equal(const Iterator& other) const {
1502 // We must compare end_ in addition to pos_, because forward traversal
1503 // requires that if two iterators are equal (a == b) and dereferenceable,
1505 return pos_ == other.pos_ && end_ == other.end_;
1509 pos_ = pos_->next();
1518 inline IOBuf::Iterator IOBuf::begin() const { return cbegin(); }
1519 inline IOBuf::Iterator IOBuf::end() const { return cend(); }
1523 #pragma GCC diagnostic pop