folly.git
6 years agoFix crashing on corrupted ELF binaries with invalid offsets in ELF header.
Andrii Nakryiko [Thu, 21 Dec 2017 20:47:30 +0000 (12:47 -0800)]
Fix crashing on corrupted ELF binaries with invalid offsets in ELF header.

Summary:
There are cases where ELF binaries are running fine, but have slightly
corrupted ELF headers, e.g., with section headers offset pointing beyond boundaries
of file. I'm guessing this is due to running strip or objdump with either some
particular combination of flags or some due to buggy version of those tools. This
change prevents from crashing on such files.

Reviewed By: yfeldblum

Differential Revision: D6616027

fbshipit-source-id: 8cb3ac4475a51d1f1045c395977a6a77fbefffb2

6 years agotest that the value remains alive even if the .then callback takes no arguments
Chad Austin [Wed, 20 Dec 2017 21:37:15 +0000 (13:37 -0800)]
test that the value remains alive even if the .then callback takes no arguments

Summary:
It was not clear to me, if a callback takes no arguments, the
underlying value is guaranteed to be alive during the execution of the
callback, so I wrote these tests.

Reviewed By: yfeldblum

Differential Revision: D6594921

fbshipit-source-id: 6a658afc1bf4d29eaa9c62269ddc21c7f897ad01

6 years agoKill FOLLY_ALIGNED etc
Yedidya Feldblum [Wed, 20 Dec 2017 19:41:29 +0000 (11:41 -0800)]
Kill FOLLY_ALIGNED etc

Summary:
[Folly] Kill `FOLLY_ALIGNED` etc.

`alignas` is standardized as of C++11. Let us just use that.

Replace:
* `FOLLY_ALIGNED` with `alignas`
* `FOLLY_ALIGNED_MAX` with `alignas(folly::max_align_v)`
* `FOLLY_ALIGN_TO_AVOID_FALSE_SHARING` with `alignas(folly::hardware_destructive_interference_size)`

Because where `alignas` may be placed is more restrictive than where attributes may be placed, we also need to move these directives in some cases on top of doing the replacement.

Reviewed By: Orvid

Differential Revision: D6555167

fbshipit-source-id: 4b05b570bace3f8c0fe810b6dd58781dd45757f4

6 years agoFix Build: propagate_const under GCC
Yedidya Feldblum [Wed, 20 Dec 2017 06:31:53 +0000 (22:31 -0800)]
Fix Build: propagate_const under GCC

Summary: [Folly] Fix Build: `propagate_const` under GCC.

Reviewed By: aary

Differential Revision: D6609722

fbshipit-source-id: 47d8b0f2d54e0e5834383ea64c93af30927c921d

6 years agoAdd readWithPriority to PriorityMPMCQueue
Zheng Yan [Wed, 20 Dec 2017 03:40:14 +0000 (19:40 -0800)]
Add readWithPriority to PriorityMPMCQueue

Summary: Add readWithPriority to PriorityMPMCQueue

Reviewed By: yfeldblum

Differential Revision: D6606604

fbshipit-source-id: 07382e3c6d0f53a8727c059410c0577cf0aac82c

6 years agoAdd hardware_destructive_interference_size
Yedidya Feldblum [Wed, 20 Dec 2017 03:02:51 +0000 (19:02 -0800)]
Add hardware_destructive_interference_size

Summary:
[Folly] Add `hardware_destructive_interference_size` and `hardware_constructive_interference_size` to `folly/lang/Align.h`.

As backports from C++17. Which may require keeping, depending on how stanard libraries choose to implement these.

And replace `CacheLocality::kFalseSharingRange` with `hardware_destructive_interference_size`.

Reviewed By: ot

Differential Revision: D6554817

fbshipit-source-id: bff49f5ca8b01d38fa806076f99201355df76cd9

6 years agopropagate_const
Yedidya Feldblum [Wed, 20 Dec 2017 02:57:03 +0000 (18:57 -0800)]
propagate_const

Summary: [Folly] `propagate_const`, backported from C++ Library Fundamentals TS v2.

Reviewed By: ericniebler

Differential Revision: D6589681

fbshipit-source-id: cdc8981d17938b99afe60e2baefff7deb5316612

6 years agoFix erase in Iterate (2)
Dave Watson [Tue, 19 Dec 2017 22:30:19 +0000 (14:30 -0800)]
Fix erase in Iterate (2)

Summary:
Previously D6579707.

Correctly advance to next item if we erase the current element.  Corner cases were slightly off if we were at the end of a hash chain.

Reviewed By: yfeldblum

Differential Revision: D6603518

fbshipit-source-id: acb959e5bcd5da1c3df642b75985d464fdd3b23d

6 years agoFix ThreadLocal races
Dave Watson [Tue, 19 Dec 2017 21:17:28 +0000 (13:17 -0800)]
Fix ThreadLocal races

Summary:
I misread the ThreadLocal docs, thread destruction functions do *not* grab the accessAllTHreads_ lock
unless you use *strict* mode, and even then, it is only a read lock.

Easy enough to make the thread-destruction global bits to be atomic / use folly::Synchronized.

Reviewed By: yfeldblum

Differential Revision: D6592905

fbshipit-source-id: 4ae600dff4c8c04751483a452ca7c07ef3f26380

6 years agoSwitch the Baton template params
Yedidya Feldblum [Tue, 19 Dec 2017 19:24:17 +0000 (11:24 -0800)]
Switch the Baton template params

Summary: [Folly] Switch the `Baton` template params for consistency with `SaturatingSemaphore`.

Reviewed By: davidtgoldblatt

Differential Revision: D6591060

fbshipit-source-id: 44d6243d3185d95364a27e497216cc02fb3bc2e8

6 years agoSecure RNG utilities
Dustin Pho [Tue, 19 Dec 2017 14:51:29 +0000 (06:51 -0800)]
Secure RNG utilities

Summary: Adding secure RNG utility functions (rand32, rand64, oneIn, randDouble01, randDouble).

Reviewed By: yfeldblum

Differential Revision: D6551975

fbshipit-source-id: 720d138de1329669b1a15eb3e9cb3fe91ce982a4

6 years agoBaton::ready, a const variant of try_wait
Yedidya Feldblum [Tue, 19 Dec 2017 01:13:12 +0000 (17:13 -0800)]
Baton::ready, a const variant of try_wait

Summary: [Folly] `Baton::ready`, a `const` variant of `try_wait`.

Reviewed By: djwatson

Differential Revision: D6508064

fbshipit-source-id: ba458577574ba58165408a93238da7eb09adf1e6

6 years agoMake global executors shutdown-safe
Giuseppe Ottaviano [Mon, 18 Dec 2017 16:40:40 +0000 (08:40 -0800)]
Make global executors shutdown-safe

Summary:
The `get*Executor()` APIs don't check whether the singletons
haven't been destroyed already. Add a check and allow to return
`nullptr` during shutdown.

Also do a general clean up of the code, there was no reason to use
three independent singletons (non-atomically destroyed) for each
executor.

Reviewed By: philippv, luciang

Differential Revision: D6589486

fbshipit-source-id: 20fb835db7e446bd811bbd6d5ddbc41db9e98b54

6 years agoRemove the folly/Assume.h shim v2017.12.18.00
Christopher Dykes [Mon, 18 Dec 2017 04:56:49 +0000 (20:56 -0800)]
Remove the folly/Assume.h shim

Summary: The shim existed so that hhvm-third-party could be updated, and it has been, so it's time to remove the shim.

Reviewed By: yfeldblum

Differential Revision: D6590439

fbshipit-source-id: 3e1b039c1df6cf1e74102f7b7b26aa90fae31ec0

6 years agoTest UnboundedBlockingQueue in ThreadPoolExecutorTest
Maged Michael [Sun, 17 Dec 2017 01:20:04 +0000 (17:20 -0800)]
Test UnboundedBlockingQueue in ThreadPoolExecutorTest

Summary: Test UnboundedBlockingQueue for the case in D3527722

Reviewed By: ot

Differential Revision: D6587163

fbshipit-source-id: 1465991018187c5f841e6e3e7a11676390e2f8f2

6 years agoUnboundedQueue: Change try_dequeue variants to fail only if the queue is empty
Maged Michael [Sat, 16 Dec 2017 22:13:47 +0000 (14:13 -0800)]
UnboundedQueue: Change try_dequeue variants to fail only if the queue is empty

Summary:
The current semantics of try_dequeue variants can cause problems in cases where based on external dependencies the consumer is guaranteed that the queue is not empty and depends on the success of try_dequeue. See https://github.com/facebook/wangle/commit/abd07a06cae09efcd4347a9a019d59ae8a013c3f

This diff changes the semantics so that try_dequeue operations fail only if the queue is empty.

Reviewed By: yfeldblum

Differential Revision: D6586156

fbshipit-source-id: 25d6085e28d3e24034ecf6a8bafab51c95464b01

6 years agoUnboundedQueue: Fix advanceHead
Maged Michael [Sat, 16 Dec 2017 17:48:36 +0000 (09:48 -0800)]
UnboundedQueue: Fix advanceHead

Summary:
Without the fix multiple consumers may update head concurrently and cause it to lag. If this persists until the destruction time of the queue, some segments may be incorrectly retired twice.

The fix is to wait for head to advance to the current segment first before advancing head to the next segment.

Reviewed By: djwatson

Differential Revision: D6588135

fbshipit-source-id: 3e916441bff5ad3f27de418601990c59a0b89bc2

6 years agoGroup the fields of UnboundedQueue
Yedidya Feldblum [Sat, 16 Dec 2017 02:52:25 +0000 (18:52 -0800)]
Group the fields of UnboundedQueue

Summary:
[Folly] Group the fields of `UnboundedQueue`.

Into helper structures which are organized by, and aligned by, access.

This is all compiled away and has no runtime cost.

Reviewed By: magedm

Differential Revision: D6555031

fbshipit-source-id: 94424747bf76a0a9a2bd59b6a94749efb391d6a7

6 years agoFix comments in UnboundedQueue and DynamicBoundedQueue
Maged Michael [Sat, 16 Dec 2017 02:18:07 +0000 (18:18 -0800)]
Fix comments in UnboundedQueue and DynamicBoundedQueue

Summary: Fix typos and wording in comments.

Reviewed By: yfeldblum

Differential Revision: D6584559

fbshipit-source-id: b4b72f6c25c9f9e6fb3f2fbee69eea4a39b5c004

6 years agoRevert D6579707: [folly/ConcurrentHashMap] Fix erase in Iterate
Yanbo Xu [Sat, 16 Dec 2017 01:10:05 +0000 (17:10 -0800)]
Revert D6579707: [folly/ConcurrentHashMap] Fix erase in Iterate

Summary:
This reverts commit a45f47a53e106d22daa9cf57be6c40c4f6a430d9

bypass-lint

Differential Revision: D6579707

fbshipit-source-id: 6f6c35de5e05f8cf37491471e1ae1272290fe61b

6 years agoFix PicoSpinLock on aarch64
Keith Daigle [Fri, 15 Dec 2017 22:17:36 +0000 (14:17 -0800)]
Fix PicoSpinLock on aarch64

Summary:
Found that building folly on aarch64 gave errors with
picospinlock.  Looked and found that MSVC used casts
so I figured I'd try that

Reviewed By: yfeldblum

Differential Revision: D6504689

fbshipit-source-id: 9565bae5ffab485da407b8609be92ef7db10ab72

6 years agoFix erase in Iterate
Yanbo Xu [Fri, 15 Dec 2017 21:43:53 +0000 (13:43 -0800)]
Fix erase in Iterate

Summary:
The iterator returned from erase api could skip nodes. The fix is to
initialize the returned iterator with right value.

Reviewed By: djwatson

Differential Revision: D6579707

fbshipit-source-id: a45f47a53e106d22daa9cf57be6c40c4f6a430d9

6 years agoFix folly::max_align_v for Clang on ARMv7
Yedidya Feldblum [Fri, 15 Dec 2017 07:26:47 +0000 (23:26 -0800)]
Fix folly::max_align_v for Clang on ARMv7

Summary:
[Folly] Fix `folly::max_align_v` for Clang on ARMv7.

There is some problem with computing the correct result of `alignof` for a compound `union` POD type, because why wouldn't there be. The result *should* just be the max of the `alignof` of the type of each field if each field is default aligned - but not on some platforms! So we must compute the max directly.

Reviewed By: mzlee, Orvid

Differential Revision: D6573548

fbshipit-source-id: 512a255fda64795104d71fde14372befa3bf41e4

6 years agoLet Baton methods be noexcept
Yedidya Feldblum [Fri, 15 Dec 2017 04:59:51 +0000 (20:59 -0800)]
Let Baton methods be noexcept

Summary: [Folly] Let `Baton` methods be `noexcept`.

Reviewed By: djwatson

Differential Revision: D6508057

fbshipit-source-id: 2138a228d8291e79e9368cb77b0f63f4412524dc

6 years agoFix zerocopy AsyncSocket memory leaks
Dan Melnic [Fri, 15 Dec 2017 03:59:27 +0000 (19:59 -0800)]
Fix zerocopy AsyncSocket memory leaks

Summary:
We currently leak ptr entries due to a missing map erase call in AsyncSocket::releaseZeroCopyBuf.
Also addZeroCopyBuf called when the bytesWritten == 0 will cause and extra id to be allocated. This will delay the deletion of the last IOBuf sent via zerocopy. The number of buffers can accumulate over time if the same bytesWritten == 0 situation is encountered.
Found when running the nbd-server zero downtime upgrade during heavy fio traffic.
Add an AsyncSocket::processZeroCopyWriteInProgress so we can drain the zerocopy msg error queue even after we set the ReadCallback to nullptr.

Reviewed By: djwatson

Differential Revision: D6552982

fbshipit-source-id: 3d2fdca84ec3b5fc46c3bed06c0c9ede66ed565a

6 years agoFix xlog docs
Chen Shen [Thu, 14 Dec 2017 23:26:53 +0000 (15:26 -0800)]
Fix xlog docs

Summary: When I was investigating the difference between glog/xlog I encountered this.

Reviewed By: yfeldblum

Differential Revision: D6572117

fbshipit-source-id: 4130deb6cfed7cfb7d2433cdab292481beb2469f

6 years agoUse explicit memory order in Baton::post
Yedidya Feldblum [Thu, 14 Dec 2017 23:07:18 +0000 (15:07 -0800)]
Use explicit memory order in Baton::post

Summary: [Folly] Use explicit memory order in `Baton::post`.

Reviewed By: djwatson

Differential Revision: D6502268

fbshipit-source-id: a379ea711f165e3645ac2127a49824f46225b80a

6 years agoMove security protocol and application protocol APIs to AsyncTransport.
Kyle Nekritz [Thu, 14 Dec 2017 19:32:57 +0000 (11:32 -0800)]
Move security protocol and application protocol APIs to AsyncTransport.

Summary: They belong here rather than AsyncTransportWrapper.

Reviewed By: siyengar

Differential Revision: D6568209

fbshipit-source-id: 0856dba0b2d527ae68d33942b28e2c485a5b1f46

6 years agoDynamic Bounded Queue
Maged Michael [Thu, 14 Dec 2017 18:33:06 +0000 (10:33 -0800)]
Dynamic Bounded Queue

Summary:
DynamicBoundedQueue supports:
- Dynamic memory usage that grows and shrink in proportion to the number of elements in the queue.
- Adjustable capacity that helps throttle pathological cases of producer-consumer imbalance that may lead to excessive memory usage.
- The adjustable capacity can also help prevent deadlock by allowing users to temporarily increase capacity substantially to guarantee accommodating producer requests that cannot wait.
- SPSC, SPMC, MPSC, MPMC variants.
- Blocking and spinning-only variants.
- Inter-operable non-waiting, timed until, timed for, and waiting variants of producer and consumer operations.
- Optional variable element weights.

Reviewed By: djwatson

Differential Revision: D6462123

fbshipit-source-id: 5c7a7d88c3fa273211702af9ff78f6302afd2276

6 years agoUnboundedQueue: Use alignas instead of FOLLY_ALIGNED
Maged Michael [Thu, 14 Dec 2017 02:54:26 +0000 (18:54 -0800)]
UnboundedQueue: Use alignas instead of FOLLY_ALIGNED

Summary: Use alignas instead of FOLLY_ALIGNED

Reviewed By: Orvid

Differential Revision: D6554984

fbshipit-source-id: 5b11ac387faa137aa8dd0c35b6e9bb999cf0bf2d

6 years agoFix AsyncSocketTest.WriteErrorCallbackBytesWritten
Yedidya Feldblum [Thu, 14 Dec 2017 02:00:55 +0000 (18:00 -0800)]
Fix AsyncSocketTest.WriteErrorCallbackBytesWritten

Summary:
[Folly] Fix `AsyncSocketTest.WriteErrorCallbackBytesWritten`.

Thanks to congestion, especially when the tests are run concurrently, the expectations in the test were too restrictive.

If only 20KB are read, it is possible that only 20KB are acknowledged. The expectation was that if 20KB are read and the recv buffer and send buffer are both 8KB and 24KB are written then all 24KB are acknowledged, but congestion control disagrees.

It is possible that any number of bytes are written to the send buffer, from 28KB up to 40KB. And we have to explicitly wait for 28KB to be written even to know that (otherwise we only know that 20KB are written).

Differential Revision: D6550804

fbshipit-source-id: 100d086972c1526b909da0dbb6e609c144d7b17b

6 years agoconstexpr_ceil
Yedidya Feldblum [Thu, 14 Dec 2017 01:08:39 +0000 (17:08 -0800)]
constexpr_ceil

Summary: [Folly] `constexpr_ceil`, an integral rounding-up util.

Reviewed By: Orvid

Differential Revision: D6558042

fbshipit-source-id: 6b42add9bf2e3605baf71391130c2a2c88cc4385

6 years agoMove max_align_v and max_align_t to folly/lang/Align.h
Yedidya Feldblum [Thu, 14 Dec 2017 00:29:37 +0000 (16:29 -0800)]
Move max_align_v and max_align_t to folly/lang/Align.h

Summary:
[Folly] Move `max_align_v` and `max_align_t` to `folly/lang/Align.h`.

And drop the macro `FOLLY_ALIGNED_MAX` - just use `alignas(max_align_v)` instead.

Reviewed By: Orvid

Differential Revision: D6554677

fbshipit-source-id: cacb335b141623506a41e95e0740613374279c6f

6 years agoRemove unused #include <string.h> from folly/Portability.h
Yedidya Feldblum [Wed, 13 Dec 2017 22:53:49 +0000 (14:53 -0800)]
Remove unused #include <string.h> from folly/Portability.h

Summary: [Folly] Remove unused `#include <string.h>` from `folly/Portability.h`.

Reviewed By: Orvid

Differential Revision: D6557374

fbshipit-source-id: 369ef0b622dc057e882ed0f4ad25b47fa3f05100

6 years agoExplicitly refer to the std::chrono namespace to avoid conflicts with the folly:...
Christopher Dykes [Wed, 13 Dec 2017 20:49:43 +0000 (12:49 -0800)]
Explicitly refer to the std::chrono namespace to avoid conflicts with the folly::chrono namespace

Summary:
This is a hard requirement due to an upcoming change in Folly.

The codemod was only run on files that referenced the `std::chrono` namespace with an unqualified `chrono` and also used both the `std` and `folly` namespaces.
```
fbgr -sl "[^:]chrono::" | xargs grep -l "using namespace std;" | xargs grep -l "using namespace folly;" | xargs sed -r -i 's/([^:])chrono::([a-z])/\1std::chrono::\2/g'
```

Reviewed By: yfeldblum

Differential Revision: D6547864

fbshipit-source-id: 4a8230d311edbaa173722a09b2773e8d053fae7e

6 years agotypo in io/Cursor.h
RcRonco [Wed, 13 Dec 2017 19:26:12 +0000 (11:26 -0800)]
typo in io/Cursor.h

Summary: Closes https://github.com/facebook/folly/pull/721

Differential Revision: D6558163

Pulled By: yfeldblum

fbshipit-source-id: ed03aa92b54c61f9980a4759da29910aa77d5db6

6 years agoRCU
Dave Watson [Wed, 13 Dec 2017 16:13:20 +0000 (08:13 -0800)]
RCU

Summary: This adds an RCU implementation, matching http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0461r1.pdf as closely as pratical.  This implementation does not require thread registration or quiescence.

Reviewed By: magedm

Differential Revision: D6330631

fbshipit-source-id: 2c729f3a4c0f151cde5d9a599ecd2a2c20c7da55

6 years agologging: minor clean up in Logger.h
Adam Simpkins [Wed, 13 Dec 2017 00:30:16 +0000 (16:30 -0800)]
logging: minor clean up in Logger.h

Summary:
Update the comment describing the Logger class, and remove a `clang-format off`
since recent versions of `clang-format` appear to do a good job formatting this
macro.

Reviewed By: yfeldblum

Differential Revision: D6525996

fbshipit-source-id: 6abbb3740afb53feb2418ca890cd78af1eb5f7a2

6 years agologging: add more documentation
Adam Simpkins [Wed, 13 Dec 2017 00:30:13 +0000 (16:30 -0800)]
logging: add more documentation

Summary: Add several new documents describing the behavior of the folly logging library.

Reviewed By: yfeldblum

Differential Revision: D6525997

fbshipit-source-id: 8ba2ff99c70c7566987b3da321526a16755e72bf

6 years agoSwitch to the try_wait_for and try_wait_until Baton APIs
Yedidya Feldblum [Tue, 12 Dec 2017 06:10:59 +0000 (22:10 -0800)]
Switch to the try_wait_for and try_wait_until Baton APIs

Summary: Switch to the `try_wait_for` and `try_wait_until` `Baton` APIs.

Reviewed By: davidtgoldblatt

Differential Revision: D6532103

fbshipit-source-id: aa3ce64152d167bb9c9cb1f266be0f9f8bd498f5

6 years agoStore filename and provide detailed message on data access assertion failure.
Andrii Nakryiko [Tue, 12 Dec 2017 06:05:33 +0000 (22:05 -0800)]
Store filename and provide detailed message on data access assertion failure.

Summary:
It looks like not having offset/size/filename information is way more
harmful, than storing filename just for the sake of this error message.

Reviewed By: yfeldblum

Differential Revision: D6536616

fbshipit-source-id: 469fbdf1deedd76ebd79cf98716c2c269cb10e4d

6 years agoAssorted tweaks to folly/String.h
Yedidya Feldblum [Tue, 12 Dec 2017 04:19:06 +0000 (20:19 -0800)]
Assorted tweaks to folly/String.h

Summary:
[Folly] Assorted tweaks to `folly/String.h`.

* Use `void_t` in the definition of `IsConvertible`.
* Remove `AllConvertible`. Callers can use `StrictConjunction` instead.
* Move the `static_assert`s from the header to the source file.

Reviewed By: andrewjcg, ot

Differential Revision: D6501515

fbshipit-source-id: 472ecf23bf7f06be211480b0aceea95f7e60dc21

6 years agoFix case where ssl cert does not match key
Neel Goyal [Tue, 12 Dec 2017 02:48:49 +0000 (18:48 -0800)]
Fix case where ssl cert does not match key

Summary: In some cases, SSLContextManager seg faults if a cert and key do not match.  This guards against that case when strictSSL = false, and throws a more useful error in the cases when SSL is required.

Reviewed By: xybu

Differential Revision: D6513964

fbshipit-source-id: 8e63a22b346fd3f2a30d558a3659ab6794c7a105

6 years agofolly::fibers::Baton API consistency with folly::Baton
Yedidya Feldblum [Tue, 12 Dec 2017 00:26:17 +0000 (16:26 -0800)]
folly::fibers::Baton API consistency with folly::Baton

Summary:
[Folly] `folly::fibers::Baton` API consistency with `folly::Baton`.

Specifically, the suite of `wait`, `try_wait`, `try_wait_for`, and `try_wait_until` members and member templates.

Hold onto the `timed_wait` members for now, but mark them deprecated.

Additionally, for consistency, offer main-context function params consistently for all `try_wait_for`, `try_wait_until`, and both variants of `timed_wait`.

Reviewed By: andriigrynenko

Differential Revision: D6531145

fbshipit-source-id: 960fba48716b12b0ef53262786eacab88d8b2375

6 years agoPass ZSTD_CONTENTSIZE_UNKNOWN
Nick Terrell [Tue, 12 Dec 2017 00:21:29 +0000 (16:21 -0800)]
Pass ZSTD_CONTENTSIZE_UNKNOWN

Summary:
Content size 0 as unknown is deprecated.
See code comments for details.

Reviewed By: felixhandte

Differential Revision: D6521582

fbshipit-source-id: 876b94cc0123c1020b456cc8f4cd86997afe0619

6 years agoRemove multi-poster support from Baton v2017.12.11.00
Yedidya Feldblum [Sun, 10 Dec 2017 20:39:24 +0000 (12:39 -0800)]
Remove multi-poster support from Baton

Summary:
[Folly] Remove multi-poster support from `Baton`.

Code which needs multi-poster support may use `SaturatingSemaphore` instead.

Reviewed By: magedm

Differential Revision: D6529661

fbshipit-source-id: d9dc053ca984ef3a404e9361910b0044817d4905

6 years agoFix some clang build failures on aarch64
Andrew Gallagher [Sun, 10 Dec 2017 08:55:21 +0000 (00:55 -0800)]
Fix some clang build failures on aarch64

Reviewed By: Orvid

Differential Revision: D6387412

fbshipit-source-id: 4c1424ed4fa49e69885de8cd72e78169314fc502

6 years agoUnboundedBlockingQueue: Remove extra include
Maged Michael [Sun, 10 Dec 2017 06:31:03 +0000 (22:31 -0800)]
UnboundedBlockingQueue: Remove extra include

Summary: Remove include not needed after recent change

Reviewed By: yfeldblum

Differential Revision: D6529608

fbshipit-source-id: 904072084e180125d68bd1ef6e4dffb2939cd1eb

6 years agoFix signed integer overflow in StaticTracepointTest.cpp
Yedidya Feldblum [Sun, 10 Dec 2017 01:09:24 +0000 (17:09 -0800)]
Fix signed integer overflow in StaticTracepointTest.cpp

Summary:
[Folly] Fix signed integer overflow in `StaticTracepointTest.cpp`.

It is undefined behavior and is caught by UBSAN in some test runs.

Reviewed By: Orvid

Differential Revision: D6528445

fbshipit-source-id: e4a199e9005b57bcd93c8c4b69f37ccb1e072bdf

6 years agoSplit Baton wait methods into fast and slow paths
Yedidya Feldblum [Sat, 9 Dec 2017 17:50:25 +0000 (09:50 -0800)]
Split Baton wait methods into fast and slow paths

Summary:
[Folly] Split `Baton` wait methods into fast and slow paths.

Force-inline the fast paths, and force-outline the slow paths.

Reviewed By: magedm

Differential Revision: D6501992

fbshipit-source-id: 611e26b3cfeef01eb8d3a3500ae3ebc26bee6e86

6 years agoFix UBSAN failure in RangeTest.cpp with libstdc++-v4.9
Yedidya Feldblum [Sat, 9 Dec 2017 17:47:06 +0000 (09:47 -0800)]
Fix UBSAN failure in RangeTest.cpp with libstdc++-v4.9

Summary:
[Folly] Fix UBSAN failure in `RangeTest.cpp` with `libstdc++-v4.9`.

The implementation of `std::array` in `libstdc++-v4.9` has undefined behavior in the case of the zero-length array - dereferencing (but immediately taking the address again) of `nullptr`. This is fixed in `libstdc++-v5`.

Reviewed By: ot

Differential Revision: D6528614

fbshipit-source-id: 26be6bdc28d9e6dc6e6293a4d4ecffd769c01413

6 years agoreplace std::dequeue with UMPMCQueue in UnboundedBlockingQueue
Yedidya Feldblum [Sat, 9 Dec 2017 05:11:03 +0000 (21:11 -0800)]
replace std::dequeue with UMPMCQueue in UnboundedBlockingQueue

Summary: As above. Thanks magedm for giving us this beautiful piece of equipment!

Reviewed By: magedm

Differential Revision: D6488661

fbshipit-source-id: 95aa9646ca1ea937bb1d055e9baa037896c3161e

6 years agoUnboundedQueue: Add LgAlign template parameter - Refactor code
Maged Michael [Fri, 8 Dec 2017 17:25:57 +0000 (09:25 -0800)]
UnboundedQueue: Add LgAlign template parameter - Refactor code

Summary:
- Add a template parameter, LgAlign, to control memory usage. The parameter is used in DynamicBoundedQueue.
- Refactor code.

Reviewed By: yfeldblum

Differential Revision: D6508015

fbshipit-source-id: 6e17b1d8fd900595147dc4217e04d379a13fbdf8

6 years agoMove folly/Bits.h to folly/lang/
Yedidya Feldblum [Fri, 8 Dec 2017 08:24:44 +0000 (00:24 -0800)]
Move folly/Bits.h to folly/lang/

Summary: [Folly] Move `folly/Bits.h` to `folly/lang/`.

Reviewed By: phoad, Orvid

Differential Revision: D6495547

fbshipit-source-id: a93159321df8277f8a4b4f10a5e4e0fc58cb6022

6 years agoprevent folly::Function<T const&()> from being initialized with a function that retur...
Eric Niebler [Fri, 8 Dec 2017 06:36:49 +0000 (22:36 -0800)]
prevent folly::Function<T const&()> from being initialized with a function that returns a prvalue

Summary: If the folly::Function returns by const& and the wrapped function returns a prvalue, it is guaranteed to return a dangling reference. Prevent it.

Reviewed By: yfeldblum, aary

Differential Revision: D6487447

fbshipit-source-id: 61700b4688e29409eefa27f546b31ecac258cfdd

6 years agologging: use raw string literals in config tests
Adam Simpkins [Fri, 8 Dec 2017 03:24:34 +0000 (19:24 -0800)]
logging: use raw string literals in config tests

Summary:
Update string literals with escaped quotes and backslashes to use raw string
literal syntax instead.

Reviewed By: yfeldblum

Differential Revision: D6501426

fbshipit-source-id: 557366f86b360596b13148d03fd33c7df5c3f890

6 years agoFlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton
Maged Michael [Thu, 7 Dec 2017 22:41:53 +0000 (14:41 -0800)]
FlatCombining: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton

Summary: Use SaturatingSemaphore instead of multi-poster and non-blocking Baton, in preparation for restoring Baton to being single-poster and with blocking wait().

Reviewed By: davidtgoldblatt

Differential Revision: D6510619

fbshipit-source-id: eed87beabc1b698080b8e41c746d570d30804ce7

6 years agoFix crash in exception_wrapper::get_exception<>
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:59 +0000 (12:38 -0800)]
Fix crash in exception_wrapper::get_exception<>

Summary:
[Folly] Fix crash in `exception_wrapper::get_exception<>`.

When the contained object is unthrown and does not inherit `std::exception`, `get_exception` templated over a type that does not inherit `std::exception` must throw and catch internally and may then return a pointer to the internally thrown object, which has since been deallocated.

Attempting to dereference that pointer is undefined behavior is correctly caught by ASAN as heap-use-after-free.

Fix it by storing objects not inheriting `std::exception` using only the `std::exception_ptr` representation.

As a downside, we no longer get the small-object optimization or the non-throwing optimization for objects which do not inherit `std::exception`. But this is not likely to be terribly concerning in practice.

Reviewed By: ericniebler

Differential Revision: D6504911

fbshipit-source-id: 0065de911733b5cab87be55e7e4e47f0a9c09140

6 years agoUse simpler tags for ctor dispatch in exception_wrapper
Yedidya Feldblum [Thu, 7 Dec 2017 20:38:57 +0000 (12:38 -0800)]
Use simpler tags for ctor dispatch in exception_wrapper

Summary:
[Folly] Use simpler tags for ctor dispatch in `exception_wrapper`.

The tags do not need to be `std::integral_constant`, but can just be arbitrary empty types.

Also format the usage sites of the tags.

Reviewed By: ericniebler

Differential Revision: D6506417

fbshipit-source-id: 377359086d94fc1a8bf8eebf6f8b058ff544829f

6 years agoFix DCHECKs in IOBufQueue
Stepan Palamarchuk [Thu, 7 Dec 2017 15:43:57 +0000 (07:43 -0800)]
Fix DCHECKs in IOBufQueue

Summary: glog actually tries to pull srtings from these pointers when the checks fail, which obfuscates the error

Reviewed By: yfeldblum

Differential Revision: D6506886

fbshipit-source-id: 0283e02aecaa248b84fca1433d3f29f92c37b2e6

6 years agoRemoved dead IsSomeVector
Mehdi Bouaziz [Thu, 7 Dec 2017 10:57:26 +0000 (02:57 -0800)]
Removed dead IsSomeVector

Summary: deadcode

Reviewed By: ddrcoder

Differential Revision: D6498754

fbshipit-source-id: 9dda0ade6928d648b8938a06b185e430753f2318

6 years agoFix SignalHandlerTest with ASAN
Andrii Grynenko [Thu, 7 Dec 2017 01:50:03 +0000 (17:50 -0800)]
Fix SignalHandlerTest with ASAN

Reviewed By: yfeldblum

Differential Revision: D6503762

fbshipit-source-id: f82b88c401979761053004eadd2e4c6a74f05f9c

6 years agologging: allow partial updates to log handler settings
Adam Simpkins [Thu, 7 Dec 2017 01:29:32 +0000 (17:29 -0800)]
logging: allow partial updates to log handler settings

Summary:
This updates the LogHandlerConfig code to allow changing the settings on an
existing log handler without listing all of its existing options from scratch.

This also changes the syntax of the basic log handler configuration string to
use a colon to separate the log handler name+type from the options list.  In
other words, rather than specifying `default=stream,stream=stderr,async=true`
you now say `default=stream:stream=stderr,async=true`.

The primary motivation for this change is to make it easy for users to switch
the async setting for the default log handler on or off.  Callers can now
specify `default:async=true` to easily enable async mode on the default log
handler without having to completely re-list the full settings for the default
handler.

Reviewed By: yfeldblum

Differential Revision: D6494228

fbshipit-source-id: 52a296f800a5456f0c3aa10546298139c8db52fc

6 years agologging: split FileHandlerFactory into two classes
Adam Simpkins [Thu, 7 Dec 2017 01:29:30 +0000 (17:29 -0800)]
logging: split FileHandlerFactory into two classes

Summary:
Split FileHandlerFactory into separate FileHandlerFactory and
StreamHandlerFactory classes, where FileHandlerFactory only handles logging to
files by path name, and StreamHandlerFactory only supports logging to stdout or
stderr.

The primary motivation for this is to allow logging to stdout or stderr in some
cases without allowing arbitrary files to be opened by FileHandlerFactory.
This can be achieved now by registering StreamHandlerFactory but not
FileHandlerFactory.  This makes it safer to allow controlling logging
configuration via command line flags even in setuid binaries.

Reviewed By: yfeldblum

Differential Revision: D6494226

fbshipit-source-id: a3ec371ca4266424d07dff20be18e6e13c057593

6 years agologging: add a StandardLogHandlerFactory helper class
Adam Simpkins [Thu, 7 Dec 2017 01:29:27 +0000 (17:29 -0800)]
logging: add a StandardLogHandlerFactory helper class

Summary:
This moves some of the FileHandlerFactory code out into a new
StandardLogHandlerFactory helper class.  This will make it easier in the future
to add new LogHandlerFactory implementations that create StandardLogHandler
objects.

In particular, I plan to use this soon to split FileHandlerFactory into two
separate classes: one for writing to files on disk and a separate class for
writing to stdout or stderr.

Reviewed By: yfeldblum

Differential Revision: D6494227

fbshipit-source-id: 52e24250d020d21a5395d2a68fa5bd40bb32fbd4

6 years agofix ubsan-detected UMR in TemporaryFile
Louis Brandy [Thu, 7 Dec 2017 01:10:31 +0000 (17:10 -0800)]
fix ubsan-detected UMR in TemporaryFile

Summary:
Move constructors are just like any other constructors. `this` is not in a valid state. You aren't supposed to destroy `this` before move-constructing it.

This was detected by ubsan stress runs.

UBSAN report:

```
[ RUN      ] TemporaryFile.moveCtor
folly/experimental/TestUtil.cpp:91:20: runtime error: load of value 3, which is not a valid value for type 'bool'
 [ failed: 1 ]
 ````

Reviewed By: yfeldblum

Differential Revision: D6501500

fbshipit-source-id: 83bd0525ac2c1f5951f70f5596034bb6cb0ca985

6 years agoRename Baton::timed_wait to try_wait_for and try_wait_until
Yedidya Feldblum [Wed, 6 Dec 2017 19:36:23 +0000 (11:36 -0800)]
Rename Baton::timed_wait to try_wait_for and try_wait_until

Summary:
[Folly] Rename `Baton::timed_wait` to `try_wait_for` and `try_wait_until`.

For consistency with `std::timed_mutex`'s suite of `lock` member functions, as well as `SaturatingSemaphore` and others.

Reviewed By: davidtgoldblatt

Differential Revision: D6493714

fbshipit-source-id: 566c6d5dc0f6db4a0308cbd1972d53794f898383

6 years agoMove folly/Baton.h to folly/synchronization/
Yedidya Feldblum [Wed, 6 Dec 2017 06:03:40 +0000 (22:03 -0800)]
Move folly/Baton.h to folly/synchronization/

Summary: [Folly] Move `folly/Baton.h` to `folly/synchronization/`.

Reviewed By: phoad, Orvid

Differential Revision: D6490282

fbshipit-source-id: 66e2d25ffe3275d576b97b81c1987709000f6649

6 years agoUse boost::filesystem::current_path in ChangeToTempDir
Yedidya Feldblum [Wed, 6 Dec 2017 05:00:02 +0000 (21:00 -0800)]
Use boost::filesystem::current_path in ChangeToTempDir

Summary:
[Folly] Use `boost::filesystem::current_path` in `ChangeToTempDir`.

It will report failures noisily.

Reviewed By: pixelb

Differential Revision: D6493243

fbshipit-source-id: 423dc0e3a46781e9af42fee69060d31085f1a7c6

6 years agoexclude Unix Domain Socket from enableTTLBANotifications
Liang Zhu [Wed, 6 Dec 2017 00:45:14 +0000 (16:45 -0800)]
exclude Unix Domain Socket from enableTTLBANotifications

Reviewed By: yfeldblum

Differential Revision: D6482348

fbshipit-source-id: 5b2d3c984b761a54d5a9eba48953c556ed22cda6

6 years agoFix RequestContext held too long issue in EventBase
Pingjia Shan [Tue, 5 Dec 2017 21:03:44 +0000 (13:03 -0800)]
Fix RequestContext held too long issue in EventBase

Summary:
Upon debugging for the attached task, it appears to me the problem are in two places:
. After a callback has been run, context_ wasn't reset
. After a callback has been canceled, context_ wasn't reset

In this diff:
. Fix these two places.
. Updating unit tests to cover these two cases.

Reviewed By: yfeldblum

Differential Revision: D6465788

fbshipit-source-id: 85b3b29dc80c9f3971c85f302385d41ded44fa0e

6 years agoSplit get_default() into two for deferred default construction and added forwarding...
Aaryaman Sagar [Tue, 5 Dec 2017 18:54:55 +0000 (10:54 -0800)]
Split get_default() into two for deferred default construction and added forwarding to speed up default element construction with rvalues

Summary:
As it stood currently folly::get_default() would unnecessarily
construct a value into the third parameter, which was unnecessary in the code
path where the element was found in the map.  Also the default value can be
forwarded to the return type in the code path where the element is not found
and an rvalue parameter is passed as the default value

Reviewed By: yfeldblum

Differential Revision: D6390315

fbshipit-source-id: ef692b827d5a36751b4eb1e12042869e8fbba2e5

6 years agobitreverse
Dave Watson [Tue, 5 Dec 2017 17:14:34 +0000 (09:14 -0800)]
bitreverse

Summary:
Adds a bitreverse function to Bits.h

AFAIK there is no great way to do this on x86, takes ~4-5ns.
arm has a single instruction we can drop in in the future.

Reviewed By: yfeldblum

Differential Revision: D6459283

fbshipit-source-id: 129db196b2fac95386f601ae57843aa87523b915

6 years agoexclude Unix Domain Socket from setErrMessageCB
Liang Zhu [Tue, 5 Dec 2017 11:49:11 +0000 (03:49 -0800)]
exclude Unix Domain Socket from setErrMessageCB

Summary:
In the latest stable kernel 4.14.3 as of 2017-12-04, unix domain socket does not support MSG_ERRQUEUE. So recvmsg(MSG_ERRQUEUE) will read application data from unix doamin socket as error message, which breaks the message flow in application. This diff disable setErrMessageCB for Unix Domain Socket.

Both [[ https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c#L1782 | tcp_recvmsg ]] and [[ https://github.com/torvalds/linux/blob/master/net/ipv4/udp.c#L1571 | udp_recvmsg ]] will check flag MSG_ERRQUEUE and read from error queue.  However, there is nothing about MSG_ERRQUEUE in [[ https://github.com/torvalds/linux/blob/master/net/unix/af_unix.c#L2249 | af_unix ]].

Reviewed By: yfeldblum

Differential Revision: D6479465

fbshipit-source-id: eba2c8650e96466f2b361a42ddf90053d65f19bd

6 years agoAdd missing check against SemiFuture in FutureBase ctor
Yedidya Feldblum [Tue, 5 Dec 2017 04:57:02 +0000 (20:57 -0800)]
Add missing check against SemiFuture in FutureBase ctor

Summary: [Folly] Add missing check against `SemiFuture` in `FutureBase` ctor.

Reviewed By: LeeHowes

Differential Revision: D6469021

fbshipit-source-id: 0ca31461afee4bf64725e065a8cb4d5e933d2013

6 years agobetter error handling in MemoryIdler for inside jails
Nathan Bronson [Tue, 5 Dec 2017 00:39:00 +0000 (16:39 -0800)]
better error handling in MemoryIdler for inside jails

Summary:
/proc may be unavailable in restricted environments, which can
cause pthread_getattr_np to fail.  This diff adds checking to the return
code from that function.

Reviewed By: yfeldblum

Differential Revision: D6473789

fbshipit-source-id: de016d32f29edf8410808dfb491987692f39e768

6 years agoImprove QueueAppender/IOBufQueue performance
Stepan Palamarchuk [Mon, 4 Dec 2017 23:08:24 +0000 (15:08 -0800)]
Improve QueueAppender/IOBufQueue performance

Summary:
Currently QueueAppender needs to follow a chain of 4 indirections (QueueAppender->IOBufQueue->IOBuf(head)->IOBuf(tail)->data).
This diff adds a cache of writable tail range in IOBufQueue and allows it to be placed externally.

Before this diff on hot path QueueAppender::write<signed char> was ~167 bytes of code (with majority being actually executed), after this diff it's down to ~30 bytes:
  0x0000000000419d10 <+0>:     mov    (%rdi),%rax
  0x0000000000419d13 <+3>:     cmp    %rax,0x8(%rdi)
  0x0000000000419d17 <+7>:     je     0x419d28 <folly::io::QueueAppender::write<signed char>(signed char)+24>
  0x0000000000419d19 <+9>:     mov    %sil,(%rax)
  0x0000000000419d1c <+12>:    addq   $0x1,(%rdi)
  0x0000000000419d20 <+16>:    retq
  0x0000000000419d21 <+17>:    nopl   0x0(%rax)
  0x0000000000419d28 <+24>:    movsbl %sil,%esi
  0x0000000000419d2c <+28>:    jmpq   0x419ca0 <folly::io::QueueAppender::writeSlow<signed char>(signed char)>

With this diff, Thrift serialization performance is improved up to 2x with production workloads (2x for compact, 3x for binary).

Thrift benchmark output:
Before:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  58.05ns   17.23M
  BinaryProtocol_write_SmallInt                               75.17ns   13.30M
  BinaryProtocol_write_BigInt                                 74.60ns   13.41M
  BinaryProtocol_write_SmallString                            85.12ns   11.75M
  BinaryProtocol_write_BigString                             802.96ns    1.25M
  BinaryProtocol_write_BigBinary                             174.69ns    5.72M
  BinaryProtocol_write_LargeBinary                           171.81ns    5.82M
  BinaryProtocol_write_Mixed                                 130.97ns    7.64M
  BinaryProtocol_write_SmallListInt                          123.99ns    8.06M
  BinaryProtocol_write_BigListInt                             40.72us   24.56K
  BinaryProtocol_write_BigListMixed                          784.78us    1.27K
  BinaryProtocol_write_LargeListMixed                         98.84ms    10.12
  CompactProtocol_write_Empty                                 64.38ns   15.53M
  CompactProtocol_write_SmallInt                              76.74ns   13.03M
  CompactProtocol_write_BigInt                                83.62ns   11.96M
  CompactProtocol_write_SmallString                           86.05ns   11.62M
  CompactProtocol_write_BigString                            786.18ns    1.27M
  CompactProtocol_write_BigBinary                            184.91ns    5.41M
  CompactProtocol_write_LargeBinary                          182.12ns    5.49M
  CompactProtocol_write_Mixed                                120.89ns    8.27M
  CompactProtocol_write_SmallListInt                         119.74ns    8.35M
  CompactProtocol_write_BigListInt                            43.76us   22.85K
  CompactProtocol_write_BigListMixed                         595.90us    1.68K
  CompactProtocol_write_LargeListMixed                        72.80ms    13.74
  ============================================================================
After:
  ============================================================================
  thrift/lib/cpp2/test/ProtocolBench.cpp          relative  time/iter  iters/s
  ============================================================================
  BinaryProtocol_write_Empty                                  65.97ns   15.16M
  BinaryProtocol_write_SmallInt                               72.31ns   13.83M
  BinaryProtocol_write_BigInt                                 72.67ns   13.76M
  BinaryProtocol_write_SmallString                            77.56ns   12.89M
  BinaryProtocol_write_BigString                             782.07ns    1.28M
  BinaryProtocol_write_BigBinary                             179.69ns    5.57M
  BinaryProtocol_write_LargeBinary                           182.62ns    5.48M
  BinaryProtocol_write_Mixed                                  91.62ns   10.92M
  BinaryProtocol_write_SmallListInt                           96.22ns   10.39M
  BinaryProtocol_write_BigListInt                             19.65us   50.90K
  BinaryProtocol_write_BigListMixed                          245.69us    4.07K
  BinaryProtocol_write_LargeListMixed                         46.56ms    21.48
  CompactProtocol_write_Empty                                 74.44ns   13.43M
  CompactProtocol_write_SmallInt                              80.35ns   12.45M
  CompactProtocol_write_BigInt                                85.30ns   11.72M
  CompactProtocol_write_SmallString                           82.61ns   12.10M
  CompactProtocol_write_BigString                            784.77ns    1.27M
  CompactProtocol_write_BigBinary                            193.20ns    5.18M
  CompactProtocol_write_LargeBinary                          192.53ns    5.19M
  CompactProtocol_write_Mixed                                 99.78ns   10.02M
  CompactProtocol_write_SmallListInt                         104.77ns    9.54M
  CompactProtocol_write_BigListInt                            25.62us   39.03K
  CompactProtocol_write_BigListMixed                         272.42us    3.67K
  CompactProtocol_write_LargeListMixed                        38.32ms    26.09
  ============================================================================

QueueAppender Benchmark output (although not very representative due to a tight loop):
Before:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                 10.50us   95.20K
  write_uint16                                                 5.48us  182.49K
  write_uint32                                                 2.73us  366.22K
  push_64b                                                     9.77us  102.36K
  push_1024b                                                 112.87us    8.86K
  append                                                      64.21us   15.57K
  preallocate_postallocate_1b                                 16.34us   61.19K
  preallocate_postallocate_4b                                 15.56us   64.26K
  preallocate_postallocate_32b                                22.17us   45.11K
  preallocate_postallocate_256b                              149.55us    6.69K
  ============================================================================

After:
  ============================================================================
  folly/io/test/QueueAppenderBenchmark.cpp        relative  time/iter  iters/s
  ============================================================================
  write_uint8                                                  8.86us  112.81K
  write_uint16                                                 3.91us  255.68K
  write_uint32                                                 2.08us  481.78K
  push_64b                                                     8.24us  121.39K
  push_1024b                                                 115.50us    8.66K
  append                                                      67.52us   14.81K
  preallocate_postallocate_1b                                 13.86us   72.17K
  preallocate_postallocate_4b                                 11.67us   85.71K
  preallocate_postallocate_32b                                20.35us   49.14K
  preallocate_postallocate_256b                              148.57us    6.73K
  ============================================================================

Reviewed By: yfeldblum

Differential Revision: D6427749

fbshipit-source-id: 8495cc74b6106b15d201e37533ae4c0a1abc9d74

6 years agoadd AtFork::unregisterHandler
Dave Watson [Mon, 4 Dec 2017 20:57:07 +0000 (12:57 -0800)]
add AtFork::unregisterHandler

Summary: Adds an AdFork::unregisterHandler command, such that we can register and unregister fork handlers.

Reviewed By: yfeldblum

Differential Revision: D6439796

fbshipit-source-id: b710152f8cf98371c330b484cd5fe42de44e74ac

6 years agoUnbounded queue
Maged Michael [Mon, 4 Dec 2017 19:07:30 +0000 (11:07 -0800)]
Unbounded queue

Summary:
UnboundedQueue supports:
- SPSC, MPSC, SCMP, MPMC
- Non-waiting, waiting, and timed consumer operations.
- Producers never wait or fail (unless out-of-memory).
- Memory usage grows and shrinks dynamically

```
/// UnboundedQueue supports a variety of options for unbounded
/// dynamically expanding an shrinking queues, including variations of:
/// - Single vs. multiple producers
/// - Single vs. multiple consumers
/// - Blocking vs. spin-waiting
/// - Non-waiting, timed, and waiting consumer operations.
/// Producer operations never wait or fail (unless out-of-memory).
///
/// Template parameters:
/// - T: element type
/// - SingleProducer: true if there can be only one producer at a
///   time.
/// - SingleConsumer: true if there can be only one consumer at a
///   time.
/// - MayBlock: true if consumers may block, false if they only
///   spins. A performance tuning parameter.
/// - LgSegmentSize (default 8): Log base 2 of number of elements per
///   segment. A performance tuning parameter. See below.
///
/// When to use UnboundedQueue:
/// - If a small bound may lead to deadlock or performance degradation
///   under bursty patterns.
/// - If there is no risk of the queue growing too much.
///
/// When not to use UnboundedQueue:
/// - If there is risk of the queue growing too much and a large bound
///   is acceptable, then use DynamicBoundedQueue.
/// - If the queue must not allocate on enqueue or it must have a
///   small bound, then use fixed-size MPMCQueue or (if non-blocking
///   SPSC) ProducerConsumerQueue.
///
/// Template Aliases:
///   USPSCQueue<T, MayBlock, LgSegmentSize>
///   UMPSCQueue<T, MayBlock, LgSegmentSize>
///   USPMCQueue<T, MayBlock, LgSegmentSize>
///   UMPMCQueue<T, MayBlock, LgSegmentSize>
///
/// Functions:
///   Producer operations never wait or fail (unless OOM)
///     void enqueue(const T&);
///     void enqueue(T&&);
///         Adds an element to the end of the queue.
///
///   Consumer operations:
///     void dequeue(T&);
///         Extracts an element from the front of the queue. Waits
///         until an element is available if needed.
///     bool try_dequeue(T&);
///         Tries to extracts an element from the front of the queue
///         if available. Returns true if successful, false otherwise.
///     bool try_dequeue_until(T&, time_point& deadline);
///         Tries to extracts an element from the front of the queue
///         if available until the specified deadline.  Returns true
///         if successful, false otherwise.
///     bool try_dequeue_for(T&, duration&);
///         Tries to extracts an element from the front of the queue
///         if available for for the specified duration.  Returns true
///         if successful, false otherwise.
///
///   Secondary functions:
///     size_t size();
///         Returns an estimate of the size of the queue.
///     bool empty();
///         Returns true only if the queue was empty during the call.
///     Note: size() and empty() are guaranteed to be accurate only if
```

Reviewed By: djwatson

Differential Revision: D6157613

fbshipit-source-id: db423f86d1d0604d22f6b9c71ea0ed08be32e2a1

6 years agoAdd wrapper for getting X509_digest from a cert
Anirudh Ramachandran [Mon, 4 Dec 2017 07:56:21 +0000 (23:56 -0800)]
Add wrapper for getting X509_digest from a cert

Summary:
X509_digest, or the "fingerprint" of the cert, is useful to identify
certificates with the same CN/SANs. This adds an API to get the SHA1 or SHA256
digest of a cert.

Reviewed By: yfeldblum

Differential Revision: D6457917

fbshipit-source-id: 148a5a2270e938b39065e00d7947c2fc57cd7f75

6 years agoStatically allocate futex array v2017.12.04.00
Alastair Daivis [Sun, 3 Dec 2017 01:35:33 +0000 (17:35 -0800)]
Statically allocate futex array

Summary:
AFAICT this is currently the only thing preventing the non-dynamic version of Folly's MPMCQueue from being allocation-free on enqueue and dequeue operations. Allocating this with a constant expression should move the allocation from runtime to link (possibly compile?) time, which will let us use it in allocation sensitive contexts.

Feel free to suggest other reviewers, I couldn't find an existing folly reviewer group.

Reviewed By: yfeldblum, agola11, nbronson

Differential Revision: D6447848

fbshipit-source-id: 86b84b19d62f1e1bcecdb9e757a6dfa90597b084

6 years agoallow fiber executor to override fiber options
David Goode [Sat, 2 Dec 2017 03:56:14 +0000 (19:56 -0800)]
allow fiber executor to override fiber options

Summary: As in title, for eg larger stack size

Reviewed By: yfeldblum

Differential Revision: D6466762

fbshipit-source-id: 303e8732c8bfb64fbd8619163779942fe5f8b5a8

6 years agoadd makeSystemError*() helper functions
Adam Simpkins [Sat, 2 Dec 2017 02:05:54 +0000 (18:05 -0800)]
add makeSystemError*() helper functions

Summary:
Add makeSystemError() helper functions that are similar to the existing
throwSystemError() functions but just return the exception rather than throwing
it.  This is helpful for callers using folly::Expected or folly::Future, where
they need to return an exception type rather than throwing it.

This also includes comments about the fact that this code is using the wrong
error category.  The C++ standard indicates that std::generic_category() for
POSIX errno values.  I am not fixing this as part of this diff, since this
change has the potential to break existing users, and would need a fair amount
of testing first.

Reviewed By: yfeldblum

Differential Revision: D6456771

fbshipit-source-id: 4724b51b8d4a7f513ae70ea1b0c4f0516cfc205f

6 years agologging: fix compilation error on older C++ compilers
Adam Simpkins [Sat, 2 Dec 2017 02:01:59 +0000 (18:01 -0800)]
logging: fix compilation error on older C++ compilers

Summary:
Add an explicit `std::move()` around the return value of
`logConfigToDynamic()`.  This explicit move is required pre-C++14,
but is not required after DR 1579:
http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_defects.html#1579

Newer versions of g++ and clang no longer require the move.  However gcc
versions earlier than 5.0 do require it.

Reviewed By: yfeldblum

Differential Revision: D6466447

fbshipit-source-id: 1b15934490d4966e9b3e5612e0e2ecbb43c979ca

6 years agologging: update initialization code to use the new LogConfig logic
Adam Simpkins [Sat, 2 Dec 2017 00:53:51 +0000 (16:53 -0800)]
logging: update initialization code to use the new LogConfig logic

Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6342086

fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242

6 years agounify FOLLY_SANITIZE existing uses and add a few new
Igor Sugak [Fri, 1 Dec 2017 19:19:44 +0000 (11:19 -0800)]
unify FOLLY_SANITIZE existing uses and add a few new

Reviewed By: yfeldblum

Differential Revision: D6453972

fbshipit-source-id: fa083c0b782188f5f45a2a354045aee03e528829

6 years agoimplement chrono conversions for unusual duration types
Adam Simpkins [Fri, 1 Dec 2017 19:18:17 +0000 (11:18 -0800)]
implement chrono conversions for unusual duration types

Summary:
Implement conversions between std::chrono::duration and POSIX-style time
structures even when neither the numerator nor the denominator of the duration
ratio are 1.

Both of these are done by first converting to an intermediate type where the
numerator is 1, and then using the conversion routines for that case.

Reviewed By: yfeldblum

Differential Revision: D6366647

fbshipit-source-id: 8f9495fb4101cac6d8b4cf0353a679107007b298

6 years agoMove threadlocal_detail::Atfork to its own file
Dave Watson [Fri, 1 Dec 2017 15:09:43 +0000 (07:09 -0800)]
Move threadlocal_detail::Atfork to its own file

Summary: As title

Reviewed By: yfeldblum

Differential Revision: D6440723

fbshipit-source-id: 3168d7bb616ae0ff3fe42f7584c5a255c4953875

6 years agofix ASAN build
Bin Liu [Thu, 30 Nov 2017 23:52:54 +0000 (15:52 -0800)]
fix ASAN build

Summary:
Fix bug in non-jemalloc build.

(Note: this ignores all push blocking failures!)

Reviewed By: igorsugak

Differential Revision: D6451342

fbshipit-source-id: 44725d4e0c685a59325fa3d08877d0f62120c5c2

6 years agoLet SaturatingSemaphore::try_wait be non-const and add ready
Yedidya Feldblum [Thu, 30 Nov 2017 22:56:07 +0000 (14:56 -0800)]
Let SaturatingSemaphore::try_wait be non-const and add ready

Summary:
[Folly] Let `SaturatingSemaphore::try_wait` be non-`const` and add `ready`.

For internal API consistency.

Reviewed By: magedm

Differential Revision: D6450089

fbshipit-source-id: 65b9b723672521710a69719b192eb2922a27b778

6 years agofix build when sanitizers are enabled and jemalloc is disabled
Igor Sugak [Thu, 30 Nov 2017 22:48:24 +0000 (14:48 -0800)]
fix build when sanitizers are enabled and jemalloc is disabled

Summary: Add preprocessor checks to guard jemalloc headers if any sanitizer is enabled.

Reviewed By: yfeldblum

Differential Revision: D6441578

fbshipit-source-id: 85364fca5af33c95cd05f232d1e3f9cbac9a4120

6 years agoSome fixes for custom conversions of enums
Yedidya Feldblum [Thu, 30 Nov 2017 06:13:39 +0000 (22:13 -0800)]
Some fixes for custom conversions of enums

Summary:
[Folly] Some fixes for custom conversions of enums.

Of note, `to` was defined for enum -> all conversions, including enum -> string conversions, but we actually want enum -> string conversions to be done via ADL-discovered toAppend.

Reviewed By: ot

Differential Revision: D6411250

fbshipit-source-id: 852b64309e6adf1c68e5153635cb29632e2d86d4

6 years agoSaturating semaphore
Maged Michael [Thu, 30 Nov 2017 05:12:16 +0000 (21:12 -0800)]
Saturating semaphore

Summary:
SaturatingSemaphore is a flag that allows:
- multiple concurrent posters
- multiple concurrent waiters
- idempotent posting
- non-destructive waiting
- blocking and spinning
- pre-block spin time control

```
/// SaturatingSemaphore is a flag that allows concurrent posting by
/// multiple posters and concurrent non-destructive waiting by
/// multiple waiters.
///
/// A SaturatingSemaphore allows one or more waiter threads to check,
/// spin, or block, indefinitely or with timeout, for a flag to be set
/// by one or more poster threads. By setting the flag, posters
/// announce to waiters (that may be already waiting or will check
/// the flag in the future) that some condition is true. Posts to an
/// already set flag are idempotent.
```

Reviewed By: djwatson

Differential Revision: D6379704

fbshipit-source-id: 59aed76caa2d159639e75425a778a9c63f18f375

6 years agoadd FOLLY_SANITIZE macro
Igor Sugak [Thu, 30 Nov 2017 02:14:04 +0000 (18:14 -0800)]
add FOLLY_SANITIZE macro

Summary: Most of the time we want to guard some code when any of the sanitizers are enabled not just one.

Reviewed By: yfeldblum

Differential Revision: D6429720

fbshipit-source-id: 7157a0a13f2b2acabd36a19c61b9fdbf38ba96b8

6 years agologging: add a LogConfig::update() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:21 +0000 (17:35 -0800)]
logging: add a LogConfig::update() method

Summary:
Add a method for merging the settings from two LogConfig objects.
This allows LogConfig objects to be merged before applying them to the
LoggerDB.  The effects are the same as two sequential LoggerDB::updateConfig()
calls, but without having to apply the intermediate state to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6342085

fbshipit-source-id: 0f8a1b7d8d195a80bc74342444dd3152d331fcb6

6 years agologging: add an assert for the FLAG_INHERIT bit
Adam Simpkins [Thu, 30 Nov 2017 01:35:19 +0000 (17:35 -0800)]
logging: add an assert for the FLAG_INHERIT bit

Summary:
Add a static_assert() to check that the FLAG_INHERIT bit does not conflict with
valid LogLevel values.

Reviewed By: bolinfest

Differential Revision: D6341242

fbshipit-source-id: cec284623c8a612a3c54f1b593d169310790616e

6 years agologging: remove comment about LogLevel::ERROR
Adam Simpkins [Thu, 30 Nov 2017 01:35:17 +0000 (17:35 -0800)]
logging: remove comment about LogLevel::ERROR

Summary:
The logging code used to support a LogLevel::ERROR value on non-Windows
platforms.  I removed it in D5288600 to make all platforms consistent, but I
forgot to remove this comment describing it.

Reviewed By: bolinfest

Differential Revision: D6341244

fbshipit-source-id: 054427d342066f354c859b12611b907dc2d4fa35

6 years agologging: add LoggerDB::updateConfig() and resetConfig()
Adam Simpkins [Thu, 30 Nov 2017 01:35:16 +0000 (17:35 -0800)]
logging: add LoggerDB::updateConfig() and resetConfig()

Summary: Add methods for applying config changes from a LogConfig object to the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200564

fbshipit-source-id: a25eb99e84b2885bf6853e2222db0d7432a6c37b

6 years agologging: add a LoggerDB::getConfig() method
Adam Simpkins [Thu, 30 Nov 2017 01:35:13 +0000 (17:35 -0800)]
logging: add a LoggerDB::getConfig() method

Summary: Add a method to get the current LogConfig state from the LoggerDB.

Reviewed By: bolinfest

Differential Revision: D6200596

fbshipit-source-id: 3bc57d498a5d25d19099d861376d71ea9f7e4039

6 years agologging: don't clamp the log level to DFATAL in debug builds
Adam Simpkins [Thu, 30 Nov 2017 01:35:11 +0000 (17:35 -0800)]
logging: don't clamp the log level to DFATAL in debug builds

Summary:
Remove the logic that clamps the log level to DFATAL in debug builds.

This behavior logically makes some sense, but results in subtle and potentially
confusing behavior changes.  It seems confusing that after calling
`setLevel(LogLevel::MAX_LEVEL)` on a log category, calling `getLevel()` on that
object would return `DFATAL` rather than `MAX_LEVEL`.  We also weren't clamping
the level to DFATAL consistently: `setLevel()` would clamp the value, but on
construction the initial level was still set to MAX_LEVEL rather than DFATAL.
This resulted in some issues when implementing `LoggerDB::getConfig()` since we
could not consistently detect if a log category was using the default level
settings.  Rather than fix this inconsistency it seems better to simply remove
this clamping behavior.

This means that it is possible for users to disable DFATAL log messages even in
debug builds if they really want to.  Previously this was only allowed in
release builds.  This protection doesn't really seem all that
valuable--presumably most developers won't really want to do this, and if they
really do request this configuration it doesn't seem all that bad to honor it.

Reviewed By: bolinfest, yfeldblum

Differential Revision: D6200569

fbshipit-source-id: 83ef8e2e4d3b61bc5b105038cbe3132979e9ac67