Elizabeth Smith [Thu, 10 Jul 2014 21:54:28 +0000 (14:54 -0700)]
Expression SFINAE issue in base.h
Summary:
MSVC 14 is still broken with expression sfinae - and the things that break are often strange
Moving this out into two expr templates solves the compilation issue
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: tjackson@fb.com
FB internal diff:
D1413297
Wez Furlong [Thu, 10 Jul 2014 18:37:17 +0000 (11:37 -0700)]
use recursive_mutex to protect State
Summary:
maelstrom destructs a Promise during an indirect call from
maybeCallback and deadlocks on itself unless we use a recursive mutex.
There may be a smarter way to proceed but at the moment I can't build
and deploy a package from trunk because the service is non-functional
:-/
Test Plan:
run
```
fbconfig -r folly/wangle messaging/maelstrom
fbmake runtests
```
Reviewed By: hannesr@fb.com
Subscribers: fugalh, fsilva
FB internal diff:
D1428504
Tudor Bosman [Wed, 9 Jul 2014 15:42:15 +0000 (08:42 -0700)]
Remove unnecessary and broken hash specializations
Summary:
No need to specialize std::hash for std::basic_string; the STL already does
this. Plus, the specializations were broken in multiple ways:
1. std::hash does not treat const char* specially, so it just hashes the
pointer value (contrasting the old __gnu_cxx::hash, which hashed a C-string)
2. Either way, using __gnu_cxx::hash<const char*> for std::string is broken,
as it won't hash anything past the first null byte.
Also, make sure fbstring gets the same (full) specializations as std::string
does in the standard.
Test Plan: fbconfig -r folly && fbmake runtests_opt, also tests in tupperware/agent which is still using __gnu_cxx::hash_map with string keys.
Reviewed By: pgriess@fb.com, andrei.alexandrescu@fb.com
Subscribers: njormrod, jhj, kma, lesha
FB internal diff:
D1426479
Gunjan Sharma [Wed, 9 Jul 2014 17:56:05 +0000 (10:56 -0700)]
Fix race in Notification Queue
Summary: When we are changing value of numActiveConsumers_ with setActive from handlerReady at the SCOPE_EXIT we have a race with reading of the same variable in putMessageImpl.
Test Plan: Had a local race which works fine now.
Reviewed By: davejwatson@fb.com
Subscribers: trunkagent
FB internal diff:
D1424674
Tudor Bosman [Tue, 8 Jul 2014 00:29:32 +0000 (17:29 -0700)]
Simplify pipe2 / O_CLOEXEC handling in Subprocess.cpp
Summary: per @rockyliu's suggestions
Test Plan: subprocess_test
Reviewed By: rockyliu4@fb.com
Subscribers: rockyliu, jhj, lesha, kma
FB internal diff:
D1423157
Paul Tarjan [Wed, 9 Jul 2014 14:07:38 +0000 (07:07 -0700)]
Revert "make `folly::Formatter` extendible"
Summary:
This reverts commit
5ea0eb0c705224b82a5c284dc5f7722e1f71199f.
This breaks the HHVM clang build test.
Test Plan: clang will pass now
Reviewed By: smith@fb.com
Subscribers: dmitrys
FB internal diff:
D1426399
Tasks:
4667712
Tyler MacDonald [Wed, 9 Jul 2014 00:27:36 +0000 (17:27 -0700)]
make `folly::Formatter` extendible
Summary:
on advice of @tudorb, move most of `folly::Formatter` into `folly::BaseFormatter` so that we can use compile-time polymorphism to provide different types of Formatters.
I wasn't able to get the recursive formatter to be polymorphic -- whenever I tried to convert `class FormatValue<Formatter<containerMode, Args...>, void>` into `class FormatValue<BaseFormatter...`, `FormatTest.cpp:Test(Format, Nested)` wouldn't compile because it couldn't find the template. @tudorb, if you have an easy fix for this, lmk, otherwise I'm (reluctantly) okay with requiring that `Formatter`s define their own nesting `FormatValue`.
phew. the last time I did this sort of metaprogramming was over 5 years ago in perl. Doing it in C++ is... interesting.
Test Plan: `fbconfig -r thrift folly cold_storage && fbmake dbg && fbmake runtests`
Reviewed By: tudorb@fb.com
Subscribers: tudorb, dgp
FB internal diff:
D1422343
Tasks:
4624268
Tudor Bosman [Sat, 5 Jul 2014 15:50:38 +0000 (08:50 -0700)]
Fix libc++ build errors
Summary:
Reported externally:
https://github.com/facebook/folly/issues/70
https://github.com/facebook/folly/issues/71
https://github.com/facebook/folly/issues/72
https://github.com/facebook/folly/issues/73
Note that I can't test on libc++ myself, but the reports suggested fixes which
sounded good.
Test Plan: fbconfig -r folly && fbmake runtests_opt
Reviewed By: marcelo.juchem@fb.com
Subscribers: jhj, ntv, lesha, kma, fugalh, jdelong
FB internal diff:
D1421029
Tudor Bosman [Mon, 7 Jul 2014 20:08:28 +0000 (13:08 -0700)]
Use pipe2 in Subprocess; platform-specific config
Summary:
We've wanted to use pipe2 in Subprocess for a while, but that's not supported
on glibc 2.5.1. This is not a problem in OSS (autoconf can solve this),
but, internally, we don't run autoconf; add an internal platform include file
with such per-platform differences.
Test Plan: fbconfig -r folly && fbmake runtests_opt
Reviewed By: meyering@fb.com
Subscribers: jhj, lesha, kma, agallagher
FB internal diff:
D1422128
Tom Jackson [Thu, 3 Jul 2014 22:28:27 +0000 (15:28 -0700)]
stride()
Summary: thatwaseasy
Test Plan: iloveunittests
Reviewed By: lucian@fb.com
Subscribers: philipp
FB internal diff:
D1419848
Tasks:
4636617
Brian Pane [Mon, 7 Jul 2014 17:27:30 +0000 (10:27 -0700)]
Move the branchless, parallel ASCII toLower into folly
Summary:
* Moved the fast toLower code into folly
* Updated the name to emphasize that it's ASCII-only
Test Plan: * Unit tests and benchmarks included.
Reviewed By: tudorb@fb.com
Subscribers: ruibalp, bmatheny
FB internal diff:
D1421056
Hans Fugal [Mon, 7 Jul 2014 16:01:47 +0000 (09:01 -0700)]
(wangle) fix shadow
Summary: Not sure why this shadow error didn't turn up in my tests, but whatevs
Test Plan: traffic manager builds without warning-errors
Reviewed By: suhas@fb.com
Subscribers: hannesr, net-systems@, fugalh, exa
FB internal diff:
D1421495
Tasks:
4653938
Tudor Bosman [Mon, 7 Jul 2014 15:59:17 +0000 (08:59 -0700)]
fix Makefile.am
Summary: wangle/detail.h -> wangle/detail/State.h
Test Plan: OSS build
Reviewed By: meyering@fb.com
Subscribers: fugalh, lesha
FB internal diff:
D1421490
Tudor Bosman [Fri, 4 Jul 2014 05:18:06 +0000 (22:18 -0700)]
gflags now likes namespace gflags, not google
Summary:
Use namespace gflags going forward, import namespace google into it if
backward.
Also added a few "checking for..." messages in autoconf.
Test Plan: fbconfig -r folly && fbmake runtests_opt, OSS build
Reviewed By: meyering@fb.com
Subscribers: fjargsto, ntv, jhj, lesha, kma, davejwatson
FB internal diff:
D1420575
Hans Fugal [Tue, 1 Jul 2014 00:29:05 +0000 (17:29 -0700)]
(wangle) cold via
Summary:
Instead of returning a Later, `via` returns a cold future.
This works without keeping a backreference like Later does, because an inactive Future will always activate on destruction. Alternatively we could have an extra Promise, a la Later, and pass that along like Later does, and require launch() at the end (though, implicit launching on destruction would be an option there too).
If you think this approach is viable I'll clean it up on Wednesday: make sure all the calling sites work, etc.
Test Plan:
new unit test
This may fail in contbuild, I haven't done the codemod for calling sites, if there are any.
Reviewed By: hannesr@fb.com
Subscribers: jsedgwick, net-systems@, fugalh, exa
FB internal diff:
D1412499
Tasks:
4480567
Hans Fugal [Mon, 30 Jun 2014 20:38:41 +0000 (13:38 -0700)]
(wangle) Future/Promise detachment
Summary:
Bring a bit more sanity to the lifetime. Now Future and Promise detach by calling the respective methods on State, and they do it during their destruction only. State manages its own lifetime.
Besides being cleaner, this also sets the stage for cancellation (where we'll need Future to hang on to its reference to State after setting the callback), and for folding in Later (we will have a bool for whether the future is hot and hold off executing the callback if it isn't).
Also cleans up various things I noticed while auditing the code for usages of `state_`.
Test Plan:
All the unit tests still pass.
Ran with ASAN (and found I had introduced a bug, then fixed it. yay)
Reviewed By: hannesr@fb.com
Subscribers: jsedgwick, net-systems@, fugalh, exa
FB internal diff:
D1412038
Tasks:
4618297
Hannes Roth [Fri, 4 Jul 2014 15:11:36 +0000 (08:11 -0700)]
(Folly/FBString) Enum class for category
Summary: Suggested by @aalexandre.
Test Plan: Compile.
Reviewed By: andrei.alexandrescu@fb.com
Subscribers: folly@lists, njormrod, aalexandre
FB internal diff:
D1246180
Lucian Grijincu [Thu, 26 Jun 2014 23:26:33 +0000 (16:26 -0700)]
folly: StringPiece: add skipWhitespace
Test Plan: copied from folly::json
Reviewed By: philipp@fb.com, soren@fb.com
FB internal diff:
D1417992
Tasks:
4527315
Hans Fugal [Mon, 30 Jun 2014 20:34:42 +0000 (13:34 -0700)]
s/setCallback_/setCallback/ for detail::State
Summary: This isn't really an internal-to-this-class method. :-/
Test Plan: builds
Reviewed By: hannesr@fb.com
Subscribers: jsedgwick, net-systems@, fugalh, exa
FB internal diff:
D1412008
Hans Fugal [Mon, 30 Jun 2014 20:32:17 +0000 (13:32 -0700)]
(wangle) s/continuation_/callback/ (missed some)
Summary: In the spirit of
D1406753
Test Plan: still builds and tests pass
Reviewed By: hannesr@fb.com
Subscribers: jsedgwick, net-systems@, fugalh, exa
FB internal diff:
D1412002
Hans Fugal [Mon, 30 Jun 2014 20:22:56 +0000 (13:22 -0700)]
(wangle) s/FutureObject/detail::State/
Summary:
codemod
I never was satisfied with this name. `State` feels a little better. More descriptive. But still pretty generic, so I'm open to alternative ideas.
Test Plan: It still builds and tests pass.
Reviewed By: hannesr@fb.com
Subscribers: jsedgwick, net-systems@, fugalh, exa
FB internal diff:
D1411506
Kevin Chou [Wed, 2 Jul 2014 16:14:24 +0000 (09:14 -0700)]
Fix string split when last part is shorter than delimiter
Summary: The bug is when ignoreEmpty is true, we use the wrong token size, which will be 0 when last token is shorter than delimiter.
Test Plan:
1. Add unit test
2. fbconfig -r folly && fbmake runtests
Reviewed By: philipp@fb.com
Subscribers: maxime, xiaol
FB internal diff:
D1415803
Tudor Bosman [Wed, 2 Jul 2014 01:03:26 +0000 (18:03 -0700)]
Add Random-inl.h to Makefile.am
Test Plan: OSS
Reviewed By: lesha@fb.com
FB internal diff:
D1415190
@override-unit-failures
Dave Watson [Fri, 27 Jun 2014 17:46:15 +0000 (10:46 -0700)]
update make check targets
Summary:
Update targets so that gtest isn't required unless you run 'make check'.
Simply make everything that uses gtest a 'check_PROGRAM' instead of a 'noinst_PROGRAM'
Test Plan: build folly with 'make -j32', gtest isn't required.
Reviewed By: andrewjcg@fb.com
Subscribers: pavlo, doug
FB internal diff:
D1408288
Tudor Bosman [Mon, 30 Jun 2014 23:12:47 +0000 (16:12 -0700)]
Simplify include path, make sure .. is in it
Summary: so #include <folly/File.h> works
Test Plan: built OSS folly, make check, built tiny program out of tree
Reviewed By: davejwatson@fb.com
FB internal diff:
D1412123
Tudor Bosman [Mon, 30 Jun 2014 18:56:53 +0000 (11:56 -0700)]
Codemod: use #include angle brackets in folly and thrift
Summary: Also changed the thrift compilers to emit code with <...>.
Test Plan: fbconfig -r folly thrift && fbmake opt && fbmake runtests_opt
Reviewed By: davejwatson@fb.com
Subscribers: ruibalp, nli, shilin, tjackson, fugalh, alandau, bmatheny, njormrod
FB internal diff:
D1411225
Philip Pronin [Sun, 29 Jun 2014 11:49:47 +0000 (04:49 -0700)]
fix AsyncIO::doWait
Summary:
As it turns out, `io_getevents` may actually return less than
`min_nr` events. According to the aio logic
(https://github.com/torvalds/linux/blob/
10b5b5361a3c2a7fff9dbfa0f127adc2531e7732/fs/aio.c#L1634),
there may be a couple of rounds required to get at least `nr_min` events, and
if interrupted after the first one, incomplete results would be returned
Test Plan:
fbconfig -r folly/experimental/io/test && fbmake runtests_opt -32
and was no longer able to repro #
4609062
Reviewed By: soren@fb.com
FB internal diff:
D1410389
Tasks:
4609062
Joel Marcey [Fri, 27 Jun 2014 21:47:54 +0000 (14:47 -0700)]
Be explicit about what we're passing to Endian::big
Summary:
This prevents a linker error on OSX:
```
Undefined symbols for architecture x86_64:
"folly::detail::EndianIntBase::swap(unsigned long)",
referenced from:
__GLOBAL__sub_I_MacAddress.cpp in libfolly.a(MacAddress.cpp.o)
ld: symbol(s) not found for architecture x86_64
```
We need folly and third-party changes in order to land a pull request for HHVM
that starts to get FastCGI running on OSX.
See the checklist of the HHVM pull request here: https://github.com/facebook/hhvm/pull/2944#issuecomment-
47281003
Closes #68
GitHub Author: Daniel Sloof <goapsychadelic@gmail.com>
@override-unit-failures
Test Plan: fbmake runtests 100%
Reviewed By: pt@fb.com, njormrod@fb.com
FB internal diff:
D1407426
Joel Marcey [Fri, 27 Jun 2014 21:46:19 +0000 (14:46 -0700)]
Fix folly on OSX and BSD in prep for FastCGI on HHVM
Summary:
A recent change in folly/MemoryMapping.cpp uses MAP_ANONYMOUS, which is
named MAP_ANON on OSX/BSD.
We need folly and third-party changes in order to land a pull request for HHVM
that starts to get FastCGI running on OSX.
See the checklist of the HHVM pull request here: https://github.com/facebook/hhvm/pull/2944#issuecomment-
47281003
Closes #67
GitHub Author: Daniel Sloof <goapsychadelic@gmail.com>
@override-unit-failures
Test Plan: fbmake runtests 100%
Reviewed By: pt@fb.com, njormrod@fb.com
FB internal diff:
D1407393
Hans Fugal [Thu, 26 Jun 2014 23:23:11 +0000 (16:23 -0700)]
(wangle) Return a Later from Future::via
Summary: Stroke of brilliance, Hannes.
Test Plan:
unit tests, including a new one
Looked through `fbgs 'via('` and all the extant `via`s are attached to `Later`s already so it shouldn't break anything. But check contbuild before commit.
Reviewed By: hannesr@fb.com
Subscribers: net-systems@, fugalh, exa
FB internal diff:
D1406976
Tasks:
4480567
Jim Meyering [Wed, 25 Jun 2014 23:47:55 +0000 (16:47 -0700)]
folly: futexWaitUntilImpl: avoid abort triggered via tao/queues tests
Summary:
* folly/test/DeterministicSchedule.cpp (folly): Tao/queues tests exposed
an unhandled case in this code: when (futex->data == expected) happens,
we would call futexErrnoToFutexResult with a futexErrno value of 0,
which would cause an abort (unhandled valued in switch).
Here's a stack trace from the abort: https://phabricator.fb.com/P12558008
showing the invalid futexErrno value in frame #4.
Test Plan:
fbconfig -r --platform-all=gcc-4.8.1-glibc-2.17 --sanitize=address tao/queues:TimeoutWorkQueueTests
_bin/tao/queues/WorkQueueTests -fbunit_test_regexp '^mt\_stress\_deterministic$'
Reviewed By: mssarang@fb.com
FB internal diff:
D1404572
Tasks:
4494871
Sarang Masti [Thu, 26 Jun 2014 22:37:13 +0000 (15:37 -0700)]
Fix use-after-free in futexWaitUntilImpl
Summary: Handle wake-ups correctly in futexWaitUntilImpl.
Test Plan:
-- ran all folly unit tests
-- ran TimeoutWorkQueue test under tao/queues
Reviewed By: ngbronson@fb.com, meyering@fb.com
FB internal diff:
D1406845
Tasks:
4494871
Hans Fugal [Thu, 26 Jun 2014 22:22:55 +0000 (15:22 -0700)]
(wangle) noexcept move constructor
Summary: to make lint happy
Test Plan: builds
Reviewed By: hannesr@fb.com
Subscribers: net-systems@, fugalh, exa
FB internal diff:
D1406766
Tasks:
4480567
Hans Fugal [Thu, 26 Jun 2014 22:20:40 +0000 (15:20 -0700)]
(wangle) s/continuation/callback
Summary: The word "continuation" is too ambiguous. Prefer callback (except where we are actually talking about CSP continuations in the README).
Test Plan: Still builds. Nothing external is or should be calling `setContinuation`. But I'll wait for contbuild anyway.
Reviewed By: hannesr@fb.com
Subscribers: net-systems@, fugalh, exa
FB internal diff:
D1406753
Tasks:
4480567
Hans Fugal [Thu, 26 Jun 2014 21:31:48 +0000 (14:31 -0700)]
nuke executeWith
Summary: Removing this crufty API. The only callsite for `executeWith` was `Later::via` so I just folded it in there.
Test Plan:
unit tests
contbuild
Reviewed By: hannesr@fb.com
Subscribers: net-systems@, fugalh, exa
FB internal diff:
D1406592
Tasks:
4480567
Hans Fugal [Thu, 26 Jun 2014 17:44:20 +0000 (10:44 -0700)]
Fix bizarre optimization
Summary:
I have no idea why the compiler is gettings its britches in a bunch because a unit test has a predictable output. Really, gcc?!
But whatever, avoiding inlining by putting it in the cpp file solves it.
Test Plan: fbmake runtests_opt
Reviewed By: davejwatson@fb.com
Subscribers: net-systems@, fugalh, exa
FB internal diff:
D1405811
Tasks:
4591823
Hans Fugal [Wed, 4 Jun 2014 22:38:24 +0000 (15:38 -0700)]
(folly) QueuedImmediateExecutor
Summary: Add the `QueuedImmediateExecutor` which behaves like `InlineExecutor` but with different (and usually better) ordering semantics for nested calls.
@override-unit-failures
Test Plan: unit tests
Reviewed By: davejwatson@fb.com
Subscribers: folly@lists, net-systems@, fugalh, exa
FB internal diff:
D1364904
Tasks:
3789661
Tudor Bosman [Tue, 24 Jun 2014 01:23:03 +0000 (18:23 -0700)]
Oops, fix README
Reviewed By: meyering@fb.com
Test Plan: no
Tudor Bosman [Sat, 21 Jun 2014 02:10:12 +0000 (19:10 -0700)]
folly OSS fixes: add ThreadName.h and compression
Summary: Also, optionalize dependencies on compression libraries.
Test Plan: fbconfig -r folly && fbmake runtests_opt
Reviewed By: meyering@fb.com
Subscribers: kma, jhj, simpkins, lesha, folly@lists
FB internal diff:
D1396573
Anton Likhtarov [Wed, 25 Jun 2014 01:52:54 +0000 (18:52 -0700)]
Fix for folly open source build on Ubuntu 12.04
Summary: On 12.04, there's both /usr/lib/libiberty.a and /usr/lib/libiberty_pic.a, and _pic is the one we want to build a Folly shared library.
Test Plan: build on Ubuntu 12.04
Reviewed By: meyering@fb.com
FB internal diff:
D1402194
Hans Fugal [Thu, 19 Jun 2014 01:03:45 +0000 (18:03 -0700)]
Scheduler interface of Executor
Summary: and ManualExecutor implementation
Test Plan: unit tests, contbuild
Reviewed By: davejwatson@fb.com
Subscribers: bmatheny, folly@lists, net-systems@, fugalh, exa, marccelani, jsedgwick
FB internal diff:
D1392999
Tasks:
4548494
Tudor Bosman [Sat, 21 Jun 2014 02:10:12 +0000 (19:10 -0700)]
folly OSS fixes: add ThreadName.h and compression
Summary: Also, optionalize dependencies on compression libraries.
Test Plan: fbconfig -r folly && fbmake runtests_opt
Reviewed By: meyering@fb.com
Subscribers: kma, jhj, simpkins, lesha, folly@lists
FB internal diff:
D1396573
Tudor Bosman [Fri, 20 Jun 2014 03:46:10 +0000 (20:46 -0700)]
Make randomNumberSeed read from /dev/urandom
Summary: Because @lesha asked "why not" and I couldn't give him an answer.
Test Plan: random_test
Reviewed By: bmaurer@fb.com
Subscribers: bmaurer, folly@lists, jhj, kma, lesha, sdoroshenko, soren
FB internal diff:
D1394401
Peter Ruibal [Sun, 22 Jun 2014 20:40:11 +0000 (13:40 -0700)]
Add ThreadName.h to folly's Makefile.am
Summary:
fbthrift depends on <folly/ThreadName.h>, which isn't currently
getting installed as part of the autotools build. Add it to Makefile.am
Test Plan:
Made this change to folly, re-autogen/configure/install, and
then was able to successfully compile fbthrift's compiler
Reviewed By: davejwatson@fb.com
Subscribers: doug, folly@lists
FB internal diff:
D1397084
Yunqi Zhang [Thu, 19 Jun 2014 01:40:19 +0000 (18:40 -0700)]
Expose EVLOOP_NONBLOCK
Summary:
This diff allows users to loop through EventBase without blocking if there are
not any events to process.
This is useful for sending and receiving requests on network, where users just
want to try if there are any events and do not want to block if not.
https://phabricator.fb.com/
D1373887 is an example where we find this feature
useful, otherwise we have to add an empty callback before loop.
event_base_.runInLoop([] {});
event_base_.loopOnce();
@davejwatson, @fugalh, @simpkins, @stepan: Could you please take a look at the
proposed changes and let me know if there is any better ways of doing this.
Thank you!
Test Plan:
I think this would not break anything, but we might want to do some performance
profiling if needed.
Reviewed By: hans@fb.com
Subscribers: simpkins, davejwatson, fugalh, stepan, folly@lists
FB internal diff:
D1383401
Nicholas Ormrod [Wed, 18 Jun 2014 16:35:10 +0000 (09:35 -0700)]
Make fbstring libgcc-safe
Summary:
Some libgcc-incompatible code has been added to fbstring.
Removed/reorganized it so that we can drop fbstring right into libgcc.
Test Plan:
fbconfig -r folly && fbmake runtests
Copied FBString.h into libgcc's basic_fbstring.h, with no modifications.
Successfully tp2_build libgcc/4.8.1. Adjusted symlink, then fbmake clean
&& fbconfig -r folly && fbmake dbg. The fbmake dbg failed with an
assertion error, which is consistent with @lucian's observations in
D1373725; the important part is that the error was at runtime, so the
compile-time changes of this diff looks good.
Reviewed By: lucian@fb.com
Subscribers: folly@lists, sdwilsh, njormrod, lucian
FB internal diff:
D1382873
Nicholas Ormrod [Sat, 14 Jun 2014 01:09:44 +0000 (18:09 -0700)]
FBString conservative additions
Summary:
Now that fbstring is conservative by default (
D1373308), we can remove
the mutability of the data members and the call to c_str() in operator[].
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: lucian@fb.com
Subscribers: folly@lists, sdwilsh, njormrod
FB internal diff:
D1382644
Anton Likhtarov [Sat, 14 Jun 2014 00:45:11 +0000 (17:45 -0700)]
Fix open source build
Test Plan: build it
Reviewed By: pavlo@fb.com
Subscribers: folly@lists
FB internal diff:
D1383722
Lucian Grijincu [Fri, 6 Jun 2014 21:48:15 +0000 (14:48 -0700)]
folly: fbstring: make it conservative-only: write '\0' in ctor and drop the c_str shenanigans
Test Plan: ran folly tests
Reviewed By: njormrod@fb.com
Subscribers: folly@lists, njormrod
FB internal diff:
D1373308
Nicholas Ormrod [Thu, 12 Jun 2014 18:11:55 +0000 (11:11 -0700)]
fbstring conservative corruption
Summary:
@lucian's
D1373308 set fbstring to conservative by default.
This breaks, eg, ti/proxygen/httpclient tests, by failing an assertion
inside of c_str(). Specifically, the terminator is not '\0'.
The fbstring_core move constructor, when sourced by a MediumLarge
string, steals the source's internal data, then resets the source by
calling ##setSmallSize(0)##. That function sets the in-situ size of the
fbstring to zero, thus requalifying the string as a small string;
however, it does nothing to the data - the previous 23 bytes now contain
garbage.
Sources of a move must be in a consistent state after the move is
complete. The source, once a MediumLarge string whose first eight bytes
were a pointer, is now a small string of size zero whose first byte is
not necessarily '\0'. This breaks the FBSTRING_CONSERVATIVE invariant.
This can be fixed by writing a terminator after the setSmallSize call.
I have fixed all setSmallSize locations that do not writeTerminator.
fbstring_core's move constructor is called exclusively from
basic_fbstring's move assignment operator, hence the odd format of the
test case.
== TMI ==
Interestingly, the source will almost certainly* contain a '\0', which
prevents this simple ##str.size() != strlen(str.c_str())## bug from
turning into a memory-trampling monster. The new size of zero is not
what saves us - the 'size' byte of a small fbstring, through a very
clever trick to allow 23-byte in-situ strings, is actually 23 minus the
actual size (now 0), so is 23! Where, then, does the '\0' byte come? A
MediumLarge string's data layout is [pointer, size, capacity]. The
pointer is not guaranteed to contain a '\0', and neither are size or
capacity. However, the size of the string needs to be very large in
order to force the top byte of the size member to be non-zero; in that
case, the string is so large that malloc is returning memory by the
page. Since page sizes are a multiple of 2^8 (almost always, and if not
then I don't think your fbstring can support large enough sizes
anyways), and we use goodMallocSize, the capacity pointer would have a
least signfigicant byte of zero.
Why the (*)? Well, when reserving extra space on a non-refcounted Large
string, the reallocation does not yield its extra goodMallocSize space.
This could be fixed, though probably isn't worth the trouble. Anyways,
since we aren't goodMallocSize-ing the user-supplied requested capacity,
it probably won't contain a '\0'.
Test Plan:
fbconfig -r folly && fbmake runtests
Modify folly/test/FBStringTest.cpp to define FBSTRING_CONSERVATIVE, then
fbconfig folly/test/:fbstring_test_using_jemalloc && fbmake runtests
Note that this fails before the patch is applied.
Note that it is possible for the tests to pass even when this bug is
present, since the top byte of the heap pointer must be non-0 in order
for this error to be triggered.
Reviewed By: lucian@fb.com
Subscribers: folly@lists, njormrod, lucian, markisaa, robbert, sdwilsh, tudorb, jdelong
FB internal diff:
D1376517
Nicholas Ormrod [Thu, 12 Jun 2014 00:34:12 +0000 (17:34 -0700)]
static-ify FBString asserts
Summary: Some asserts could be static_asserts. Make it so!
Test Plan: fbconfig -r folly && fbmke opt && fbmake runtests_opt
Reviewed By: lucian@fb.com
Subscribers: folly@lists, sdwilsh, njormrod
FB internal diff:
D1378670
Vojin Katic [Wed, 11 Jun 2014 20:53:16 +0000 (13:53 -0700)]
folly::gen::splitByLine
Summary:
I made it work, but please send your feedback how to improve code quality.
splitByLine will split on \r, \n, and \r\n.
Test Plan: add new test, arc unit
Reviewed By: tjackson@fb.com
Subscribers: folly@lists, crawler-diffs@
FB internal diff:
D1322212
Jim Meyering [Mon, 9 Jun 2014 20:32:26 +0000 (13:32 -0700)]
folly: do not disable RW_SPINLOCK_USE_X86_INTRINSIC_ for clang
Summary:
Without this, we'd see problems like this in tao, when building with clang:
With this change, this now works with clang-3.4 and clang.dev (3.4+).
This change reverts
D950285, which change appears to have been made
to accommodate weakness in clang-3.3 or older.
In file included from tao/data_providers/common/simpledp.cpp:7:
./tao/data_providers/common/stats.h:175:18: error: no type named 'RWTicketSpinLockT' in namespace 'folly'
typedef folly::RWTicketSpinLockT<64, true> RWLockType;
~~~~~~~^
./tao/data_providers/common/stats.h:175:35: error: expected member name or ';' after declaration specifiers
typedef folly::RWTicketSpinLockT<64, true> RWLockType;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Test Plan:
ensure that all of these pass:
fbconfig -r --clang folly/test:rw_spinlock_test && fbmake runtests
fbconfig -r --clang folly/test:rw_spinlock_test && fbmake runtests_opt
fbconfig -r folly/test:rw_spinlock_test && fbmake runtests_opt
fbconfig -r --clang --with-project-version clang:dev \
folly/test:rw_spinlock_test && fbmake runtests_opt
Reviewed By: delong.j@fb.com
Subscribers: folly@lists, mathieubaudet
FB internal diff:
D1370024
Tasks:
4090011
Zejun Wu [Mon, 9 Jun 2014 17:16:31 +0000 (10:16 -0700)]
Specialize string to identical string conversion
Summary:
Avoid copying underlying char array in to<string>(const string&) and
to<fbstring>(const fbstring&).
Test Plan: fbconfig -r $redteamisthebestteam/$nekomikoreimu && fbmake
Reviewed By: ldbrandy@fb.com
Subscribers: jonp, folly@lists
FB internal diff:
D1368183
Tasks:
4263125
Marcus Holland-Moritz [Fri, 6 Jun 2014 22:57:21 +0000 (15:57 -0700)]
Add support for returning number of benchmark iterations
Summary:
I'm looping through a large number of test cases in a benchmark and I'm
interested in the average time per test case rather than the total time,
which is what I'm currently seeing. In order to get the average time,
I need to be able to tell the benchmark module how many iterations have
been run.
This change adds _MULTI variants of the different BENCHMARK_ macros that
allow for returning the actual number of iterations that have been run,
e.g.:
BENCHMARK_MULTI(benchmarkSomething) {
std::vector<int> testCases { 0, 1, 1, 2, 3, 5 };
for (int c : testCases) {
doSomething(c);
}
return testCases.size();
}
Test Plan:
* fbconfig -r folly && fbmake runtests
* added new test cases to example benchmark code
Reviewed By: simpkins@fb.com
Subscribers: folly@lists, london_search@
FB internal diff:
D1356437
Philip Pronin [Fri, 6 Jun 2014 17:53:51 +0000 (10:53 -0700)]
revert "conditionally write terminator in c_str()"
Summary:
D1318048#21
@override-unit-failures
Test Plan: fbconfig -r folly && fbmake runtests_opt -j32
Reviewed By: njormrod@fb.com
Subscribers: folly@lists, njormrod
FB internal diff:
D1368939
Tasks:
4466412
Blame Revision:
D1318048
Stepan Palamarchuk [Thu, 5 Jun 2014 05:33:39 +0000 (22:33 -0700)]
Introduce destruction callbacks
Summary:
This change allows users to track lifetime of EventBase and perform clean shutdown when EventBase gets destructed.
It is useful for users that rely on EventBase lifetime, but don't have any feedback mechanism with the owner of EventBase.
For instance some part of code might remain running in background on the EventBase after the main object was destroyed (e.g. it might be finalizing some async requests). In such case the original owner doesn't know that there's something still running and may try to destroy EventBase. In that case such background code will remain zombie forever.
AsyncMcClient changes are presented just as an example of usage.
@davejwatson, @simpkins: Could you please take a look at the proposed changes for the EventBase? If this is something not worth adding into EventBase, could you recommend a better way of doing things?
Test Plan: fbmake runtests
Reviewed By: alikhtarov@fb.com
Subscribers: folly@lists, simpkins, davejwatson
FB internal diff:
D1353101
Marcelo Juchem [Tue, 3 Jun 2014 21:40:29 +0000 (14:40 -0700)]
Allowing additional arguments to be passed to split_step's functor
Summary: more flexibility for using functors with split_step
Test Plan: unit tests added + arc unit
Reviewed By: ldbrandy@fb.com
Subscribers: folly@lists
FB internal diff:
D1362644
Tudor Bosman [Mon, 2 Jun 2014 19:08:05 +0000 (12:08 -0700)]
Hasher and equality comparison for IOBuf
Test Plan: test added
Reviewed By: davejwatson@fb.com
Subscribers: folly@lists
FB internal diff:
D1359469
Matt Dordal [Tue, 3 Jun 2014 17:57:38 +0000 (10:57 -0700)]
Timed wait for futures
Summary:
It might be useful to be able to wait for some time (but not forever) on a
future, so this is a shot at doing that. It's a very heavyweight implementation, however.
Since the current interface for waitWithSemaphore doesn't really make sense if
the timeout fires, change it to return a Future<T>.
Test Plan: unit tests
Reviewed By: hans@fb.com
Subscribers: trunkagent, folly@lists, fugalh
FB internal diff:
D1358230
Jon Purdy [Sat, 31 May 2014 00:10:38 +0000 (17:10 -0700)]
Add missing make_unique overload.
Summary: C++14 adds this overload but I wanted it today.
Test Plan: It compiles, and this is the definition described in the standard.
Reviewed By: xning@fb.com
Subscribers: folly@lists
FB internal diff:
D1338839
Philip Pronin [Fri, 30 May 2014 07:05:28 +0000 (00:05 -0700)]
use NoInt() wrappers in FileUtil
Summary:
Accidentally spotted this problem. `folly/FileUtil.h` and
`common/files/FileUtil.h` are now using `*NoInt` wrappers where appropriate.
Test Plan: fbconfig -r common/files folly && fbmake opt -j32
Reviewed By: lucian@fb.com
Subscribers: folly@lists, fbcode-common-diffs@lists
FB internal diff:
D1356261
Tom Jackson [Fri, 30 May 2014 18:52:55 +0000 (11:52 -0700)]
Minor edit to comment
Summary: I was confused by it, thought rephrasing might help.
Test Plan: Read
Reviewed By: tudorb@fb.com
Subscribers: folly@lists
FB internal diff:
D1315612
Dave Watson [Fri, 11 Apr 2014 18:46:40 +0000 (11:46 -0700)]
Remove extraneous syscalls if NotificationQueue size() > 1
Summary:
Currently notification queue does 2 syscalls per item: one read, one write. We only need the eventfd to notify to wake up the thread, so instead, if the thread is already awake, don't bother writing to the fd.
Benchmark shows that when the queue size() > 1, this is ~4x faster.
Note that this might be unfair if there are multiple consumers: I could imagine a situation where one thread eats all the wakeups written to the fd, so only one thread is actually working. However, multiple consumers is a bad idea anyway, and I'd consider removing it entirely: If the same fd is in multiple epoll() loops, _all_ epolls will wake up, resulting in a thundering herd problem. I don't see any multiConsumer cases in fbcode
Using EFD_SEMAPHORE or not doesn't seem to matter, since hopefully we're only writing 1 wakeup per thread - and it wouldn't work at all for multiConsumer case.
Test Plan:
fbconfig thrift/lib/cpp/test:TNotificationQueueTest; fbamke runtests
fbconfig common/concurrency:QueueBenchmark
fbmake opt
QueueBenchmark --bm_min_iters=10000
Reviewed By: afrind@fb.com
Subscribers: doug, folly@lists, fbcode-common-diffs@lists, alandau, bmatheny, haijunz
FB internal diff:
D1272872
Tasks:
2802758
Jim Meyering [Thu, 29 May 2014 20:44:06 +0000 (13:44 -0700)]
folly/wangle: temporarily disable compilation of Thens.cpp
Summary:
This code fails to compile with clang:dev, so don't try for now.
* folly/wangle/test/Thens.cpp: Don't attempt to compile test/Thens.cpp.
See
4412111 for details. Prompted by clang:dev+MSAN effort,
4090011.
Test Plan:
Run this:
fbconfig --clang --with-project-version clang:dev -r folly/wangle
fbmake runtests
Failed before, passes with this patch.
Reviewed By: hans@fb.com
Subscribers: folly@lists, fugalh
FB internal diff:
D1354751
Tasks:
4090011,
4412111
Yedidya Feldblum [Sat, 24 May 2014 18:45:38 +0000 (11:45 -0700)]
Add shorthand functions to Format.h.
Summary:
[Folly] Add shorthand functions to Format.h.
This makes calling code simpler when it does not depend on the raw performance of writing to a stream.
Test Plan:
$ fbconfig -r folly/test
$ fbmake runtests
Reviewed By: tudorb@fb.com
Subscribers: folly@lists, dougw
FB internal diff:
D1347353
Tasks:
4391110
Matt Dordal [Fri, 23 May 2014 16:19:20 +0000 (09:19 -0700)]
add support for whenAll to waitWithSemaphore
Summary:
waitWithSemaphore currently doesn't support vector<Try<T>>, unless T is void.
Fix that, and also add a now-required void specialization.
Test Plan:
Add a test that uses vector<Try<bool>>, ensure that the tests compile
(and pass).
Reviewed By: hans@fb.com
Subscribers: folly@lists, fugalh
FB internal diff:
D1338528
Tasks:
4389473
Simon Martin [Wed, 21 May 2014 20:31:54 +0000 (13:31 -0700)]
Future::value() should throw when unset
Summary:
Added a test to call Future::value() before the Promise value is set, expecting an exception.
In a dbg build the test failed due on the assertion in Optional::value().
In a opt build the test failed due as no exception was thrown.
There are 2 points where we could throw our exception:
a) Optional::value() - replacing the assertion
b) Future::value()
I'm not sure which location makes the most sense.
With the assertion in Optional it seems that adding the throw here would not be unexpected but this is outside the wangle code.
So as a first pass I've added the throw in Future::value(), and made a new WangleException for this.
Test Plan:
$ fbconfig folly/wangle
$ fbmake runtests
Reviewed By: hans@fb.com
Subscribers: folly@lists, fugalh
FB internal diff:
D1340886
Daniil Burdakov [Wed, 21 May 2014 17:52:03 +0000 (17:52 +0000)]
added missing includes; also fixed lint issue with noexcept
Summary: subj
Test Plan: unit tests
Reviewed By: tjackson@fb.com
Subscribers: folly@lists
FB internal diff:
D1341693
Tudor Bosman [Tue, 20 May 2014 21:32:43 +0000 (14:32 -0700)]
More opensource build fixes
Summary:
- libtool version
- get rid of tiny libraries
- add folly/gen and a bunch of stuff from experimental
Test Plan: built, built a program against it in a ubuntu vm
Reviewed By: davejwatson@fb.com
Subscribers: folly@lists, fugalh
FB internal diff:
D1339920
Adam Simpkins [Wed, 14 May 2014 20:40:33 +0000 (13:40 -0700)]
make BucketedTimeSeries::addValue() honor old timestamps
Summary:
Previously BucketedTimeSeries()::addValue() documented that it required
time to move forwards. If it was ever called with a timestamp older
than the most recent one it had seen, it would just use latestTime_ as
the time, and add the value to the most recent bucket.
This changes addValue() so that it always uses the timestamp passed in
by the caller. If this time value refers to an old bucket that is still
being tracked, the data point will be added to that bucket. If the time
value is older than the oldest currently tracked bucket, the data point
will be ignored, and false will be returned.
I did consider leaving the current addValue() behavior as-is, and
requiring a separate addHistoricalValue() for when users intentionally
want to try adding old data points. However, it seems nicer to build
this into the existing addValue() function. The old behavior of just
replacing the supplied time value seems potentially surprising to users.
This does change the behavior of addValue(), and therefore could affect
the behavior of some programs. However, up until now no-one should have
been calling addValue() with old time values, as it wouldn't have done
what they want anyway. I did a brief search through our code base, and
all call sites I saw always called addValue() with the current time.
(Most of the callers use wall clock time, so this change might affect
program behavior if the system time changes after the program starts.
We should ideally change our programs to use monotonic clocks instead.)
Test Plan:
Included a new unit test.
Also compared the timeseries_benchmark results before and after this
change. Overall this new logic seems to be faster. For the "all time"
case, the new code is over 2x as fast. For the normal, non-all-time
case the new code is around 5% faster.
Reviewed By: hans@fb.com
Subscribers: doug, folly@lists, net-systems@, exa
FB internal diff:
D1338466
Tudor Bosman [Tue, 20 May 2014 15:49:16 +0000 (08:49 -0700)]
Some opensource build fixes
Summary:
- switch to new versions of ax_boost_*.m4
- versioning in libtool
- better checks in configure.ac
Test Plan: built in an Ubuntu VM
Reviewed By: davejwatson@fb.com
Subscribers: folly@lists
FB internal diff:
D1338957
Tudor Bosman [Mon, 19 May 2014 19:13:46 +0000 (12:13 -0700)]
Build up signal handler message before writing
Summary: So it doesn't interleave with whatever other threads write to stderr.
Test Plan: folly/experimental/symbolizer/test
Reviewed By: lucian@fb.com
Subscribers: folly@lists
FB internal diff:
D1337029
Matt Dordal [Mon, 19 May 2014 20:03:28 +0000 (13:03 -0700)]
fix waitWithSemaphore return type
Summary:
waitWithSemaphore always tried to return a value, which is not what the
underlying implementation did. If the value_type was an object, it would
fail to compile.
Test Plan: unit tests (added one to compile all the variants)
Reviewed By: hans@fb.com
Subscribers: folly@lists, fugalh
FB internal diff:
D1326916
Alex Landau [Thu, 15 May 2014 20:23:23 +0000 (13:23 -0700)]
Make EventHandler::isPending const
Summary: Because it just queries state
Test Plan: fbmake
Reviewed By: haijunz@fb.com
Subscribers: folly@lists
FB internal diff:
D1332397
Nicholas Ormrod [Thu, 15 May 2014 17:36:38 +0000 (10:36 -0700)]
Removed old FBVector compatibility functions
Summary: FBVector still has some code for gcc-4.6. Removed it.
Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig folly/test/stl_test && fbmake runtests (after enabling)
Reviewed By: robbert@fb.com
Subscribers: folly@lists, sdwilsh
FB internal diff:
D1320358
Rocky Liu [Tue, 13 May 2014 23:40:54 +0000 (16:40 -0700)]
Revert "[folly::Subprocess] Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations"
Summary: This reverts commit
c2f089cf080f2b3effa9efa5e4708b9674437d45.
Test Plan: Compile && folly::Subprocess unit tests
Reviewed By: tudorb@fb.com
FB internal diff:
D1327952
Rocky Liu [Tue, 13 May 2014 21:35:37 +0000 (14:35 -0700)]
Always #define _GNU_SOURCE to pull in pipe2() declarations
Summary: [folly::Subprocess] Always #define _GNU_SOURCE to pull in pipe2() declarations
Test Plan: Compile
Reviewed By: tudorb@fb.com
FB internal diff:
D1327004
Akshay Vaidya [Fri, 9 May 2014 17:07:05 +0000 (10:07 -0700)]
Adding a release function for ThreadLocalPtr.
Summary:
ThreadLocalPtr manages the lifecycle of the object that is
stored with it. We have a use case where we sometimes want to transfer ownership
of the stored object to another thread by wrapping them with
unique_ptrs. Adding a release function, similar to to the
unique_ptr::release is the cleanest way for us to transfer ownership.
Test Plan:
I can do some on off testing using a command line tool, but I
was wondering about how to add some unit tests. Not sure when the folly
unit tests were.
Reviewed By: njormrod@fb.com
FB internal diff:
D1321588
Rocky Liu [Tue, 13 May 2014 18:43:59 +0000 (11:43 -0700)]
Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations
Summary:
[folly::Subprocess] Set O_CLOEXEC by default when creating pipes to avoid race conditions resulting from concurrent Subprocess creations
If multiple threads are creating Subprocess objects concurrently, the
write side file descriptor of the pipe created in the parent process
might be inherited into other child processes unintentionally and never
closed, causing the parent process to hang while reading from the read
side of its pipe, thinking the other side must have been closed.
The fix to the problem is to create the pipes and set O_CLOEXEC in
a single pipe2 call. Then the child could clear the O_CLOEXEC flag
selectively before calling exec().
Test Plan:
Existing unit tests of Subprocess
Added a new unit test which will hang in Subprocess constructor without
this fix.
Reviewed By: tudorb@fb.com
FB internal diff:
D1267396
Matt Dordal [Tue, 13 May 2014 17:37:45 +0000 (10:37 -0700)]
add waitWithSemaphore to folly::wangle
Summary:
It may be useful to wait for a future to finish. This adds a utility function
to do so, returning a completed future.
NB: While it doesn't matter which thread executes the `then`, there does need
to be two threads. If not, this will deadlock forever. I'm not sure if there's
a way to detect/prevent that.
Test Plan: added some unit tests.
Reviewed By: hans@fb.com
FB internal diff:
D1319330
Elizabeth Smith [Tue, 13 May 2014 15:15:07 +0000 (08:15 -0700)]
missing exception
Summary: windows is also missing __throw_bad_alloc
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: delong.j@fb.com
FB internal diff:
D1318921
Nicholas Ormrod [Mon, 12 May 2014 18:02:00 +0000 (11:02 -0700)]
small_vector exception safety, part 2
Summary:
small_vector is now object-exception safe for all container
functions, except for input-iterators.
That's a bold claim; probably incorrect. At the very least, it passes
the same test suite as std::vector and fbvector.
Aside: Clearly, no one uses erase(q1, q2) in the wild.
Facebook: Nothing special.
Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig -r experimental/njormrod/stltest && fbmake runtests
Reviewed By: andrei.alexandrescu@fb.com
FB internal diff:
D1319787
Nicholas Ormrod [Mon, 12 May 2014 17:57:21 +0000 (10:57 -0700)]
small_vector exception safety, part 1
Summary:
small_vector is now object-exception safe for the general
container functions (N3337 table 96).
An amusing bug: in debug mode, swap would trigger an out-of-bounds
operator[] access. This has been fixed.
Facebook:
Nothing fancy in the non-OSS files.
Test Plan:
fbconfig -r folly && fbmake runtests
fbconfig -r experimental/njormrod/stltest && fbmake runtests
Reviewed By: delong.j@fb.com
FB internal diff:
D1319574
Hans Fugal [Fri, 9 May 2014 22:00:48 +0000 (15:00 -0700)]
(wangle) ManualExecutor::waitFor(F&&)
Summary:
@override-unit-failures
Test Plan: Using it in
D1322350
Reviewed By: hannesr@fb.com
FB internal diff:
D1322475
James Sedgwick [Fri, 9 May 2014 22:37:42 +0000 (15:37 -0700)]
makeFuture(Try<T>&&)
Summary:
@override-unit-failures
Add makeFuture variant which extracts the result contained in a Try and sticks it in a Future
One use case:
```
template <typename Result, typename Op, typename... Args>
Future<Result> wrapper(Op op, Args&&... args) {
// ... do some stuff before...
return op(std::forward<Args>(args)...).then([] (Try<Result>&& t) {
// ... do some stuff after...
return makeFuture<Result>(std::move(t));
});
}
```
With this makeFuture variant, "wrapper" doesn't need to be specialized for when
Result is void
Test Plan: employed in my code, will link to diff when ready
Reviewed By: hans@fb.com
FB internal diff:
D1318047
Nicholas Ormrod [Fri, 9 May 2014 21:33:26 +0000 (14:33 -0700)]
Delete small_vector's OneBitMutex policy
Summary:
It is unused.
Facebook: All hits from fbgs OneBitMutex have been deleted in this diff.
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: delong.j@fb.com
FB internal diff:
D1319624
Nicholas Ormrod [Thu, 8 May 2014 23:23:13 +0000 (16:23 -0700)]
Housekeeping
Summary:
Remvoed old fbvector folly/test/stl_test files.
Have kept StlVectorTest, since it is still impressive and useful.
Facebook: n/a
Test Plan:
enable StlVectorTest in the TARGETS
fbconfig -r folly && fbmake runtests
Reviewed By: robbert@fb.com
FB internal diff:
D1320254
Daniil Burdakov [Thu, 8 May 2014 11:57:20 +0000 (11:57 +0000)]
made folly::gen::member accept pointers to objects as well as references
Summary: subj
Test Plan: tests
Reviewed By: tjackson@fb.com
FB internal diff:
D1318719
Lucian Grijincu [Thu, 8 May 2014 18:50:50 +0000 (11:50 -0700)]
folly: fbstring: conditionally write terminator in c_str()
Summary:
c_str/data writes a terminator '\0' on each call. When
multiple threads call c_str/data on the same string (e.g. small global
constant) the string's cache line will bounce from cpu to cpu.
Cpus report that most instructions are stalled
(stalled-cycles-frontend/stalled-cycles-backend near 100%).
Fix: check before setting '\0'.
Test Plan:
** BEFORE **
- nothing defined ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
4.10s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.40s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:27:39.373522 28745 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 934.10ps 1.07G
privat_std_1t 100.32% 931.08ps 1.07G
static_fbs_1t 100.30% 931.33ps 1.07G
privat_fbs_1t 100.42% 930.22ps 1.08G
static_sp__1t 86986.52% 1.07ps 931.24G
privat_sp__1t 81459.50% 1.15ps 872.07G
static_std_32t 392.41ns 2.55M
privat_std_32t 15072.69% 2.60ns 384.10M
static_fbs_32t 102.41% 383.17ns 2.61M
privat_fbs_32t 13643.61% 2.88ns 347.68M
static_sp__32t 61032.25% 642.96ps 1.56G
privat_sp__32t 74172.72% 529.06ps 1.89G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
2417478.451740 task-clock # 27.005 CPUs utilized
65,374 context-switches # 0.027 K/sec
4,270 CPU-migrations # 0.002 K/sec
2,594 page-faults # 0.001 K/sec
5,261,195,623,299 cycles # 2.176 GHz [25.09%]
5,072,634,235,091 stalled-cycles-frontend # 96.42% frontend cycles idle [30.01%]
3,848,759,938,052 stalled-cycles-backend # 73.15% backend cycles idle [30.01%]
614,983,033,652 instructions # 0.12 insns per cycle
# 8.25 stalled cycles per insn [25.09%]
152,995,596,390 branches # 63.287 M/sec [30.01%]
24,501,018 branch-misses # 0.02% of all branches [25.94%]
66,298,370,215 L1-dcache-loads # 27.425 M/sec [30.03%]
1,642,188,850 L1-dcache-load-misses # 2.48% of all L1-dcache hits [30.02%]
968,142,977 LLC-loads # 0.400 M/sec [30.02%]
480,786,261 LLC-load-misses # 49.66% of all LL-cache hits [30.02%]
89.
520960614 seconds time elapsed
- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
4.25s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.45s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:24:26.487848 23720 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 932.63ps 1.07G
privat_std_1t 100.95% 923.85ps 1.08G
static_fbs_1t 100.04% 932.23ps 1.07G
privat_fbs_1t 101.14% 922.07ps 1.08G
static_sp__1t 30418.15% 3.07ps 326.16G
privat_sp__1t 56775.39% 1.64ps 608.77G
static_std_32t 388.57ns 2.57M
privat_std_32t 16465.70% 2.36ns 423.75M
static_fbs_32t 96.19% 403.98ns 2.48M
privat_fbs_32t 16248.69% 2.39ns 418.16M
static_sp__32t 81185.06% 478.63ps 2.09G
privat_sp__32t 81470.69% 476.95ps 2.10G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
2486009.171842 task-clock # 27.241 CPUs utilized
64,538 context-switches # 0.026 K/sec
4,501 CPU-migrations # 0.002 K/sec
2,863 page-faults # 0.001 K/sec
5,413,507,764,726 cycles # 2.178 GHz [25.62%]
5,207,550,928,974 stalled-cycles-frontend # 96.20% frontend cycles idle [30.01%]
3,919,627,007,209 stalled-cycles-backend # 72.40% backend cycles idle [30.01%]
685,365,706,158 instructions # 0.13 insns per cycle
# 7.60 stalled cycles per insn [25.62%]
148,798,151,350 branches # 59.854 M/sec [30.01%]
24,386,492 branch-misses # 0.02% of all branches [26.08%]
72,683,962,699 L1-dcache-loads # 29.237 M/sec [30.02%]
1,687,684,775 L1-dcache-load-misses # 2.32% of all L1-dcache hits [30.01%]
989,352,938 LLC-loads # 0.398 M/sec [30.02%]
484,825,665 LLC-load-misses # 49.00% of all LL-cache hits [30.02%]
91.
259265010 seconds time elapsed
- #define FBSTRING_CONSERVATIVE ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
0.18s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.43s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 17:30:58.246834 1571 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 1.44ps 692.61G
privat_std_1t 193.01% 748.06fs 1.34T
static_fbs_1t 399.47% 361.43fs 2.77T
privat_fbs_1t 129.29% 1.12ps 895.50G
static_sp__1t 78.99% 1.83ps 547.10G
privat_sp__1t 56.22% 2.57ps 389.36G
static_std_32t 478.67ps 2.09G
privat_std_32t 100.30% 477.26ps 2.10G
static_fbs_32t 100.52% 476.21ps 2.10G
privat_fbs_32t 100.52% 476.22ps 2.10G
static_sp__32t 99.57% 480.76ps 2.08G
privat_sp__32t 100.06% 478.40ps 2.09G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
198000.887222 task-clock # 14.738 CPUs utilized
5,016 context-switches # 0.025 K/sec
2,536 CPU-migrations # 0.013 K/sec
4,717 page-faults # 0.024 K/sec
421,850,242,358 cycles # 2.131 GHz [25.31%]
209,633,195,159 stalled-cycles-frontend # 49.69% frontend cycles idle [30.32%]
6,252,451,713 stalled-cycles-backend # 1.48% backend cycles idle [30.21%]
874,096,942,441 instructions # 2.07 insns per cycle
# 0.24 stalled cycles per insn [25.37%]
218,114,341,006 branches # 1101.583 M/sec [30.34%]
3,080,267 branch-misses # 0.00% of all branches [26.23%]
240,532,859 L1-dcache-loads # 1.215 M/sec [30.73%]
23,151,832 L1-dcache-load-misses # 9.63% of all L1-dcache hits [30.61%]
5,283,803 LLC-loads # 0.027 M/sec [30.60%]
1,079,973 LLC-load-misses # 20.44% of all LL-cache hits [30.60%]
13.
434898734 seconds time elapsed
- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
I0507 17:37:01.095785 28744 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 2.80ps 357.44G
privat_std_1t 118.65% 2.36ps 424.09G
static_fbs_1t 0.30% 937.44ps 1.07G
privat_fbs_1t 0.30% 924.33ps 1.08G
static_sp__1t 229.14% 1.22ps 819.03G
privat_sp__1t 212.22% 1.32ps 758.55G
static_std_32t 662.00ps 1.51G
privat_std_32t 134.15% 493.47ps 2.03G
static_fbs_32t 1.27% 52.24ns 19.14M
privat_fbs_32t 28.02% 2.36ns 423.31M
static_sp__32t 138.51% 477.94ps 2.09G
privat_sp__32t 138.39% 478.38ps 2.09G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
316918.699522 task-clock # 17.561 CPUs utilized
8,761 context-switches # 0.028 K/sec
2,190 CPU-migrations # 0.007 K/sec
5,361 page-faults # 0.017 K/sec
683,482,498,044 cycles # 2.157 GHz [25.13%]
488,046,572,692 stalled-cycles-frontend # 71.41% frontend cycles idle [30.18%]
269,765,079,972 stalled-cycles-backend # 39.47% backend cycles idle [30.17%]
756,213,442,723 instructions # 1.11 insns per cycle
# 0.65 stalled cycles per insn [25.17%]
175,643,190,326 branches # 554.222 M/sec [30.14%]
3,737,621 branch-misses # 0.00% of all branches [25.63%]
35,169,313,417 L1-dcache-loads # 110.973 M/sec [30.31%]
215,454,174 L1-dcache-load-misses # 0.61% of all L1-dcache hits [30.25%]
148,883,234 LLC-loads # 0.470 M/sec [30.25%]
48,768,585 LLC-load-misses # 32.76% of all LL-cache hits [30.26%]
18.
046902132 seconds time elapsed
** AFTER **
- define nothing ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
I0507 20:34:15.943284 16852 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 936.05ps 1.07G
privat_std_1t 100.67% 929.82ps 1.08G
static_fbs_1t 98.63% 949.07ps 1.05G
privat_fbs_1t 101.56% 921.69ps 1.08G
static_sp__1t 40342.12% 2.32ps 430.98G
privat_sp__1t 16370.35% 5.72ps 174.89G
static_std_32t 409.26ns 2.44M
privat_std_32t 16777.41% 2.44ns 409.95M
static_fbs_32t 17398.58% 2.35ns 425.13M
privat_fbs_32t 17475.47% 2.34ns 427.01M
static_sp__32t 85426.96% 479.07ps 2.09G
privat_sp__32t 85033.14% 481.29ps 2.08G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
1387268.222731 task-clock # 25.553 CPUs utilized
34,698 context-switches # 0.025 K/sec
2,770 CPU-migrations # 0.002 K/sec
3,039 page-faults # 0.002 K/sec
3,019,637,816,074 cycles # 2.177 GHz [25.21%]
2,801,192,631,479 stalled-cycles-frontend # 92.77% frontend cycles idle [30.02%]
2,020,857,118,698 stalled-cycles-backend # 66.92% backend cycles idle [30.00%]
801,090,224,478 instructions # 0.27 insns per cycle
# 3.50 stalled cycles per insn [25.20%]
206,099,842,649 branches # 148.565 M/sec [30.03%]
12,954,936 branch-misses # 0.01% of all branches [25.64%]
104,129,435,254 L1-dcache-loads # 75.061 M/sec [30.07%]
883,390,641 L1-dcache-load-misses # 0.85% of all L1-dcache hits [30.06%]
516,975,218 LLC-loads # 0.373 M/sec [30.04%]
255,887,523 LLC-load-misses # 49.50% of all LL-cache hits [30.04%]
54.
289185806 seconds time elapsed
- #define FBSTRING_PERVERSE ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
4.20s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.47s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 20:19:07.272021 18610 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 936.15ps 1.07G
privat_std_1t 100.84% 928.34ps 1.08G
static_fbs_1t 97.92% 956.06ps 1.05G
privat_fbs_1t 101.41% 923.14ps 1.08G
static_sp__1t inf% 0.00fs infinity
privat_sp__1t inf% 0.00fs infinity
static_std_32t 413.49ns 2.42M
privat_std_32t 17375.60% 2.38ns 420.22M
static_fbs_32t 17443.07% 2.37ns 421.85M
privat_fbs_32t 15354.32% 2.69ns 371.33M
static_sp__32t 82034.91% 504.05ps 1.98G
privat_sp__32t 63367.22% 652.53ps 1.53G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
1390538.848103 task-clock # 25.373 CPUs utilized
37,417 context-switches # 0.027 K/sec
3,147 CPU-migrations # 0.002 K/sec
2,876 page-faults # 0.002 K/sec
3,024,513,016,946 cycles # 2.175 GHz [25.08%]
2,813,021,108,191 stalled-cycles-frontend # 93.01% frontend cycles idle [30.02%]
2,043,124,392,473 stalled-cycles-backend # 67.55% backend cycles idle [30.01%]
774,663,686,661 instructions # 0.26 insns per cycle
# 3.63 stalled cycles per insn [25.09%]
197,666,485,664 branches # 142.151 M/sec [30.03%]
15,077,576 branch-misses # 0.01% of all branches [25.73%]
104,720,369,589 L1-dcache-loads # 75.309 M/sec [30.05%]
886,090,434 L1-dcache-load-misses # 0.85% of all L1-dcache hits [30.04%]
520,015,584 LLC-loads # 0.374 M/sec [30.03%]
256,990,100 LLC-load-misses # 49.42% of all LL-cache hits [30.04%]
54.
804099454 seconds time elapsed
- #define FBSTRING_CONSERVATIVE ** gcc-4.8.1-glibc-2.17-fb
$ perf stat --detailed _build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000
4.02s experimental/lucian/bench/StringTerminatorBenchmark.cpp
Linking _build/opt/experimental/lucian/bench/string_terminator_benchmark...
2.45s _build/opt/experimental/lucian/bench/string_terminator_benchmark
I0507 20:31:31.120209 10543 StringTerminatorBenchmark.cpp:98] --bm_min_iters=
100000000
============================================================================
experimental/lucian/bench/StringTerminatorBenchmark.cpprelative time/iter iters/s
============================================================================
static_std_1t 929.96ps 1.08G
privat_std_1t 100.37% 926.53ps 1.08G
static_fbs_1t inf% 0.00fs infinity
privat_fbs_1t inf% 0.00fs infinity
static_sp__1t inf% 0.00fs infinity
privat_sp__1t inf% 0.00fs infinity
static_std_32t 381.41ns 2.62M
privat_std_32t 16080.74% 2.37ns 421.62M
static_fbs_32t 80498.05% 473.81ps 2.11G
privat_fbs_32t 80368.84% 474.57ps 2.11G
static_sp__32t 80287.07% 475.05ps 2.11G
privat_sp__32t 80410.51% 474.33ps 2.11G
============================================================================
Performance counter stats for '_build/opt/experimental/lucian/bench/string_terminator_benchmark --bm_min_iters=
100000000':
1293727.223839 task-clock # 25.403 CPUs utilized
33,881 context-switches # 0.026 K/sec
3,718 CPU-migrations # 0.003 K/sec
3,829 page-faults # 0.003 K/sec
2,813,078,184,066 cycles # 2.174 GHz [25.11%]
2,597,561,032,630 stalled-cycles-frontend # 92.34% frontend cycles idle [30.04%]
1,883,652,860,583 stalled-cycles-backend # 66.96% backend cycles idle [30.02%]
800,150,465,648 instructions # 0.28 insns per cycle
# 3.25 stalled cycles per insn [25.10%]
197,970,559,157 branches # 153.023 M/sec [30.02%]
15,385,773 branch-misses # 0.01% of all branches [25.95%]
36,541,946,374 L1-dcache-loads # 28.245 M/sec [30.08%]
813,994,917 L1-dcache-load-misses # 2.23% of all L1-dcache hits [30.07%]
467,241,616 LLC-loads # 0.361 M/sec [30.07%]
247,980,582 LLC-load-misses # 53.07% of all LL-cache hits [30.08%]
50.
928965545 seconds time elapsed
Reviewed By: njormrod@fb.com
FB internal diff:
D1318048
Dave Watson [Wed, 7 May 2014 19:58:31 +0000 (12:58 -0700)]
Fix buid, missing getHugePageSizeForDevice
Summary:
Folly jenkins build has been failing for a couple days: http://ci-builds.fb.com/job/folly/
Due to a dep on an experimental file in
D1307044. Added file to Makefile.am, also needs boost_filesystem, added configure and link check for that too.
Test Plan: build works on ubuntu
Reviewed By: alandau@fb.com
FB internal diff:
D1316855
Elizabeth Smith [Thu, 8 May 2014 15:20:48 +0000 (08:20 -0700)]
Expression SFINAE fixes in ApplyTuple
Summary:
MSVC does not support Expression SFINAE
http://stackoverflow.com/questions/
12654067
this is a very nice c++11 feature that makes for some nice clean templating
But of course MSVC can't have nice things - it partially implements this when it feels like it, so some will work and some will need the nonsense
@override-unit-failures
There will be more of these little template helper fixes - they make the code a bit more complex but don't actually change anything when compiled
The accompanying fix in the test also does nothing but work around an MSVC compiler bug where it loses it's mind over the global namespace
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: delong.j@fb.com
FB internal diff:
D1312700
Marc Celani [Sat, 3 May 2014 01:17:32 +0000 (18:17 -0700)]
try_and_catch
Summary: A helper function to do try/catch on multiple exception types and store the ressult in an exception_wrapper. The purpose of this function is to best effort mimic std::current_exception(). Unlike std::current_exception(), we need to specify the types that we expect to see. Rather than writing macros or several lines per exception type to capture the exception into an exception_wrapper, this function makes writing try/catch blocks for this purpose very easy.
Test Plan: unit test
Reviewed By: mhorowitz@fb.com
FB internal diff:
D1308511
@override-unit-failures
Alexey Spiridonov [Fri, 25 Apr 2014 22:38:38 +0000 (15:38 -0700)]
Small readability improvements
Summary: I got confused by the error message, and wasn't quite clear on the intent of communicateIOBuf()
Test Plan: waiting for auto-unittests
@override-unit-failures
Reviewed By: tudorb@fb.com
FB internal diff:
D1297525
Philip Pronin [Tue, 6 May 2014 22:53:39 +0000 (15:53 -0700)]
fix IOBuf self move-assignment
Summary: Properly handle `this == &other` case.
Test Plan: fbconfig -r folly/test && fbmake runtests_opt -j32
Reviewed By: simpkins@fb.com
FB internal diff:
D1314916
Marc Celani [Wed, 7 May 2014 03:43:41 +0000 (20:43 -0700)]
Fix rebase fail
Summary: whoops
Test Plan: reran unit tests
Reviewed By: davejwatson@fb.com
Blame Rev:
Marc Celani [Wed, 7 May 2014 02:05:02 +0000 (19:05 -0700)]
Allow for folly::exception_wrapper in ClientReceiveState
Summary:
This diff allows us to use folly::exception_wrapper in ClientReceiveState. Existing use cases are still supported (crs.exception() still returns an exception_ptr no matter what), but we can now choose to set an exception without throwing first.
On the folly side, add some new functions for making an exception_ptr from an exception_wrapper.
Test Plan: Reran unit tests
Reviewed By: davejwatson@fb.com
FB internal diff:
D1307027
@override-unit-failures
Elizabeth Smith [Wed, 7 May 2014 00:41:09 +0000 (17:41 -0700)]
Benchmark specific fixes
Summary:
benchmark is necessary to get the tests running
primarily replacing a few inline asm items with MSVC intrinsics and do not optimize tricks
also a fix for use of the gcc specific ## with __VAR_ARGS__
although that is primarily intended as a workaround for trailing commas, gcc apparently cuts off all items in the macro afterwards
this was being used for a clever/dirty trick to do one or none for a macro
replaced instead with a version that will work for all other compilers (might need to be expanded for more args, but for now it's working with all current usage)
@override-unit-failures
also fixed a use of max without specifying a template type that was making msvc barf - specifying it (as in the min case) made it compile cleanly
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: andrei.alexandrescu@fb.com
FB internal diff:
D1313609
Hans Fugal [Wed, 30 Apr 2014 16:40:08 +0000 (09:40 -0700)]
race in Future destructor
Summary:
@mnd wrote some Wangle code that would trip up with a `bad_function_call` exception. This Shouldn't Happen™ but the exception comes from trying to call a `std::function` which is null. We pretty thoroughly examined his usage and didn't find any problems, and this patch seems to make the error go away. See #
4207781 for more details.
And reasoning about it, it makes sense. Inline comments explain the race.
Test Plan: Alas, I haven't been able to get a minimal repro and therefore a regression unit test. It's a hard race to trigger. I still don't understand why Matt's code does it.
Reviewed By: davejwatson@fb.com
FB internal diff:
D1304001
Elizabeth Smith [Tue, 6 May 2014 18:44:31 +0000 (11:44 -0700)]
printf format checking for msvc
Summary:
sal annotations can be used to do similar (not exact) checking to the functionality provided by the format attribute in gcc
the annotations are done by prefixing the format string with a value which makes the macro definitions a bit messy
@override-unit-failures
Test Plan: fbconfig -r folly && fbmake runtests
Reviewed By: delong.j@fb.com
FB internal diff:
D1313653
projects / folly.git / log