+The docker image `tsan11-tsan11rec-image.tar.gz` contains prebuilt LLVMs for tsan11 and tsan11rec. Tsan11 was built based on the instructions given at: https://github.com/mc-imperial/tsan11/. And the source code for tsan11rec was obtained from https://github.com/ChrisLidbury/tsan11/ with commit hash `ecbd6b8`.
+
+## Building the docker container and sshing in:
+
+In the repository root:
+
+```bash
# Load image:
docker image load -i tsan11-tsan11rec-image.tar.gz
# Create the container:
docker create -it --name tsan11-tsan11rec-container tsan11-tsan11rec-image:asplos
-# Start the container
+# Start the container:
docker start -ia tsan11-tsan11rec-container
+```
+
+Then in a different terminal in the same directory:
+```bash
# To find the IP address of the container:
-$ docker inspect tsan11-tsan11rec-container
+docker inspect tsan11-tsan11rec-container
+
+# Then copy the `scripts` folder in this directory
+# to the container (replace 172.17.0.2 by the container’s IP address):
+scp -i insecure_key -r scripts root@172.17.0.2:/data
+
+# ssh into the container as root (replace 172.17.0.2 by the container’s IP address):
+ssh -i insecure_key root@172.17.0.2
+```
+
+## Compiling benchmarks for tsan11 and tsan11rec:
+
+Inside the docker container:
+```bash
+cd /data/scripts
+
+# Start the shell that downloads and compiles benchmarks
+./setup.sh
+```
+
+The `setup.sh` will create two benchmark directories `/data/tsan11-benchmarks` and `/data/tsan11rec-benchmarks` for tsan11 and tsan11rec respectively.
+
+## To run benchmarks for tsan11 and tsan11rec:
+The benchmark directories `tsan11-benchmarks` and `tsan11rec-benchmarks` have the same structures. So we will use `tsan11-benchmarks` as an example.
+
+Our benchmarks fall into three categories: application benchmarks, data structure benchmarks (CDSChecker data structure benchmarks) used to evaluate CDSChecker, and data structure benchmarks with injected bugs that both tsan11 and tsan11rec miss.
+
+In the `tsan11-benchmarks` directory, the application benchmarks include Gdax (`gdax-orderbook-hpp`), Iris (`iris`), Mabain (`mabain`), Silo (`silo`), and the Javascript Engine of Firefox that runs Jsbench (`jsbench-2013.1`). The `tsan11-benchmarks` repository does not contain the Javascript Engine of Firefox, but the setup scripts download Firefox release 50.0.1, compile the Javascript Engine, and copy the Javascript Engine binary into the benchmark directory (`tsan11-benchmarks/js`) by running the script `build_firefox_jsshell.sh`.
+
+The `tsan11-benchmarks/cdschecker_modified_benchmarks` directory contains data structure benchmarks used to evaluate CDSChecker. The `tsan11-benchmarks/tsan11-missingbug` directory contains data structure benchmarks with injected bugs that tsan11 and tsan11rec fail to detect.
+
+After the setup, some scripts are copied into the `tsan11-benchmarks` directory, including a unified script `do_test_all.sh` that runs all tests.
+
+```bash
+# To run all tests
+cd /data/tsan11-benchmarks
+./do_test_all.sh
+```
+
+We also provide instructions on how to run each individual test below.
+
+### To run application benchmarks:
+
+```bash
+ cd /data/tsan11-benchmarks
+ ./app_test_all.sh [number of runs]
+```
+
+The `tsan11-benchmarks/app_test_all.sh` script runs all of five application benchmarks in both the all-core and single-core configurations. The `app_test_all.sh` script also accepts an integer as an optional parameter that specifies how many times each application benchmark is run, such as `./app_test_all.sh 5`. It runs all of five application benchmarks 10 times by default. After finish running the application benchmarks, the `app_test_all.sh` script executes `python calculator.py all-core` or `python calculator.py single-core` in the `tsan11-benchmarks` directory to print out results.
+
+### To run CDSChecker data structure benchmarks:
+
+```bash
+ cd /data/tsan11-benchmarks/cdschecker_modified_benchmarks
+ ./test_all.sh
+```
+
+The `cdschecker_modified_benchmarks/test_all.sh` script tests seven data structures for 500 runs and reports data race detection rates and execution time for each data structure. The results are printed in the console.
+
+### To run data structure benchmarks with bugs that tsan11 and tsan11rec miss:
+
+```bash
+ cd /data/tsan11-benchmarks/tsan11-missingbug
+ ./test_all.sh
+```
+
+The `tsan11-missingbug/test_all.sh` script tests two buggy data structure implementations for 1000 runs and reports assertion detection rates and execution time for each data structure. The results are printed in the console.
+
+### To test assertion failures in Silo and Mabain:
-# Then use scp to copy the scripts directory in the tsan11-tsan11rec-docker repository to the container (replace 172.17.0.2 by the container’s IP address):
-$ scp -i insecure\_key -r scripts root@172.17.0.2:/data
+```bash
+ cd /data/tsan11-benchmarks/
+ ./app_assertion_test.sh [number of runs]
+```
-# Logging into the container as root (replace 172.17.0.2 by the container’s IP address):
-$ ssh -i insecure\_key root@172.17.0.2
+The `app_assertion_test.sh` script also accepts an optional integer parameter that specifies how many times Silo and Mabian are tested. Both benchmarks are tested 10 time for assertion failures by default. The assertion detection rates are reported and printed in the console. Both tsan11 and tsan11 detect the assertion failures in Mabain but fail to do so in Silo.
projects / tsan11-tsan11rec-docker.git / commitdiff