To update perf expectations run the DEPS'ed in from chromium tools.

=======
Example
=======

Suppose we have the following line in nacl_perf_expectations.json

{...
"nacl-lucid-32-bare-newlib-opt/nacl-perf/browser_startup_time.out/Total_x86-32_nnacl_newlib_static_irt": {"reva": 5823, "revb": 6098, "type": "absolute", "improve": 147, "regress": 227, "sha1": "0d2c4397"},
}

The JSON dictionary key is roughly "bot/test-suite/graph/trace".
Thus, this refers to the lucid-32 newlib bot, which is measuring browser
startup time. Startup time is expected to run somewhere between
147 (improve) and 227 (regress) milliseconds (lower is better).
This range is based on samples from "reva" (svn -r 5823) and "revb" (6098).

If at svn rev 7000 we improve startup time beyond 147 milliseconds, the bot
will emit a warning. We can "lock in" the improvement by changing "reva"
beyond 7000 and re-running the "make_expectations.py" tool to grab the new
values (it leaves some padding by guessing the variance).

Update procedure, from this directory:

(1) update reva to 7000 and revb to 7010.

(2) RUN THIS COMMAND:
    ../perf_expectations/make_expectations.py -c ./nacl_perf_expectations.cfg

    This will poll the bots for samples and update the json file.
    TIP: "cat README", then copy paste the command line!

(3) make a CL to check in the new nacl_perf_expectations.json file.


Utilities
=========

A handy tool to update *all* perf expectations matching a regex is
"update_perf_expectations.py", found under tools/perf_expectations.
For example, to set reva and revb for TestCondvarSignalNoOp over
all bots (Windows, Mac, Linux) x (32-bit, 64-bit), run the following

  python ../perf_expectations/update_perf_expectations.py \
    -f .*TestCondvarSignalNoOp.* \
    --reva <new_reva> --revb <new_revb> \
    --config ./nacl_perf_expecations.cfg

That will only reset the reva and revb fields. To actually poll the
bots for the improve and regress values, use the usual command:

  ../perf_expectations/make_expectations.py -c ./nacl_perf_expectations.cfg


More information
================

For more info on the format see:
http://dev.chromium.org/developers/testing/chromium-build-infrastructure/performance-test-plots#TOC-Format-of-expectations-file


