The Best Guess logo, a jar of colored marbles

BestGuess

Command-line benchmarking tool

Source code repository

News

  • 2026-09-05. BestGuess 0.8.0 released.
    • Command-line arguments can also be stored in a file. The -f <file> option reads additional arguments from <file>. This facilitates reliably executing the same benchmarking experiment repeatedly. It can also help with shell quoting issues when invoking BestGuess from a shell script.
    • A CSV reader bug was fixed. Bestguess currently supports two ways of including a double quote in a quoted field. The standard way is to write two consecutive double quote characters ("") , but some CSV libraries use backslash-quote instead (\").
    • Some “high scale” tests have been added, such as to execute two commands 50,000 times, and rank them, where ranking takes O(n^2) time. Use make NOHIGHSCALE=1 test to skip tests that take several minutes to complete.

  • 2024-12-06. BestGuess 0.7.5 released.

BestGuess

BestGuess is a tool for command-line benchmarking, sometimes called “macro-benchmarking” because entire programs are measured.

BestGuess does these things:

  1. Run commands, both warm-up runs and measured executions.
  2. Capture raw data, such as user, system, and wall-clock times, memory usage, context switches and RSS size.
  3. Save the raw data, for record-keeping or for analysis using bestreport or proper statistics tools.
  4. Show the data collected in various ways (both when executing an experiment with bestguess or reporting later using the collected data with bestreport). For example:
    • (Flag -M) Shows only total CPU time and wall clock time.
    • (Flag -G) Shows a sideways bar graph of total CPU time in the order that executions happened.
    • (Flag -B) Shows a (coarse) box plot on the terminal that summarizes the distribution of total CPU times for each command.
  5. Analyze the distribution of CPU times, showing these calculations:
    • (Flag -D) How much does the total CPU time distribution deviate from a normal distribution? (Anderson-Darling, skew, and kurtosis are measured.)
    • (Flag -T) What does the tail of the distribution look like?
    • (Flag -E) Explain the how the commands are ranked, showing the $U$ value of the Mann-Whitney test, the adjusted $p$ value, the Hodges-Lehmann $\Delta$ value, and the probability of superiority.
  6. Rank the benchmarked commands from fastest to slowest, showing when the fastest command is in a statistical tie with others.

It’s easy to use. For instance, run bestguess -r 100 'ps' 'ps A' to see how much slower ps runs when it needs to list all processes. The -r 100 specifies 100 timed executions.

Because performance distributions are rarely (if ever) regular, BestGuess uses non-parametric statistics to analyze the distribution of run times (total CPU time) for each command. See the help (bestguess -h) or the list above.