I have setup a benchmark to measure running time for real and synthetic Sage programs that use Pynac. We will use these to see how the following iterations of the project will compare to the previous ones.
For benchmarking, Burcin (the project mentor) suggested to use pybench benchmarking suite. It is a small easy to use program. It repeats the benchmark multiple times and computes the mean and minimum timings. Each program can be repeated multiple times to avoid noise when measuring the time. For each program the overhead timing of the initialisation is measured separately and subtracted from the whole program, to reflect what actually needs to be measured. The results from different runs can be compared easily. New tests can be added to the benchmark.
The benchmark programs for Pynac are selected from those on the Sage wiki page symbenchand adapted from Arithmetic.py from the original pybench test suite. While adapting Arithmetic.py program to use symbolics, we encountered an infinite loop error when dividing Python longs. We’ve added ticket #13107 and a patch that solves the issue.
Benchmark repository https://github.com/titusnicolae/pybench
Benchmark output https://github.com/titusnicolae/pybench/blob/master/output1
