benchbuild.experiments package

Experiments module.

By default, only experiments that are listed in the configuration are loaded automatically. See configuration variables:

*_PLUGINS_AUTOLOAD *_PLUGINS_EXPERIMENTS

benchbuild.experiments.discover()

Import all experiments listed in PLUGINS_EXPERIMENTS.

Tests:

>>> from benchbuild.settings import CFG
>>> from benchbuild.experiments import discover
>>> import logging as lg
>>> import sys
>>> l = lg.getLogger('benchbuild')
>>> lg.getLogger('benchbuild').setLevel(lg.DEBUG)
>>> lg.getLogger('benchbuild').handlers = [lg.StreamHandler(stream=sys.stdout)]
>>> CFG["plugins"]["experiments"] = ["benchbuild.non.existing", "benchbuild.experiments.raw"]
>>> discover()
Could not find 'benchbuild.non.existing'
ImportError: No module named 'benchbuild.non'
Found experiment: benchbuild.experiments.raw

Subpackages

• benchbuild.experiments.polly package
  • Submodules
  • benchbuild.experiments.polly.openmp module
    • Measurements
  • benchbuild.experiments.polly.openmpvect module
    • Measurements
  • benchbuild.experiments.polly.polly module
    • The ‘polly’ Experiment
      • Measurements
  • benchbuild.experiments.polly.pollyperformance module
    • The ‘polly’ Experiment
      • Measurements
  • benchbuild.experiments.polly.vectorize module
    • The ‘polly-vectorize’ Experiment
      • Measurements

Submodules

benchbuild.experiments.compilestats module

The ‘compilestats’ Experiment.

This experiment is a basic experiment in the benchbuild study. It simply runs all projects after compiling it with -O3 and catches all statistics emitted by llvm

class benchbuild.experiments.compilestats.CompilestatsExperiment(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The compilestats experiment.

NAME = 'cs'

actions_for_project(p)

class benchbuild.experiments.compilestats.PollyCompilestatsExperiment(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The compilestats experiment with polly enabled.

NAME = 'p-cs'

actions_for_project(p)

benchbuild.experiments.compilestats.collect_compilestats(project, experiment, config, clang, **kwargs)

Collect compilestats.

benchbuild.experiments.compilestats.get_compilestats(prog_out)

Get the LLVM compilation stats from :prog_out:.

benchbuild.experiments.compilestats_ewpt module

The ‘compilestats_ewpt’ Experiment.

Gathers compilation statistics for compiling the project with the EWPT alias analysis enabled.

class benchbuild.experiments.compilestats_ewpt.EWPTCompilestatsExperiment(projects=None, group=None)

Bases: benchbuild.experiments.compilestats.CompilestatsExperiment

NAME = 'ewpt'

extra_cflags()

benchbuild.experiments.empty module

The ‘empty’ Experiment.

This experiment is for debugging purposes. It only prepares the basic directories for benchbuild. No compilation & no run can be done with it.

class benchbuild.experiments.empty.Empty(projects=None, group=None)

Bases: benchbuild.experiment.Experiment

The empty experiment.

NAME = 'empty'

actions_for_project(p)

Do nothing.

benchbuild.experiments.papi module

PAPI based experiments.

These types of experiments (papi & papi-std) need to instrument the project with libbenchbuild support to work.

class benchbuild.experiments.papi.PapiScopCoverage(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

PAPI-based dynamic SCoP coverage measurement.

NAME = 'papi'

actions_for_project(p)

Create & Run a papi-instrumented version of the project.

This experiment uses the -jitable flag of libPolyJIT to generate dynamic SCoP coverage.

run()

Do the postprocessing, after all projects are done.

class benchbuild.experiments.papi.PapiStandardScopCoverage(projects=None, group=None)

Bases: benchbuild.experiments.papi.PapiScopCoverage

PAPI Scop Coverage, without JIT.

NAME = 'papi-std'

actions_for_project(p)

Create & Run a papi-instrumented version of the project.

This experiment uses the -jitable flag of libPolyJIT to generate dynamic SCoP coverage.

benchbuild.experiments.papi.collect_compilestats(project, experiment, clang, **kwargs)

Collect compilestats.

benchbuild.experiments.papi.get_compilestats(prog_out)

Get the LLVM compilation stats from :prog_out:.

benchbuild.experiments.pjtest module

A test experiment for PolyJIT.

This experiment should only be used to test various features of PolyJIT. It provides only 1 configuration (maximum number of cores) and tests 2 run-time execution profiles of PolyJIT:

1. PolyJIT enabled, with specialization
2. PolyJIT enabled, without specialization

class benchbuild.experiments.pjtest.Test(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj-test'

actions_for_project(p)

class benchbuild.experiments.pjtest.TestReport(exp_ids, outfile)

Bases: benchbuild.reports.Report

SUPPORTED_EXPERIMENTS = ['pj-test']

generate()

report()

benchbuild.experiments.pjtest.time_polyjit_and_polly(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given binary wrapped with time.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit module

The ‘polyjit’ experiment.

This experiment uses likwid to measure the performance of all binaries when running with polyjit support enabled.

class benchbuild.experiments.polyjit.Compilestats(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

Gather compilestats, with enabled JIT.

NAME = 'pj-cs'

actions_for_project(p)

class benchbuild.experiments.polyjit.PJITRaw(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj-raw'

actions_for_project(p)

class benchbuild.experiments.polyjit.PJITRegression(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

This experiment will generate a series of regression tests.

This can be used every time a new revision is produced for PolyJIT, as it will automatically collect any new SCoPs detected, using the JIT.

The collection of the tests itself is intgrated into the JIT, so this experiment looks a lot like a RAW experiment, except we don’t run anything.

NAME = 'pj-collect'

actions_for_project(p)

class benchbuild.experiments.polyjit.PJITlikwid(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that uses likwid’s instrumentation API for profiling.

This instruments all projects with likwid instrumentation API calls in key regions of the JIT.

This allows for arbitrary profiling of PolyJIT’s overhead and run-time

NAME = 'pj-likwid'

actions_for_project(p)

class benchbuild.experiments.polyjit.PJITpapi(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

Experiment that uses PolyJIT’s instrumentation facilities.

This uses PolyJIT to instrument all projects with libPAPI based region measurements. In the end the region measurements are aggregated and metrics like the dynamic SCoP coverage are extracted.

This uses the same set of flags as all other PolyJIT based experiments.

NAME = 'pj-papi'

actions_for_project(p)

run()

Do the postprocessing, after all projects are done.

class benchbuild.experiments.polyjit.PJITperf(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that uses linux perf tools to generate flamegraphs.

NAME = 'pj-perf'

actions_for_project(p)

class benchbuild.experiments.polyjit.PolyJIT(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The polyjit experiment.

actions_for_project(p)

classmethod init_project(project)

Execute the benchbuild experiment.

We perform this experiment in 2 steps:

1. with likwid disabled.
2. with likwid enabled.

Parameters:project – The project we initialize. Returns:The initialized project.

class benchbuild.experiments.polyjit.PolyJITFull(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj'

actions_for_project(p)

benchbuild.experiments.polyjit.run_raw(project, experiment, config, run_f, args, **kwargs)

Run the given binary wrapped with nothing.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit.run_with_likwid(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given file wrapped by likwid.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit.run_with_papi(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given file with PAPI support.

This just runs the project as PAPI support should be compiled in already. If not, this won’t do a lot.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit.run_with_perf(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given binary wrapped with time.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit.run_with_time(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given binary wrapped with time.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.polyjit.run_without_recompile(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given binary wrapped with time.

Parameters:

• project – The benchbuild.project.
• experiment – The benchbuild.experiment.
• config – The benchbuild.settings.config.
• jobs – Number of cores we should use for this exection.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin.

benchbuild.experiments.raw module

The ‘raw’ Experiment.

This experiment is the basic experiment in the benchbuild study. It simply runs all projects after compiling it with -O3. The binaries are wrapped with the time command and results are written to the database.

This forms the baseline numbers for the other experiments.

Measurements

3 Metrics are generated during this experiment:

time.user_s - The time spent in user space in seconds (aka virtual time) time.system_s - The time spent in kernel space in seconds (aka system time) time.real_s - The time spent overall in seconds (aka Wall clock)

class benchbuild.experiments.raw.RawRuntime(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The polyjit experiment.

NAME = 'raw'

actions_for_project(project)

Compile & Run the experiment with -O3 enabled.

benchbuild.experiments.raw.run_with_time(project, experiment, config, jobs, run_f, args, **kwargs)

Run the given binary wrapped with time.

Parameters:

• project – The benchbuild project that has called us.
• experiment – The benchbuild experiment which we operate under.
• config – The benchbuild configuration we are running with.
• jobs – The number of cores we are allowed to use. This may differ from the actual amount of available cores, obey it. We should enforce this from the outside. However, at the moment we do not do this.
• run_f – The file we want to execute.
• args – List of arguments that should be passed to the wrapped binary.
• **kwargs –

  Dictionary with our keyword args. We support the following entries:

  project_name: The real name of our project. This might not

  be the same as the configured project name, if we got wrapped with ::benchbuild.project.wrap_dynamic

  has_stdin: Signals whether we should take care of stdin. may_wrap:

  Project may signal that it they are not suitable for wrapping. Usually because they scan/parse the output, which may interfere with the output of the wrapper binary.