Score-P Metric Plugins

The Score-P metric plugin interface makes it possible for programmers to increase the event stream with metric data from additional data sources that are otherwise not accessible for Score-P.

READEX currently supports various backends for energy measurement. This includes: Intel RAPl, AMD RAPL, and AMD APM (via the Processor Energy Event Plugin), Bull/ATOS HDEEM and ANTAREX EXAMON. In the following, we describe, how they can be used and installed.

Processor Energy Event Plugin

Version 2 of this metric plugin provides energy measurement for most contemporary x86-based processors and platforms.

Installation

Prerequisites

To compile this plugin, you need:

  • A C++ 14 compiler
  • CMake
  • Score-P
  • A processor of the following architectures:
    • Intel with Sandy Bridge architecture or newer
    • AMD Family15h (Bulldozer) or AMD Zen
  • Each of the following kernel modules will grant you energy measurement access:
    • intel_powerclamp kernel module (Intel architectures, recommended, part of recent Linux kernels. There’s a backport for older kernels available at http://content.allinea.com/downloads/allinea-powercap-backport-20150601.tar.bz2)
    • intel_rapl_perf kernel module (Intel architectures, part of recent Linux kernels)
    • msr and msr-safe kernel module (Intel architectures, link msr-safe)
    • x86_adapt kernel module (Intel architectures, AMD Zen, link x86_adapt)
    • likwid plus msr/msr-safe kernel module (Intel architectures, link likwid)
    •  fam15h_power kernel module (AMD Bulldozer, part of recent Linux kernels)

Build Options

Building

  1. Create a build directory
    mkdir build
    cd build
  2. Invoking CMake
    cmake ..
    The following settings can be customized:

      • SCOREP_CONFIG
        Path to the scorep-config tool including the file name
        Note: If you have scorep-config in your PATH, it should be found by CMake.
      • CMAKE_INSTALL_PREFIX
        Directory where the resulting plugin will be installed (lib/ suffix will be added)
      • ENABLE_MPI (only applicable to the examon_sync_plugin)
        Enables MPI communication for the sync plugin, and allows to run more than one MPI process per node. This is mandatory for READEX!
  3. Invoking make
    make
  4. Install the files
    make install

Usage in READEX

For READEX purposes, we use thex86_energy_sync_plugin, which enables us to attribute energy to specific regions. To use the plugin, it must be enabled during runs with Periscope. To do so, use the following environment variables:

export SCOREP_METRIC_PLUGINS=x86_energy_sync_plugin
export SCOREP_METRIC_PLUGINS_SEP=";"
export SCOREP_METRIC_X86_ENERGY_SYNC_PLUGIN="BLADE/E"
export SCOREP_METRIC_X86_ENERGY_PLUGIN_INTERVALL_US=0
export SCOREP_METRIC_X86_ENERGY_SYNC_PLUGIN_OFFSET=70 # This is the offset that is set to the sum of package and DRAM consumptions to compute BLADE power and highly system dependent

Furthermore, the metric source definition of your readex_config.xml should look like this:

<metricPlugin>
<name>x86_energy_sync_plugin</name>
</metricPlugin>
<metrics>
<node_energy>x86_energy/BLADE/E</node_energy>
</metrics>

HDEEM Energy Measurement Plugin

High Definition Energy Efficiency Monitoring (HDEEM) is the power measurement infrastructure provided by contemporary blades from the HPC vendor Bull(Atos). It provides a high resolution with inband and out-of band interfaces for accessing energy and power data. More details can be found here.

Installation

To compile this plugin, you need:

  • A C++ 14 compiler
  • Score-P Version 2+ (SCOREP_METRIC_PLUGIN_VERSION >= 1)
  • The hdeem library and header files by BULL/ATOS

Building

  1. Create a build directory
    mkdir build
    cd build
  2. Invoke CMake
    cmake ..
    The following settings can be customized:

      • HDEEM_BMC_USER HDEEM_BMC_PASS
        Required for out of band access
      • HDEEM_INCLUDE_DIRS
        Directory where hdeem.h is located
      • HDEEM_LIBRARIES
        The full path of libhdeem.so, including the file name, e.g. /usr/local/hdeem/libhdeem.so.1
      • SCOREP_CONFIG
        Path to the scorep-config tool including the file name
        Note: If you have scorep-config in your PATH, it should be found by CMake.
      • CMAKE_INSTALL_PREFIX
        Directory where the resulting plugin will be installed (lib/ suffix will be added)
      • ENABLE_MPI (only applicable to the hdeem_sync_plugin)
        Enables MPI communication for the sync plugin, and allows to run more than one MPI process per node. This is mandatory for READEX!
  3. Invoke make
    make
  4. Install the files
    make install

Usage in READEX

For READEX purposes, we use thehdeem_sync_plugin, which enables us to attribute energy to specific regions. To use the plugin, it must be enabled during runs with Periscope. To do so, use the following environment variables:

export SCOREP_METRIC_PLUGINS=hdeem_sync_plugin
export SCOREP_METRIC_PLUGINS_SEP=";"
export SCOREP_METRIC_HDEEM_SYNC_PLUGIN_CONNECTION="INBAND"
export SCOREP_METRIC_HDEEM_SYNC_PLUGIN_VERBOSE="WARN"
export SCOREP_METRIC_HDEEM_SYNC_PLUGIN_STATS_TIMEOUT_MS=1000

Furthermore, the metric source definition of your readex_config.xml should look like this:

<metricPlugin>
<name>hdeem_sync_plugin</name>
</metricPlugin>
<metrics>
<node_energy>hdeem/BLADE/E</node_energy>
<cpu0_energy>hdeem/CPU0/E</cpu0_energy>
<cpu1_energy>hdeem/CPU1/E</cpu1_energy>
</metrics>

EXAMON Energy Event Plugin

EXAMON is a scalable HPC measurement infrastructure with different data sources and plugins. More details can be found here. EXAMON was developed as part of the ANTAREX project

Installation

To compile this plugin, you need:

  • A C++ 14 compiler
  • Score-P Version 2+ (SCOREP_METRIC_PLUGIN_VERSION >= 1)
  • CMake 3.8+
  • Other dependencies will be downloaded during the cmake/make process

Building

  1. Create a build directory
    mkdir build
    cd build
  2. Invoke CMake
    cmake ..
    The following settings can be customized:

      • SCOREP_CONFIG
        Path to the scorep-config tool including the file name
        Note: If you have scorep-config in your PATH, it should be found by CMake.
      • CMAKE_INSTALL_PREFIX
        Directory where the resulting plugin will be installed (lib/ suffix will be added)
      • ENABLE_MPI (only applicable to the examon_sync_plugin)
        Enables MPI communication for the sync plugin, and allows to run more than one MPI process per node. This is mandatory for READEX!
  3. Invoke make
    make
  4. Install the files
    make install

Usage in READEX

For READEX purposes, we use theexamon_sync_plugin, which enables us to attribute energy to specific regions. To use the plugin, it must be enabled during runs with Periscope. To do so, use the following environment variables:

export SCOREP_METRIC_PLUGINS=examon_sync_plugin
export SCOREP_METRIC_PLUGINS_SEP=";"
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN_BROKER=<MQTT broker address>
#
# You can use this option, if your blade counter does not use the systems hostname but something else;
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN_EXAMON_HOST=<hostname override&>
#
# the default is org/antarex/cluster/testcluster
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN_CHANNEL=<the default channel configured in examon's pmu_pub.conf key "topic">
#
# lower means more overhead, but higher granularity.
# The granularity is also limited from the EXAMON publisher, which provides the data!
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN_INTERVAL=<Readout delay in seconds (e.g., 0.001 for 1 ms)>
#
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN="EXAMON/BLADE/E"
#
# the whole counter name used with MQTT is:
# $SCOREP_METRIC_EXAMON_SYNC_PLUGIN_EXAMON_HOST/$SCOREP_METRIC_EXAMON_SYNC_PLUGIN_CHANNEL/$SCOREP_METRIC_EXAMON_SYNC_PLUGIN_READEX_BLADE
export SCOREP_METRIC_EXAMON_SYNC_PLUGIN_READEX_BLADE=<The systems blade counter>

Furthermore, the metric source definition of your readex_config.xml should look like this:

<metricPlugin>
<name>examon_sync_plugin</name>
</metricPlugin>
<metrics>
<node_energy>EXAMON/BLADE/E</node_energy>
</metrics>