User-defined metrics

Autosubmit allows you to define your own metrics. This is useful if you want to track specific parameters or performance indicators that need to be calculated on the fly by a job when it is COMPLETED. Then, these metrics can be visualized using the Autosubmit API or the Autosubmit GUI.

Define your own metrics

To define your own metrics, you need to update your JOBS sections in your experiment configuration files. For each job section, you need to add a METRICS section. This section should contain the following parameters:

• NAME: The name of the metric. This name will be used to identify the metric in the Autosubmit API and GUI.

• FILENAME: The name of the file that contains the metric. This file should be generated by the job and should contain the metric value. To see where this file should be generated, see the Prepare your script template section.

• SELECTOR: The selector defines the strategy used to extract the metric value from the file. If not set, the type will be TEXT and it will store the entire file content.

  • TYPE: The type of the metric. This can be TEXT or JSON. The type will be used to determine how the metric value is extracted from the file. If not set, the type will be TEXT.

  • KEY: Needs to be set if the type is JSON. This will be the key used to extract the value from the JSON file. For deeper keys, you can use the dot notation. For example, key1.key2 will extract the value of key2 inside the key1 object.

Example:

JOBS:
  SIM:
    METRICS:
      - NAME: metric1
        FILENAME: my_metric.out
        SELECTOR:
          TYPE: TEXT
      - NAME: metric2
        FILENAME: output.json
        SELECTOR:
          TYPE: JSON
          KEY: job

Prepare your script template

Autosubmit will look for the metric files in a specific location. This location will be available to the template by using the %CURRENT_METRIC_FOLDER% variable. This variable will be replaced by the path to the metrics folder of the job.

For example, a valid script template for the specification above will look like this:

METRIC_FOLDER=%CURRENT_METRIC_FOLDER%

mkdir -p $METRIC_FOLDER

echo "%JOBNAME% $(date -Iseconds)" > $METRIC_FOLDER/my_metric.out

cat <<EOF > $METRIC_FOLDER/output.json
{
"job": "%JOBNAME%",
"timestamp": "$(date)"
}
EOF

The %CURRENT_METRIC_FOLDER% variable is by default set to the %CURRENT_ROOTDIR%/%JOBNAME% location. However, you can change this location by setting the CONFIG.METRIC_FOLDER variable in your experiment configuration file, and then the job name will be appended to it. This is useful if you want to store the metrics in a different location.

CONFIG:
  METRIC_FOLDER: /path/to/metrics/folder
  # This will update CURRENT_METRIC_FOLDER=%CONFIG.METRIC_FOLDER%/%JOBNAME%

Important

The %CURRENT_METRIC_FOLDER% will append the job name to the path. So, if you set the CONFIG.METRIC_FOLDER to /path/to/metrics/folder, the metric files should be stored in /path/to/metrics/folder/%JOBNAME%.