Variables reference#

Autosubmit uses a variable substitution system to facilitate the development of the templates. These variables can be used on templates with the syntax %VARIABLE_NAME%.

All configuration variables that are not related to the current job or platform are available by accessing first their parents, e.g. %PROJECT.PROJECT_TYPE% or %DEFAULT.EXPID%.

You can review all variables at any given time by using the report command, as illustrated below.

4 Example usage of autosubmit report#

$ autosubmit report <EXPID> -all

The command will save the list of variables available to a file in the experiment area. The groups of variables of Autosubmit are detailed in the next sections on this page.

Note

All the variable tables are displayed in alphabetical order.

Note

Custom configuration files (e.g. my-file.yml) may contain configuration like this example:

MYAPP:
  MYPARAMETER: 42
  ANOTHER_PARAMETER: 1984

If you configure Autosubmit to include this file with the rest of your configuration, then those variables will be available to each job as %MYAPP.MYPARAMETER% and %MYAPP.ANOTHER_PARAMETER%.

Job variables#

These variables are relatives to the current job. These variables appear in the output of the report command with the pattern JOBS.${JOB_ID}.${JOB_VARIABLE}=${VALUE}. They can be used in templates with %JOB_VARIABLE%.

Variable	Description
CHECKPOINT	Generates a checkpoint step for this job based on job.type.
CHUNK	Current chunk.
CHUNK_SIZE	Chunk size used to compute CPMIP metrics.
CHUNK_SIZE_UNIT	Chunk size unit used to compute CPMIP metrics.
CPMIP_THRESHOLDS	Thresholds for CPMIP metrics.
CPUS_PER_TASK	Number of threads that the job will use.
CURRENT_QUEUE	Returns the queue to be used by the job. Chooses between serial and parallel platforms.
CUSTOM_DIRECTIVES	List of custom directives.
DELAY	Current delay.
DELAY_RETRIALS	TODO
DEPENDENCIES	Current job dependencies.
EXPORT	TODO.
FAIL_COUNT	Number of failed attempts to run this job.
FREQUENCY	TODO.
HYPERTHREADING	Detects if hyperthreading is enabled or not.
JOBNAME	Current job full name.
MEMBER	Current member.
MEMORY	Memory requested for the job.
MEMORY_PER_TASK	Memory requested per task.
NODES	Number of nodes that the job will use.
NOTIFY_ON	Send mail notification on job status change.
NUMMEMBERS	Number of members of the experiment.
NUMPROC	Number of processors that the job will use.
NUMTASK	Number of tasks that the job will use.
NUMTHREADS	Number of threads that the job will use.
PACKED	TODO
PROCESSORS	Number of processors that the job will use.
PROCESSORS_PER_NODE	Number of processors per node that the job can use.
PROJDIR	Project folder path.
RETRIALS	Max amount of retrials to run this job.
ROOTDIR	Experiment folder path.
SCRATCH_FREE_SPACE	Percentage of free space required on the `scratch`.
SCRIPT	Allows to launch inline code instead of using the file parameter
SDATE	Current start date.
SPLIT	Current split.
SPLITS	Max number of splits.
STATUS	None
SYNCHRONIZE	TODO.
TASKS	Number of tasks that the job will use.
TASKS_PER_NODE	Number of tasks that the job will use.
TASKTYPE	Type of the job, as given on job configuration file.
THREADS	Number of threads that the job will use.
VALIDATE_TEMPLATE	Whether to print validate information about the job.
WALLCLOCK	Duration for which nodes used by job will remain allocated.
X11	Whether to use X11 forwarding
X11_OPTIONS	Allows to set salloc parameters for x11

The following variables are present only in jobs that contain a date (e.g. RUNNING=date).

Variable	Description
CHUNK_END_DATE	Chunk end date.
CHUNK_END_DATE_LAST	End date of the last chunk, i.e. the experiment end boundary. Available to any date-aware job (notably RUNNING: date jobs).
CHUNK_END_DAY	Chunk end day.
CHUNK_END_HOUR	Chunk end hour.
CHUNK_END_IN_DAYS	Days passed from the start of the simulation until the end of the chunk.
CHUNK_END_MONTH	Chunk end month.
CHUNK_END_YEAR	Chunk end year.
CHUNK_FIRST	True if the current chunk is the first, false otherwise.
CHUNK_LAST	True if the current chunk is the last, false otherwise.
CHUNK_SECOND_TO_LAST_DATE	Chunk second to last date.
CHUNK_SECOND_TO_LAST_DAY	Chunk second to last day.
CHUNK_SECOND_TO_LAST_HOUR	Chunk second to last hour.
CHUNK_SECOND_TO_LAST_MONTH	Chunk second to last month.
CHUNK_SECOND_TO_LAST_YEAR	Chunk second to last year.
CHUNK_START_DATE	Chunk start date.
CHUNK_START_DAY	Chunk start day.
CHUNK_START_HOUR	Chunk start hour.
CHUNK_START_MONTH	Chunk start month.
CHUNK_START_YEAR	Chunk start year.
DAY_BEFORE	Day before the start date.
LDATE	Last date of the experiment (the run’s final day, parallel to SDATE).
NOTIFY_ON	Determine the job statuses you want to be notified.
PREV	Days since start date at the chunk’s start.
RUN_DAYS	Chunk length in days.

Custom directives#

There are job variables that Autosubmit automatically converts into directives for your batch server. For example, THREADS will be set in a Slurm platform as --SBATCH --cpus-per-task=$THREADS.

However, the variables in Autosubmit do not contain all the directives available in each platform like Slurm. For values that do not have a direct variable, you can use CUSTOM_DIRECTIVES to define them in your target platform. For instance, to set the number of GPU’s in a Slurm job, you can use CUSTOM_DIRECTIVES=--gpus-per-node=10.

Platform variables#

Platform variables come from the PLATFORMS section of the configuration files. Each JOBS entry can reference any of these platforms via its PLATFORM key. If no platform is specified, the job uses the experiment’s default platform (DEFAULT.HPCARCH).

There are three ways platform variables reach your job templates: raw keys, HPC-prefixed keys, and CURRENT_-prefixed keys.

Example configuration#

Consider the following experiment configuration:

DEFAULT:
  EXPID: a000
  HPCARCH: PS

PLATFORMS:
  MARENOSTRUM5:
    TYPE: slurm
    HOST: glogin1.bsc.es, glogin2.bsc.es
    USER: root
    PROJECT: bsc32
    SCRATCH_DIR: /gpfs/scratch
  SLURM:
    TYPE: slurm
    HOST: slurm-test
    USER: root
    PROJECT: group
    SCRATCH_DIR: /tmp/scratch
  PS:
    TYPE: pbs
    HOST: ps-test
    USER: root
    PROJECT: bsc-es
    SCRATCH_DIR: /tmp/scratch


JOBS:
  SIM:
    FILE: sim.sh
    RUNNING: once
    PLATFORM: MARENOSTRUM5
  POST:
    FILE: post.sh
    RUNNING: once
    PLATFORM: SLURM
  LOCAL_JOB:
    FILE: local_test.sh
    RUNNING: once
     # No PLATFORM key, uses DEFAULT.HPCARCH (PS)

In this example:

SIM uses MARENOSTRUM5 as its platform, so it will have access to all the keys under PLATFORMS.MARENOSTRUM5.
POST uses SLURM as its platform, so it will have access to all the keys under PLATFORMS.SLURM.
LOCAL_JOB does not specify a platform, so it will use the default platform DEFAULT.HPCARCH (PS). It will have access to all the keys under PLATFORMS.PS.

—

1. Raw platform keys (global)

Every key under a platform is available globally as PLATFORMS.<PLATFORM_ID>.<KEY>. You can reference them in templates as %PLATFORMS.MARENOSTRUM5.HOST% for example.

In the output of report you will find them with the pattern PLATFORMS.<PLATFORM_ID>.<KEY>=<VALUE>.

5 Excerpt from autosubmit report a000 -all#

PLATFORMS.MARENOSTRUM5.TYPE=slurm
PLATFORMS.MARENOSTRUM5.HOST=glogin1.bsc.es, glogin2.bsc.es
PLATFORMS.MARENOSTRUM5.USER=root
PLATFORMS.MARENOSTRUM5.PROJECT=bsc32
PLATFORMS.MARENOSTRUM5.SCRATCH_DIR=/gpfs/scratch
PLATFORMS.SLURM.TYPE=slurm
...

—

2. HPC prefixed keys (default platform, global)

Every key under the default platform (DEFAULT.HPCARCH) is available with an HPC prefix. These are also global, as they have the same value for every job.

From the example above (default platform is PS):

Template variable	Source key	Value
`%HPCHOST%`	`HOST`	`ps-test`
`%HPCUSER%`	`USER`	`root`
`%HPCPROJECT%`	`PROJECT`	`bsc-es`
`%HPCSCRATCH_DIR%`	`SCRATCH_DIR`	`/tmp/scratch`

Note

The key name is used as-is. E.g. HPCPROJECT (not HPCPROJ). If you add a custom key to your platform (e.g. GPU_MODEL: A100), it becomes available as %HPCGPU_MODEL%.

In addition, Autosubmit automatically computes the following derived variables:

Variable	Description
HPCARCH	Name of the default platform (`PS`).
HPCROOTDIR	`<SCRATCH_DIR>/<PROJECT>/<USER>/<EXPID>` e.g. `/tmp/scratch/bsc-es/root/a000`.
HPCLOGDIR	`<HPCROOTDIR>/LOG_<EXPID>` e.g. `/tmp/scratch/bsc-es/root/a000/LOG_a000`.

6 Excerpt from autosubmit report a000 -all#

HPCARCH=PS
HPCHOST=ps-test
HPCLOGDIR=/tmp/scratch/bsc-es/root/a000/LOG_a000
HPCPROJECT=bsc-es
HPCROOTDIR=/tmp/scratch/bsc-es/root/a000
HPCSCRATCH_DIR=/tmp/scratch
HPCUSER=root

—

3. CURRENT_ prefixed keys (per-job)

Each job has access to the keys of its own platform (the one it runs on) with a CURRENT_ prefix. This means different jobs may have different values for the same CURRENT_<KEY> variable.

Continuing with the example above:

Template variable	In `SIM`	In `POST`	In `LOCAL_JOB`
`%CURRENT_HOST%`	`glogin1.bsc.es, glogin2.bsc.es`	`slurm-test`	`ps-test`
`%CURRENT_USER%`	`root`	`root`	`root`
`%CURRENT_PROJECT%`	`bsc32`	`group`	`bsc-es`
`%CURRENT_SCRATCH_DIR%`	`/gpfs/scratch`	`/tmp/scratch`	`/tmp/scratch`

7 Excerpt from autosubmit report a000 -all#

JOBS.LOCAL_JOB.CURRENT_HOST=ps-test
JOBS.LOCAL_JOB.CURRENT_PROJECT=bsc-es
JOBS.POST.CURRENT_HOST=slurm-test
JOBS.POST.CURRENT_PROJECT=group
JOBS.SIM.CURRENT_HOST=glogin1.bsc.es, glogin2.bsc.es
JOBS.SIM.CURRENT_PROJECT=bsc32

The CURRENT_ prefix is also populated from the job’s section keys (JOBS.<section>). This allows you to inject job-specific information into your templates.

Auto-generated CURRENT_ variables

Below is a list of the auto-generated CURRENT_ variables that are available in each job. These are always available regardless of the raw YAML keys defined.

Variable	Description
CURRENT_ARCH	Platform name.
CURRENT_BUDG	Platform budget.
CURRENT_EXCLUSIVITY	True if you want to request exclusivity nodes.
CURRENT_HOST	Platform url.
CURRENT_HYPERTHREADING	True if hyperthreading is enabled on this platform.
CURRENT_LOGDIR	The platform’s LOG directory.
CURRENT_PARTITION	Partition to use for jobs.
CURRENT_PROJ	Platform project.
CURRENT_PROJ_DIR	Platform’s project folder path.
CURRENT_RESERVATION	You can configure your reservation id for the given platform.
CURRENT_ROOTDIR	Platform’s experiment folder path.
CURRENT_SCRATCH_DIR	Platform’s scratch folder path.
CURRENT_USER	Platform user.

Note

The table above lists the CURRENT_ variables that Autosubmit exposes from the platform section. Not all of them are always populated. Some are filled automatically by Autosubmit, others come from the YAML configuration, and a few are only used by certain schedulers.

CURRENT_ARCH, CURRENT_HOST, CURRENT_TYPE, CURRENT_SCRATCH_DIR, CURRENT_ROOTDIR, CURRENT_LOGDIR and CURRENT_HYPERTHREADING are automatically filled by Autosubmit.
CURRENT_USER, CURRENT_PROJ, CURRENT_QUEUE, CURRENT_PARTITION and CURRENT_TEMP_DIR are only filled if the corresponding YAML key (USER, PROJECT, QUEUE, PARTITION, TEMP_DIR) is defined in the platform configuration. On the LOCAL platform they remain empty.
CURRENT_BUDG, CURRENT_RESERVATION and CURRENT_EXCLUSIVITY are only populated if the corresponding YAML keys (BUDGET, RESERVATION, EXCLUSIVITY) are defined in the platform configuration. They are only used by certain schedulers: CURRENT_BUDG is used by the ECMWF header, CURRENT_RESERVATION by SLURM and PBS, and CURRENT_EXCLUSIVITY by SLURM.

Other variables#

Variable	Description
CONFIG.AUTOSUBMIT_VERSION	Current version of Autosubmit.
CONFIG.MAXWAITINGJOBS	Maximum number of jobs permitted in the waiting status.
CONFIG.TOTALJOBS	Total number of jobs in the workflow.

Variable	Description
DEFAULT.CUSTOM_CONFIG	Custom configuration location.
DEFAULT.EXPID	Job experiment ID.
DEFAULT.HPCARCH	Default HPC platform name.

Variable	Description
EXPERIMENT.CALENDAR	Calendar used for the experiment. Can be standard or noleap.
EXPERIMENT.CHUNKSIZE	Size of each chunk.
EXPERIMENT.CHUNKSIZEUNIT	Unit of the chunk size. Can be hour, day, month, or year.
EXPERIMENT.DATELIST	List of start dates
EXPERIMENT.MEMBERS	List of members.
EXPERIMENT.NUMCHUNKS	Number of chunks of the experiment.

Variable	Description
PROJECT.PROJECT_DESTINATION	Folder to hold the project sources.
PROJECT.PROJECT_TYPE	Type of the project.

Note

Depending on your project type other variables may be available. For example, if you choose Git, then you should have %PROJECT_ORIGIN%. If you choose Subversion, then you will have %PROJECT_URL%.

Performance Metrics variables#

These variables apply only to the report subcommand.

Variable	Description
ASYPD	Actual simulated years per day.
CHSY	Core hours per simulated year.
JPSY	Joules per simulated year.
Parallelization	Number of cores requested for the simulation job.
RSYPD	Raw simulated years per day.
SYPD	Simulated years per day.

Variables reference#

Job variables#

Custom directives#

Platform variables#

Example configuration#

Other variables#

Performance Metrics variables#

This Page