Wrappers

Job packages, or “wrappers”, are jobs created as bundles of different tasks (submitted at once in a single script to the platform) assembled by Autosubmit to maximize the usage of platforms managed by a scheduler (by minimizing the queuing time between consecutive or concurrent tasks). Autosubmit supports four wrapper types that can be used depending on the experiment’s workflow.

Note

To have a preview of wrappers, you must use the parameter -cw available on inspect, monitor, and create.

autosubmit create  <expid>  -cw  # Unstarted experiment
autosubmit monitor <expid> -cw # Ongoing experiment
autosubmit inspect <expid> -cw -f # Visualize wrapper cmds

Basic configuration

To configure a new wrapper, the user has to define a WRAPPERS section in any configuration file. When using the standard configuration, this one is autosubmit.yml.

WRAPPERS:
 WRAPPER_0:
  TYPE: "horizontal"

By default, Autosubmit will try to bundle jobs of the same type. The user can alter this behavior by setting the JOBS_IN_WRAPPER parameter directive in the wrapper section.

When using multiple wrappers or 2-dim wrappers is essential to define the JOBS_IN_WRAPPER parameter.

WRAPPERS:
  WRAPPER_H:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_V:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM2"
  WRAPPER_VH:
    TYPE: "vertical-horizontal"
    JOBS_IN_WRAPPER: "SIM3 SIM4"
  WRAPPER_HV:
    TYPE: "horizontal-vertical"
    JOBS_IN_WRAPPER: "SIM5 SIM6"
wrapper all

Important

Autosubmit will not wrap tasks with external and non-fulfilled dependencies.

Wrapper parameters description

Type

The type parameter allow the user to determine the wrapper algorithm.

It affects tasks in wrapper order executions, and in hybrid cases, it adds some internal logic.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"

Jobs_in_wrapper

The jobs_in_wrapper parameter allow the user to determine the tasks inside a wrapper by giving the job_section name. It can group multiple tasks by providing more than one job_section name.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"

Method

The method parameter allow the user to determine if the wrapper will use machine files or threads.

This allows to form a wrapper with that relies on machinefiles to work.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    METHOD: ASTHREAD

or

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"

This allows to form a wrapper with shared-memory paradigm instead of rely in machinefiles to work in parallel.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    METHOD: SRUN

Extend_wallclock

The extend_wallclock parameter allow the users to provide extra headroom for the wrapper. The accepted value is an integer. Autosubmit will translate this value automatically to the max_wallclock of the sum of wrapper inner-tasks wallclock at the horizontal level.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    extend_wallclock: 1

Retrials

The retrials parameter allows the users to enable or disable the wrapper’s retrial mechanism. This value overrides the general tasks defined.

Vertical wrappers will retry the jobs without resubmitting the wrapper.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    RETRIALS: 2

Queue

The queue parameter allows the users to define a different queue for the wrapper. This value overrides the platform queue and job queue.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    QUEUE: BSC_ES

Export

The queue parameter allows the users to define a path to a script that will load environment scripts before running the wrapper tasks. This value overrides the job queue.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    EXPORT: "%CURRENT_ROOTDIR%/envmodules.sh"

Check_time_wrapper

The CHECK_TIME_WRAPPER parameter defines the frequency, in seconds, on which Autosubmit will check the remote platform status of all the wrapper tasks. This affects all wrappers.

WRAPPERS:
  CHECK_TIME_WRAPPER: 10
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_1:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM1"

Number of jobs in a wrapper({MIN/MAX}_WRAPPED{_H/_V}

Users can configure the maximum and the minimum number of jobs in each wrapper by configuring MAX_WRAPPED and MIN_WRAPPED inside the wrapper section. If the user doesn’t set them, Autosubmit will default to MAX_WRAPPED: “infinite” and MIN_WRAPPED: 2.

WRAPPERS:
  MIN_WRAPPED: 2
  MAX_WRAPPED: 999999
  WRAPPER_0:
    MAX_WRAPPED: 2
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_1:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM1"

For 2-dim wrappers, {MAX_MIN}_WRAPPED_{V/H} must be used instead of the general one.

WRAPPERS:
 MIN_WRAPPED: 2
 MAX_WRAPPED: 999999
 WRAPPER_0:
  MAX_WRAPPED_H: 2
  MAX_WRAPPED_V: 4
  MIN_WRAPPED_H: 2
  MIN_WRAPPED_V: 2
  TYPE: "horizontal-vertical"
  JOBS_IN_WRAPPER: "SIM SIM1"

Policy

Autosubmit will wrap as many tasks as possible while respecting the limits set in the configuration(MAX_WRAPPED, MAX_WRAPPED_H, MAX_WRAPPED_V, MIN_WRAPPED, MIN_WRAPPED_V, and MIN_WRAPPED_H parameters). However, users have three different policies available to tune the behavior in situations where there aren’t enough tasks in general, or there are uncompleted tasks remaining from a failed wrapper job:

  • Flexible: if there aren’t at least MIN_WRAPPED tasks to be grouped, Autosubmit will submit them as individual jobs.

  • Mixed: will wait for MIN_WRAPPED jobs to be available to create a wrapper, except if one of the wrapped tasks had failed beforehand. In this case, Autosubmit will submit them individually.

  • Strict: will always wait for MIN_WRAPPED tasks to be ready to create a wrapper.

WRAPPERS:
  POLICY: "flexible"
  WRAPPER_0:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM SIM1"

Vertical wrapper

Vertical wrappers are suited for sequential dependent jobs (e.x. chunks of SIM tasks that depend on the previous chunk). Defining the platform’s MAX_WALLCLOCK is essential since the wrapper’s total wallclock time will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the wallclock time.

Autosubmit supports wrapping together vertically jobs of different types.

WRAPPERS:
  WRAPPER_V:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM"
wrapper vertical

Horizontal wrapper

Horizontal wrappers are suited for jobs that must run parallel (e.x. members of SIM tasks). Defining the platform’s MAX_PROCESSORS is essential since the wrapper processor amount will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the MAX_PROCESSORS of the platform.

WRAPPERS:
  WRAPPER_H:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
wrapper horizontal

Vertical-horizontal wrapper

The vertical-horizontal wrapper allows bundling together a vertical sequence of tasks independent of the horizontal ones. Therefore, all horizontal tasks do not need to finish to progress to the next horizontal level.

wrapper vertical-horizontal

Horizontal-vertical wrapper

The horizontal-vertical wrapper allows bundling together tasks that could run simultaneously but need to communicate before progressing to the next horizontal level.

wrapper horizontal-vertical

Advanced example: Set-up an crossdate wrapper

Considering the following configuration:

experiment:
  DATELIST: 20120101 20120201
  MEMBERS: "000 001"
  CHUNKSIZEUNIT: day
  CHUNKSIZE: '1'
  NUMCHUNKS: '3'

JOBS:
  LOCAL_SETUP:
    FILE: templates/local_setup.sh
    PLATFORM: marenostrum_archive
    RUNNING: once
    NOTIFY_ON: COMPLETED
  LOCAL_SEND_SOURCE:
    FILE: templates/01_local_send_source.sh
    PLATFORM: marenostrum_archive
    DEPENDENCIES: LOCAL_SETUP
    RUNNING: once
    NOTIFY_ON: FAILED
  LOCAL_SEND_STATIC:
    FILE: templates/01b_local_send_static.sh
    PLATFORM: marenostrum_archive
    DEPENDENCIES: LOCAL_SETUP
    RUNNING: once
    NOTIFY_ON: FAILED
  REMOTE_COMPILE:
    FILE: templates/02_compile.sh
    DEPENDENCIES: LOCAL_SEND_SOURCE
    RUNNING: once
    PROCESSORS: '4'
    WALLCLOCK: 00:50
    NOTIFY_ON: COMPLETED
  SIM:
    FILE: templates/05b_sim.sh
    DEPENDENCIES:
      LOCAL_SEND_STATIC:
      REMOTE_COMPILE:
      SIM-1:
      DA-1:
    RUNNING: chunk
    PROCESSORS: '68'
    WALLCLOCK: 00:12
    NOTIFY_ON: FAILED
  LOCAL_SEND_INITIAL_DA:
    FILE: templates/00b_local_send_initial_DA.sh
    PLATFORM: marenostrum_archive
    DEPENDENCIES: LOCAL_SETUP LOCAL_SEND_INITIAL_DA-1
    RUNNING: chunk
    SYNCHRONIZE: member
    DELAY: '0'
  COMPILE_DA:
    FILE: templates/02b_compile_da.sh
    DEPENDENCIES: LOCAL_SEND_SOURCE
    RUNNING: once
    WALLCLOCK: 00:20
    NOTIFY_ON: FAILED
  DA:
    FILE: templates/05c_da.sh
    DEPENDENCIES:
      SIM:
      LOCAL_SEND_INITIAL_DA:
        CHUNKS_TO: "all"
        DATES_TO: "all"
        MEMBERS_TO: "all"
      COMPILE_DA:
      DA:
        DATES_FROM:
          "20120201":
            CHUNKS_FROM:
            1:
              DATES_TO: "20120101"
              CHUNKS_TO: "1"
    RUNNING: chunk
    SYNCHRONIZE: member
    DELAY: '0'
    WALLCLOCK: 00:12
    PROCESSORS: '256'
    NOTIFY_ON: FAILED
wrappers:
  wrapper_simda:
    TYPE: "horizontal-vertical"
    JOBS_IN_WRAPPER: "SIM DA"
crossdate-example