Wrappers#

Job packages, or “wrappers”, are jobs created as bundles of different tasks (submitted at once in a single script to the platform) assembled by Autosubmit to maximize the usage of platforms managed by a scheduler (by minimizing the queuing time between consecutive or concurrent tasks). Autosubmit supports four wrapper types that can be used depending on the experiment’s workflow.

Note

To have a preview of wrappers, you must use the parameter -cw available on inspect, monitor, and create.

autosubmit create  EXPID -cw  # Unstarted experiment
autosubmit monitor EXPID -cw # Ongoing experiment
autosubmit inspect EXPID -cw -f # Visualize wrapper cmds

Basic configuration#

To configure a new wrapper, the user has to define a WRAPPERS section in any configuration file. When using the standard configuration, this one is autosubmit_<EXPID>.yml.

WRAPPERS:
    WRAPPER_0:
        TYPE: "horizontal"

By default, Autosubmit will try to bundle jobs of the same type. The user can alter this behavior by setting the JOBS_IN_WRAPPER parameter directive in the wrapper section.

When using multiple wrappers or 2-dim wrappers is essential to define the JOBS_IN_WRAPPER parameter.

EXPERIMENT:
  DATELIST: 20220101
  MEMBERS: "fc0 fc1"
  CHUNKSIZEUNIT: day
  CHUNKSIZE: '1'
  NUMCHUNKS: '4'
  CALENDAR: standard

JOBS:
    SIM:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM-1
      WALLCLOCK: 00:15
    SIM2:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM2-1
      WALLCLOCK: 00:15
    SIM3:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM3-1
      WALLCLOCK: 00:15
    SIM4:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM4-1
      WALLCLOCK: 00:15
    SIM5:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM5-1
      WALLCLOCK: 00:15
    SIM6:
      FILE: sim.sh
      RUNNING: chunk
      QUEUE: debug
      DEPENDENCIES: SIM6-1
      WALLCLOCK: 00:15

WRAPPERS:
  WRAPPER_H:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_V:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM2"
  WRAPPER_VH:
    TYPE: "vertical-horizontal"
    JOBS_IN_WRAPPER: "SIM3 SIM4"
  WRAPPER_HV:
    TYPE: "horizontal-vertical"
    JOBS_IN_WRAPPER: "SIM5&SIM6"
wrapper all

Important

Autosubmit will not wrap tasks with external and non-fulfilled dependencies.

Warning

Start conditions and wrappers are not compatible.

Jobs that use start conditions (START_CONDITIONS) cannot be reliably bundled inside a wrapper. When a job with start conditions is part of a wrapper, Autosubmit ignores the start conditions and enforces a stricter rule: the wrapper will only be submitted once all parent jobs have reached ``COMPLETED`` status. This avoids downstream issues caused by partially satisfied start conditions inside a wrapper package.

Wrapper parameters description#

Type#

The type parameter allows the user to determine the wrapper algorithm.

It affects task execution order in wrappers, and in hybrid cases, it adds some internal logic.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"

Jobs_in_wrapper#

The jobs_in_wrapper parameter allows the user to determine the tasks inside a wrapper by specifying the job_section name. It can group multiple tasks by providing more than one job_section name.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"

Method#

The method parameter allows the user to determine if the wrapper will use machine files or threads.

This allows forming a wrapper that relies on machinefiles to work.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    METHOD: ASTHREAD

or

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"

This allows forming a wrapper with shared-memory paradigm instead of relying on machinefiles to work in parallel.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    METHOD: SRUN

Extend_wallclock#

The extend_wallclock parameter allows users to provide extra headroom for the wrapper. The accepted value is an integer. Autosubmit will translate this value automatically to the max_wallclock of the sum of wrapper inner-task wallclocks at the horizontal level.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    extend_wallclock: 1

Retrials#

This parameter allows the users to enable or disable the wrapper’s retrying mechanism. This value overrides the general tasks defined.

Vertical wrappers will retry the jobs without resubmitting the wrapper.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    RETRIALS: 2

Queue#

The queue parameter allows the users to define a different queue for the wrapper. This value overrides the platform queue and job queue.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    QUEUE: BSC_ES

Export#

The export parameter allows users to define a path to a script that will load environment scripts before running the wrapper tasks. This value overrides the job export setting.

WRAPPERS:
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
    EXPORT: "%CURRENT_ROOTDIR%/envmodules.sh"

Check_time_wrapper#

The CHECK_TIME_WRAPPER parameter defines the frequency, in seconds, on which Autosubmit will check the remote platform status of all the wrapper tasks. This affects all wrappers.

WRAPPERS:
  CHECK_TIME_WRAPPER: 10
  WRAPPER_0:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_1:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM1"

Number of jobs in a wrapper({MIN/MAX}_WRAPPED{_H/_V}#

Users can configure the maximum and the minimum number of jobs in each wrapper by configuring MAX_WRAPPED and MIN_WRAPPED inside the wrapper section. If the user doesn’t set them, Autosubmit will default to MAX_WRAPPED: “infinite” and MIN_WRAPPED: 2.

WRAPPERS:
  MIN_WRAPPED: 2
  MAX_WRAPPED: 999999
  WRAPPER_0:
    MAX_WRAPPED: 2
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
  WRAPPER_1:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM1"

For 2-dim wrappers, {MAX_MIN}_WRAPPED_{V/H} must be used instead of the general one.

WRAPPERS:
 MIN_WRAPPED: 2
 MAX_WRAPPED: 999999
 WRAPPER_0:
  MAX_WRAPPED_H: 2
  MAX_WRAPPED_V: 4
  MIN_WRAPPED_H: 2
  MIN_WRAPPED_V: 2
  TYPE: "horizontal-vertical"
  JOBS_IN_WRAPPER: "SIM SIM1"

Policy#

Autosubmit will wrap as many tasks as possible while respecting the limits set in the configuration(MAX_WRAPPED, MAX_WRAPPED_H, MAX_WRAPPED_V, MIN_WRAPPED, MIN_WRAPPED_V, and MIN_WRAPPED_H parameters). However, users have three different policies available to tune the behavior in situations where there aren’t enough tasks in general, or there are uncompleted tasks remaining from a failed wrapper job:

  • Flexible: if there aren’t at least MIN_WRAPPED tasks to be grouped, Autosubmit will submit them as individual jobs.

  • Mixed: will wait for MIN_WRAPPED jobs to be available to create a wrapper, except if one of the wrapped tasks had failed beforehand. In this case, Autosubmit will submit them individually.

  • Strict: will always wait for MIN_WRAPPED tasks to be ready to create a wrapper.

Warning

Mixed and strict policies can cause deadlocks.

WRAPPERS:
  POLICY: "flexible"
  WRAPPER_0:
    TYPE: "vertical"
    JOBS_IN_WRAPPER: "SIM SIM1"

Vertical wrapper#

Vertical wrappers are suited for sequential dependent jobs (e.x. chunks of SIM tasks that depend on the previous chunk). Defining the platform’s MAX_WALLCLOCK is essential since the wrapper’s total wallclock time will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the wallclock time.

Autosubmit supports wrapping together vertically jobs of different types.

JOBS:
  SIM:
    FILE: sim.sh
    RUNNING: chunk
    QUEUE: debug
    DEPENDENCIES: SIM-1
    WALLCLOCK: 00:15

  WRAPPERS:
    WRAPPER_V:
      TYPE: "vertical"
      JOBS_IN_WRAPPER: "SIM"
wrapper vertical

Horizontal wrapper#

Horizontal wrappers are suited for jobs that must run parallel (e.x. members of SIM tasks). Defining the platform’s MAX_PROCESSORS is essential since the wrapper processor amount will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the MAX_PROCESSORS of the platform.

WRAPPERS:
  WRAPPER_H:
    TYPE: "horizontal"
    JOBS_IN_WRAPPER: "SIM"
wrapper horizontal

Vertical-horizontal wrapper#

The vertical-horizontal wrapper allows bundling together a vertical sequence of tasks independent of the horizontal ones. Therefore, all horizontal tasks do not need to finish to progress to the next horizontal level.

wrapper vertical-horizontal

Horizontal-vertical wrapper#

The horizontal-vertical wrapper allows bundling together tasks that could run simultaneously but need to communicate before progressing to the next horizontal level.

wrapper horizontal-vertical

Advanced example: Set-up an crossdate wrapper#

Considering the following configuration:

EXPERIMENT:
  DATELIST: "20120101 20120201"
  MEMBERS: "000 001"
  CHUNKSIZEUNIT: day
  CHUNKSIZE: '1'
  NUMCHUNKS: '3'

JOBS:
LOCAL_SETUP:
  FILE: local_setup.sh
  RUNNING: once
  NOTIFY_ON: COMPLETED
LOCAL_SEND_SOURCE:
  FILE: local_send_source.sh
  DEPENDENCIES: LOCAL_SETUP
  RUNNING: once
  NOTIFY_ON: FAILED
LOCAL_SEND_STATIC:
  FILE: local_send_static.sh
  DEPENDENCIES: LOCAL_SETUP
  RUNNING: once
  NOTIFY_ON: FAILED
REMOTE_COMPILE:
  FILE: compile.sh
  DEPENDENCIES: LOCAL_SEND_SOURCE
  RUNNING: once
  PROCESSORS: '4'
  WALLCLOCK: 00:50
  NOTIFY_ON: COMPLETED
SIM:
  FILE: sim.sh
  DEPENDENCIES:
    LOCAL_SEND_STATIC:
    REMOTE_COMPILE:
    SIM-1:
    DA-1:
  RUNNING: chunk
  PROCESSORS: '68'
  WALLCLOCK: 00:12
  NOTIFY_ON: FAILED
LOCAL_SEND_INITIAL_DA:
  FILE: local_send_initial_DA.sh
  DEPENDENCIES: LOCAL_SETUP LOCAL_SEND_INITIAL_DA-1
  RUNNING: chunk
  SYNCHRONIZE: member
  DELAY: '0'
COMPILE_DA:
  FILE: compile_da.sh
  DEPENDENCIES: LOCAL_SEND_SOURCE
  RUNNING: once
  WALLCLOCK: 00:20
  NOTIFY_ON: FAILED
DA:
  FILE: da.sh
  DEPENDENCIES:
    SIM:
    LOCAL_SEND_INITIAL_DA:
      CHUNKS_TO: "all"
      DATES_TO: "all"
      MEMBERS_TO: "all"
    COMPILE_DA:
    DA:
      DATES_FROM:
        "20120201":
          CHUNKS_FROM:
            1:
              DATES_TO: "20120101"
              CHUNKS_TO: "1"
  RUNNING: chunk
  SYNCHRONIZE: member
  DELAY: '0'
  WALLCLOCK: 00:12
  PROCESSORS: '256'
  NOTIFY_ON: FAILED

WRAPPERS:
  WRAPPER_SIMDA:
    TYPE: "horizontal-vertical"
    JOBS_IN_WRAPPER: "SIM DA"

wrappers:
  wrapper_simda:
    TYPE: "horizontal-vertical"
    JOBS_IN_WRAPPER: "SIM DA"
crossdate-example