Wrappers ======== Job packages, or "wrappers", are jobs created as bundles of different tasks (submitted at once in a single script to the platform) assembled by Autosubmit to maximize the usage of platforms managed by a scheduler (by minimizing the queuing time between consecutive or concurrent tasks). Autosubmit supports four wrapper types that can be used depending on the experiment’s workflow. * Horizontal_ * Vertical_ * Horizontal-vertical_ * Vertical-horizontal_ .. note:: To have a preview of wrappers, you must use the parameter `-cw` available on inspect, monitor, and create. .. code-block:: bash autosubmit create -cw # Unstarted experiment autosubmit monitor -cw # Ongoing experiment autosubmit inspect -cw -f # Visualize wrapper cmds Basic configuration ------------------- To configure a new wrapper, the user has to define a `WRAPPERS` section in any configuration file. When using the standard configuration, this one is autosubmit.yml. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" By default, Autosubmit will try to bundle jobs of the same type. The user can alter this behavior by setting the `JOBS_IN_WRAPPER` parameter directive in the wrapper section. When using multiple wrappers or 2-dim wrappers is essential to define the `JOBS_IN_WRAPPER` parameter. .. code-block:: YAML WRAPPERS: WRAPPER_H: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" WRAPPER_V: TYPE: "vertical" JOBS_IN_WRAPPER: "SIM2" WRAPPER_VH: TYPE: "vertical-horizontal" JOBS_IN_WRAPPER: "SIM3 SIM4" WRAPPER_HV: TYPE: "horizontal-vertical" JOBS_IN_WRAPPER: "SIM5 SIM6" experiment: DATELIST: 20220101 MEMBERS: "fc0 fc1" CHUNKSIZEUNIT: day CHUNKSIZE: '1' NUMCHUNKS: '4' CALENDAR: standard JOBS: SIM: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM-1 WALLCLOCK: 00:15 SIM2: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM2-1 WALLCLOCK: 00:15 SIM3: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM3-1 WALLCLOCK: 00:15 SIM4: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM4-1 WALLCLOCK: 00:15 SIM5: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM5-1 WALLCLOCK: 00:15 SIM6: FILE: sim.sh RUNNING: chunk QUEUE: debug DEPENDENCIES: SIM6-1 WALLCLOCK: 00:15 .. figure:: fig/wrapper_all.png :name: wrapper all :align: center :alt: wrapper all .. important:: Autosubmit will not wrap tasks with external and non-fulfilled dependencies. Wrapper parameters description ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Type ^^^^ The type parameter allow the user to determine the wrapper algorithm. It affects tasks in wrapper order executions, and in hybrid cases, it adds some internal logic. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" Jobs_in_wrapper ^^^^^^^^^^^^^^^ The jobs_in_wrapper parameter allow the user to determine the tasks inside a wrapper by giving the job_section name. It can group multiple tasks by providing more than one job_section name. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" Method ^^^^^^ The method parameter allow the user to determine if the wrapper will use machine files or threads. This allows to form a wrapper with that relies on machinefiles to work. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" METHOD: ASTHREAD or .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" This allows to form a wrapper with shared-memory paradigm instead of rely in machinefiles to work in parallel. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" METHOD: SRUN Extend_wallclock ^^^^^^^^^^^^^^^^ The extend_wallclock parameter allow the users to provide extra headroom for the wrapper. The accepted value is an integer. Autosubmit will translate this value automatically to the max_wallclock of the sum of wrapper inner-tasks wallclock at the horizontal level. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" extend_wallclock: 1 Retrials ^^^^^^^^ The retrials parameter allows the users to enable or disable the wrapper's retrial mechanism. This value overrides the general tasks defined. Vertical wrappers will retry the jobs without resubmitting the wrapper. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" RETRIALS: 2 Queue ^^^^^ The queue parameter allows the users to define a different queue for the wrapper. This value overrides the platform queue and job queue. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" QUEUE: BSC_ES Export ^^^^^^ The queue parameter allows the users to define a path to a script that will load environment scripts before running the wrapper tasks. This value overrides the job queue. .. code-block:: YAML WRAPPERS: WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" EXPORT: "%CURRENT_ROOTDIR%/envmodules.sh" Check_time_wrapper ^^^^^^^^^^^^^^^^^^ The CHECK_TIME_WRAPPER parameter defines the frequency, in seconds, on which Autosubmit will check the remote platform status of all the wrapper tasks. This affects all wrappers. .. code-block:: YAML WRAPPERS: CHECK_TIME_WRAPPER: 10 WRAPPER_0: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" WRAPPER_1: TYPE: "vertical" JOBS_IN_WRAPPER: "SIM1" Number of jobs in a wrapper({MIN/MAX}_WRAPPED{_H/_V} ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Users can configure the maximum and the minimum number of jobs in each wrapper by configuring MAX_WRAPPED and MIN_WRAPPED inside the wrapper section. If the user doesn't set them, Autosubmit will default to MAX_WRAPPED: “infinite” and MIN_WRAPPED: 2. .. code-block:: YAML WRAPPERS: MIN_WRAPPED: 2 MAX_WRAPPED: 999999 WRAPPER_0: MAX_WRAPPED: 2 TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" WRAPPER_1: TYPE: "vertical" JOBS_IN_WRAPPER: "SIM1" For 2-dim wrappers, {MAX_MIN}_WRAPPED_{V/H} must be used instead of the general one. .. code-block:: YAML WRAPPERS: MIN_WRAPPED: 2 MAX_WRAPPED: 999999 WRAPPER_0: MAX_WRAPPED_H: 2 MAX_WRAPPED_V: 4 MIN_WRAPPED_H: 2 MIN_WRAPPED_V: 2 TYPE: "horizontal-vertical" JOBS_IN_WRAPPER: "SIM SIM1" Policy ^^^^^^ Autosubmit will wrap as many tasks as possible while respecting the limits set in the configuration(MAX_WRAPPED, MAX_WRAPPED_H, MAX_WRAPPED_V, MIN_WRAPPED, MIN_WRAPPED_V, and MIN_WRAPPED_H parameters). However, users have three different policies available to tune the behavior in situations where there aren’t enough tasks in general, or there are uncompleted tasks remaining from a failed wrapper job: * Flexible: if there aren’t at least MIN_WRAPPED tasks to be grouped, Autosubmit will submit them as individual jobs. * Mixed: will wait for MIN_WRAPPED jobs to be available to create a wrapper, except if one of the wrapped tasks had failed beforehand. In this case, Autosubmit will submit them individually. * Strict: will always wait for MIN_WRAPPED tasks to be ready to create a wrapper. .. warning: Mixed and strict policies can cause deadlocks. .. code-block:: YAML WRAPPERS: POLICY: "flexible" WRAPPER_0: TYPE: "vertical" JOBS_IN_WRAPPER: "SIM SIM1" .. _Vertical: Vertical wrapper ---------------- Vertical wrappers are suited for sequential dependent jobs (e.x. chunks of SIM tasks that depend on the previous chunk). Defining the platform’s `MAX_WALLCLOCK` is essential since the wrapper's total wallclock time will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the wallclock time. Autosubmit supports wrapping together vertically jobs of different types. .. code-block:: YAML WRAPPERS: WRAPPER_V: TYPE: "vertical" JOBS_IN_WRAPPER: "SIM" .. figure:: fig/wrapper_v.png :name: wrapper vertical :align: center :alt: wrapper vertical .. _Horizontal: Horizontal wrapper ------------------ Horizontal wrappers are suited for jobs that must run parallel (e.x. members of SIM tasks). Defining the platform’s `MAX_PROCESSORS` is essential since the wrapper processor amount will be the sum of each job and will be a limiting factor for the creation of the wrapper, which will not bundle more jobs than the ones fitting in the `MAX_PROCESSORS` of the platform. .. code-block:: YAML WRAPPERS: WRAPPER_H: TYPE: "horizontal" JOBS_IN_WRAPPER: "SIM" .. figure:: fig/wrapper_h.png :name: wrapper horizontal :align: center :alt: wrapper horizontal .. _Vertical-horizontal: Vertical-horizontal wrapper --------------------------- The vertical-horizontal wrapper allows bundling together a vertical sequence of tasks independent of the horizontal ones. Therefore, all horizontal tasks do not need to finish to progress to the next horizontal level. .. figure:: fig/wrapper_vh.png :name: wrapper vertical-horizontal :align: center :alt: wrapper vertical-horizontal .. _Horizontal-vertical: Horizontal-vertical wrapper --------------------------- The horizontal-vertical wrapper allows bundling together tasks that could run simultaneously but need to communicate before progressing to the next horizontal level. .. figure:: fig/wrapper_hv.png :name: wrapper horizontal-vertical :align: center :alt: wrapper horizontal-vertical Advanced example: Set-up an crossdate wrapper ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Considering the following configuration: .. code-block:: yaml experiment: DATELIST: 20120101 20120201 MEMBERS: "000 001" CHUNKSIZEUNIT: day CHUNKSIZE: '1' NUMCHUNKS: '3' JOBS: LOCAL_SETUP: FILE: templates/local_setup.sh PLATFORM: marenostrum_archive RUNNING: once NOTIFY_ON: COMPLETED LOCAL_SEND_SOURCE: FILE: templates/01_local_send_source.sh PLATFORM: marenostrum_archive DEPENDENCIES: LOCAL_SETUP RUNNING: once NOTIFY_ON: FAILED LOCAL_SEND_STATIC: FILE: templates/01b_local_send_static.sh PLATFORM: marenostrum_archive DEPENDENCIES: LOCAL_SETUP RUNNING: once NOTIFY_ON: FAILED REMOTE_COMPILE: FILE: templates/02_compile.sh DEPENDENCIES: LOCAL_SEND_SOURCE RUNNING: once PROCESSORS: '4' WALLCLOCK: 00:50 NOTIFY_ON: COMPLETED SIM: FILE: templates/05b_sim.sh DEPENDENCIES: LOCAL_SEND_STATIC: REMOTE_COMPILE: SIM-1: DA-1: RUNNING: chunk PROCESSORS: '68' WALLCLOCK: 00:12 NOTIFY_ON: FAILED LOCAL_SEND_INITIAL_DA: FILE: templates/00b_local_send_initial_DA.sh PLATFORM: marenostrum_archive DEPENDENCIES: LOCAL_SETUP LOCAL_SEND_INITIAL_DA-1 RUNNING: chunk SYNCHRONIZE: member DELAY: '0' COMPILE_DA: FILE: templates/02b_compile_da.sh DEPENDENCIES: LOCAL_SEND_SOURCE RUNNING: once WALLCLOCK: 00:20 NOTIFY_ON: FAILED DA: FILE: templates/05c_da.sh DEPENDENCIES: SIM: LOCAL_SEND_INITIAL_DA: CHUNKS_TO: "all" DATES_TO: "all" MEMBERS_TO: "all" COMPILE_DA: DA: DATES_FROM: "20120201": CHUNKS_FROM: 1: DATES_TO: "20120101" CHUNKS_TO: "1" RUNNING: chunk SYNCHRONIZE: member DELAY: '0' WALLCLOCK: 00:12 PROCESSORS: '256' NOTIFY_ON: FAILED .. code-block:: yaml wrappers: wrapper_simda: TYPE: "horizontal-vertical" JOBS_IN_WRAPPER: "SIM DA" .. figure:: fig/monarch-da.png :name: crossdate-example :align: center :alt: crossdate-example