Since the beginning, Autosubmit has always been composed of five files in the folder $expid/conf that define the experiment configuration.
However, from Autosubmit 4, the configuration is no longer bound to one specific location. And it is composed of YAML files.
This document will teach you how to set up an experiment configuration using the different available methods and what Autosubmit expects to find in the configuration.
The standard configuration is the one that is used by default. It is composed of five files in the folder $expid/conf that define the experiment configuration.
This configuration is generated by the expid command without any opcional flag or when using the -dm flag.
The following table summarizes what configuration files Autosubmit expects and what parameters you can define.
File
Content
expdef.yml
It contains the default platform, the one set with -H.
Allows changing the start dates, members and chunks.
Allows changing the experiment project source ( git, local, svn or dummy)
platforms.yml
It contains the list of platforms to use in the experiment.
This file must be filled-up with the platform(s) configuration(s).
Several platforms can be defined and used in the same experiment.
jobs.yml
It contains the tasks’ definitions in sections.
This file must be filled-up with the tasks’ definitions.
Several sections can be defined and used in the same experiment.
autosubmit.yml
Parameters that control workflow behavior.
Parameters that activate extra functionalities.
proj.yml
Project-dependent parameters.
version.yml
Current AS version for this experiment.
It is worth mentioning that for Autosubmit 4, these files are seen as one. Therefore, the sections and parameters can be defined in any of the files.
Note
The version.yml file is automatically generated by Autosubmit. It is not necessary to create it.
Note
Autosubmit only admits the use of the .yml and .yaml ( lowercase ) extensions for the configuration files.
Advanced configuration structure and restrictions#
From Autosubmit4, the configuration structure can be split into multiple locations and different files.
The experiment must have a *-minimal.yml file in $expid/conf in the $expid/conf folder. This file is used to define the location of the configuration files and can be generated by the expid command when using with -min flag.
This location can be defined by the user in the DEFAULT.CONFIG_DIR parameter inside a file ending with minimal.yml or minimal.yaml file.
You would define the model-specific parameters inside your git or local repository. So when you push/pull the changes from git, they will be updated automatically.
You would define the experiment-specific parameters under $expid/conf.
You would define your user-specific parameters, for example, platform user, in a different location.
The expid command can generate a sample structure containing all the parameters that Autosubmit needs to work correctly.
#Create a new experiment.
autosubmitexpid-H"LOCAL"-d"Standard configuration."# Get the expid from the output. Ex. expid=a000cd$autosubmit_experiment_folder/a000
lsconf
autosubmit_a01y.ymlexpdef_a01y.ymlplatforms_a01y.yml
jobs_a01y.ymlproj_a01y.yml
The expid command can copy another user’s existing expid to work correctly.
#Create a new experiment.
autosubmitexpid--copya000-H"LOCAL"-d"Standard configuration. --copy of a000"# Get the expid from the output. Ex. expid=a001cd$autosubmit_experiment_folder/a001
lsconf
autosubmit_a001.ymlexpdef_a001.ymlplatforms_a001.yml
jobs_a001.ymlproj_a001.yml
Warning
You must share the same Autosubmit experiment database for this to work.
Autosubmit is now able to find the configuration files in different locations. The user can define the location of the configuration files in the DEFAULT.CONFIG_DIR parameter inside a file ending with minimal.yml or minimal.yaml file.
An skeleton of the advanced configuration can be generated through the expid command when using the -min flag.
#Create a new experiment.
autosubmitexpid-min-d"Test minimal conf"# Get the expid from the output. Ex. expid=a002cd$autosubmit_experiment_folder/a002
lsconf
minimal.yml
To give a practical example, we will show an example using git. However, using a non-git folder is also possible.
Edit or generate a minimal.yml with the following parameters, leaving the rest untouched.
DEFAULT:#ADD, note that %PROJDIR% is an special AS_PLACEHOLDER that points to the `$expid/proj/proj_destination` folder.CUSTOM_CONFIG:PRE:"%PROJDIR%/<path_to_model_as_conf>"POST:<path_to_user_conf>PROJECT:PROJECT_TYPE:"git"PROJECT_DESTINATION:"git_project"GIT:PROJECT_ORIGIN:"TO_FILL"PROJECT_BRANCH:"TO_FILL"PROJECT_COMMIT:"TO_FILL"PROJECT_SUBMODULES:"TO_FILL"FETCH_SINGLE_BRANCH:True
Important
The final configuration will be loaded in the following order: PRE:$expid/%PROJDIR%/$as_proj_config_path -> $expid/conf -> POST. Overwriting the parameters in the order they are loaded.
The %DEFAULT.CUSTOM_CONFIG% parameter is used to define the location of the model/project or user files. The paths can be absolute or relative to the %PROJDIR%.
It has two different syntaxes:
Simple a list of paths to the model or project yaml files. This can be a file or a folder. If it is a folder, all the files inside will be loaded in a non-recursive way.
Advanced a dictionary with two keys: PRE and POST. The PRE key is used to define the files that will be loaded before the $EXPID/CONF ones. The POST key is used to define user configuration.
Note
With the simple syntax, the outcome is the same as the advanced one, but with the POST key empty.
Note
If a list of path is provided, the paths will be loaded in the order they are provided and in a recursive way. Meaning that in the case there are additional DEFAULT.CUSTOM_CONFIG parameter inside the files, they will be also loaded.
# Download the git projectautosubmit create a002autosubmit refresh a002
Warning
Keep in mind that no parameters are disabled when custom_config is activated, including the jobs definitions.
Advanced configuration - Full dummy example (reproducible)#
#Create a new experiment.
autosubmitexpid-min-repohttps://earth.bsc.es/gitlab/ces/auto-advanced_config_example-bmain-confas_conf-d"Test minimal conf"# expid=a04b
dbeltran@bsces107894:cd~/autosubmit/a04b
dbeltran@bsces107894:~/autosubmit/a04b$lsconf
minimal.yml
cat~/autosubmit/conf/minimal.yml
CONFIG:AUTOSUBMIT_VERSION:"4.0.0b"DEFAULT:EXPID:"a04b"HPCARCH:"local"#ADD, note that %PROJDIR% is an special AS_PLACEHOLDER that points to the expid folder.#hint: use %PROJDIR% to point to the project folder (where the project is cloned)CUSTOM_CONFIG:"%PROJDIR%/as_conf"PROJECT:PROJECT_TYPE:"git"PROJECT_DESTINATION:"git_project"GIT:PROJECT_ORIGIN:"https://earth.bsc.es/gitlab/ces/auto-advanced_config_example"PROJECT_BRANCH:"main"PROJECT_COMMIT:""PROJECT_SUBMODULES:""
# Download the git project to obtain the distributed configuration
dbeltran@bsces107894:autosubmitrefresha04b
# Check the downloaded model-configuration
dbeltran@bsces107894:~/autosubmit/a04b$lsproj/git_project/as_conf/
autosubmit.ymlexpdef.ymljobs.ymlplatforms.yml
The user configuration is not distributed, it is a local file that must be edited by the user.
# Create and run the experiment, since it contains all the info!autosubmit create a04b# if $expid/proj doesn't existsautosubmit refresh a04bautosubmit run a04b
The following figure shows the flow of the execution.
The expid command can copy another user’s existing expid to work correctly.
Note
This only copies the $expid/conf/{*.yml,*yaml} experiment configuration files.
#Create a new experiment.
autosubmitexpid--copya002-H"LOCAL"-d"Advanced configuration. --copy of a002"# Get the expid from the output. Ex. expid=a004cd$autosubmit_experiment_folder/a004
lsconf
minimal.yml
autosubmitcreatea004
Warning
All users must share the same experiment autosubmit.db for this to work. More info at shared-db
Sharing an experiment configuration across filesystems is possible only by including the same DEFAULT.CUSTOM_CONFIG and GIT.PROJECT_ORIGIN, GIT.PROJECT_BRANCH and GIT.PROJECT_TAG inside the expdef.yml file.