Configuration details, setup and sharing ======================================== Experiment configuration ------------------------ Since the beginning, Autosubmit has always been composed of five files in the folder ``$expid/conf`` that define the experiment configuration. However, from Autosubmit 4, the configuration is no longer bound to one specific location. And it is composed of YAML files. This document will teach you how to set up an experiment configuration using the different available methods and what Autosubmit expects to find in the configuration. Standard configuration structure --------------------------------- The standard configuration is the one that is used by default. It is composed of five files in the folder ``$expid/conf`` that define the experiment configuration. This configuration is generated by the ``expid`` command without any opcional flag or when using the -dm flag. The following table summarizes what configuration files Autosubmit expects and what parameters you can define. .. list-table:: :header-rows: 1 :widths: 20 80 * - File - Content * - ``expdef.yml`` - * It contains the default platform, the one set with -H. * Allows changing the start dates, members and chunks. * Allows changing the experiment project source ( git, local, svn or dummy) * - ``platforms.yml`` - * It contains the list of platforms to use in the experiment. * This file must be filled-up with the platform(s) configuration(s). * Several platforms can be defined and used in the same experiment. * - ``jobs.yml`` - - It contains the tasks' definitions in sections. - This file must be filled-up with the tasks' definitions. - Several sections can be defined and used in the same experiment. * - ``autosubmit.yml`` - - Parameters that control workflow behavior. - Parameters that activate extra functionalities. * - ``proj.yml`` - - Project-dependent parameters. * - ``version.yml`` - - Current AS version for this experiment. It is worth mentioning that for Autosubmit 4, these files are seen as one. Therefore, the sections and parameters can be defined in any of the files. .. note:: The ``version.yml`` file is automatically generated by Autosubmit. It is not necessary to create it. .. note:: Autosubmit only admits the use of the ``.yml`` and ``.yaml`` ( lowercase ) extensions for the configuration files. Advanced configuration structure and restrictions ------------------------------------------------- From Autosubmit4, the configuration structure can be split into multiple locations and different files. The experiment must have a `*-minimal.yml` file in `$expid/conf` in the `$expid/conf` folder. This file is used to define the location of the configuration files and can be generated by the `expid` command when using with -min flag. This location can be defined by the user in the `DEFAULT.CONFIG_DIR` parameter inside a file ending with `minimal.yml` or `minimal.yaml` file. * You would define the model-specific parameters inside your git or local repository. So when you push/pull the changes from git, they will be updated automatically. * You would define the experiment-specific parameters under `$expid/conf`. * You would define your user-specific parameters, for example, platform user, in a different location. How to create and share the configuration ------------------------------------------- This section contains examples of creating a standard configuration and an advanced one from a newly made experiment. Standard Configuration ~~~~~~~~~~~~~~~~~~~~~~ The expid command can generate a sample structure containing all the parameters that Autosubmit needs to work correctly. .. code-block:: bash #Create a new experiment. autosubmit expid -H "LOCAL" -d "Standard configuration." # Get the expid from the output. Ex. expid=a000 cd $autosubmit_experiment_folder/a000 ls conf autosubmit_a01y.yml expdef_a01y.yml platforms_a01y.yml jobs_a01y.yml proj_a01y.yml Sharing a standard Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The expid command can copy another user's existing expid to work correctly. .. code-block:: bash #Create a new experiment. autosubmit expid --copy a000 -H "LOCAL" -d "Standard configuration. --copy of a000" # Get the expid from the output. Ex. expid=a001 cd $autosubmit_experiment_folder/a001 ls conf autosubmit_a001.yml expdef_a001.yml platforms_a001.yml jobs_a001.yml proj_a001.yml .. warning:: You must share the same Autosubmit experiment database for this to work. .. _advanced_features: Advanced Configuration ~~~~~~~~~~~~~~~~~~~~~~ Autosubmit is now able to find the configuration files in different locations. The user can define the location of the configuration files in the `DEFAULT.CONFIG_DIR` parameter inside a file ending with `minimal.yml` or `minimal.yaml` file. An skeleton of the advanced configuration can be generated through the `expid` command when using the `-min` flag. .. code-block:: bash #Create a new experiment. autosubmit expid -min -d "Test minimal conf" # Get the expid from the output. Ex. expid=a002 cd $autosubmit_experiment_folder/a002 ls conf minimal.yml To give a practical example, we will show an example using git. However, using a non-git folder is also possible. Edit or generate a `minimal.yml` with the following parameters, leaving the rest untouched. .. code-block:: yaml DEFAULT: #ADD, note that %PROJDIR% is an special AS_PLACEHOLDER that points to the `$expid/proj/proj_destination` folder. CUSTOM_CONFIG: PRE: "%PROJDIR%/" POST: PROJECT: PROJECT_TYPE: "git" PROJECT_DESTINATION: "git_project" GIT: PROJECT_ORIGIN: "TO_FILL" PROJECT_BRANCH: "TO_FILL" PROJECT_COMMIT: "TO_FILL" PROJECT_SUBMODULES: "TO_FILL" FETCH_SINGLE_BRANCH: True .. important:: The final configuration will be loaded in the following order: `PRE`:`$expid/%PROJDIR%/$as_proj_config_path` -> `$expid/conf` -> `POST`. Overwriting the parameters in the order they are loaded. CUSTOM_CONFIG: Syntax ^^^^^^^^^^^^^^^^^^^^^ The %DEFAULT.CUSTOM_CONFIG% parameter is used to define the location of the model/project or user files. The paths can be absolute or relative to the ``%PROJDIR%``. It has two different syntaxes: * *Simple* a list of paths to the model or project yaml files. This can be a file or a folder. If it is a folder, all the files inside will be loaded in a non-recursive way. * *Advanced* a dictionary with two keys: ``PRE`` and ``POST``. The ``PRE`` key is used to define the files that will be loaded before the ``$EXPID/CONF`` ones. The ``POST`` key is used to define user configuration. .. note:: With the simple syntax, the outcome is the same as the advanced one, but with the ``POST`` key empty. .. note:: If a list of path is provided, the paths will be loaded in the order they are provided and in a recursive way. Meaning that in the case there are additional ``DEFAULT.CUSTOM_CONFIG`` parameter inside the files, they will be also loaded. .. code-block:: yaml # Download the git project autosubmit create a002 autosubmit refresh a002 .. warning:: Keep in mind that no parameters are disabled when custom_config is activated, including the jobs definitions. Advanced configuration - Full dummy example (reproducible) ---------------------------------------------------------- .. code-block:: bash #Create a new experiment. autosubmit expid -min -repo https://earth.bsc.es/gitlab/ces/auto-advanced_config_example -b main -conf as_conf -d "Test minimal conf" # expid=a04b dbeltran@bsces107894: cd ~/autosubmit/a04b dbeltran@bsces107894:~/autosubmit/a04b$ ls conf minimal.yml .. code-block:: bash cat ~/autosubmit/conf/minimal.yml .. code-block:: yaml CONFIG: AUTOSUBMIT_VERSION: "4.0.0b" DEFAULT: EXPID: "a04b" HPCARCH: "local" #ADD, note that %PROJDIR% is an special AS_PLACEHOLDER that points to the expid folder. #hint: use %PROJDIR% to point to the project folder (where the project is cloned) CUSTOM_CONFIG: "%PROJDIR%/as_conf" PROJECT: PROJECT_TYPE: "git" PROJECT_DESTINATION: "git_project" GIT: PROJECT_ORIGIN: "https://earth.bsc.es/gitlab/ces/auto-advanced_config_example" PROJECT_BRANCH: "main" PROJECT_COMMIT: "" PROJECT_SUBMODULES: "" .. code-block:: bash # Download the git project to obtain the distributed configuration dbeltran@bsces107894: autosubmit refresh a04b # Check the downloaded model-configuration dbeltran@bsces107894:~/autosubmit/a04b$ ls proj/git_project/as_conf/ autosubmit.yml expdef.yml jobs.yml platforms.yml Model configuration is distributed at `git. `_ .. code-block:: bash dbeltran@bsces107894:~/autosubmit/a04b$ cat ~/as_user_conf/platforms.yml .. code-block:: yaml Platforms: MARENOSTRUM4: USER: bsc32xxx QUEUE: debug MAX_WALLCLOCK: "02:00" marenostrum_archive: USER: bsc32xxx transfer_node: USER: bsc32xxx transfer_node_bscearth000: USER: dbeltran bscearth000: USER: dbeltran nord3: USER: bsc32xxx ecmwf-xc40: USER: c3d .. Note:: The user configuration is not distributed, it is a local file that must be edited by the user. .. code-block:: yaml # Create and run the experiment, since it contains all the info! autosubmit create a04b # if $expid/proj doesn't exists autosubmit refresh a04b autosubmit run a04b The following figure shows the flow of the execution. .. figure:: fig/advanced_conf.jpg :width: 400px :align: center :alt: Advanced configuration example Advanced configuration example Sharing an advanced configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The expid command can copy another user's existing expid to work correctly. .. note:: This only copies the ``$expid/conf/{*.yml,*yaml}`` experiment configuration files. .. code-block:: bash #Create a new experiment. autosubmit expid --copy a002 -H "LOCAL" -d "Advanced configuration. --copy of a002" # Get the expid from the output. Ex. expid=a004 cd $autosubmit_experiment_folder/a004 ls conf minimal.yml autosubmit create a004 .. warning:: All users must share the same experiment autosubmit.db for this to work. More info at `shared-db `_ Sharing an experiment configuration across filesystems is possible only by including the same `DEFAULT.CUSTOM_CONFIG` and `GIT.PROJECT_ORIGIN`, `GIT.PROJECT_BRANCH` and `GIT.PROJECT_TAG` inside the expdef.yml file.