Manage Experiments#

Cleaning the experiment#

This procedure allows you to save space after finalising an experiment. You must execute:

autosubmit clean <EXPID>

Options:

$ autosubmit clean -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit clean -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1202
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit clean [-h] [-pr] [-p] [-s] [-v] [--profile] expid

Cleans an experiment, reducing disk space usage.

It removes the project directory and outdated plots or stats.

When it deletes plots or stats, it keeps the two newest files
(i.e. two newest statistics files, and two newest plot files).

positional arguments:
  expid                 Experiment identifier.

options:
  -h, --help            Show available options and exit.
  -pr, --project        Clean project.
  -p, --plot            Keep only the two most recently created plots.
  -s, --stats           Keep only the most recently created statistics.
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

Examples:

    # clean an experiment.
    $ autosubmit clean a000

    # clean the Autosubmit project folder of an experiment.
    $ autosubmit clean --project a000

    # clean the plot and stat files of an experiment.
    $ autosubmit clean --plot --stats a000
  • The -p and -s flags are used to clean our experiment plot folder to save disk space. Only the two latest plots will be kept. Older plots will be removed.

Example:

autosubmit clean <EXPID> -p
  • The -pr flag is used to clean our experiment proj locally in order to save space (it could be particularly big).

Caution

Bear in mind that if you have not synchronized your experiment project folder with the information available on the remote repository (i.e.: commit and push any changes we may have), or in case new files are found, the clean procedure will be failing although you specify the -pr option.

Example:

autosubmit clean <EXPID> -pr

A bare copy (which occupies less space on disk) will be automatically made.

Hint

That bare clone can be always reconverted in a working clone if we want to run again the experiment by using git clone bare_clone original_clone.

Note

In addition, every time you run this command with the -pr option, it will check the commit unique identifier for the local working tree existing in the proj directory. If that commit identifier exists, clean will register it in the expdef_<EXPID>.yml file.

Archiving an experiment#

When you archive an experiment in Autosubmit, it automatically cleans the experiment as well. This means the experiment will not be available for use, unless it is unarchived.

autosubmit archive EXPID

Options:

$ autosubmit archive -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit archive -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1204
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit archive [-h] [-nclean] [-uc] [--rocrate] [-v] [--profile]
                          expid

Archives an experiment.

Compresses the experiment directory and moves it to the archival location.

positional arguments:
  expid                 Experiment identifier.

options:
  -h, --help            Show available options and exit.
  -nclean, --noclean    Do not clean the experiment folder.
  -uc, --uncompress     Create the container without compression.
  --rocrate             produce an RO-Crate file
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

The archived experiment will be stored as a tar.gz file, under a directory named after the year of the last _COMPLETED file date or, if no _COMPLETED job is present, it will use the year of the date the autosubmit archive was run (e.g. for the selected year 2023, the location will be $HOME/autosubmit/2023/<EXPID>.tar.gz).

Unarchiving an experiment#

To unarchive an experiment, use the command:

autosubmit unarchive <EXPID>

Options:

$ autosubmit unarchive -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit unarchive -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1206
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit unarchive [-h] [-nclean] [-uc] [--rocrate] [-v] [--profile]
                            expid

Unarchives an experiment.

Decompresses the experiment archive, ensuring the experiment can be used again.

positional arguments:
  expid                 Experiment identifier.

options:
  -h, --help            Show available options and exit.
  -nclean, --noclean    Avoid cleaning of experiment folder.
  -uc, --uncompressed   Extract files of the tar file without gzip
                        compression.
  --rocrate             Unarchive an RO-Crate file.
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

Deleting the experiment#

To delete the experiment, use the command:

autosubmit delete <EXPID>

<EXPID> is the experiment identifier. You can pass a single identifier, or a comma-separated list of identifiers.

Warning

DO NOT USE THIS COMMAND IF YOU ARE NOT SURE ! It deletes the experiment from database and experiment’s folder.

Options:

$ autosubmit delete -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit delete -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1208
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit delete [-h] [-f] [--profile] expid

Deletes an experiment.

Removes the experiment from the database and file system.

positional arguments:
  expid        Experiment identifiers separated by commas.

options:
  -h, --help   Show available options and exit.
  -f, --force  Delete experiment without confirmation.
  --profile    Print performance parameters of the execution of this command.

Example:

autosubmit delete <EXPID>

Warning

Be careful ! force option does not ask for your confirmation.

Synchronizing with the project’s latest changes#

Autosubmit supports directly fetching files from the repository, which could be local or remote.

In order to synchronize with remote, use the command:

autosubmit refresh <EXPID>

where <EXPID> is the experiment identifier.

It checks the experiment configuration and copies code from the original repository to project directory.

Warning

THIS WILL OVERWRITE LOCAL CHANGES! Project directory <EXPID>/proj will be overwritten and you may loose local changes.

Options:

$ autosubmit refresh -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit refresh -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1210
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit refresh [-h] [-mc] [-jc] [-v] [--profile] expid

Refreshes the project files for an experiment.

Synchronises the experiment's project files with their configured source.
When the project uses Git, the project is cloned again, replacing the local
copy and discarding any local modifications.

Use --model_conf or --jobs_conf to also overwrite the corresponding
configuration files from the refreshed project.

positional arguments:
  expid                 Experiment identifier.

options:
  -h, --help            Show available options and exit.
  -mc, --model_conf     Overwrite model conf file.
  -jc, --jobs_conf      Overwrite jobs conf file.
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

Examples:

    # refresh the project files for an experiment.
    $ autosubmit refresh a000

    # refresh the project and overwrite the model configuration file.
    $ autosubmit refresh a000 --model_conf

    # refresh the project and overwrite the jobs configuration file.
    $ autosubmit refresh a000 --jobs_conf

    # refresh the project and overwrite both configuration files.
    $ autosubmit refresh a000 --model_conf --jobs_conf

Example:

autosubmit refresh <EXPID>

Updating the experiment description#

Use the command:

autosubmit updatedescrip <EXPID> "DESCRIPTION"

EXPID is the experiment identifier.

DESCRIPTION is the new description of your experiment.

Options:

$ autosubmit updatedescrip -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit updatedescrip -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1212
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit updatedescrip [-h] [-v] [--profile] expid description

Updates the description of an experiment.

positional arguments:
  expid                 Experiment identifier.
  description           New description.

options:
  -h, --help            Show available options and exit.
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

Examples:

    # change the experiment description to "Testing"
    $ autosubmit updatedescrip a000 "Testing"

Autosubmit will validate the provided data and print the results in the command line.

Example:

autosubmit a29z "Updated using Autosubmit updatedescrip"

Changing the job status#

This procedure allows you to modify the status of your jobs.

Warning

Beware that Autosubmit must be stopped to use setstatus. Otherwise a running instance of Autosubmit, at some point, will overwrite any changes you may have done.

You must execute:

autosubmit setstatus <EXPID> <FILTER> <VALUE_TO_FILTER> -t <STATUS_FINAL> -s

By default, plots are not generated when changing status. To generate plots showing the updated job statuses, use the -plt or --plot option.

autosubmit setstatus <EXPID> <FILTER> <VALUE_TO_FILTER> -t <STATUS_FINAL> -s -plt

Where:

FILTER

Meaning

Example of VALUE_TO_FILTER

-fl

filter by job name

-fl "a000_20101101_fc3_21_SIM"

-fs

filter by job status

-fs FAILED

-ft

filter by job type (and optionally split)

-ft TRANSFER

-fc

filter by chunk/section/split

-fc "[ 19601101 [ fc1 [1] ] ]"

If multiple filters are provided (-fl, -fs, -ft, -fc), they will be combined as logical AND, meaning that only jobs matching ALL specified filters will have their status changed.

Mandatory arguments:

  • <EXPID>: experiment identifier

  • <FILTER> <VALUE_TO_FILTER>: at least one filter is required to select jobs (see below for filter options)

  • -t STATUS_FINAL: target status (READY, COMPLETED, WAITING, SUSPENDED, HELD, UNKNOWN)

Optional filter arguments (combine multiple for granular selection):

  • -fl: space-separated list of job names

autosubmit setstatus <EXPID> -fl "<EXPID>_20101101_fc3_21_SIM <EXPID>_20111101_fc4_26_SIM" -t READY -s
  • -fc: chunk/section/split filter (JSON-like format, takes precedence over legacy filters)

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc1 [1] ] ]" -t READY -s
  • -fs: space-separated job statuses. The available statuses are: READY, COMPLETED, WAITING, SUSPENDED, HELD, and UNKNOWN.

autosubmit setstatus <EXPID> -fs FAILED -t READY -s
  • -ft: space-separated job types with optional split selection

autosubmit setstatus <EXPID> -ft TRANSFER -t SUSPENDED -s
autosubmit setstatus <EXPID> -ft "TRANSFER[1 2]" -t SUSPENDED -s
  • -ftc: [DEPRECATED] legacy chunk/type filter (use a combination of -fc and -ft instead).

Command:

autosubmit setstatus <EXPID> -ftc "[ 19601101 [ fc0 [1 2 3 4] ] ],SIM" -t READY -s

Can be replaced with:

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc0 [1 2 3 4] ] ]" -ft SIM -t READY -s
  • -ftcs: [DEPRECATED] legacy chunk/type/split filter (use a combination of -fc and -ft instead).

Command:

autosubmit setstatus <EXPID> -ftcs "[ 19601101 [ fc0 [1 2 3 4] ] ],SIM,1" -t READY -s

Can be replaced with:

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc0 [1 2 3 4] ] ]" -ft SIM -t READY -s

Options:

$ autosubmit setstatus -h

Output:
Autosubmit v4.2.0

Command traceability:
  Command: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/envs/latest/bin/autosubmit setstatus -h
  Host: build-35786016-project-45339-autosubmit
  User: docs
  PID: 1214
  Python: 3.12.13
  Platform: Linux-7.0.0-1004-aws-x86_64-with-glibc2.35
  Timezone: UTC (UTC+00:00)
  Working directory: /home/docs/checkouts/readthedocs.org/user_builds/autosubmit/checkouts/latest/docs

usage: autosubmit setstatus [-h] [-np | -plt] [-s] -t
                            {READY,COMPLETED,WAITING,SUSPENDED,UNKNOWN,HELD}
                            [-fl LIST] [-fc FILTER_CHUNKS] [-fs FILTER_STATUS]
                            [-ft FILTER_TYPE] [-ftc FILTER_TYPE_CHUNK]
                            [-ftcs FILTER_TYPE_CHUNK_SPLIT] [--hide]
                            [-group_by {date,member,chunk,split,automatic}]
                            [-expand EXPAND] [-expand_status EXPAND_STATUS]
                            [-nt] [-cw] [-d] [-v] [--profile]
                            expid

Changes the status of selected jobs in the database.

NOTE: Use --save to persist the changes to disk!

This can be used to skip completed parts of a workflow and allow downstream
jobs to run.

For example, jobs that have already been completed outside Autosubmit can be
marked as COMPLETED so that a downstream job, such as CLEAN, can run.

positional arguments:
  expid                 Experiment identifier.

options:
  -h, --help            Show available options and exit.
  -np, --noplot         Omit plot (default).
  -plt, --plot          Generate plot.
  -s, --save            Save changes to disk.
  -t {READY,COMPLETED,WAITING,SUSPENDED,UNKNOWN,HELD}, --status_final {READY,COMPLETED,WAITING,SUSPENDED,UNKNOWN,HELD}
                        Supply the target status.
  -fl LIST, --list LIST
                        Supply the list of job names to be changed. Default =
                        "Any". LIST = "b037_20101101_fc3_21_sim
                        b037_20111101_fc4_26_sim".
  -fc FILTER_CHUNKS, --filter_chunks FILTER_CHUNKS
                        Supply the list of chunks to change the status.
                        Default = "Any". LIST = "[ 19601101 [ fc0 [1 2 3 4]
                        fc1 [1] ] 19651101 [ fc0 [16-30] ] ]".
  -fs FILTER_STATUS, --filter_status FILTER_STATUS
                        Select the status (one or more) to filter the list of
                        jobs.Valid values = ['Any', 'READY', 'COMPLETED',
                        'WAITING', 'SUSPENDED', 'FAILED', 'UNKNOWN'].
  -ft FILTER_TYPE, --filter_type FILTER_TYPE
                        Select the job type and split to filter the list of
                        jobs. Default split = "Any". LIST = "LOCALJOB [5-10]
                        SIM".
  -ftc FILTER_TYPE_CHUNK, --filter_type_chunk FILTER_TYPE_CHUNK
                        [Deprecated] Equivalent behaviour can be achieved by
                        combining -ft and -fc. Supply the list of chunks to
                        change the status. Default = "Any". When the member
                        name "all" is set, all the chunks selected from for
                        that member will be updated for all the members.
                        Example: all [1], will have as a result that the
                        chunks 1 for all the members will be updated. Follow
                        the format: "[ 19601101 [ fc0 [1 2 3 4] Any [1] ]
                        19651101 [ fc0 [16-30] ] ],SIM,SIM2,SIM3".
  -ftcs FILTER_TYPE_CHUNK_SPLIT, --filter_type_chunk_split FILTER_TYPE_CHUNK_SPLIT
                        [Deprecated] Equivalent behaviour can be achieved by
                        combining -ft and -fc. Supply the list of chunks &
                        splits to change the status. Default = "Any". When the
                        member name "all" is set, all the chunks selected from
                        for that member will be updated for all the members.
                        Example: all [1], will have as a result that the
                        chunks 1 for all the members will be updated. Follow
                        the format: "[ 19601101 [ fc0 [1 [1 2] 2 3 4] Any [1]
                        ] 19651101 [ fc0 [16-30] ] ],SIM,SIM2,SIM3".
  --hide                Hide the plot window.
  -group_by {date,member,chunk,split,automatic}
                        Group the jobs automatically or by date, member, chunk
                        or split.
  -expand EXPAND        Supply the list of dates/members/chunks to filter the
                        list of jobs. Default = "Any". LIST = "[ 19601101 [
                        fc0 [1 2 3 4] fc1 [1] ] 19651101 [ fc0 [16-30] ] ]".
  -expand_status EXPAND_STATUS
                        Select the statuses to be expanded.
  -nt, --notransitive   Disable transitive reduction.
  -cw, --check_wrapper  Generate possible wrapper in the current workflow.
  -d, --detail          Generate detailed view of changes.
  -v, --update_version  Update experiment version.
  --profile             Print performance parameters of the execution of this
                        command.

Examples:

    # mark selected jobs as completed and save the changes.
    $ autosubmit setstatus a000 --status_final COMPLETED --save

    # mark specific jobs as completed.
    $ autosubmit setstatus a000 --status_final COMPLETED --list "JOB1 JOB2" --save

    # mark jobs in a specific status as completed.
    $ autosubmit setstatus a000 --status_final COMPLETED --filter_status WAITING --save

    # mark jobs as completed for specific chunks.
    $ autosubmit setstatus a000 --status_final COMPLETED --filter_chunks "[ 19601101 [ fc0 [1 2 3 4] ] ]" --save

Filter precedence (when multiple chunk filters are specified):

When -fc, -ftc, and -ftcs are combined, precedence is: -fc → -ftc → -ftcs. A warning will be logged if multiple chunk filters are detected.

Filter Examples#

Single filter, by job list:

autosubmit setstatus <EXPID> -fl "<EXPID>_20101101_fc3_21_SIM <EXPID>_20111101_fc4_26_SIM" -t READY -s

Single filter, by chunk/section/split:

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc1 [1] ] ]" -t READY -s

Single filter, by current status:

autosubmit setstatus <EXPID> -fs FAILED -t READY -s

Single filter, by job type:

autosubmit setstatus <EXPID> -ft TRANSFER -t SUSPENDED -s

Single filter, by job type and split:

autosubmit setstatus <EXPID> -ft "TRANSFER[1:5]" -t SUSPENDED -s

Multiple filters combined (AND logic, selects jobs matching ALL filters):

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc1 [1] ] ]" -ft "SIM" -t SUSPENDED -s

This selects jobs that are in both the chunk filter AND have type “SIM”.

Chunk/Section/Split Filter Details#

Format:

[ 19601101 [ fc0 [1 2 3 4] fc1 [1] ] 19651101 [ fc0 [16-30] ] ]

Date (month) range example:

autosubmit setstatus <EXPID> -fc "[ 1960(1101-1201) [ fc1 [1] ] ]" -ft "SIM" -t SUSPENDED -s

Result:

<EXPID>_19601101_fc1_1_SIM
<EXPID>_19601201_fc1_1_SIM

Date (day) range example:

autosubmit setstatus <EXPID> -fc "[ 1960(1101-1105) [ fc1 [1] ] ]" -ft "SIM" -t SUSPENDED -s

Result:

<EXPID>_19601101_fc1_1_SIM
<EXPID>_19601102_fc1_1_SIM
<EXPID>_19601103_fc1_1_SIM
<EXPID>_19601104_fc1_1_SIM
<EXPID>_19601105_fc1_1_SIM

Using the “Any” Keyword#

The keyword Any (case-insensitive) means “no restriction” in that filter:

  • -fl Any: all jobs (no job list restriction)

  • -fs Any: all job statuses (no status restriction)

  • -ft Any: all job types (no type restriction)

Example:

autosubmit setstatus <EXPID> -fs Any -fc "[ 19601101 [ fc1 [1] ] ]" -t READY -s

This changes all jobs in the chunk filter to READY, regardless of their current status.

Deprecated Filters#

Warning

The -ftc and -ftcs filters are deprecated and will be removed in future versions. Use -fc instead.

-ftc (deprecated) is similar to -fc but also accepts job types without separate -ft:

autosubmit setstatus <EXPID> -ftc "[ 19601101 [ fc0 [1 2 3 4] ] ],SIM" -t READY -s

Use -fc with -ft instead:

autosubmit setstatus <EXPID> -fc "[ 19601101 [ fc0 [1 2 3 4] ] ]" -ft SIM -t READY -s

Hint

When satisfied with your filter selections, use the parameter -s to save changes to the pkl file. In order to understand more the grouping options, which are used for visualization purposes, please check Grouping jobs.

Changing the job status without stopping Autosubmit#

This procedure allows you to modify the status of your jobs without having to stop Autosubmit.

You must create a file in <experiments_directory>/<EXPID>/status/ named:

updated_list_<EXPID>.txt

Format:

This file should have two columns: the first one has to be the job_name and the second one the status.

Options:

READY,COMPLETED,WAITING,SUSPENDED,FAILED,UNKNOWN

Example:

vi updated_list_<EXPID>.txt
<EXPID>_20101101_fc3_21_SIM    READY
<EXPID>_20111101_fc4_26_SIM    READY

If Autosubmit finds the above file, it will process it. You can check that the processing was OK at a given date and time, if you see that the file name has changed to:

updated_list_<EXPID>.txt_<YYYYMMDD_HHMM>

Note

A running instance of Autosubmit will check the existence of the above file after checking already submitted jobs. It may take some time, depending on the setting SAFETYSLEEPTIME.

Note

Job names and statuses are matched case-insensitively. Malformed lines, unknown jobs/statuses and active statuses (SUBMITTED, QUEUING, RUNNING) are skipped with a warning. When a job is active and its platform is reachable, Autosubmit cancels it on the platform before applying the requested change.