autosubmit#

class autosubmit.autosubmit.Autosubmit#

Bases: object

Interface class for autosubmit.

static archive(expid, noclean=True, uncompress=True, rocrate=False)#

Archives an experiment: call clean (if experiment is of version 3 or later), compress folder to tar.gz and moves to year’s folder

Parameters:

expid (str) – experiment identifier
noclean (bool) – flag telling it whether to clean the experiment or not.
uncompress (bool) – flag telling it whether to decompress or not.
rocrate (bool) – flag to enable RO-Crate

Returns:

True if the experiment has been successfully archived. False otherwise.

Return type:

bool

static cat_log(exp_or_job_id: str, file: None | str, mode: None | str, inspect: bool = False) → bool#

The cat-log command allows users to view Autosubmit logs using the command-line.

It is possible to use autosubmit cat-log for Workflow and for Job logs. It decides whether to show Workflow or Job logs based on the ID given. Shorter ID’s, such as a000` are considered Workflow ID's, so it will display logs for that workflow. For longer ID's, such as ``a000_20220401_fc0_1_GSV, the command will display logs for that specific job.

Users can choose the log file using the FILE parameter, to display an error or output log file, for instance.

Finally, the MODE parameter allows users to choose whether to display the complete file contents (similar to the cat command) or to start tailing its output (akin to tail -f).

Parameters:

exp_or_job_id (str) – A workflow or job ID.
file (Union[None, str]) – the type of the file to be printed (not the file path!).
mode (Union[None, str]) – the mode to print the file (e.g. cat, tail).
inspect (bool) – when True it will use job files in tmp/ instead of tmp/LOG_a000/.

Returns:

Return type:

bool

static change_status(final: str, final_status: int, final_list: list[Job], save: bool, definitive_platforms: list[str], platforms: dict[str, ParamikoPlatform]) → dict[str, str]#

Apply a status change to all jobs in final_list and cancel active jobs on their platforms.

Iterates over final_list, skipping jobs already at final_status. For jobs moving out of an active status, the platform connection is verified and a batch cancel command is queued. All queued cancels are dispatched once after the loop.

Parameters:

final – Human-readable name of the target status.
final_status – Numeric status value to assign.
final_list – Jobs selected for the status change.
save – Whether changes should be persisted.
definitive_platforms – Names of platforms with a confirmed connection.
platforms – Mapping of platform name to Platform instance.

Returns:

Mapping of job name to "old_status -> new_status" strings.

static check(experiment_id: str) → bool#

Checks experiment configuration and warns about any detected error or inconsistency.

Parameters:: experiment_id – experiment identifier:

static check_deadlock(wrapper_errors: dict, any_job_submitted: bool, job_list: JobList) → None#: Check for deadlock situations and raise an exception if detected. :param wrapper_errors: Dictionary of wrapper errors. :param any_job_submitted: Boolean indicating if any job was submitted. :param job_list: JobList object containing the jobs.

static check_wrapper_stored_status(as_conf: AutosubmitConfig, job_list: JobList, wrapper_wallclock: str) → JobList#

Check if the wrapper job has been submitted and the inner jobs are in the queue after a load.

Parameters:

as_conf (BasicConfig) – A BasicConfig object.
job_list (JobList) – A JobList object.
wrapper_wallclock (str) – The wallclock of the wrapper.

Returns:

Updated JobList object.

Return type:

JobList

static check_wrappers(as_conf: AutosubmitConfig, job_list: JobList, platforms_to_test: set[Platform], expid: str) → tuple[dict[str, list[list[Job]]], dict[str, tuple[Status, Status]]]#

Check wrappers and inner jobs status also order the non-wrapped jobs to be submitted by active platforms.

Parameters:

as_conf – a AutosubmitConfig object
job_list – a JobList object
platforms_to_test – a list of Platform
expid – a string with the experiment id

Returns:

non-wrapped jobs to check and a dictionary with the changes in the jobs status

static clean(expid: str, project: bool, plot: bool, stats: bool) → bool#

Clean the experiment directory to save storage space.

It removes project directory and outdated plots or stats.

When it deletes plots or stats, it keeps the two newest files (i.e. two newest statistics files, and two newest plot files).

Parameters:

expid – The experiment ID.
project (bool) – Whether to clean the experiment project folder or not.
plot (bool) – Whether to delete plot files (keeping two newest) or not.
stats (bool) – Whether to delete statistics files (keeping two newest) or not.

Type:

str

Returns:

True is the command ran successfully and False otherwise.

Return type:

bool

Raises:

AutosubmitCritical if anything goes wrong cleaning the experiment folders.

static configure(advanced, database_path, database_filename, local_root_path, platforms_conf_path, jobs_conf_path, smtp_hostname, mail_from, machine: bool, local: bool, database_backend: str = 'sqlite', database_conn_url: str = '')#

Configure several paths for autosubmit: database, local root and others. Can be configured at system, user or local levels. Local level configuration precedes user level and user level precedes system configuration.

Parameters:

advanced
database_path (str) – path to autosubmit database
database_filename (str) – database filename
local_root_path (str) – path to autosubmit’s experiments’ directory
platforms_conf_path (str) – path to platforms conf file to be used as model for new experiments
jobs_conf_path (str) – path to jobs conf file to be used as model for new experiments
machine (bool) – True if this configuration has to be stored for all the machine users
local (bool) – True if this configuration has to be stored in the local path
mail_from (str)
smtp_hostname (str)

static create(expid: str, noplot: bool, hide: bool, output='pdf', group_by: str | None = None, expand: list | None = [], expand_status: str | None = [], check_wrappers=False, detail=False, profile=False, force=False) → int#

Creates job list for given experiment. Configuration files must be valid before executing this process.

Parameters:

detail – Show Job List view in terminal
check_wrappers – Generate possible wrapper in the current workflow
expand_status – Select the statuses to be expanded
expand – Supply the list of dates/members/chunks to filter the list of jobs.
group_by – Groups the jobs automatically by date, member, chunk or split
expid – Experiment identifier
noplot – if True, method omits final plotting of the jobs list. Only needed on large experiments when plotting time can be much larger than creation time.
hide – hides plot window
hide – hides plot window
output – plot’s file format. It can be pdf, png, ps or svg
profile – Whether to use a profiler during the execution of the command or not.
force – Whether to force the creation of a new job object or not.

Returns:

True if successful, False if not

static database_fix(expid: str) → bool#

Database methods. Performs a sql dump of the database and restores it.

Parameters:: expid (str) – experiment identifier
Returns:
Return type:

static describe(input_experiment_list='*', get_from_user='')#

Show details for specified experiment

Parameters:

input_experiment_list (str) – experiments identifier:
get_from_user (str) – user to get the experiments from

Returns:

tuple with user, created time, model, branch, and HPC

Return type:

bool | (str, str, str, str, str)

static environ_init()#: Initialise AS environment.

property experiment_data#: Get the current voltage.

static expid(description, hpc='', copy_id='', dummy=False, minimal_configuration=False, git_repo='', git_branch='', git_as_conf='', operational=False, testcase=False, evaluation=False, use_local_minimal=False) → str#

Creates a new experiment for given HPC.

Parameters:

description (str) – description of the experiment
hpc (str) – HPC where the experiment will be executed
copy_id (str) – if specified, experiment id to copy
dummy (bool) – if true, creates a dummy experiment
minimal_configuration (bool) – if true, creates a minimal configuration
git_repo (str) – git repository to clone
git_branch (str) – git branch to clone
git_as_conf (str) – path to as_conf file in git repository
operational (bool) – if true, creates an operational experiment
testcase (bool)
evaluation (bool) – if true, creates an evaluation experiment
use_local_minimal (bool) – Gets local minimal instead of git minimal

Returns:

return the expid of the experiment.

Return type:

str

static generate_as_config(exp_id: str, dummy: bool = False, minimal_configuration: bool = False, local: bool = False, parameters: dict[str, dict | list | str] | None = None) → None#

Retrieve the configuration from autosubmit.config package.

Parameters:

exp_id – Experiment ID
dummy – Whether the experiment is a dummy one or not.
minimal_configuration – Whether the experiment is configured with minimal configuration or not.
local – Whether the experiment project type is local or not.
parameters – Optional list of parameters to be used when processing the configuration files.

Returns:

None

static generate_scripts_andor_wrappers(as_conf, job_list, jobs_filtered, packages_persistence, only_wrappers=False)#

Parameters:

as_conf (AutosubmitConfig() Object) – Class that handles basic configuration parameters of Autosubmit.
job_list (JobList() Object) – Representation of the jobs of the experiment, keeps the list of jobs inside.
jobs_filtered (List() of Job Objects) – list of jobs that are relevant to the process.
packages_persistence (JobPackagePersistence() Object) – Object that handles local db persistence.
only_wrappers (Boolean) – True when coming from Autosubmit.create(). False when coming from Autosubmit.inspect(),

Returns:

Nothing

Return type:

static get_historical_database(expid, job_list, as_conf)#

Get the historical database for the experiment.

Parameters:

expid – a string with the experiment id
job_list – a JobList object
as_conf – a AutosubmitConfig object

Returns:

an experiment history object

static get_iteration_info(as_conf, job_list) → tuple[int, int, int, bool]#

Prints the current iteration information.

Parameters:

as_conf – autosubmit configuration object
job_list – job list object

Returns:

common parameters for the iteration

static inspect(expid: str, lst: str, filter_chunks: str, filter_status: str, filter_section: str, force=False, check_wrapper=False, quick=False) → bool#

Generates cmd files experiment.

Parameters:

expid (str) – Identifier of experiment to be run
lst (Optional[str]) – Optional list of job names to filter for inspect.
filter_chunks (Optional[str]) – Optional list of chunk identifiers to filter for inspect.
filter_status (Optional[str]) – Optional list of job statuses to filter for inspect.
filter_section (Optional[str]) – Optional list of job sections to filter for inspect.
force (bool) – If true, forces the generation of all cmd files.
check_wrapper (bool) – If true, checks the wrapper.
quick (bool) – If true, performs a quick inspect.

Returns:

True if run to the end, False otherwise

Return type:

bool

Raises:

AutosubmitCritical – If there is a critical error during the inspection process.

static install()#: Creates a new database instance for autosubmit at the configured path.

static load_job_list(expid: str, as_conf: AutosubmitConfig, monitor: bool = False, new: bool = True) → JobList#

Load and generate the job list for an experiment.

Builds a JobList from the experiment configuration. When new is True (the default), a fresh job list is generated from the current configuration, discarding any previously persisted state. Set new to False to load the previously persisted job list state — this is the expected usage when inspecting or operating on an already-created experiment (e.g. recovery, setstatus, statistics).

Parameters:

expid – Experiment identifier (e.g. "a000").
as_conf – Autosubmit configuration object for the experiment.
monitor – When True, the job list is generated in monitor mode. This flag is forwarded to JobList.generate() and JobList.rerun().
new – When True (default), generates a brand-new job list from the experiment configuration. When False, loads the previously saved/persisted job list state instead of regenerating it. Use new=False when the experiment already exists and you need its current state (e.g. for recovery or status queries).

Returns:

The populated job list for the experiment.

static monitor(expid: str, file_format: str, lst: str, filter_chunks: str, filter_status: str, filter_section: str, hide: bool, txt_only=False, group_by: str | None = None, expand='', expand_status='', hide_groups=False, check_wrapper=False, txt_logfiles=False, profile=False) → bool#

Plots workflow graph for a given experiment with status of each job coded by node color.

Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:

txt_logfiles (bool) – Whether to include log file paths in the text output.
expid (str) – Identifier of the experiment to plot.
file_format (str) – Plot file format. It can be pdf, png, ps or svg
lst (str) – list of jobs to change status
filter_chunks (str) – chunks to change status
filter_status (str) – current status of the jobs to change status
filter_section (str) – sections to change status
hide (bool) – hides plot window
txt_only (bool) – workflow will only be written as text
group_by (Optional[str]) – workflow will only be written as text
expand (str) – Filtering of jobs for its visualization
expand_status (str) – Filtering of jobs for its visualization
hide_groups (bool) – Simplified workflow illustration by encapsulating the jobs.
check_wrapper (bool) – Shows a preview of how the wrappers will look
profile (bool) – Whether to enable a profiler during the command execution or not.

Returns:

True if monitor was executed successfully, False otherwise

Return type:

bool

static online_recovery(as_conf: AutosubmitConfig, platforms: list[ParamikoPlatform], job_list: JobList, offline: bool = False) → list[str]#

Return a list of completed job names recovered from the given platforms.

Test each platform connection and collect completed job names. If a platform is not reachable and offline is True, recover completed jobs from the experiment history for that platform. On unrecoverable connection errors raise AutosubmitCritical.

Parameters:

as_conf – Autosubmit configuration object.
platforms – Sequence of Platform objects to query.
job_list – JobList used to recover completed jobs when offline.
offline – If True, proceed with offline recovery when a platform is not reachable.

Returns:

List of completed job names (may be empty).

Raises:

AutosubmitCritical – If a platform is unreachable and offline is False, or if fetching completed job names fails.

static parse_args() → tuple[int, Namespace | None]#

Parse arguments given to an executable and start execution of command given.

Returns a tuple with the exit code (status), and an optional list of arguments for argparse. The list of arguments is only ever returned when the arguments are valid for the execution of a subcommand. Otherwise, they will be None.

static pkl_fix(expid: str, force: bool = False) → int#

Tries to find a backup of the pkl file and restores it. Verifies that autosubmit is not running on this experiment.

Parameters:

expid (str) – experiment identifier
force (bool) – force the operation without confirmation

Returns:

Return type:

static prepare_run(expid: str, start_time: str | None = None, start_after: str | None = None, run_only_members: str | None = None, recover: bool = False, check_scripts: bool = False, submitter: ParamikoSubmitter | None = None) → tuple[JobList, ParamikoSubmitter, ExperimentHistory | None, str | None, AutosubmitConfig, set[Platform], JobPackagePersistence, bool]#

Prepare the run of the experiment.

Parameters:

expid – a string with the experiment id.
start_time – a string with the starting time of the experiment.
start_after – a string with the experiment id to start after.
run_only_members – a string with the members to run.
recover – a boolean to indicate if the experiment is recovering from a failure.
check_scripts – Whether to check the scripts before submitting.
submitter – the actual loaded platforms if any

Returns:

a Union

static process_historical_data_iteration(job_list, job_changes_tracker, expid)#

Process the historical data for the current iteration.

Parameters:

job_list – a JobList object.
job_changes_tracker – a dictionary with the changes in the job status.
expid – a string with the experiment id.

Returns:

an ExperimentHistory object.

static provenance(expid: str, rocrate: bool = False) → bool#

Create the experiment provenance archive.

Parameters:

expid (str) – experiment identifier
rocrate (bool) – flag to enable RO-Crate

static recovery(expid: str, noplot: bool, save: bool, all_jobs: bool, hide: bool, group_by: str | None = None, expand: list[str] | None = None, expand_status: list[str] | None = None, detail: bool = False, force: bool = False, offline: bool = False, filter_list: str | None = None, filter_chunks: str | None = None, filter_status: str | None = None, filter_section: str | None = None) → bool#

Recover job statuses for an experiment and update the job list.

Return True when the recovery completed successfully.

Parameters:

expid (str) – Experiment identifier.
noplot (bool) – If True, do not generate a plot.
save (bool) – If True, persist changes to the job list.
all_jobs (bool) – If True, recover all jobs; otherwise only active jobs.
hide (bool) – If True, hide GUI/windows when generating plots.
group_by (Optional[str]) – Optional grouping key for display.
expand (Optional[list[str]]) – Optional list of job names/sections to expand in the view.
expand_status (Optional[list[str]]) – Optional list of statuses to expand in the view.
detail (bool) – If True, produce a more detailed (and more expensive) textual representation.
force (bool) – If True, cancel active jobs before recovery.
offline (bool) – If True, avoid connecting to remote platforms and use offline recovery.
filter_list (Optional[str]) – Optional list of job names to filter for recovery.
filter_chunks (Optional[str]) – Optional list of chunk identifiers to filter for recovery.
filter_status (Optional[str]) – Optional list of job statuses to filter for recovery.
filter_section (Optional[str]) – Optional list of job sections to filter for recovery.

Returns:

True if recovery ran successfully, False otherwise.

Return type:

bool

Raises:

AutosubmitCritical – On configuration/IO failures.

static refresh(expid, model_conf, jobs_conf)#

Refresh project folder for given experiment

Parameters:

model_conf (bool)
jobs_conf (bool)
expid (str) – experiment identifier

static refresh_log_recovery_process(platforms: list[Platform], as_conf: AutosubmitConfig) → None#: Relaunch the log recovery processes for each platform if necessary.

static report(expid: str, template_file_path='', show_all_parameters=False, folder_path='', placeholders=False) → bool#

Show report for specified experiment.

Parameters:

expid – experiment identifier
template_file_path – path to template file
show_all_parameters – show all parameters
folder_path – path to folder
placeholders – show placeholders

static rocrate(expid: str, path: Path) → ROCrate | None#

Produces an RO-Crate archive for an Autosubmit experiment.

Skips other crate ZIP archive files in tmp/ASLOGS. It ignores files that start with <EXPID>-crate and end with .zip.

Parameters:

expid – experiment ID
path – path to save the RO-Crate in

Returns:

True if successful, False otherwise

static run_experiment(expid: str, start_time: str | None = None, start_after: str | None = None, run_only_members: str | None = None, profile: int | None = None, trace: bool = False) → int#

Runs and experiment (submitting all the jobs properly and repeating its execution in case of failure).

Parameters:

expid – the experiment id
start_time – the time at which the experiment should start
start_after – the expid after which the experiment should start
run_only_members – the members to run
profile – if >0, profile will stop after N iterations, if 0, profile will not stop until the experiment finishes or is stopped by the user
trace – if True, the function will be traced

Returns:

exit status

static set_status(expid: str, noplot: bool, save: bool, final: str, filter_list: str, filter_chunks: str, filter_status: str, filter_section: str, filter_type_chunk: str, filter_type_chunk_split: str, hide: bool, group_by: str | None = None, expand: list | None = None, expand_status: str | None = None, check_wrapper=False, detail=False) → bool#

Set status of jobs.

Parameters:

expid (str) – experiment id
noplot (bool) – do not plot
save (bool) – save
final (str) – final status
filter_list (str) – list of jobs
filter_chunks (str) – filter chunks
filter_status (str) – filter status
filter_section (str) – filter section
filter_type_chunk (str) – filter type chunk
filter_type_chunk_split (str) – filter chunk split
hide (bool) – hide
group_by (Optional[str]) – group by
expand (Optional[list]) – Whether to expand during job grouping or not.
expand_status (Optional[str]) – The status to use when expanding.
check_wrapper (bool) – check wrapper
detail (bool) – detail

Returns:

True if executed successfully.

Return type:

bool

Raises:

AutosubmitCritical – if any of the filters is malformed or if no filter is provided, with a message describing the errors found.

static statistics(expid: str, filter_type: str, filter_period: int, file_format: str, section_summary: bool, jobs_summary: bool, hide: bool) → bool#

Plots statistics graph for a given experiment. Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:

expid – Identifier of the experiment to plot.
filter_type – section of the jobs to plot
filter_period – period to plot
file_format – plot’s file format. It can be pdf, png, ps or svg
section_summary – shows summary statistics
jobs_summary – shows jobs statistics
hide – hides plot window

static stop(expids: str, force=False, all_expids=False, force_all=False, cancel=False, current_status='', status='FAILED', force_yes=False) → bool#

The stop command allows users to stop the desired experiments.

Parameters:

expids (str) – expids to stop
force (bool) – force the stop of the experiment
all_expids (bool) – stop all experiments
force_all (bool) – force the stop of all experiments
cancel (bool) – cancel the jobs of the experiment
current_status (str) – what status to change # defaults to all active jobs.
status (str) – status to change the active jobs to
force_yes (bool) – force yes answer to prompts

static submit_ready_jobs(as_conf: AutosubmitConfig, job_list: JobList, platforms_to_test: list[ParamikoPlatform], packages_persistence: JobPackagePersistence, inspect=False, only_wrappers=False) → None#

Gets READY jobs and send them to the platforms if there is available space on the queues.

Parameters:

as_conf (AutosubmitConfig object) – autosubmit config object
job_list (JobList object) – job list to check
platforms_to_test (set of Platform Objects, e.g. SgePlatform(), SlurmPlatform().) – platforms used
packages_persistence (JobPackagePersistence object) – Handles database per experiment.
inspect (Boolean) – True if coming from generate_scripts_andor_wrappers().
only_wrappers (Boolean) – True if it comes from create -cw, False if it comes from inspect -cw.

Returns:

True if at least one job was submitted, False otherwise

static testcase(description, hpc=None, copy_id=None, minimal_configuration=False, git_repo=None, git_branch=None, git_as_conf=None, use_local_minimal=False)#: Method to conduct a test for a given experiment. It creates a new experiment for a given experiment with a given number of chunks with a random start date and a random member to be run on a random HPC. :param description: description of the experiment :param hpc: HPC to be used by the test. If None, a random HPC will be chosen :param copy_id: copy id to be used by the test. If None, a random copy id will be chosen :param minimal_configuration: if True, the experiment will be run with a minimal configuration :param git_repo: git repository to be used by the test. If None, a random git repository will be chosen :param git_branch: git branch to be used by the test. If None, a random git branch will be chosen :param git_as_conf: git autosubmit configuration to be used by the test. If None, a random git autosubmit configuration will be chosen. :param use_local_minimal: if True, the experiment will be run with a local minimal configuration :return: Whether the test case ID was generated successfully or not.

static unarchive(experiment_id, uncompressed=True, rocrate=False)#

Unarchives an experiment: uncompress folder from tar.gz and moves to experiment root folder.

Parameters:

experiment_id (str) – experiment identifier
uncompressed (bool) – if True, the tar file is uncompressed
rocrate (bool) – flag to enable RO-Crate

static update_version(expid)#

Refresh experiment version with the current autosubmit version.

Parameters:: expid (str) – experiment identifier

class autosubmit.autosubmit.CancelAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)#: Bases: Action

class autosubmit.autosubmit.MyParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True)#

Bases: ArgumentParser

add_argument(dest, ..., name=value, ...)#
add_argument(option_string, option_string, ..., name=value, ...) → None

error(message: string)#

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.

autosubmit.autosubmit.signal_handler(signal_received, frame)#

Used to handle interrupt signals, allowing autosubmit to clean before exit

Parameters:

signal_received
frame

Returns:

autosubmit.autosubmit.signal_handler_create(signal_received, frame)#

Used to handle KeyboardInterrupt signals while the create method is being executed

Parameters:

signal_received
frame

Returns:

autosubmit#

This Page