Ctrl+K

autosubmit#

class autosubmit.autosubmit.Autosubmit#

Bases: object

Interface class for autosubmit.

static archive(expid, noclean=True, uncompress=True, rocrate=False)#

Archives an experiment: call clean (if experiment is of version 3 or later), compress folder to tar.gz and moves to year’s folder

Parameters:

  • expid (str) – experiment identifier

  • noclean (bool) – flag telling it whether to clean the experiment or not.

  • uncompress (bool) – flag telling it whether to decompress or not.

  • rocrate (bool) – flag to enable RO-Crate

Returns:

True if the experiment has been successfully archived. False otherwise.

Return type:

bool

static as_conf_default_values(exp_id, hpc='local', minimal_configuration=False, git_repo='', git_branch='main', git_as_conf='')#

Replace default values in as_conf files :param exp_id: experiment id :param hpc: platform :param minimal_configuration: minimal configuration :param git_repo: path to project git repository :param git_branch: main branch :param git_as_conf: path to as_conf file in git repository :return: None

static cat_log(exp_or_job_id: str, file: Union[None, str], mode: Union[None, str], inspect: bool = False) → bool#

The cat-log command allows users to view Autosubmit logs using the command-line.

It is possible to use autosubmit cat-log for Workflow and for Job logs. It decides whether to show Workflow or Job logs based on the ID given. Shorter ID’s, such as a000` are considered Workflow ID's, so it will display logs for that workflow. For longer ID's, such as ``a000_20220401_fc0_1_GSV, the command will display logs for that specific job.

Users can choose the log file using the FILE parameter, to display an error or output log file, for instance.

Finally, the MODE parameter allows users to choose whether to display the complete file contents (similar to the cat command) or to start tailing its output (akin to tail -f).

Args:

exp_or_job_id: A workflow or job ID. file: the type of the file to be printed (not the file path!). mode: the mode to print the file (e.g. cat, tail). inspect: when True it will use job files in tmp/ instead of tmp/LOG_a000/.

static change_status(final, final_status, job, save)#

Set job status to final

Parameters:

  • save –

  • final –

  • final_status –

  • job –

static check(experiment_id, notransitive=False)#

Checks experiment configuration and warns about any detected error or inconsistency.

Parameters:

  • notransitive –

  • experiment_id (str) – experiment identifier:

static check_wrapper_stored_status(as_conf, job_list)#

Check if the wrapper job has been submitted and the inner jobs are in the queue. :param as_conf: a BasicConfig object :param job_list: a JobList object :return: JobList object updated

static check_wrappers(as_conf, job_list, platforms_to_test, expid)#

Check wrappers and inner jobs status also order the non-wrapped jobs to be submitted by active platforms :param as_conf: a AutosubmitConfig object :param job_list: a JobList object :param platforms_to_test: a list of Platform :param expid: a string with the experiment id :return: non-wrapped jobs to check and a dictionary with the changes in the jobs status

static clean(expid, project, plot, stats)#

Clean experiment’s directory to save storage space. It removes project directory and outdated plots or stats.

Parameters:

  • expid (str) – identifier of experiment to clean

  • project (bool) – set True to delete project directory

  • plot (bool) – set True to delete outdated plots

  • stats (bool) – set True to delete outdated stats

static configure(advanced, database_path, database_filename, local_root_path, platforms_conf_path, jobs_conf_path, smtp_hostname, mail_from, machine, local)#

Configure several paths for autosubmit: database, local root and others. Can be configured at system, user or local levels. Local level configuration precedes user level and user level precedes system configuration.

Parameters:

  • advanced –

  • database_path (str) – path to autosubmit database

  • database_filename (str) – database filename

  • local_root_path (str) – path to autosubmit’s experiments’ directory

  • platforms_conf_path (str) – path to platforms conf file to be used as model for new experiments

  • jobs_conf_path (str) – path to jobs conf file to be used as model for new experiments

  • machine (bool) – True if this configuration has to be stored for all the machine users

  • local (bool) – True if this configuration has to be stored in the local path

  • mail_from (str) –

  • smtp_hostname (str) –

static configure_dialog()#

Configure several paths for autosubmit interactively: database, local root and others. Can be configured at system, user or local levels. Local level configuration precedes user level and user level precedes system configuration.

static create(expid, noplot, hide, output='pdf', group_by=None, expand=[], expand_status=[], notransitive=False, check_wrappers=False, detail=False, profile=False, force=False)#

Creates job list for given experiment. Configuration files must be valid before executing this process.

Parameters:

  • detail –

  • check_wrappers –

  • notransitive –

  • expand_status –

  • expand –

  • group_by –

  • expid (str) – experiment identifier

  • noplot (bool) – if True, method omits final plotting of the jobs list. Only needed on large experiments when plotting time can be much larger than creation time.

  • hide (bool) – hides plot window

  • hide – hides plot window

  • output (str) – plot’s file format. It can be pdf, png, ps or svg

Returns:

True if successful, False if not

Return type:

bool

static database_fix(expid)#

Database methods. Performs a sql dump of the database and restores it.

Parameters:

expid (str) – experiment identifier

Returns:

Return type:

static delete(expid, force)#

Deletes and experiment from database and experiment’s folder

Parameters:

  • expid (str) – identifier of the experiment to delete

  • force (bool) – if True, does not ask for confirmation

Returns:

True if successful, False if not

Return type:

bool

static describe(input_experiment_list='*', get_from_user='')#

Show details for specified experiment

Parameters:

  • experiments_id (str) – experiments identifier:

  • get_from_user (str) – user to get the experiments from

Returns:

str,str,str,str

static environ_init()#

Initialise AS environment.

property experiment_data#

Get the current voltage.

static expid(description, hpc='', copy_id='', dummy=False, minimal_configuration=False, git_repo='', git_branch='', git_as_conf='', operational=False, testcase=False, use_local_minimal=False)#

Creates a new experiment for given HPC description: description of the experiment hpc: HPC where the experiment will be executed copy_id: if specified, experiment id to copy dummy: if true, creates a dummy experiment minimal_configuration: if true, creates a minimal configuration git_repo: git repository to clone git_branch: git branch to clone git_as_conf: path to as_conf file in git repository operational: if true, creates an operational experiment local: Gets local minimal instead of git minimal

static generate_as_config(exp_id: str, dummy: bool = False, minimal_configuration: bool = False, local: bool = False, parameters: Optional[Dict[str, Union[Dict, List, str]]] = None) → None#

Retrieve the configuration from autosubmitconfigparser package.

Parameters:

  • exp_id – Experiment ID

  • dummy – Whether the experiment is a dummy one or not.

  • minimal_configuration – Whether the experiment is configured with minimal configuration or not.

  • local – Whether the experiment project type is local or not.

  • parameters – Optional list of parameters to be used when processing the configuration files.

Returns:

None

static generate_scripts_andor_wrappers(as_conf, job_list, jobs_filtered, packages_persistence, only_wrappers=False)#
Parameters:

  • as_conf (AutosubmitConfig() Object) – Class that handles basic configuration parameters of Autosubmit.

  • job_list (JobList() Object) – Representation of the jobs of the experiment, keeps the list of jobs inside.

  • jobs_filtered (List() of Job Objects) – list of jobs that are relevant to the process.

  • packages_persistence (JobPackagePersistence() Object) – Object that handles local db persistence.

  • only_wrappers (Boolean) – True when coming from Autosubmit.create(). False when coming from Autosubmit.inspect(),

Returns:

Nothing

Return type:

static get_historical_database(expid, job_list, as_conf)#

Get the historical database for the experiment :param expid: a string with the experiment id :param job_list: a JobList object :param as_conf: a AutosubmitConfig object :return: an experiment history object

static get_iteration_info(as_conf, job_list)#

Prints the current iteration information :param as_conf: autosubmit configuration object :param job_list: job list object :return: common parameters for the iteration

static inspect(expid, lst, filter_chunks, filter_status, filter_section, notransitive=False, force=False, check_wrapper=False, quick=False)#

Generates cmd files experiment.

Parameters:

  • check_wrapper –

  • force –

  • notransitive –

  • filter_section –

  • filter_status –

  • filter_chunks –

  • lst –

  • expid (str) – identifier of experiment to be run

Returns:

True if run to the end, False otherwise

Return type:

bool

static install()#

Creates a new database instance for autosubmit at the configured path

static migrate(experiment_id, offer, pickup, only_remote)#

Migrates experiment files from current to other user. It takes mapping information for new user from config files.

Parameters:

  • experiment_id – experiment identifier:

  • pickup –

  • offer –

  • only_remote –

static monitor(expid, file_format, lst, filter_chunks, filter_status, filter_section, hide, txt_only=False, group_by=None, expand='', expand_status=[], hide_groups=False, notransitive=False, check_wrapper=False, txt_logfiles=False, profile=False, detail=False)#

Plots workflow graph for a given experiment with status of each job coded by node color. Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:

  • txt_logfiles –

  • expid (str) – identifier of the experiment to plot

  • file_format (str) – plot’s file format. It can be pdf, png, ps or svg

  • lst (str) – list of jobs to change status

  • filter_chunks (str) – chunks to change status

  • filter_status (str) – current status of the jobs to change status

  • filter_section (str) – sections to change status

  • hide (bool) – hides plot window

  • txt_only (bool) – workflow will only be written as text

  • group_by (bool) – workflow will only be written as text

  • expand (str) – Filtering of jobs for its visualization

  • expand_status (str) – Filtering of jobs for its visualization

  • hide_groups (bool) – Simplified workflow illustration by encapsulating the jobs.

  • notransitive (bool) – workflow will only be written as text

  • check_wrapper (bool) – Shows a preview of how the wrappers will look

  • notransitive – Some dependencies will be omitted

  • detail (bool) – better text format representation but more expensive

static parse_args()#

Parse arguments given to an executable and start execution of command given

static pkl_fix(expid)#

Tries to find a backup of the pkl file and restores it. Verifies that autosubmit is not running on this experiment.

Parameters:

expid (str) – experiment identifier

Returns:

Return type:

static prepare_run(expid, notransitive=False, start_time=None, start_after=None, run_only_members=None, recover=False)#

Prepare the run of the experiment. :param expid: a string with the experiment id. :param notransitive: a boolean to indicate for the experiment to not use transitive dependencies. :param start_time: a string with the starting time of the experiment. :param start_after: a string with the experiment id to start after. :param run_only_members: a string with the members to run. :param recover: a boolean to indicate if the experiment is recovering from a failure. :return: a tuple

static process_historical_data_iteration(job_list, job_changes_tracker, expid)#

Process the historical data for the current iteration. :param job_list: a JobList object. :param job_changes_tracker: a dictionary with the changes in the job status. :param expid: a string with the experiment id. :return: an ExperimentHistory object.

static recovery(expid, noplot, save, all_jobs, hide, group_by=None, expand=[], expand_status=[], notransitive=False, no_recover_logs=False, detail=False, force=False)#

Method to check all active jobs. If COMPLETED file is found, job status will be changed to COMPLETED, otherwise it will be set to WAITING. It will also update the jobs list.

Parameters:

  • detail –

  • no_recover_logs –

  • notransitive –

  • expand_status –

  • expand –

  • group_by –

  • noplot –

  • expid (str) – identifier of the experiment to recover

  • save (bool) – If true, recovery saves changes to the jobs list

  • all_jobs (bool) – if True, it tries to get completed files for all jobs, not only active.

  • hide (bool) – hides plot window

  • force (bool) – Allows to restore the workflow even if there are running jobs

static refresh(expid, model_conf, jobs_conf)#

Refresh project folder for given experiment

Parameters:

  • model_conf (bool) –

  • jobs_conf (bool) –

  • expid (str) – experiment identifier

static report(expid, template_file_path='', show_all_parameters=False, folder_path='', placeholders=False)#

Show report for specified experiment :param expid: experiment identifier :type expid: str :param template_file_path: path to template file :type template_file_path: str :param show_all_parameters: show all parameters :type show_all_parameters: bool :param folder_path: path to folder :type folder_path: str :param placeholders: show placeholders :type placeholders: bool

static rerun_recovery(expid, job_list, rerun_list, as_conf)#

Method to check all active jobs. If COMPLETED file is found, job status will be changed to COMPLETED, otherwise it will be set to WAITING. It will also update the jobs list.

Parameters:

  • expid (str) – identifier of the experiment to recover

  • job_list (JobList) – job list to update

  • rerun_list (list) – list of jobs to rerun

  • as_conf (AutosubmitConfig) – AutosubmitConfig object

Returns:

static rocrate(expid, path: Path)#

Produces an RO-Crate archive for an Autosubmit experiment.

Parameters:

  • expid (str) – experiment ID

  • path (Path) – path to save the RO-Crate in

Returns:

True if successful, False otherwise

Return type:

bool

static run_experiment(expid, notransitive=False, start_time=None, start_after=None, run_only_members=None, profile=False)#

Runs and experiment (submitting all the jobs properly and repeating its execution in case of failure). :param expid: the experiment id :param notransitive: if True, the transitive closure of the graph is not computed :param start_time: the time at which the experiment should start :param start_after: the expid after which the experiment should start :param run_only_members: the members to run :param profile: if True, the function will be profiled :return: None

static set_status(expid, noplot, save, final, filter_list, filter_chunks, filter_status, filter_section, filter_type_chunk, filter_type_chunk_split, hide, group_by=None, expand=[], expand_status=[], notransitive=False, check_wrapper=False, detail=False)#

Set status of jobs :param expid: experiment id :param noplot: do not plot :param save: save :param final: final status :param filter_list: list of jobs :param filter_chunks: filter chunks :param filter_status: filter status :param filter_section: filter section :param filter_type_chunk: filter type chunk :param filter_chunk_split: filter chunk split :param hide: hide :param group_by: group by :param expand: expand :param expand_status: expand status :param notransitive: notransitive :param check_wrapper: check wrapper :param detail: detail :return:

static statistics(expid, filter_type, filter_period, file_format, hide, notransitive=False, db=False)#

Plots statistics graph for a given experiment. Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:

  • expid (str) – identifier of the experiment to plot

  • filter_type – type of the jobs to plot

  • filter_period – period to plot

  • file_format (str) – plot’s file format. It can be pdf, png, ps or svg

  • hide (bool) – hides plot window

  • notransitive – Reduces workflow linkage complexity

  • db (bool) – Use database to get the statistics

static submit_ready_jobs(as_conf: AutosubmitConfig, job_list: JobList, platforms_to_test: Set[Platform], packages_persistence: JobPackagePersistence, inspect: bool = False, only_wrappers: bool = False, hold: bool = False) → bool#

Gets READY jobs and send them to the platforms if there is available space on the queues

Parameters:

  • hold –

  • as_conf (AutosubmitConfig object) – autosubmit config object

  • job_list (JobList object) – job list to check

  • platforms_to_test (set of Platform Objects, e.g. SgePlatform(), LsfPlatform().) – platforms used

  • packages_persistence (JobPackagePersistence object) – Handles database per experiment.

  • inspect (Boolean) – True if coming from generate_scripts_andor_wrappers().

  • only_wrappers (Boolean) – True if it comes from create -cw, False if it comes from inspect -cw.

Returns:

True if at least one job was submitted, False otherwise

Return type:

Boolean

static test(expid, chunks, member=None, start_date=None, hpc=None, branch=None)#

Method to conduct a test for a given experiment. It creates a new experiment for a given experiment with a given number of chunks with a random start date and a random member to be run on a random HPC.

Parameters:

  • expid (str) – experiment identifier

  • chunks (int) – number of chunks to be run by the experiment

  • member (str) – member to be used by the test. If None, it uses a random one from which are defined on the experiment.

  • start_date (str) – start date to be used by the test. If None, it uses a random one from which are defined on the experiment.

  • hpc (str) – HPC to be used by the test. If None, it uses a random one from which are defined on the experiment.

  • branch (str) – branch or revision to be used by the test. If None, it uses configured branch.

Returns:

True if test was successful, False otherwise

Return type:

bool

static testcase(description, chunks=None, member=None, start_date=None, hpc=None, copy_id=None, minimal_configuration=False, git_repo=None, git_branch=None, git_as_conf=None, use_local_minimal=False)#

Method to conduct a test for a given experiment. It creates a new experiment for a given experiment with a given number of chunks with a random start date and a random member to be run on a random HPC. :param description: description of the experiment :type description: str :param chunks: number of chunks to be run by the experiment :type chunks: int :param member: member to be used by the test. If None, a random member will be chosen :type member: str :param start_date: start date of the experiment. If None, a random start date will be chosen :type start_date: str :param hpc: HPC to be used by the test. If None, a random HPC will be chosen :type hpc: str :param copy_id: copy id to be used by the test. If None, a random copy id will be chosen :type copy_id: str :param minimal_configuration: if True, the experiment will be run with a minimal configuration :type minimal_configuration: bool :param git_repo: git repository to be used by the test. If None, a random git repository will be chosen :type git_repo: str :param git_branch: git branch to be used by the test. If None, a random git branch will be chosen :type git_branch: str :param git_as_conf: git autosubmit configuration to be used by the test. If None, a random git autosubmit configuration will be chosen :type git_as_conf: str :param use_local_minimal: if True, the experiment will be run with a local minimal configuration :type use_local_minimal: bool :return: experiment identifier :rtype: str

static unarchive(experiment_id, uncompressed=True, rocrate=False)#

Unarchives an experiment: uncompress folder from tar.gz and moves to experiment root folder

Parameters:

  • experiment_id (str) – experiment identifier

  • uncompressed (bool) – if True, the tar file is uncompressed

  • rocrate (bool) – flag to enable RO-Crate

static update_version(expid)#

Refresh experiment version with the current autosubmit version :param expid: experiment identifier :type expid: str

class autosubmit.autosubmit.MyParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True)#

Bases: ArgumentParser

add_argument(dest, ..., name=value, ...)#
add_argument(option_string, option_string, ..., name=value, ...) → None
error(message: string)#

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.

autosubmit.autosubmit.signal_handler(signal_received, frame)#

Used to handle interrupt signals, allowing autosubmit to clean before exit

Parameters:

  • signal_received –

  • frame –

autosubmit.autosubmit.signal_handler_create(signal_received, frame)#

Used to handle KeyboardInterrupt signals while the create method is being executed

Parameters:

  • signal_received –

  • frame –

On this page
Edit on GitLab
Show Source