autosubmit

class autosubmit.autosubmit.Autosubmit

Interface class for autosubmit.

static archive(expid, noclean=True, uncompress=True)

Archives an experiment: call clean (if experiment is of version 3 or later), compress folder to tar.gz and moves to year’s folder

Parameters:
  • clean,compress –
  • expid (str) – experiment identifier
Returns:
static change_status(final, final_status, job, save)

Set job status to final

Parameters:
  • final –
  • final_status –
  • job –
static check(experiment_id, notransitive=False)

Checks experiment configuration and warns about any detected error or inconsistency.

Parameters:experiment_id (str) – experiment identifier:
static clean(expid, project, plot, stats)

Clean experiment’s directory to save storage space. It removes project directory and outdated plots or stats.

Parameters:
  • create_log_file (bool) – if true, creates log file
  • expid (str) – identifier of experiment to clean
  • project (bool) – set True to delete project directory
  • plot (bool) – set True to delete outdated plots
  • stats (bool) – set True to delete outdated stats
static configure(advanced, database_path, database_filename, local_root_path, platforms_conf_path, jobs_conf_path, smtp_hostname, mail_from, machine, local)

Configure several paths for autosubmit: database, local root and others. Can be configured at system, user or local levels. Local level configuration precedes user level and user level precedes system configuration.

Parameters:
  • database_path (str) – path to autosubmit database
  • database_filename (str) – database filename
  • local_root_path (str) – path to autosubmit’s experiments’ directory
  • platforms_conf_path (str) – path to platforms conf file to be used as model for new experiments
  • jobs_conf_path (str) – path to jobs conf file to be used as model for new experiments
  • machine (bool) – True if this configuration has to be stored for all the machine users
  • local (bool) – True if this configuration has to be stored in the local path
  • mail_from (str) –
  • smtp_hostname (str) –
static configure_dialog()

Configure several paths for autosubmit interactively: database, local root and others. Can be configured at system, user or local levels. Local level configuration precedes user level and user level precedes system configuration.

static create(expid, noplot, hide, output='pdf', group_by=None, expand=[], expand_status=[], notransitive=False, check_wrappers=False, detail=False)

Creates job list for given experiment. Configuration files must be valid before executing this process.

Parameters:
  • expid (str) – experiment identifier
  • noplot – if True, method omits final plotting of the jobs list. Only needed on large experiments when

plotting time can be much larger than creation time. :type noplot: bool :return: True if successful, False if not :rtype: bool :param hide: hides plot window :type hide: bool :param hide: hides plot window :type hide: bool :param output: plot’s file format. It can be pdf, png, ps or svg :type output: str

static database_fix(expid)

Database methods. Performs a sql dump of the database and restores it.

Parameters:expid (str) – experiment identifier
Returns:
Return type:
static delete(expid, force)

Deletes and experiment from database and experiment’s folder

Parameters:
  • expid (str) – identifier of the experiment to delete
  • force (bool) – if True, does not ask for confirmation
Returns:

True if succesful, False if not
Return type:

bool
static describe(experiment_id)

Show details for specified experiment

Parameters:experiment_id (str) – experiment identifier:
static expid(hpc, description, copy_id='', dummy=False, test=False, operational=False, root_folder='')

Creates a new experiment for given HPC

Parameters:
  • operational (bool) – if true, creates an operational experiment
  • hpc (str) – name of the main HPC for the experiment
  • description (str) – short experiment’s description.
  • copy_id (str) – experiment identifier of experiment to copy
  • dummy (bool) – if true, writes a default dummy configuration for testing
  • test – if true, creates an experiment for testing
Returns:

experiment identifier. If method fails, returns ‘’.
Return type:

str
static generate_scripts_andor_wrappers(as_conf, job_list, jobs_filtered, packages_persistence, only_wrappers=False)
Parameters:
  • as_conf (AutosubmitConfig() Object) – Class that handles basic configuration parameters of Autosubmit.
  • job_list (JobList() Object) – Representation of the jobs of the experiment, keeps the list of jobs inside.
  • jobs_filtered (List() of Job Objects) – list of jobs that are relevant to the process.
  • packages_persistence (JobPackagePersistence() Object) – Object that handles local db persistence.
  • only_wrappers (Boolean) – True when coming from Autosubmit.create(). False when coming from Autosubmit.inspect(),
Returns:

Nothing
Return type:
static inspect(expid, lst, filter_chunks, filter_status, filter_section, notransitive=False, force=False, check_wrapper=False)

Generates cmd files experiment.

Parameters:expid (str) – identifier of experiment to be run
Returns:True if run to the end, False otherwise
Return type:bool
static install()

Creates a new database instance for autosubmit at the configured path

static migrate(experiment_id, offer, pickup, only_remote)

Migrates experiment files from current to other user. It takes mapping information for new user from config files.

Parameters:
  • experiment_id – experiment identifier:
  • pickup –
  • offer –
  • only_remote –
static monitor(expid, file_format, lst, filter_chunks, filter_status, filter_section, hide, txt_only=False, group_by=None, expand='', expand_status=[], hide_groups=False, notransitive=False, check_wrapper=False, txt_logfiles=False, detail=False)

Plots workflow graph for a given experiment with status of each job coded by node color. Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:
  • expid (str) – identifier of the experiment to plot
  • file_format (str) – plot’s file format. It can be pdf, png, ps or svg
  • lst (str) – list of jobs to change status
  • filter_chunks (str) – chunks to change status
  • filter_status (str) – current status of the jobs to change status
  • filter_section (str) – sections to change status
  • hide (bool) – hides plot window
  • txt_only (bool) – workflow will only be written as text
  • group_by (bool) – workflow will only be written as text
  • expand (str) – Filtering of jobs for it’s visualization
  • expand_status (str) – Filtering of jobs for it’s visualization
  • hide_groups (bool) – Simplified workflow illustration by encapsulating the jobs.
  • notransitive (bool) – workflow will only be written as text
  • check_wrapper (bool) – Shows a preview of how the wrappers will look
  • notransitive – Some dependencies will be omitted
  • detail (bool) – better text format representation but more expensive
static parse_args()

Parse arguments given to an executable and start execution of command given

static pkl_fix(expid)

Tries to find a backup of the pkl file and restores it. Verifies that autosubmit is not running on this experiment.

Parameters:expid (str) – experiment identifier
Returns:
Return type:
static recovery(expid, noplot, save, all_jobs, hide, group_by=None, expand=[], expand_status=[], notransitive=False, no_recover_logs=False, detail=False, force=False)

Method to check all active jobs. If COMPLETED file is found, job status will be changed to COMPLETED, otherwise it will be set to WAITING. It will also update the jobs list.

Parameters:
  • expid (str) – identifier of the experiment to recover
  • save (bool) – If true, recovery saves changes to the jobs list
  • all_jobs (bool) – if True, it tries to get completed files for all jobs, not only active.
  • hide (bool) – hides plot window
  • force (bool) – Allows to restore the workflow even if there are running jobs
static refresh(expid, model_conf, jobs_conf)

Refresh project folder for given experiment

Parameters:
  • model_conf (bool) –
  • jobs_conf (bool) –
  • expid (str) – experiment identifier
static report(expid, template_file_path='', show_all_parameters=False, folder_path='', placeholders=False)

Show report for specified experiment :param expid: experiment identifier: :type str :param template_file_path: template filepath :type str :param show_all_parameters: Write all parameters of the experiment :type bool :param folder_path: Allows to put the report files on another folder :type str

static rerun_recovery(expid, job_list, rerun_list, as_conf)

Method to check all active jobs. If COMPLETED file is found, job status will be changed to COMPLETED, otherwise it will be set to WAITING. It will also update the jobs list.

Parameters:
  • expid (str) – identifier of the experiment to recover
  • save (bool) – If true, recovery saves changes to the jobs list
  • all_jobs (bool) – if True, it tries to get completed files for all jobs, not only active.
  • hide (bool) – hides plot window
static run_experiment(expid, notransitive=False, update_version=False, start_time=None, start_after=None, run_members=None)

Runs and experiment (submitting all the jobs properly and repeating its execution in case of failure).

Parameters:expid (str) – identifier of experiment to be run
Returns:True if run to the end, False otherwise
Return type:bool
static set_status(expid, noplot, save, final, lst, filter_chunks, filter_status, filter_section, filter_type_chunk, hide, group_by=None, expand=[], expand_status=[], notransitive=False, check_wrapper=False, detail=False)

Set status

Parameters:
  • expid (str) – experiment identifier
  • save (bool) – if true, saves the new jobs list
  • final (str) – status to set on jobs
  • lst (str) – list of jobs to change status
  • filter_chunks (str) – chunks to change status
  • filter_status (str) – current status of the jobs to change status
  • filter_section (str) – sections to change status
  • hide (bool) – hides plot window
static statistics(expid, filter_type, filter_period, file_format, hide, notransitive=False)

Plots statistics graph for a given experiment. Plot is created in experiment’s plot folder with name <expid>_<date>_<time>.<file_format>

Parameters:
  • expid (str) – identifier of the experiment to plot
  • filter_type – type of the jobs to plot
  • filter_period – period to plot
  • file_format (str) – plot’s file format. It can be pdf, png, ps or svg
  • hide (bool) – hides plot window
  • notransitive – Reduces workflow linkage complexity
static submit_ready_jobs(as_conf, job_list, platforms_to_test, packages_persistence, inspect=False, only_wrappers=False, hold=False)

Gets READY jobs and send them to the platforms if there is available space on the queues

Parameters:
  • as_conf (AutosubmitConfig object) – autosubmit config object
  • job_list (JobList object) – job list to check
  • platforms_to_test (set of Platform Objects, e.g. SgePlatform(), LsfPlatform()) – platforms used
  • packages_persistence (JobPackagePersistence object) – Handles database per experiment.
  • inspect (Boolean) – True if coming from generate_scripts_andor_wrappers().
  • only_wrappers (Boolean) – True if it comes from create -cw, False if it comes from inspect -cw.
Returns:

True if at least one job was submitted, False otherwise
Return type:

Boolean
static test(expid, chunks, member=None, start_date=None, hpc=None, branch=None)

Method to conduct a test for a given experiment. It creates a new experiment for a given experiment with a given number of chunks with a random start date and a random member to be run on a random HPC.

Parameters:
  • expid (str) – experiment identifier
  • chunks (int) – number of chunks to be run by the experiment
  • member (str) – member to be used by the test. If None, it uses a random one from which are defined on the experiment.
  • start_date (str) – start date to be used by the test. If None, it uses a random one from which are defined on the experiment.
  • hpc (str) – HPC to be used by the test. If None, it uses a random one from which are defined on the experiment.
  • branch (str) – branch or revision to be used by the test. If None, it uses configured branch.
Returns:

True if test was succesful, False otherwise
Return type:

bool
static testcase(copy_id, description, chunks=None, member=None, start_date=None, hpc=None, branch=None)

Method to create a test case. It creates a new experiment whose id starts by ‘t’.

Parameters:
  • copy_id (str) – experiment identifier
  • description (str) – test case experiment description
  • chunks (int) – number of chunks to be run by the experiment. If None, it uses configured chunk(s).
  • member (str) – member to be used by the test. If None, it uses configured member(s).
  • start_date (str) – start date to be used by the test. If None, it uses configured start date(s).
  • hpc (str) – HPC to be used by the test. If None, it uses configured HPC.
  • branch (str) – branch or revision to be used by the test. If None, it uses configured branch.
Returns:

test case id
Return type:

str
static unarchive(experiment_id, uncompressed=True)

Unarchives an experiment: uncompress folder from tar.gz and moves to experiments root folder

Parameters:experiment_id (str) – experiment identifier
static update_version(expid)

Refresh experiment version with the current autosubmit version :param expid: experiment identifier :type expid: str

class autosubmit.autosubmit.MyParser(prog=None, usage=None, description=None, epilog=None, version=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True)

Bases: argparse.ArgumentParser

add_argument(dest, ..., name=value, ...)

add_argument(option_string, option_string, …, name=value, …)

error(message: string)

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.

autosubmit.autosubmit.signal_handler(signal_received, frame)

Used to handle interrupt signals, allowing autosubmit to clean before exit

Parameters:
  • signal_received –
  • frame –
autosubmit.autosubmit.signal_handler_create(signal_received, frame)

Used to handle KeyboardInterrumpt signals while the create method is being executed

Parameters:
  • signal_received –
  • frame –