autosubmit.config

autosubmit.config.basicConfig

class autosubmit.config.basicConfig.BasicConfig

Class to manage configuration for Autosubmit path, database and default values for new experiments

static read()

Reads configuration from .autosubmitrc files, first from /etc, then for user directory and last for current path.

autosubmit.config.config_common

class autosubmit.config.config_common.AutosubmitConfig(expid, basic_config, parser_factory)

Bases: object

Class to handle experiment configuration coming from file or database

Parameters:expid (str) – experiment identifier
check_autosubmit_conf()

Checks experiment’s autosubmit configuration file.

Returns:True if everything is correct, False if it founds any error
Return type:bool
check_conf_files(running_time=False)

Checks configuration files (autosubmit, experiment jobs and platforms), looking for invalid values, missing required options. Prints results in log

Returns:True if everything is correct, False if it finds any error
Return type:bool
check_expdef_conf()

Checks experiment’s experiment configuration file.

Returns:True if everything is correct, False if it founds any error
Return type:bool
check_jobs_conf()

Checks experiment’s jobs configuration file.

Returns:True if everything is correct, False if it founds any error
Return type:bool
check_platforms_conf()

Checks experiment’s queues configuration file.

check_proj()

Checks project config file

Returns:True if everything is correct, False if it founds any error
Return type:bool
check_proj_file()

Add a section header to the project’s configuration file (if not exists)

experiment_file

Returns experiment’s config file name

get_chunk_ini(default=1)

Returns the first chunk from where the experiment will start

Parameters:default –
Returns:initial chunk
Return type:int
get_chunk_size(default=1)

Chunk Size as defined in the expdef file.

Returns:Chunksize, 1 as default.
Return type:int
get_chunk_size_unit()

Unit for the chunk length

Returns:Unit for the chunk length Options: {hour, day, month, year}
Return type:str
get_communications_library()

Returns the communications library from autosubmit’s config file. Paramiko by default.

Returns:communications library
Return type:str
get_copy_remote_logs()

Returns if the user has enabled the logs local copy from autosubmit’s config file

Returns:if logs local copy
Return type:bool
get_current_host(section)

Returns the user to be changed from platform config file.

Returns:migrate user to
Return type:str
get_current_project(section)

Returns the project to be changed from platform config file.

Returns:migrate user to
Return type:str
get_current_user(section)

Returns the user to be changed from platform config file.

Returns:migrate user to
Return type:str
get_custom_directives(section)

Gets custom directives needed for the given job type :param section: job type :type section: str :return: custom directives needed :rtype: str

get_date_list()

Returns startdates list from experiment’s config file

Returns:experiment’s startdates
Return type:list
get_default_job_type()

Returns the default job type from experiment’s config file

Returns:default type such as bash, python, r..
Return type:str
get_delay_retry_time()

Returns delay time from autosubmit’s config file

Returns:safety sleep time
Return type:int
get_dependencies(section='None')

Returns dependencies list from jobs config file

Returns:experiment’s members
Return type:list
get_disable_recovery_logs(section)

Returns FALSE/TRUE :return: recovery_threads_option :rtype: str

get_disable_recovery_threads(section)

Returns FALSE/TRUE :return: recovery_threads_option :rtype: str

get_ec_queue(job)

Gets ec_queue needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str

get_ec_queue_platform(section)

Gets ec_queue needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str

get_export(section)

Gets command line for being submitted with :param section: job type :type section: str :return: wallclock time :rtype: str

get_extensible_wallclock(wrapper_section_name='wrapper')

Gets extend_wallclock for the given wrapper :param section: job type :type section: str :return: wallclock time :rtype: str

get_fetch_single_branch()

Returns fetch single branch from experiment’s config file Default is -single-branch :return: fetch_single_branch(Y/N) :rtype: boolean

get_file_jobs_conf()

Returns path to project config file from experiment config file

Returns:path to project config file
Return type:str
get_file_project_conf()

Returns path to project config file from experiment config file

Returns:path to project config file
Return type:str
get_full_config_as_dict()

Returns full configuration as json object

get_full_config_as_json()

Return config as json object

get_git_project_branch()

Returns git branch from experiment’s config file

Returns:git branch
Return type:str
get_git_project_commit()

Returns git commit from experiment’s config file

Returns:git commit
Return type:str
get_git_project_origin()

Returns git origin from experiment config file

Returns:git origin
Return type:str
get_git_remote_project_root()

Returns remote machine ROOT PATH

Returns:git commit
Return type:str
get_hyperthreading(section)

Gets hyperthreading status :param section: job type :type section: str :return: true/false/none :rtype: str

get_job_shape(section)

Returns shape

Return type:str
get_jobs_sections()

Returns the list of sections defined in the jobs config file

Returns:sections
Return type:list
get_local_project_path()

Gets path to origin for local project

Returns:path to local project
Return type:str
get_mails_to()

Returns the address where notifications will be sent from autosubmit’s config file

Returns:mail address
Return type:[str]
get_max_processors()

Returns max processors from autosubmit’s config file

Return type:str
get_max_waiting_jobs()

Returns max number of waiting jobs from autosubmit’s config file

Returns:main platforms
Return type:int
get_max_wallclock()

Returns max wallclock

Return type:str
get_max_wrapped_jobs(wrapper_section_name='wrapper')

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:maximum number of jobs (or total jobs)
Return type:int
get_max_wrapped_jobs_horizontal(wrapper_section_name='wrapper')

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:maximum number of jobs (or total jobs)
Return type:int
get_max_wrapped_jobs_vertical(wrapper_section_name='wrapper')

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:maximum number of jobs (or total jobs)
Return type:int
get_member_list(run_only=False)

Returns members list from experiment’s config file

Returns:experiment’s members
Return type:list
get_memory(section)

Gets memory needed for the given job type :param section: job type :type section: str :return: memory needed :rtype: str

get_memory_per_task(section)

Gets memory per task needed for the given job type :param section: job type :type section: str :return: memory per task needed :rtype: str

get_migrate_duplicate(section)

Returns the user to change to from platform config file.

Returns:migrate user to
Return type:str
get_migrate_host_to(section)

Returns the host to change to from platform config file.

Returns:host_to
Return type:str
get_migrate_project_to(section)

Returns the project to change to from platform config file.

Returns:migrate project to
Return type:str
get_migrate_user_to(section)

Returns the user to change to from platform config file.

Returns:migrate user to
Return type:str
get_min_wrapped_jobs(wrapper_section_name='wrapper')
Returns the minim number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns:minim number of jobs (or total jobs)
Return type:int
get_min_wrapped_jobs_horizontal(wrapper_section_name='wrapper')

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:maximum number of jobs (or total jobs)
Return type:int
get_min_wrapped_jobs_vertical(wrapper_section_name='wrapper')

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:maximum number of jobs (or total jobs)
Return type:int
get_nodes(section)

Gets tasks needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str

get_notifications()

Returns if the user has enabled the notifications from autosubmit’s config file

Returns:if notifications
Return type:string
get_notifications_crash()

Returns if the user has enabled the notifications from autosubmit’s config file

Returns:if notifications
Return type:string
get_num_chunks()

Returns number of chunks to run for each member

Returns:number of chunks
Return type:int
get_output_type()

Returns default output type, pdf if none

Returns:output type
Return type:string
get_parse_two_step_start()

Returns two step start jobs

Returns:jobs_list
Return type:str
static get_parser(parser_factory, file_path)

Gets parser for given file

Parameters:
  • parser_factory –
  • file_path (str) – path to file to be parsed
Returns:

parser
Return type:

SafeConfigParser
get_platform()

Returns main platforms from experiment’s config file

Returns:main platforms
Return type:str
get_processors(section)

Gets processors needed for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str

get_project_destination()

Returns git commit from experiment’s config file

Returns:git commit
Return type:str
get_project_dir()

Returns experiment’s project directory

Returns:experiment’s project directory
Return type:str
get_project_submodules_depth()

Returns the max depth of submodule at the moment of cloning Default is -1 (no limit) :return: depth :rtype: list

get_project_type()

Returns project type from experiment config file

Returns:project type
Return type:str
get_remote_dependencies()

Returns if the user has enabled the PRESUBMISSION configuration parameter from autosubmit’s config file

Returns:if remote dependencies
Return type:bool
get_rerun()

Returns startdates list from experiment’s config file

Returns:rerurn value
Return type:list
get_rerun_jobs()

Returns rerun jobs

Returns:jobs_list
Return type:str
get_retrials()

Returns max number of retrials for job from autosubmit’s config file

Returns:safety sleep time
Return type:int
get_safetysleeptime()

Returns safety sleep time from autosubmit’s config file

Returns:safety sleep time
Return type:int
get_scratch_free_space(section)

Gets scratch free space needed for the given job type :param section: job type :type section: str :return: percentage of scratch free space needed :rtype: int

get_shape(section)

Returns shape

Return type:str
get_storage_type()

Returns the communications library from autosubmit’s config file. Paramiko by default.

Returns:communications library
Return type:str
get_submodules_list()

Returns submodules list from experiment’s config file Default is –recursive :return: submodules to load :rtype: list

get_svn_project_revision()

Get revision for subversion project

Returns:revision for subversion project
Return type:str
get_svn_project_url()

Gets subversion project url

Returns:subversion project url
Return type:str
get_synchronize(section)

Gets wallclock for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str

get_tasks(section)

Gets tasks needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str

get_threads(section)

Gets threads needed for the given job type :param section: job type :type section: str :return: threads needed :rtype: str

get_total_jobs()

Returns max number of running jobs from autosubmit’s config file

Returns:max number of running jobs
Return type:int
get_version()

Returns version number of the current experiment from autosubmit’s config file

Returns:version
Return type:str
get_wallclock(section)

Gets wallclock for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str

get_wchunkinc(section)

Gets the chunk increase to wallclock :param section: job type :type section: str :return: wallclock increase per chunk :rtype: str

get_wrapper_check_time(wrapper_section_name='wrapper')

Returns time to check the status of jobs in the wrapper

Returns:wrapper check time
Return type:int
get_wrapper_export(wrapper_section_name='wrapper')

Returns modules variable from wrapper

Returns:string
Return type:string
get_wrapper_jobs(wrapper_section_name='wrapper')

Returns the jobs that should be wrapped, configured in the autosubmit’s config

Returns:expression (or none)
Return type:string
get_wrapper_language(wrapper_section_name='wrapper')

Returns the method of make the wrapper

Returns:method
Return type:string
get_wrapper_machinefiles(wrapper_section_name='wrapper')

Returns the strategy for creating the machinefiles in wrapper jobs

Returns:machinefiles function to use
Return type:string
get_wrapper_method(wrapper_section_name='wrapper')

Returns the method of make the wrapper

Returns:method
Return type:string
get_wrapper_multi()

return the section name of the wrappers

Returns:wrapper section list
Return type:string
get_wrapper_policy(wrapper_section_name='wrapper')

Returns what kind of policy (flexible, strict, mixed ) the user has configured in the autosubmit’s config

Returns:wrapper type (or none)
Return type:string
get_wrapper_queue(wrapper_section_name='wrapper')

Returns the wrapper queue if not defined, will be the one of the first job wrapped

Returns:expression (or none)
Return type:string
get_wrapper_retrials(wrapper_section_name='wrapper')

Returns max number of retrials for job from autosubmit’s config file

Returns:safety sleep time
Return type:int
get_wrapper_type(wrapper_section_name='wrapper')

Returns what kind of wrapper (VERTICAL, MIXED-VERTICAL, HORIZONTAL, HYBRID, MULTI NONE) the user has configured in the autosubmit’s config

Returns:wrapper type (or none)
Return type:string
get_x11(section)

Active X11 for this section :param section: job type :type section: str :return: false/true :rtype: str

get_x11_jobs()

Returns the jobs that should support x11, configured in the autosubmit’s config

Returns:expression (or none)
Return type:string
jobs_file

Returns project’s jobs file name

load_parameters()

Load parameters from experiment and autosubmit config files. If experiment’s type is not none, also load parameters from model’s config file

Returns:a dictionary containing tuples [parameter_name, parameter_value]
Return type:dict
load_platform_parameters()

Load parameters from platform config files.

Returns:a dictionary containing tuples [parameter_name, parameter_value]
Return type:dict
load_project_parameters()

Loads parameters from model config file

Returns:dictionary containing tuples [parameter_name, parameter_value]
Return type:dict
load_section_parameters(job_list, as_conf, submitter)

Load parameters from job config files.

Returns:a dictionary containing tuples [parameter_name, parameter_value]
Return type:dict
parse_githooks()

Parse githooks section in configuration file

Returns:dictionary with githooks configuration
Return type:dict
platforms_file

Returns experiment’s platforms config file name

Returns:platforms config file’s name
Return type:str
platforms_parser

Returns experiment’s platforms parser object

Returns:platforms config parser object
Return type:SafeConfigParser
project_file

Returns project’s config file name

reload()

Creates parser objects for configuration files

set_expid(exp_id)

Set experiment identifier in autosubmit and experiment config files

Parameters:exp_id (str) – experiment identifier to store
set_git_project_commit(as_conf)

Function to register in the configuration the commit SHA of the git project version. :param as_conf: Configuration class for exteriment :type as_conf: AutosubmitConfig

set_new_host(section, new_host)

Sets new host for given platform :param new_host: :param section: platform name :type: str

set_new_project(section, new_project)

Sets new project for given platform :param new_project: :param section: platform name :type: str

set_new_user(section, new_user)

Sets new user for given platform :param new_user: :param section: platform name :type: str

set_platform(hpc)

Sets main platforms in experiment’s config file

Parameters:hpc – main platforms
Type:str
set_safetysleeptime(sleep_time)

Sets autosubmit’s version in autosubmit’s config file

Parameters:sleep_time (int) – value to set
set_version(autosubmit_version)

Sets autosubmit’s version in autosubmit’s config file

Parameters:autosubmit_version (str) – autosubmit’s version