autosubmit.config#

autosubmitconfigparser.config.basicConfig#

class autosubmitconfigparser.config.basicconfig.BasicConfig#

Bases: object

Class to manage configuration for Autosubmit path, database and default values for new experiments

static read()#: Reads configuration from .autosubmitrc files, first from /etc., then for user directory and last for current path.

autosubmitconfigparser.config.config_common#

class autosubmitconfigparser.config.configcommon.AutosubmitConfig(expid, basic_config=<class 'autosubmitconfigparser.config.basicconfig.BasicConfig'>, parser_factory=<autosubmitconfigparser.config.yamlparser.YAMLParserFactory object>)#

Bases: object

Class to handle experiment configuration coming from file or database

Parameters:: expid (str) – experiment identifier

check_autosubmit_conf(no_log=False)#: Checks experiment’s autosubmit configuration file. :param refresh: True if the function is called during the refresh of the program :type refresh: bool :param no_log: True if the function is called during describe :type no_log: bool :return: True if everything is correct, False if it founds any error :rtype: bool

check_conf_files(running_time=False, force_load=True, no_log=False)#: Checks configuration files (autosubmit, experiment jobs and platforms), looking for invalid values, missing required options. Print results in log :param running_time: True if the function is called during the execution of the program :type running_time: bool :param force_load: True if the function is called during the first load of the program :type force_load: bool :param refresh: True if the function is called during the refresh of the program :type refresh: bool :param no_log: True if the function is called during describe :type no_log: bool :return: True if everything is correct, False if it finds any error :rtype: bool

check_dict_keys_type(parameters)#: Check if keys are plain into 1 dimension, checks for 33% of dict to ensure it. :param parameters: experiment parameters :return:

check_expdef_conf(no_log=False)#: Checks experiment’s experiment configuration file. :param refresh: if True, it doesn’t check the mandatory parameters :type refresh: bool :param no_log: if True, it doesn’t print any log message :type no_log: bool :return: True if everything is correct, False if it founds any error :rtype: bool

check_jobs_conf(no_log=False)#: Checks experiment’s jobs configuration file. :param no_log: if True, it doesn’t print any log message :type no_log: bool :return: True if everything is correct, False if it founds any error :rtype: bool

check_platforms_conf(no_log=False)#: Checks experiment’s platforms configuration file.

check_wrapper_conf(wrappers={}, no_log=False)#

Checks wrapper config file

Parameters:

wrappers –
no_log –

Returns:

clean_dynamic_variables(pattern, in_the_end=False)#: Clean dynamic variables :param pattern: :param in_the_end: :return:

convert_list_to_string(data)#: Convert a list to a string

deep_add_missing_starter_conf(experiment_data, starter_conf)#: Add the missing keys from starter_conf to experiment_data :param experiment_data: :param starter_conf: :return:

deep_normalize(data)#: normalize a nested dictionary or similar mapping to uppercase. Modify source in place.

deep_parameters_export(data)#: Export all variables of this experiment. Resultant format will be Section.{subsections1…subsectionN} = Value. In other words, it plain the dictionary into one level

deep_read_loops(data, for_keys=[], long_key='')#: Update a nested dictionary or similar mapping. Modify source in place.

deep_update(unified_config, new_dict)#: Update a nested dictionary or similar mapping. Modify source in place.

detailed_deep_diff(current_data, last_run_data, level=0)#: Returns a dictionary with for each key, the difference between the current configuration and the last_run_data :param current_data: dictionary with the current data :param last_run_data: dictionary with the last_run_data data :return: differences: dictionary

file_modified(file, prev_mod_time)#: Function to check if a file has been modified. :param file: path :return: bool,new_time

get_chunk_ini(default=1)#

Returns the first chunk from where the experiment will start

Parameters:: default –
Returns:: initial chunk
Return type:: int

get_chunk_size(default=1)#

Chunk Size as defined in the expdef file.

Returns:: Chunksize, 1 as default.
Return type:: int

get_chunk_size_unit()#

Unit for the chunk length

Returns:: Unit for the chunk length Options: {hour, day, month, year}
Return type:: str

get_communications_library()#

Returns the communications library from autosubmit’s config file. Paramiko by default.

Returns:: communications library
Return type:: str

get_copy_remote_logs()#

Returns if the user has enabled the logs local copy from autosubmit’s config file

Returns:: if logs local copy
Return type:: str

get_current_host(section)#

Returns the user to be changed from platform config file.

Returns:: migrate user to
Return type:: str

get_current_project(section)#

Returns the project to be changed from platform config file.

Returns:: migrate user to
Return type:: str

get_current_user(section)#

Returns the user to be changed from platform config file.

Returns:: migrate user to
Return type:: str

get_custom_directives(section)#: Gets custom directives needed for the given job type :param section: job type :type section: str :return: custom directives needed :rtype: str

get_date_list()#

Returns startdates list from experiment’s config file

Returns:: experiment’s startdates
Return type:: list

get_default_job_type()#

Returns the default job type from experiment’s config file

Returns:: default type such as bash, python, r…
Return type:: str

get_delay_retry_time()#

Returns delay time from autosubmit’s config file

Returns:: safety sleep time
Return type:: int

get_dependencies(section='None')#

Returns dependencies list from jobs config file

Returns:: experiment’s members
Return type:: list

get_disable_recovery_threads(section)#: Returns FALSE/TRUE :return: recovery_threads_option :rtype: str

get_export(section)#: Gets command line for being submitted with :param section: job type :type section: str :return: wallclock time :rtype: str

get_extensible_wallclock(wrapper={})#

Gets extend_wallclock for the given wrapper

Parameters:: wrapper (dict) – wrapper
Returns:: extend_wallclock
Return type:: int

get_fetch_single_branch()#: Returns fetch single branch from experiment’s config file Default is -single-branch :return: fetch_single_branch(Y/N) :rtype: str

get_file_jobs_conf()#

Returns path to project config file from experiment config file

Returns:: path to project config file
Return type:: str

get_file_project_conf()#

Returns path to project config file from experiment config file

Returns:: path to project config file
Return type:: str

get_full_config_as_json()#: Return config as json object

get_git_project_branch()#

Returns git branch from experiment’s config file

Returns:: git branch
Return type:: str

get_git_project_commit()#

Returns git commit from experiment’s config file

Returns:: git commit
Return type:: str

get_git_project_origin()#

Returns git origin from experiment config file

Returns:: git origin
Return type:: str

get_git_remote_project_root()#

Returns remote machine ROOT PATH

Returns:: git commit
Return type:: str

get_local_project_path()#

Gets path to origin for local project

Returns:: path to local project
Return type:: str

get_mails_to()#

Returns the address where notifications will be sent from autosubmit’s config file

Returns:: mail address
Return type:: [str]

get_max_processors()#

Returns max processors from autosubmit’s config file

Return type:: str

get_max_waiting_jobs()#

Returns max number of waiting jobs from autosubmit’s config file

Returns:: main platforms
Return type:: int

get_max_wallclock()#

Returns max wallclock

Return type:: str

get_max_wrapped_jobs(wrapper={})#

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: maximum number of jobs (or total jobs)
Return type:: int

get_max_wrapped_jobs_horizontal(wrapper={})#

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: maximum number of jobs (or total jobs)
Return type:: int

get_max_wrapped_jobs_vertical(wrapper={})#

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: maximum number of jobs (or total jobs)
Return type:: int

get_member_list(run_only=False)#

Returns members list from experiment’s config file

Returns:: experiment’s members
Return type:: list

get_memory(section)#: Gets memory needed for the given job type :param section: job type :type section: str :return: memory needed :rtype: str

get_memory_per_task(section)#: Gets memory per task needed for the given job type :param section: job type :type section: str :return: memory per task needed :rtype: str

get_migrate_duplicate(section)#

Returns the user to change to from platform config file.

Returns:: migrate user to
Return type:: str

get_migrate_host_to(section)#

Returns the host to change to from platform config file.

Returns:: host_to
Return type:: str

get_migrate_project_to(section)#

Returns the project to change to from platform config file.

Returns:: migrate project to
Return type:: str

get_migrate_user_to(section)#

Returns the user to change to from platform config file.

Returns:: migrate user to
Return type:: str

get_min_wrapped_jobs(wrapper={})#

Returns the minium number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: minim number of jobs (or total jobs)
Return type:: int

get_min_wrapped_jobs_horizontal(wrapper={})#

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: maximum number of jobs (or total jobs)
Return type:: int

get_min_wrapped_jobs_vertical(wrapper={})#

Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file

Returns:: maximum number of jobs (or total jobs)
Return type:: int

get_notifications()#

Returns if the user has enabled the notifications from autosubmit’s config file

Returns:: if notifications
Return type:: string

get_notifications_crash()#

Returns if the user has enabled the notifications from autosubmit’s config file

Returns:: if notifications
Return type:: string

get_num_chunks()#

Returns number of chunks to run for each member

Returns:: number of chunks
Return type:: int

get_output_type()#

Returns default output type, pdf if none

Returns:: output type
Return type:: string

get_parse_two_step_start()#

Returns two-step start jobs

Returns:: jobs_list
Return type:: str

static get_parser(parser_factory, file_path)#

Gets parser for given file

Parameters:

parser_factory –
file_path (Path) – path to file to be parsed

Returns:

parser

Return type:

YAMLParser

get_platform()#

Returns main platforms from experiment’s config file

Returns:: main platforms
Return type:: str

get_processors(section)#: Gets processors needed for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str

get_project_destination()#

Returns git commit from experiment’s config file

Returns:: git commit
Return type:: str

get_project_dir()#

Returns experiment’s project directory

Returns:: experiment’s project directory
Return type:: str

get_project_submodules_depth()#: Returns the max depth of submodule at the moment of cloning Default is -1 (no limit) :return: depth :rtype: list

get_project_type()#

Returns project type from experiment config file

Returns:: project type
Return type:: str

get_remote_dependencies()#

Returns if the user has enabled the PRESUBMISSION configuration parameter from autosubmit’s config file

Returns:: if remote dependencies
Return type:: string

get_rerun()#

Returns startdates list from experiment’s config file

Returns:: rerurn value
Return type:: bool

get_rerun_jobs()#

Returns rerun jobs

Returns:: jobs_list
Return type:: str

get_retrials()#

Returns max number of retrials for job from autosubmit’s config file

Returns:: safety sleep time
Return type:: int

get_safetysleeptime()#

Returns safety sleep time from autosubmit’s config file

Returns:: safety sleep time
Return type:: int

get_scratch_free_space(section)#: Gets scratch free space needed for the given job type :param section: job type :type section: str :return: percentage of scratch free space needed :rtype: int

get_section(section, d_value='', must_exists=False)#: Gets any section if it exists within the dictionary, else returns None or error if must exist. :param section: section to get :type section: list :param d_value: default value to return if section does not exist :type d_value: str :param must_exists: if true, error is raised if section does not exist :type must_exists: bool :return: section value :rtype: str

get_storage_type()#

Returns the storage system from autosubmit’s config file. Pkl by default.

Returns:: communications library
Return type:: str

get_submodules_list() → Union[List[str], bool]#: Returns submodules list from experiment’s config file. Default is –recursive. Can be disabled by setting the configuration key to False. :return: submodules to load :rtype: Union[List[str], bool]

get_svn_project_revision()#

Get revision for subversion project

Returns:: revision for subversion project
Return type:: str

get_svn_project_url()#

Gets subversion project url

Returns:: subversion project url
Return type:: str

get_synchronize(section)#: Gets wallclock for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str

get_tasks(section)#: Gets tasks needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str

get_threads(section)#: Gets threads needed for the given job type :param section: job type :type section: str :return: threads needed :rtype: str

get_total_jobs()#

Returns max number of running jobs from autosubmit’s config file

Returns:: max number of running jobs
Return type:: int

get_version()#

Returns version number of the current experiment from autosubmit’s config file

Returns:: version
Return type:: str

get_wchunkinc(section)#: Gets the chunk increase to wallclock :param section: job type :type section: str :return: wallclock increase per chunk :rtype: str

get_wrapper_check_time()#

Returns time to check the status of jobs in the wrapper

Returns:: wrapper check time
Return type:: int

get_wrapper_export(wrapper={})#

Returns modules variable from wrapper

Returns:: string
Return type:: string

get_wrapper_jobs(wrapper=None)#

Returns the jobs that should be wrapped, configured in the autosubmit’s config

Returns:: expression (or none)
Return type:: string

get_wrapper_machinefiles(wrapper={})#

Returns the strategy for creating the machinefiles in wrapper jobs

Returns:: machinefiles function to use
Return type:: string

get_wrapper_method(wrapper={})#

Returns the method of make the wrapper

Returns:: method
Return type:: string

get_wrapper_partition(wrapper={})#

Returns the wrapper queue if not defined, will be the one of the first job wrapped

Returns:: expression (or none)
Return type:: string

get_wrapper_policy(wrapper={})#

Returns what kind of policy (flexible, strict, mixed ) the user has configured in the autosubmit’s config

Returns:: wrapper type (or none)
Return type:: string

get_wrapper_queue(wrapper={})#

Returns the wrapper queue if not defined, will be the one of the first job wrapped

Returns:: expression (or none)
Return type:: string

get_wrapper_retrials(wrapper={})#

Returns max number of retrials for job from autosubmit’s config file

Returns:: safety sleep time
Return type:: int

get_wrapper_type(wrapper={})#

Returns what kind of wrapper (VERTICAL, MIXED-VERTICAL, HORIZONTAL, HYBRID, MULTI NONE) the user has configured in the autosubmit’s config

Returns:: wrapper type (or none)
Return type:: string

get_wrappers()#

Returns the jobs that should be wrapped, configured in the autosubmit’s config

Returns:: expression
Return type:: dict

get_x11(section)#: Active X11 for this section :param section: job type :type section: str :return: false/true :rtype: str

get_x11_jobs()#

Returns the jobs that should support x11, configured in the autosubmit’s config

Returns:: expression (or none)
Return type:: string

get_yaml_filenames_to_load(yaml_folder, ignore_minimal=False)#: Get all yaml files in a folder and return a list with the filenames :param yaml_folder: folder to search for yaml files :param ignore_minimal: ignore minimal files :return: list of filenames

load_common_parameters(parameters)#: Loads common parameters not specific to a job neither a platform :param parameters: :return:

load_config_file(current_folder_data, yaml_file)#: Load a config file and parse it :param current_folder_data: current folder data :param yaml_file: yaml file to load :return: unified config file

load_config_folder(current_data, yaml_folder, ignore_minimal=False)#: Load a config folder and return pre and post config :param current_data: current data to be updated :param yaml_folder: folder to load config :param ignore_minimal: ignore minimal config files :return: pre and post config

load_custom_config(current_data, filenames_to_load)#: Loads custom config files :param current_data: dict with current data :param filenames_to_load: list of filenames to load :return: current_data_pre,current_data_post with unified data

load_custom_config_section(current_data, filenames_to_load)#: Loads a section (PRE or POST ), simple str are also PRE data of the custom config files :param current_data: data until now :param filenames_to_load: files to load in this section :return:

load_parameters()#: Load all experiment data :return: a dictionary containing tuples [parameter_name, parameter_value] :rtype: dict

load_platform_parameters()#

Load parameters from platform config files.

Returns:: a dictionary containing tuples [parameter_name, parameter_value]
Return type:: dict

load_section_parameters(job_list, as_conf, submitter)#

Load parameters from job config files.

Returns:: a dictionary containing tuples [parameter_name, parameter_value]
Return type:: dict

normalize_parameters_keys(parameters, default_parameters={})#: Normalize the parameters keys to be exportable in the templates case-insensitive. :param parameters: dictionary containing the parameters :param default_parameters: dictionary containing the default parameters, they must remain in lower-case :return: upper-case parameters

normalize_variables(data)#: Apply some memory internal variables to normalize it format. (right now only dependencies)

parse_data_loops(experiment_data, data_loops)#: This function, looks for the FOR keyword, to generates N amount of subsections of the same section. Looks for the “NAME” keyword, inside this FOR keyword to determine the name of the new sections Experiment_data is the dictionary that contains all the sections, a subsection could be located at the root but also in a nested section :param experiment_data: dictionary with all the sections :param data_loops: list of lists with the path to the section that contains the FOR keyword :return: Original experiment_data with the sections in the data_loops updated changing the FOR by multiple new sections

parse_githooks()#

Parse githooks section in configuration file

Returns:: dictionary with githooks configuration
Return type:: dict

static parse_placeholders(content, parameters)#

Parse placeholders in content

Parameters:

content (str) – content to be parsed
parameters (dict) – parameters to be used in parsing

Returns:

parsed content

Return type:

str

quick_deep_diff(current_data, last_run_data, changed=False)#: Returns if there is any difference between the current configuration and the stored one :param last_run_data: dictionary with the stored data :return: changed: boolean, True if the configuration has changed

reload(force_load=False, only_experiment_data=False, save=False)#: Reloads the configuration files :param force_load: If True, reloads all the files, if False, reloads only the modified files

save()#: Saves the experiment data into the experiment_folder/conf/metadata folder as a yaml file :return: True if the data has changed, False otherwise

set_expid(exp_id)#

Set experiment identifier in autosubmit and experiment config files

Parameters:: exp_id (str) – experiment identifier to store

set_git_project_commit(as_conf)#: Function to register in the configuration the commit SHA of the git project version. :param as_conf: Configuration class for exteriment :type as_conf: AutosubmitConfig

set_new_host(section, new_host)#: Sets new host for given platform :param new_host: :param section: platform name :type: str

set_new_project(section, new_project)#: Sets new project for given platform :param new_project: :param section: platform name :type: str

set_new_user(section, new_user)#: Sets new user for given platform :param new_user: :param section: platform name :type: str

set_platform(hpc)#

Sets main platforms in experiment’s config file

Parameters:: hpc – main platforms
Type:: str

set_safetysleeptime(sleep_time)#

Sets autosubmit’s version in autosubmit’s config file

Parameters:: sleep_time (int) – value to set

set_version(autosubmit_version)#

Sets autosubmit’s version in autosubmit’s config file

Parameters:: autosubmit_version (str) – autosubmit’s version

substitute_dynamic_variables(parameters=None, max_deep=25, dict_keys_type=None, not_in_data='', in_the_end=False)#: Substitute dynamic variables in the experiment data :parameter :return:

unify_conf(current_data, new_data)#: Unifies all configuration files into a single dictionary. :param current_data: dict with current configuration :param new_data: dict with new configuration :return: dict with new configuration taking priority over current configuration