autosubmit.config¶
autosubmit.config.basicConfig¶
autosubmit.config.config_common¶
-
class
autosubmit.config.config_common.
AutosubmitConfig
(expid, basic_config, parser_factory)¶ Bases:
object
Class to handle experiment configuration coming from file or database
Parameters: expid (str) – experiment identifier -
check_autosubmit_conf
()¶ Checks experiment’s autosubmit configuration file.
Returns: True if everything is correct, False if it founds any error Return type: bool
-
check_conf_files
(running_time=False, first_load=True)¶ Checks configuration files (autosubmit, experiment jobs and platforms), looking for invalid values, missing required options. Print results in log
Returns: True if everything is correct, False if it finds any error Return type: bool
-
check_expdef_conf
()¶ Checks experiment’s experiment configuration file.
Returns: True if everything is correct, False if it founds any error Return type: bool
-
check_jobs_conf
()¶ Checks experiment’s jobs configuration file.
Returns: True if everything is correct, False if it founds any error Return type: bool
-
check_platforms_conf
()¶ Checks experiment’s queues configuration file.
-
check_proj
()¶ Checks project config file
Returns: True if everything is correct, False if it founds any error Return type: bool
-
check_proj_file
()¶ Add a section header to the project’s configuration file (if not exists)
-
deep_normalize
(data)¶ normalize a nested dictionary or similar mapping to uppercase. Modify
source
in place.
-
deep_parameters_export
(data)¶ Export all variables of this experiment. Resultant format will be Section.{subsections1…subsectionN} = Value. In other words, it plain the dictionary into one level
-
deep_read_loops
(data, for_keys=[], long_key='')¶ Update a nested dictionary or similar mapping. Modify
source
in place.
-
deep_update
(unified_config, new_dict)¶ Update a nested dictionary or similar mapping. Modify
source
in place.
-
experiment_file
¶ Returns experiment’s config file name
-
file_modified
(file, prev_mod_time)¶ Function to check if a file has been modified. :param file: path :return: bool,new_time
-
get_chunk_ini
(default=1)¶ Returns the first chunk from where the experiment will start
Parameters: default – Returns: initial chunk Return type: int
-
get_chunk_size
(default=1)¶ Chunk Size as defined in the expdef file.
Returns: Chunksize, 1 as default. Return type: int
-
get_chunk_size_unit
()¶ Unit for the chunk length
Returns: Unit for the chunk length Options: {hour, day, month, year} Return type: str
-
get_communications_library
()¶ Returns the communications library from autosubmit’s config file. Paramiko by default.
Returns: communications library Return type: str
-
get_copy_remote_logs
()¶ Returns if the user has enabled the logs local copy from autosubmit’s config file
Returns: if logs local copy Return type: str
-
get_current_host
(section)¶ Returns the user to be changed from platform config file.
Returns: migrate user to Return type: str
-
get_current_project
(section)¶ Returns the project to be changed from platform config file.
Returns: migrate user to Return type: str
-
get_current_user
(section)¶ Returns the user to be changed from platform config file.
Returns: migrate user to Return type: str
-
get_custom_directives
(section)¶ Gets custom directives needed for the given job type :param section: job type :type section: str :return: custom directives needed :rtype: str
-
get_date_list
()¶ Returns startdates list from experiment’s config file
Returns: experiment’s startdates Return type: list
-
get_default_job_type
()¶ Returns the default job type from experiment’s config file
Returns: default type such as bash, python, r… Return type: str
-
get_delay_retry_time
()¶ Returns delay time from autosubmit’s config file
Returns: safety sleep time Return type: int
-
get_dependencies
(section='None')¶ Returns dependencies list from jobs config file
Returns: experiment’s members Return type: list
-
get_disable_recovery_threads
(section)¶ Returns FALSE/TRUE :return: recovery_threads_option :rtype: str
-
get_export
(section)¶ Gets command line for being submitted with :param section: job type :type section: str :return: wallclock time :rtype: str
-
get_extensible_wallclock
(wrapper={})¶ Gets extend_wallclock for the given wrapper
Parameters: wrapper (dict) – wrapper Returns: extend_wallclock Return type: int
-
get_fetch_single_branch
()¶ Returns fetch single branch from experiment’s config file Default is -single-branch :return: fetch_single_branch(Y/N) :rtype: str
-
get_file_jobs_conf
()¶ Returns path to project config file from experiment config file
Returns: path to project config file Return type: str
-
get_file_project_conf
()¶ Returns path to project config file from experiment config file
Returns: path to project config file Return type: str
-
get_full_config_as_json
()¶ Return config as json object
-
get_git_project_branch
()¶ Returns git branch from experiment’s config file
Returns: git branch Return type: str
-
get_git_project_commit
()¶ Returns git commit from experiment’s config file
Returns: git commit Return type: str
-
get_git_project_origin
()¶ Returns git origin from experiment config file
Returns: git origin Return type: str
-
get_git_remote_project_root
()¶ Returns remote machine ROOT PATH
Returns: git commit Return type: str
-
get_jobs_sections
()¶ Returns the list of sections defined in the job’s config file
Returns: sections Return type: list
-
get_local_project_path
()¶ Gets path to origin for local project
Returns: path to local project Return type: str
-
get_mails_to
()¶ Returns the address where notifications will be sent from autosubmit’s config file
Returns: mail address Return type: [str]
-
get_max_processors
()¶ Returns max processors from autosubmit’s config file
Return type: str
-
get_max_waiting_jobs
()¶ Returns max number of waiting jobs from autosubmit’s config file
Returns: main platforms Return type: int
-
get_max_wallclock
()¶ Returns max wallclock
Return type: str
-
get_max_wrapped_jobs
(wrapper={})¶ Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: maximum number of jobs (or total jobs) Return type: int
-
get_max_wrapped_jobs_horizontal
(wrapper={})¶ Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: maximum number of jobs (or total jobs) Return type: int
-
get_max_wrapped_jobs_vertical
(wrapper={})¶ Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: maximum number of jobs (or total jobs) Return type: int
-
get_member_list
(run_only=False)¶ Returns members list from experiment’s config file
Returns: experiment’s members Return type: list
-
get_memory
(section)¶ Gets memory needed for the given job type :param section: job type :type section: str :return: memory needed :rtype: str
-
get_memory_per_task
(section)¶ Gets memory per task needed for the given job type :param section: job type :type section: str :return: memory per task needed :rtype: str
-
get_migrate_duplicate
(section)¶ Returns the user to change to from platform config file.
Returns: migrate user to Return type: str
-
get_migrate_host_to
(section)¶ Returns the host to change to from platform config file.
Returns: host_to Return type: str
-
get_migrate_project_to
(section)¶ Returns the project to change to from platform config file.
Returns: migrate project to Return type: str
-
get_migrate_user_to
(section)¶ Returns the user to change to from platform config file.
Returns: migrate user to Return type: str
-
get_min_wrapped_jobs
(wrapper={})¶ - Returns the minium number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: minim number of jobs (or total jobs) Return type: int
-
get_min_wrapped_jobs_horizontal
(wrapper={})¶ Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: maximum number of jobs (or total jobs) Return type: int
-
get_min_wrapped_jobs_vertical
(wrapper={})¶ Returns the maximum number of jobs that can be wrapped together as configured in autosubmit’s config file
Returns: maximum number of jobs (or total jobs) Return type: int
-
get_notifications
()¶ Returns if the user has enabled the notifications from autosubmit’s config file
Returns: if notifications Return type: string
-
get_notifications_crash
()¶ Returns if the user has enabled the notifications from autosubmit’s config file
Returns: if notifications Return type: string
-
get_num_chunks
()¶ Returns number of chunks to run for each member
Returns: number of chunks Return type: int
-
get_output_type
()¶ Returns default output type, pdf if none
Returns: output type Return type: string
-
get_parse_two_step_start
()¶ Returns two-step start jobs
Returns: jobs_list Return type: str
-
static
get_parser
(parser_factory, file_path)¶ Gets parser for given file
Parameters: - parser_factory –
- file_path (Path) – path to file to be parsed
Returns: parser
Return type: YAMLParser
-
get_platform
()¶ Returns main platforms from experiment’s config file
Returns: main platforms Return type: str
-
get_processors
(section)¶ Gets processors needed for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str
-
get_project_destination
()¶ Returns git commit from experiment’s config file
Returns: git commit Return type: str
-
get_project_dir
()¶ Returns experiment’s project directory
Returns: experiment’s project directory Return type: str
-
get_project_type
()¶ Returns project type from experiment config file
Returns: project type Return type: str
-
get_remote_dependencies
()¶ Returns if the user has enabled the PRESUBMISSION configuration parameter from autosubmit’s config file
Returns: if remote dependencies Return type: string
-
get_rerun
()¶ Returns startdates list from experiment’s config file
Returns: rerurn value Return type: bool
-
get_rerun_jobs
()¶ Returns rerun jobs
Returns: jobs_list Return type: str
-
get_retrials
()¶ Returns max number of retrials for job from autosubmit’s config file
Returns: safety sleep time Return type: int
-
get_safetysleeptime
()¶ Returns safety sleep time from autosubmit’s config file
Returns: safety sleep time Return type: int
-
get_scratch_free_space
(section)¶ Gets scratch free space needed for the given job type :param section: job type :type section: str :return: percentage of scratch free space needed :rtype: int
-
get_section
(section, d_value='', must_exists=False)¶ Gets any section if it exists within the dictionary, else returns None or error if must exist. :param section: section to get :type section: list :param d_value: default value to return if section does not exist :type d_value: str :param must_exists: if true, error is raised if section does not exist :type must_exists: bool :return: section value :rtype: str
-
get_storage_type
()¶ Returns the storage system from autosubmit’s config file. Pkl by default.
Returns: communications library Return type: str
-
get_submodules_list
()¶ Returns submodules list from experiment’s config file Default is –recursive :return: submodules to load :rtype: list
-
get_svn_project_revision
()¶ Get revision for subversion project
Returns: revision for subversion project Return type: str
-
get_svn_project_url
()¶ Gets subversion project url
Returns: subversion project url Return type: str
-
get_synchronize
(section)¶ Gets wallclock for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str
-
get_tasks
(section)¶ Gets tasks needed for the given job type :param section: job type :type section: str :return: tasks (processes) per host :rtype: str
-
get_threads
(section)¶ Gets threads needed for the given job type :param section: job type :type section: str :return: threads needed :rtype: str
-
get_total_jobs
()¶ Returns max number of running jobs from autosubmit’s config file
Returns: max number of running jobs Return type: int
-
get_version
()¶ Returns version number of the current experiment from autosubmit’s config file
Returns: version Return type: str
-
get_wallclock
(section)¶ Gets wallclock for the given job type :param section: job type :type section: str :return: wallclock time :rtype: str
-
get_wchunkinc
(section)¶ Gets the chunk increase to wallclock :param section: job type :type section: str :return: wallclock increase per chunk :rtype: str
-
get_wrapper_check_time
(wrapper=None)¶ Returns time to check the status of jobs in the wrapper
Returns: wrapper check time Return type: int
-
get_wrapper_export
(wrapper={})¶ Returns modules variable from wrapper
Returns: string Return type: string
-
get_wrapper_jobs
(wrapper=None)¶ Returns the jobs that should be wrapped, configured in the autosubmit’s config
Returns: expression (or none) Return type: string
-
get_wrapper_machinefiles
(wrapper={})¶ Returns the strategy for creating the machinefiles in wrapper jobs
Returns: machinefiles function to use Return type: string
-
get_wrapper_method
(wrapper={})¶ Returns the method of make the wrapper
Returns: method Return type: string
-
get_wrapper_policy
(wrapper={})¶ Returns what kind of policy (flexible, strict, mixed ) the user has configured in the autosubmit’s config
Returns: wrapper type (or none) Return type: string
-
get_wrapper_queue
(wrapper={})¶ Returns the wrapper queue if not defined, will be the one of the first job wrapped
Returns: expression (or none) Return type: string
-
get_wrapper_retrials
(wrapper={})¶ Returns max number of retrials for job from autosubmit’s config file
Returns: safety sleep time Return type: int
-
get_wrapper_type
(wrapper={})¶ Returns what kind of wrapper (VERTICAL, MIXED-VERTICAL, HORIZONTAL, HYBRID, MULTI NONE) the user has configured in the autosubmit’s config
Returns: wrapper type (or none) Return type: string
-
get_wrappers
()¶ Returns the jobs that should be wrapped, configured in the autosubmit’s config
Returns: expression Return type: dict
-
get_x11
(section)¶ Active X11 for this section :param section: job type :type section: str :return: false/true :rtype: str
-
get_x11_jobs
()¶ Returns the jobs that should support x11, configured in the autosubmit’s config
Returns: expression (or none) Return type: string
-
jobs_file
¶ Returns project’s jobs file name
-
load_parameters
()¶ Load all experiment data :return: a dictionary containing tuples [parameter_name, parameter_value] :rtype: dict
-
load_platform_parameters
()¶ Load parameters from platform config files.
Returns: a dictionary containing tuples [parameter_name, parameter_value] Return type: dict
-
load_section_parameters
(job_list, as_conf, submitter)¶ Load parameters from job config files.
Returns: a dictionary containing tuples [parameter_name, parameter_value] Return type: dict
-
normalize_variables
(data)¶ Apply some memory internal variables to normalize it format. (right now only dependencies)
-
platforms_file
¶ Returns experiment’s platforms config file name
Returns: platforms config file’s name Return type: str
-
platforms_parser
¶ Returns experiment’s platforms parser object
Returns: platforms config parser object Return type: SafeConfigParser
-
project_file
¶ Returns project’s config file name
-
reload
(first_load=False)¶ Creates parser objects for configuration files
-
set_expid
(exp_id)¶ Set experiment identifier in autosubmit and experiment config files
Parameters: exp_id (str) – experiment identifier to store
-
set_git_project_commit
(as_conf)¶ Function to register in the configuration the commit SHA of the git project version. :param as_conf: Configuration class for exteriment :type as_conf: AutosubmitConfig
-
set_new_host
(section, new_host)¶ Sets new host for given platform :param new_host: :param section: platform name :type: str
-
set_new_project
(section, new_project)¶ Sets new project for given platform :param new_project: :param section: platform name :type: str
-
set_new_user
(section, new_user)¶ Sets new user for given platform :param new_user: :param section: platform name :type: str
-
set_platform
(hpc)¶ Sets main platforms in experiment’s config file
Parameters: hpc – main platforms Type: str
-
set_safetysleeptime
(sleep_time)¶ Sets autosubmit’s version in autosubmit’s config file
Parameters: sleep_time (int) – value to set
-
set_version
(autosubmit_version)¶ Sets autosubmit’s version in autosubmit’s config file
Parameters: autosubmit_version (str) – autosubmit’s version
-
unify_conf
()¶ Unifies all configuration files into a single dictionary. Custom files will be able to override the default configuration.
-