autosubmit.platform#

class autosubmit.platforms.ecplatform.EcPlatform(expid, name, config, scheduler)#

Bases: ParamikoPlatform

Class to manage queues with ecaccess

Parameters:

  • expid (str) – experiment’s identifier

  • scheduler (str (pbs, loadleveler)) – scheduler to use

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

connect(as_conf, reconnect=False)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

delete_file(filename)#

Deletes a file from this platform

Parameters:

filename (str) – file name

Returns:

True if successful or file does not exist

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False)#

Copies a file from the current platform to experiment’s tmp folder

Parameters:

  • wrapper_failed

  • ignore_log

  • filename (str) – file name

  • must_exist (bool) – If True, raises an exception if file can not be copied

  • relative_path (str) – path inside the tmp folder

Returns:

True if file is copied successfully, false otherwise

Return type:

bool

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_ssh_output()#

Gets output from last command executed

Returns:

output from last command

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

jobs_in_queue()#

Returns empty list because ecacces does not support this command

Returns:

empty list

Return type:

list

move_file(src, dest, must_exist=False)#

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

restore_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

send_command(command, ignore_log=False, x11=False)#

Sends given command to HPC

Parameters:

  • x11

  • ignore_log

  • command (str) – command to send

Returns:

True if executed, False if failed

Return type:

bool

send_file(filename, check=True)#

Sends a local file to the platform :param check: :param filename: name of the file to send :type filename: str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf: Any) → None#

Tests the connection using the provided configuration.

Args:

as_conf (AutosubmitConfig): The configuration to use for testing the connection.

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.pbsplatform.PBSPlatform(expid, name, config, version)#

Bases: ParamikoPlatform

Class to manage jobs to host using PBS scheduler

Parameters:

  • expid (str) – experiment’s identifier

  • version (str) – scheduler version

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.sgeplatform.SgePlatform(expid, name, config)#

Bases: ParamikoPlatform

Class to manage jobs to host using SGE scheduler

Parameters:

expid (str) – experiment’s identifier

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

connect(as_conf, reconnect=False)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

restore_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.slurmplatform.SlurmPlatform(expid, name, config, auth_password=None)#

Bases: ParamikoPlatform

Class to manage jobs to host using SLURM scheduler

Parameters:

expid (str) – experiment’s identifier

check_file_exists(filename: str, wrapper_failed: bool = False, sleeptime: int = 5, max_retries: int = 3) → bool#

Checks if a file exists on the FTP server.

Args:

filename (str): The name of the file to check. wrapper_failed (bool): Whether the wrapper has failed. Defaults to False. sleeptime (int): Time to sleep between retries in seconds. Defaults to 5. max_retries (int): Maximum number of retries. Defaults to 3.

Returns:

bool: True if the file exists, False otherwise.

check_remote_log_dir()#

Creates log dir on remote host

generate_submit_script()#

Opens Submit script file

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_estimated_queue_time_cmd(job_id)#

Returns command to get estimated queue time on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – str

Returns:

command to get estimated queue time

get_jobid_by_jobname_cmd(job_name)#

Returns command to get job id by job name on remote platforms :param job_name: :return: str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_queue_status(in_queue_jobs, list_queue_jobid, as_conf)#

Get queue status for a list of jobs.

The job statuses are normally found via a command sent to the remote platform.

Each job in in_queue_jobs must be updated. Implementations may check for the reason for queueing cancellation, or if the job is held, and update the job status appropriately.

get_queue_status_cmd(job_id)#

Returns command to get queue status on remote platforms :return: str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submit_cmd_x11(args, script_name, job)#

Returns the submit command for the platform

get_submitted_job_id(outputlines, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

parse_queue_reason(output, job_id)#

Parses the queue reason from the output of the command :param output: output of the command :param job_id: job id :return: queue reason :rtype: str

process_batch_ready_jobs(valid_packages_to_submit, failed_packages, error_message='', hold=False)#

Retrieve multiple jobs identifiers. :param valid_packages_to_submit: :param failed_packages: :param error_message: :param hold: :return:

submit_Script(hold: bool = False) → List[str] | str#

Sends a Submit file Script, execute it in the platform and retrieves the Jobs_ID of all jobs at once.

Parameters:

hold (bool) – if True, the job will be held

Returns:

job id for submitted jobs

Return type:

list(str)

submit_job(job, script_name, hold=False, export='none')#

Submit a job from a given job object.

Parameters:

  • export

  • job (autosubmit.job.job.Job) – job object

  • script_name – job script’s name

  • hold (boolean) – send job hold

Rtype scriptname:

str

Returns:

job id for the submitted job

Return type:

int

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.locplatform.LocalPlatform(expid, name, config, auth_password=None)#

Bases: ParamikoPlatform

Class to manage jobs to localhost

Parameters:

expid (str) – experiment’s identifier

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

check_completed_files(sections: str | None = None) → str#

Checks for completed files in the remote log directory. This function is used to check inner_jobs of a wrapper.

Args:

sections[str]: Space-separated string of sections to check for completed files. Defaults to None.

Returns:

str: The output if the command is successful, None otherwise.

check_file_exists(src: str, wrapper_failed: bool = False, sleeptime: int = 1, max_retries: int = 1) → bool#

Checks if a file exists in the platform.

Args:

src (str): Source name. wrapper_failed (bool): Checks inner jobs files. Defaults to False. sleeptime (int): Time to sleep between retries. Defaults to 1. max_retries (int): Maximum number of retries. Defaults to 1.

Returns:

bool: True if the file exists, False otherwise.

connect(as_conf={}, reconnect=False)#

Creates ssh connection to host

Returns:

True if connection is created, False otherwise

Return type:

bool

delete_file(filename, del_cmd=False)#

Deletes a file from this platform

Parameters:

filename (str) – file name

Returns:

True if successful or file does not exist

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False)#

Copies a file from the current platform to experiment’s tmp folder

Parameters:

  • wrapper_failed

  • ignore_log

  • filename (str) – file name

  • must_exist (bool) – If True, raises an exception if file can not be copied

  • relative_path (str) – path inside the tmp folder

Returns:

True if file is copied successfully, false otherwise

Return type:

bool

get_logs_files(exp_id, remote_logs)#

Overriding the parent’s implementation. Do nothing because the log files are already in the local platform (redundancy).

Parameters:

  • exp_id (str) – experiment id

  • remote_logs ((str, str)) – names of the log files

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_ssh_output()#

Gets output from last command executed

Returns:

output from last command

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

move_file(src, dest, must_exist=False)#

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

remove_multiple_files(filenames: str) → str#

Creates a shell script to remove multiple files in the remote and sets the appropriate permissions.

Args:

filenames (str): A string containing the filenames to be removed.

Returns:

str: An empty string.

send_command(command, ignore_log=False, x11=False)#

Sends given command to HPC

Parameters:

  • x11

  • ignore_log

  • command (str) – command to send

Returns:

True if executed, False if failed

Return type:

bool

send_file(filename: str, check: bool = True) → bool#

Sends a file to a specified location using a command.

Args:

filename (str): The name of the file to send. check (bool): Unused in this platform

Returns:

bool: True if the file was sent successfully.

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf)#

Test if the connection is still alive, reconnect if not.

update_cmds()#

Updates commands for platforms

On this page
Edit on GitHub
Show Source