Ctrl+K

autosubmit.platform#

class autosubmit.platforms.ecplatform.EcPlatform(expid, name, config, scheduler)#

Bases: ParamikoPlatform

Class to manage queues with ecaccess

Parameters:

  • expid (str) – experiment’s identifier

  • scheduler (str (pbs, loadleveler)) – scheduler to use

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

connect(as_conf, reconnect=False)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

delete_file(filename)#

Deletes a file from this platform

Parameters:

filename (str) – file name

Returns:

True if successful or file does not exist

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False)#

Copies a file from the current platform to experiment’s tmp folder

Parameters:

  • wrapper_failed –

  • ignore_log –

  • filename (str) – file name

  • must_exist (bool) – If True, raises an exception if file can not be copied

  • relative_path (str) – path inside the tmp folder

Returns:

True if file is copied successfully, false otherwise

Return type:

bool

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_ssh_output()#

Gets output from last command executed

Returns:

output from last command

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

jobs_in_queue()#

Returns empty list because ecacces does not support this command

Returns:

empty list

Return type:

list

move_file(src, dest, must_exist=False)#

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

restore_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

send_command(command, ignore_log=False, x11=False)#

Sends given command to HPC

Parameters:

  • x11 –

  • ignore_log –

  • command (str) – command to send

Returns:

True if executed, False if failed

Return type:

bool

send_file(filename, check=True)#

Sends a local file to the platform :param check: :param filename: name of the file to send :type filename: str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.lsfplatform.LsfPlatform(expid, name, config)#

Bases: ParamikoPlatform

Class to manage jobs to host using LSF scheduler

Parameters:

expid (str) – experiment’s identifier

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.pbsplatform.PBSPlatform(expid, name, config, version)#

Bases: ParamikoPlatform

Class to manage jobs to host using PBS scheduler

Parameters:

  • expid (str) – experiment’s identifier

  • version (str) – scheduler version

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.sgeplatform.SgePlatform(expid, name, config)#

Bases: ParamikoPlatform

Class to manage jobs to host using SGE scheduler

Parameters:

expid (str) – experiment’s identifier

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

connect(as_conf, reconnect=False)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

restore_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf)#

In this case, it does nothing because connection is established for each command

Returns:

True

Return type:

bool

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.slurmplatform.SlurmPlatform(expid, name, config, auth_password=None)#

Bases: ParamikoPlatform

Class to manage jobs to host using SLURM scheduler

Parameters:

expid (str) – experiment’s identifier

check_remote_log_dir()#

Creates log dir on remote host

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_estimated_queue_time_cmd(job_id)#

Returns command to get estimated queue time on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – str

Returns:

command to get estimated queue time

get_jobid_by_jobname_cmd(job_name)#

Returns command to get job id by job name on remote platforms :param job_name: :return: str

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_queue_status(in_queue_jobs, list_queue_jobid, as_conf)#

Get queue status for a list of jobs.

The job statuses are normally found via a command sent to the remote platform.

Each job in in_queue_jobs must be updated. Implementations may check for the reason for queueing cancellation, or if the job is held, and update the job status appropriately.

get_queue_status_cmd(job_id)#

Returns command to get queue status on remote platforms :return: str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(outputlines, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

open_submit_script()#

Opens Submit script file

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_finish_data(output, packed)#

Parses the context of the sacct query to SLURM for a single job. Only normal jobs return submit, start, finish, joules, ncpus, nnodes.

When a wrapper has finished, capture finish time.

Parameters:

  • output (str) – The sacct output

  • packed (bool) – true if job belongs to package

Returns:

submit, start, finish, joules, ncpus, nnodes, detailed_data

Return type:

int, int, int, int, int, int, json object (str)

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

parse_queue_reason(output, job_id)#

Parses the queue reason from the output of the command :param output: output of the command :param job_id: job id :return: queue reason :rtype: str

process_batch_ready_jobs(valid_packages_to_submit, failed_packages, error_message='', hold=False)#

Retrieve multiple jobs identifiers. :param valid_packages_to_submit: :param failed_packages: :param error_message: :param hold: :return:

submit_Script(hold: bool = False) → Union[List[str], str]#

Sends a Submit file Script, execute it in the platform and retrieves the Jobs_ID of all jobs at once.

Parameters:

hold (bool) – if True, the job will be held

Returns:

job id for submitted jobs

Return type:

list(str)

submit_job(job, script_name, hold=False, export='none')#

Submit a job from a given job object.

Parameters:

  • export –

  • job (autosubmit.job.job.Job) – job object

  • script_name – job script’s name

  • hold (boolean) – send job hold

Rtype scriptname:

str

Returns:

job id for the submitted job

Return type:

int

update_cmds()#

Updates commands for platforms

class autosubmit.platforms.locplatform.LocalPlatform(expid, name, config, auth_password=None)#

Bases: ParamikoPlatform

Class to manage jobs to localhost

Parameters:

expid (str) – experiment’s identifier

check_Alljobs(job_list, as_conf, retries=5)#

Checks jobs running status :param job_list: list of jobs :type job_list: list :param job_list_cmd: list of jobs in the queue system :type job_list_cmd: str :param remote_logs: remote logs :type remote_logs: str :param retries: retries :type default_status: bool :return: current job status :rtype: autosubmit.job.job_common.Status

check_file_exists(src, wrapper_failed=False, sleeptime=5, max_retries=3, first=True)#

Moves a file on the platform :param src: source name :type src: str :param: wrapper_failed: if True, the wrapper failed. :type wrapper_failed: bool

connect(as_conf, reconnect=False)#

Creates ssh connection to host

Returns:

True if connection is created, False otherwise

Return type:

bool

delete_file(filename, del_cmd=False)#

Deletes a file from this platform

Parameters:

filename (str) – file name

Returns:

True if successful or file does not exist

Return type:

bool

get_checkAlljobs_cmd(jobs_id)#

Returns command to check jobs status on remote platforms

Parameters:

  • jobs_id – id of jobs to check

  • jobs_id – str

Returns:

command to check job status

Return type:

str

get_checkjob_cmd(job_id)#

Returns command to check job status on remote platforms

Parameters:

  • job_id – id of job to check

  • job_id – int

Returns:

command to check job status

Return type:

str

get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False)#

Copies a file from the current platform to experiment’s tmp folder

Parameters:

  • wrapper_failed –

  • ignore_log –

  • filename (str) – file name

  • must_exist (bool) – If True, raises an exception if file can not be copied

  • relative_path (str) – path inside the tmp folder

Returns:

True if file is copied successfully, false otherwise

Return type:

bool

get_logs_files(exp_id, remote_logs)#

Overriding the parent’s implementation. Do nothing because the log files are already in the local platform (redundancy).

Parameters:

  • exp_id (str) – experiment id

  • remote_logs ((str, str)) – names of the log files

get_mkdir_cmd()#

Gets command to create directories on HPC

Returns:

command to create directories on HPC

Return type:

str

get_ssh_output()#

Gets output from last command executed

Returns:

output from last command

Return type:

str

get_submit_cmd(job_script, job, hold=False, export='')#

Get command to add job to scheduler

Parameters:

  • job –

  • job_script – path to job script

  • job_script – str

  • hold – submit a job in a held status

  • hold – boolean

  • export – modules that should’ve downloaded

  • export – string

Returns:

command to submit job to platforms

Return type:

str

get_submitted_job_id(output, x11=False)#

Parses submit command output to extract job id :param x11: :param output: output to parse :type output: str :return: job id :rtype: str

move_file(src, dest, must_exist=False)#

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

parse_Alljobs_output(output, job_id)#

Parses check jobs command output, so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_output(output)#

Parses check job command output, so it can be interpreted by autosubmit

Parameters:

output (str) – output to parse

Returns:

job status

Return type:

str

send_command(command, ignore_log=False, x11=False)#

Sends given command to HPC

Parameters:

  • x11 –

  • ignore_log –

  • command (str) – command to send

Returns:

True if executed, False if failed

Return type:

bool

send_file(filename, check=True)#

Sends a local file to the platform :param check: :param filename: name of the file to send :type filename: str

submit_Script(hold=False)#

Sends a Submitfile Script, exec in platform and retrieve the Jobs_ID. :param hold: send job hold :type hold: boolean :return: job id for the submitted job :rtype: int

test_connection(as_conf)#

Test if the connection is still alive, reconnect if not.

update_cmds()#

Updates commands for platforms

On this page
Edit on GitLab
Show Source