autosubmit.platform

class autosubmit.platforms.ecplatform.EcPlatform(expid, name, config, scheduler)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage queues with ecaccess

Parameters:
  • expid (str) – experiment’s identifier
  • scheduler (str (pbs, loadleveler)) – scheduler to use
connect()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
delete_file(filename)

Deletes a file from this platform

Parameters:filename (str) – file name
Returns:True if successful or file does no exists
Return type:bool
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False, max_tries=6)

Copies a file from the current platform to experiment’s tmp folder

Parameters:
  • filename (str) – file name
  • must_exist (bool) – If True, raises an exception if file can not be copied
  • relative_path (str) – path inside the tmp folder
Returns:

True if file is copied successfully, false otherwise
Return type:

bool
get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_ssh_output()

Gets output from last command executed

Returns:output from last command
Return type:str
get_submit_cmd(job_script, job, hold=False, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(output, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

jobs_in_queue()

Returns empty list because ecacces does not support this command

Returns:empty list
Return type:list
move_file(src, dest, must_exist=False, path_root=None)

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str :param path_root: root path :type path_root: str

parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
restore_connection()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
send_command(command, ignore_log=False, x11=False)

Sends given command to HPC

Parameters:command (str) – command to send
Returns:True if executed, False if failed
Return type:bool
send_file(filename, check=True)

Sends a local file to the platform :param filename: name of the file to send :type filename: str

test_connection()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
update_cmds()

Updates commands for platforms

class autosubmit.platforms.lsfplatform.LsfPlatform(expid, name, config)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage jobs to host using LSF scheduler

Parameters:expid (str) – experiment’s identifier
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_submit_cmd(job_script, job, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(output, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
update_cmds()

Updates commands for platforms

class autosubmit.platforms.pbsplatform.PBSPlatform(expid, name, config, version)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage jobs to host using PBS scheduler

Parameters:
  • expid (str) – experiment’s identifier
  • version (str) – scheduler version
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_submit_cmd(job_script, job, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(output, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
update_cmds()

Updates commands for platforms

class autosubmit.platforms.sgeplatform.SgePlatform(expid, name, config)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage jobs to host using SGE scheduler

Parameters:expid (str) – experiment’s identifier
connect()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_submit_cmd(job_script, job, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(output, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
restore_connection()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
test_connection()

In this case, it does nothing because connection is established for each command

Returns:True
Return type:bool
update_cmds()

Updates commands for platforms

class autosubmit.platforms.slurmplatform.SlurmPlatform(expid, name, config)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage jobs to host using SLURM scheduler

Parameters:expid (str) – experiment’s identifier
get_checkAlljobs_cmd(jobs_id)

Returns command to check jobs status on remote platforms

Parameters:
  • jobs_id – id of jobs to check
  • job_id – str
Returns:

command to check job status
Return type:

str
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_jobid_by_jobname_cmd(job_name, minutes='5')

Returns command to get job id by job name on remote platforms :param job_name: :return: str

get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_queue_status(in_queue_jobs, list_queue_jobid, as_conf)

Get queue status for a list of jobs.

The job statuses are normally found via a command sent to the remote platform.

Each job in in_queue_jobs must be updated. Implementations may check for the reason for queueing cancellation, or if the job is held, and update the job status appropriately.

get_queue_status_cmd(job_id)

Returns command to get queue status on remote platforms :return: str

get_submit_cmd(job_script, job, hold=False, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(outputlines, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

open_submit_script()

Opens Submit script file

parse_Alljobs_output(output, job_id)

Parses check jobs command output so it can be interpreted by autosubmit :param output: output to parse :param job_id: select the job to parse :type output: str :return: job status :rtype: str

parse_job_finish_data(output, packed)

Parses the context of the sacct query to SLURM for a single job. Only normal jobs return submit, start, finish, joules, ncpus, nnodes.

When a wrapper has finished, capture finish time.

Parameters:
  • output (str) – The sacct output
  • job_id (int) – Id in SLURM for the job
  • packed (bool) – true if job belongs to package
Returns:

submit, start, finish, joules, ncpus, nnodes, detailed_data
Return type:

int, int, int, int, int, int, json object (str)
parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
submit_Script(hold=False)

Sends a Submit file Script, execute it in the platform and retrieves the Jobs_ID of all jobs at once.

Parameters:job (autosubmit.job.job.Job) – job object
Returns:job id for submitted jobs
Return type:list(str)
update_cmds()

Updates commands for platforms

class autosubmit.platforms.locplatform.LocalPlatform(expid, name, config)

Bases: autosubmit.platforms.paramiko_platform.ParamikoPlatform

Class to manage jobs to localhost

Parameters:expid (str) – experiment’s identifier
check_file_exists(src, wrapper_failed=False)

Moves a file on the platform :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

connect()

Creates ssh connection to host

Returns:True if connection is created, False otherwise
Return type:bool
delete_file(filename, del_cmd=False)

Deletes a file from this platform

Parameters:filename (str) – file name
Returns:True if successful or file does no exists
Return type:bool
get_checkjob_cmd(job_id)

Returns command to check job status on remote platforms

Parameters:
  • job_id – id of job to check
  • job_id – int
Returns:

command to check job status
Return type:

str
get_file(filename, must_exist=True, relative_path='', ignore_log=False, wrapper_failed=False, max_tries=1)

Copies a file from the current platform to experiment’s tmp folder

Parameters:
  • filename (str) – file name
  • must_exist (bool) – If True, raises an exception if file can not be copied
  • relative_path (str) – path inside the tmp folder
Returns:

True if file is copied successfully, false otherwise
Return type:

bool
get_logs_files(exp_id, remote_logs)

Overriding the parent’s implementation. Do nothing because the log files are already in the local platform (redundancy).

Parameters:
  • exp_id (str) – experiment id
  • remote_logs ((str, str)) – names of the log files
get_mkdir_cmd()

Gets command to create directories on HPC

Returns:command to create directories on HPC
Return type:str
get_ssh_output()

Gets output from last command executed

Returns:output from last command
Return type:str
get_submit_cmd(job_script, job, hold=False, export='')

Get command to add job to scheduler

Parameters:
  • job_type –
  • job_script – path to job script
  • job_script – str
  • hold – submit a job in a held status
  • hold – boolean
  • export – modules that should’ve downloaded
  • export – string
Returns:

command to submit job to platforms
Return type:

str
get_submitted_job_id(output, x11=False)

Parses submit command output to extract job id :param output: output to parse :type output: str :return: job id :rtype: str

move_file(src, dest, must_exist=False, path_root=None)

Moves a file on the platform (includes .err and .out) :param src: source name :type src: str :param dest: destination name :param must_exist: ignore if file exist or not :type dest: str

parse_job_output(output)

Parses check job command output so it can be interpreted by autosubmit

Parameters:output (str) – output to parse
Returns:job status
Return type:str
send_command(command, ignore_log=False, x11=False)

Sends given command to HPC

Parameters:command (str) – command to send
Returns:True if executed, False if failed
Return type:bool
send_file(filename)

Sends a local file to the platform :param filename: name of the file to send :type filename: str

test_connection()

Test if the connection is still alive, reconnect if not.

update_cmds()

Updates commands for platforms