.. _workflow_recovery:

How to restart the experiment
=============================

How to recover an experiment
----------------------------

We utilize the recovery command to restart Autosubmit when it is killed in an ungraceful way. 

.. warning:: You can only recover when the workflow is not running (i.e. there no QUEUING, SUBMITTED, or RUNNING tasks) 

The recovery command will change the state of all the tasks that are (or can be) in READY status to COMPLETED if a completed file for that task is found. 

To recover a running workflow, we must issue the recovery command with -f, so Autosubmit kills the tasks in remote before resetting their status.

Because tasks might have completed in remote, we can ask Autosubmit to check for the completion file there. To do so, use the recovery command with the --all flag.

.. warning:: without the -s Autosubmit will only perform a dry-run (i.e. it will not take effect) of the command  

Before commiting to changes on the state, check the .txt file generated by Autosubmit with the states of the tasks. When we are satisfied with the results we can use the parameter -s, which will save the change to the pkl file and rename the update file.

Examples:

::
    # check locally for completion files, do not kill running tasks, and do not commit changes
    autosubmit recovery cxxx
    # commit to the previous states
    autosubmit recovery cxxx -s
    # check locally for completion file AND issue kill commands in remote
    autosubmit recovery cxxx -f -s
    # check in remote for completion files 
    autosubmit recovery cxxx --all -s

Options:
::

    usage: autosubmit recovery [-h] [-np] [--all] [-s] [--hide] [-group_by {date,member,chunk,split,automatic}] [-expand EXPAND]
                           [-expand_status EXPAND_STATUS] [-nt] [-nl] [-d] [-f] [-v]
                           EXPID

    recover specified experiment

    positional arguments:
      EXPID                 experiment identifier

    options:
      -h, --help            show this help message and exit
      -np, --noplot         omit plot
      --all                 Get completed files to synchronize pkl
      -s, --save            Save changes to disk
      --hide                hides plot window
      -group_by {date,member,chunk,split,automatic}
                        Groups the jobs automatically or by date, member, chunk or split
      -expand EXPAND        Supply the list of dates/members/chunks to filter the list of jobs. Default = "Any". LIST = "[
                        19601101 [ fc0 [1 2 3 4] fc1 [1] ] 19651101 [ fc0 [16-30] ] ]"
      -expand_status EXPAND_STATUS
                            Select the statuses to be expanded
      -nt, --notransitive   Disable transitive reduction
      -nl, --no_recover_logs
                            Disable logs recovery
      -d, --detail          Show Job List view in terminal
      -f, --force           Cancel active jobs
      -v, --update_version  Update experiment version

How to rerun a part of the experiment
-------------------------------------

This procedure allows you to create automatically a new pickle with a list of jobs of the experiment to rerun.

Using the ``expdef_<expid>.yml`` the ``create`` command will generate the rerun if the variable RERUN is set to TRUE and a RERUN_JOBLIST is provided.

Additionally, you can have re-run only jobs that won't be include in the default job_list. In order to do that, you have to set RERUN_ONLY in the jobs conf of the corresponding job.

::

    autosubmit create cxxx

It will read the list of jobs specified in the RERUN_JOBLIST and will generate a new plot.

Example:
::

    vi <experiments_directory>/cxxx/conf/expdef_cxxx.yml

.. code-block:: yaml

    ...

    rerun:
        RERUN: TRUE
        RERUN_JOBLIST: RERUN_TEST_INI;SIM[19600101[C:3]],RERUN_TEST_INI_chunks[19600101[C:3]]
    ...

    vi <experiments_directory>/cxxx/conf/jobs_cxxx.yml

.. code-block:: yaml

    PREPROCVAR:
        FILE: templates/04_preproc_var.sh
        RUNNING: chunk
        PROCESSORS: 8

    RERUN_TEST_INI_chunks:
        FILE: templates/05b_sim.sh
        RUNNING: chunk
        RERUN_ONLY: true

    RERUN_TEST_INI:
        FILE: templates/05b_sim.sh
        RUNNING: once
        RERUN_ONLY: true

    SIM:
        DEPENDENCIES: RERUN_TEST_INI RERUN_TEST_INI_chunks PREPROCVAR SIM-1
        RUNNING: chunk
        PROCESSORS: 10

    .. figure:: fig/rerun.png
       :name: rerun_result
       :align: center
       :alt: rerun_result

Run the command:

.. code-block:: bash

    # Add your key to ssh agent ( if encrypted )
    ssh-add ~/.ssh/id_rsa
    nohup autosubmit run cxxx &