Manage failover

Failover and revert (redundancy)

Failover mechanisms ensure service continuity. If a whole server fails, or if critical alarms are detected by the processing application, services can be moved to a backup server. The Controller automatically manages failover when the automatic mode is configured. You can also manually failover, or revert once the primary server is repaired. Both server-level and service-level failover are supported.

• For server-level failover, when an alarm is received which is configured to trigger failover:
  • All services on the involved server will be moved to one of the configured backups.
  • That backup server will no longer be available for further failovers
• For service-level failover, when an alarm is received which is configured to trigger failover:
  • If it is a server-level alarm: all services on the involved server will be moved to one of the configured backups (as for a server-level failover).
  • If it is a service-level alarm: only the affected service will be moved.
  • The backup server in use will be reserved for further failovers from the same primary

Packager redundancy is managed by the Packager application (cluster of servers). The Controller is not involved in Packager redundancy.

Timestamps

• All timestamps are in Universal Time (UTC).
• Failover servers are selected and grouped according to processing type.
• User permissions are required to edit failover groups, or to failover or revert.
• If the server has no services assigned, then default user rights for failover apply.

Triggers

• Automatic failovers are triggered by alarms that are marked as "triggers failover".
• By default, all critical alarms will trigger failovers. See Alarm overrides for more details.

Display failover groups

1. Click Failover from the left-side menu pane. Failover groups display in the table.

The alarm icon displays the alarm status for the highest severity alarm based on the servers in the failover group.

2. Click in the Actions column to display details.

All time stamps are in Universal Time (UTC).

Display a failover summary

The failover summary table shows progress and status for failover procedures.

1. Click Failover from the left-side menu pane. Failover groups display.

2. Scroll to the bottom of the page to display the Failover summary table.

All time stamps are in Universal Time (UTC).

3. Click in the Actions column to display details.

Create a failover group

1. Display failover groups.

2. Click Add group…

3. Enter a Group name.

4. Select a Failover level.

5. Select a Group processing type for the servers in the group.

6. Select a Group failover mode.

The default Group failover mode is Automatic. You can manually trigger a failover, or revert, even in Automatic group failover mode.

7. Select available servers, and use the arrows to define the list of primary and backup servers. The new failover group displays in the list.

You can use the search option to display and select specific servers (for example, per processing type).

Edit a failover group

1. Display failover groups.

2. Click . The failover setup page displays.

Edit options are only available when Server settings are modifiable. If there is no button, check user rights or contact an administrator.

3. Edit the group settings configuration.

• Failover level can only be changed if there is no active failover for this group (no services assigned to a backup).
• Group processing type cannot be changed.

4. Edit the server selection for primary and backup servers, then click Save and exit. The group is added to the list.

Request a failover or a revert between servers manually

Before you start: At least one Failover group exists.

User permissions are required to request a failover or a revert. If the server has no services assigned, then default user rights for failover apply.

1. Display failover groups.

2. Click to display details.

3. Click '%3e%3cpath%20d='M3.98896%206.4287H12.2989L12.3009%208.16543C12.3009%208.39572%2012.4477%208.6042%2012.6752%208.6961C12.9026%208.78805%2013.167%208.74622%2013.3488%208.58982L16.2582%206.08121C16.5242%205.85206%2016.5242%205.46153%2016.2582%205.23238L13.3488%202.72377C13.1672%202.56736%2012.9028%202.52556%2012.6752%202.61744C12.4477%202.70949%2012.3009%202.91799%2012.3009%203.14811L12.2989%204.88494H3.98896C3.52944%204.88494%203.15771%205.22987%203.15771%205.63511C3.15771%206.04035%203.52944%206.4287%203.98896%206.4287ZM15.6265%2011.06H7.31656L7.31396%209.30154C7.31396%209.07126%207.16715%208.86278%206.93964%208.77088C6.71391%208.70092%206.44895%208.74193%206.26711%208.89872L3.35773%2011.4073C3.09173%2011.6365%203.09173%2012.027%203.35773%2012.2562L6.26711%2014.7648C6.44874%2014.9212%206.71313%2014.963%206.94068%2014.8711C7.1685%2014.7795%207.31396%2014.5696%207.31396%2014.3405L7.31656%2012.6037H15.6265C16.086%2012.6037%2016.4577%2012.2586%2016.4577%2011.8319C16.4577%2011.4052%2016.0862%2011.06%2015.6265%2011.06Z'%20fill='%23878787'%20fill-opacity='0.9'/%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_590_376'%3e%3crect%20width='19'%20height='17.5385'%20fill='white'%20transform='translate(0.307739%200.230835)'/%3e%3c/clipPath%3e%3c/svg%3e) to failover to backup servers, or revert to primary servers. You are prompted to continue.

4. A modal pop-up is displayed, showing progress and success/failure of this operation.

5. When complete, confirm that the assigned servers are correct, by checking the services page.

For a service-level group, manual failover/revert behave exactly as for a server-level group; all services on the primary server will be moved to the backup. All services on the backup server will be moved back to the configured primary (whether all services due to a manual failover, or service(s) which have been automatically moved in response to an alarm).

If the failover group is service-level, manual failover and revert can be done from the services page.

Manual service failover/revert is only available for services that are assigned to a single server.

1. From the Services page, click to show more actions on the service you want to failover or revert.

2. Click on the (failover) or '%20fill-opacity='0.9'%20fill='%23878787'%20d='M%204.3%2010.2%20H%2012.6%20L%2012.6%2012%20C%2012.6%2012.2%2012.7%2012.4%2013%2012.5%20C%2013.2%2012.6%2013.5%2012.5%2013.6%2012.4%20L%2016.6%209.9%20C%2016.8%209.7%2016.8%209.3%2016.6%209%20L%2013.6%206.5%20C%2013.5%206.4%2013.2%206.3%2013%206.4%20C%2012.7%206.5%2012.6%206.7%2012.6%206.9%20L%2012.6%208.7%20H%204.3%20C%203.8%208.7%203.5%209%203.5%209.4%20C%203.5%209.8%203.8%2010.2%204.3%2010.2%20Z'%3e%3c/path%3e%3c/svg%3e) (revert) icon to perform the action. Only the icon for the available action is displayed.

3. Click Yes to confirm the action.

4. A modal pop-up is displayed, showing progress and success/failure of this operation.

If multiple services are selected, the failover/revert action is only available if all selected services are on the same server.

Select failover trigger settings

By default, a failover is triggered for any critical alarm. You can select the failover trigger option for any alarm. This also applies to alarms that you have changed from the default alarm severity.

Changes to severity are only applied to future occurrences. Any current or previously existing alarms are unchanged.

1. Click Alarms in the left hand menu.

2. Click Alarm overrides to display options. The following page displays.

3. Select a response for Trigger failover.