PTS - How to configure a spare automation server

PTS uses Etere solutions to carry out its entire broadcast chain, from ingest to playout, connecting all devices present in the station under a fully integrated workflow system.

INTRODUCTION
PTS was founded in 1998 as a terrestrial television channel, and in 2006, was charged by the Taiwanese government with developing an HDTV (High Definition TV), a set of digital television (DTV) standards that offer the highest resolution and sharpest picture. PTS today produces a range of high quality, independent content, including local and national news programs, documentaries, TV series, children contents and other critically acclaimed programs.

PTS uses Etere solutions to carry out its entire broadcast chain, from ingest to playout, connecting all devices present in the station under a fully integrated workflow system. PTS playout system is composed by 3 main/clone redundant playlists with an extra server available for maintenance, that is, for replacing the main and backup servers in case one of them needs to be shutdown for hardware maintenance.
This document describes how to implement and reconfigure a spare Etere Automation server to keep the availability of two automation servers during routine maintenance operations, this spare server will be able to replace either the master or clone automation without altering the redundancy structure of the playout system.

REQUESTED FEATURES
PTS has requested a document that will guide the playout supervisor on reconfiguring a spare server holding an Etere Automation installation that will be able to replace either the master or the clone automations when they are temporally offline due to maintenance operations, thus keeping the playout system redundant for the entire downtime.
In order to implement a solution able to completely suite the requested feature, here below will be listed the characteristics of the automation channels managed by:
■ The playout system is composed by three redundant on-air playlists,
■ Each on-air playlist is managed by a master and a clone automation,
■ Master and clone automations can replace them mutually in case of unavailability,
■ The Master and Clone automations are installed in different main and a backup servers,
■ Each automation pilots a main video server and a virtual router (conformed by a master control and a backup router),
■ One between Main and Backup servers can be temporally down due to hardware maintenance; during this time a third server must be able to replace the down server.

On the basis of what exposed above, a quick guide must be elaborated explaining how to support the replacement of a temporally-down server by using a spare server, this spare server will have to be able to work exactly as the replaced server (i.e.: identical configuration and functioning) piloting the same devices (i.e.: video servers, master controls, backup routers, logo generators and subtitlers). A special mention is needed for the PolyStream subtitling device used by PTS, it requires precise parameters (i.e.: hostname, IP address) regarding the automation server, so it will be required to update these parameters each time the spare server is used to replace an main or backup server.

In the following chapter, it will be described the implementation of a spare server able to replace either the main and clone automation during their hardware maintenance, this spare server must be reconfigured at each use since its configuration changes depending on the kind of automation it is going to replace.

PROPOSED SOLUTION: USING A SPARE SERVER
The “spare automation server” solution requires a third server (for each on-air playlist) which will use a static IP address different from the ones used by the main and clone automations. The spare server will count with all software required to replace a down master/clone automation, being necessary to reconfigure it each time it will be used to replace either a master or clone automation.

As shown in the diagram above, software automation and hardware servers involved in the playout system will be referenced in the following text using the following names:
■ Master Automation: It’s the primary automation system, that is, the instance of Etere Automation that transmit the playlist,
■ Clone Automation: It’s the secondary automation system, that is, the instance of Etere Automation that is ready to replace the Master Automation
■ Main Server: The computer intended to hold the Master Automation,
■ Backup Server: The computer intended to hold the Clone Automation,
■ Spare Server: The computer used to replace either the main or backup server, it holds an automation system that can be master or clone depending on the eventual needs.

Preliminary configurations
Before passing to the replacement procedure, some important configurations must be set in order to ensure the complete functioning of the spare server as either a master or clone automation:

Spare server login account
The user that will log into the spare server must be a local administrator. It must belong to the domain group “Etere” thus sharing the same rights assigned to other onair users on windows Active Directory and on SQL/Security/Logins. Etere rights must be equivalent to other onair users as well.

Enable the resources for the Spare Server
The resource configuration described below must be performed only the first time before using the spare server, paying attention to do not remove any already existing resources. In order to assign resources to the spare server just follow the next procedure for each channel:

1. Launch “Etere Configuration” and open the “Resources” module,
2. Select in the “User’s resources” and “Module” dropdown lists the domain-username (e.g.: AUMAIN1) of the master automation server and the function (e.g.: automation) respectively, click on the “update” button
3. All resources used by the master automation will be highlighted with a yellow balloon
4. Make click on the used Omneon video server channel and press the “Users” button to include the spare server to the list of users allowed to use this device
5. Press the “Add” button and select the spare server’s domain-username (e.g.: AUSPARE) and the function (e.g.: automation). Also configure the “router” section
6. Once done, make click on the “Ok” button and then the “Close” button
7. Once the master automation video server channel has been configured, set also the Omneon video server channel used by the clone automation to be used also by the spare server, this by repeating the steps 2 (selecting the automation clone function), 3, 4, 5 and 6

At this point, it is required to set the rest of resources also piloted by the master and clone automations, to do this, just repeat the entire procedure described above for each one of the involved resources, that is, for the master control, backup router, virtual router, Deko, logo generator, and any other resource used by the automations.

■Set permissions for the spare publishing path
The path on which the playlist will be published (e.g.: C:\Program Files\ETERE\user.001) must be shared to the network and count with the proper ‘read/write’ permissions for the scheduling users that will publish the on-air playlist to it.

4. REPLACING THE MAIN SERVER
In this chapter it will be explained in a step-by-step method, how to reconfigure the spare server to be used for replacing a main server during its downtime (e.g..: due to hardware maintenance) without altering the redundancy structure of the system.

Main Server
■ In the main server, set Etere Automation to operate in clone-mode. This action will make the backup server to operate in master-mode, thus releasing the main server that now can be stopped and closed. The backup server, running now in master-mode, will remain as the sole automation while the spare server is being reconfigured and activated. -Now it is possible to turn-off the main server-

■ Spare Server
In the Spare Server, launch Etere Configuration, select Settings / Executive Editor / On Air, and set the “spare” publishing path of the on-air playlist in the relative “automation path” field. For example, change the \\AUMAIN1\ETERE\USER.001 path into \\AUSPARE\ETERE\USER.001

Once done, press the “Apply” button and close Etere Configuration. NB: It will be also necessary to restart all instances of Executive Editor to load the new parameter. Publish the schedule and check it is present in the new configured path.

■ Launch Etere Automation, go to the “parameters” section, and under the “Main/Backup” tab set the IP address and communication port of the backup server and press the “OK” button.

In order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module. As usual, it will start operating in clone-mode, cueing the next event of the playlist.

After clip change, when the spare automation already started playing, click on the “Clone” button to make it operate in master-mode, this action will make the backup server to operate in clone-mode. NB: It is important to check that the spare server is operating in master-mode (i.e.: currently transmitting the playlist) before passing to the next step, otherwise when the backup server will be restarted to load the new settings, you will have black going on air.

Backup Server
■ In the backup server, stop Etere Automation and close the “On-Air” window, go to the “parameters” section and under the “Main/Backup” tab set the IP address and communication port of the -now active- spare server.
Once configured, in order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module, Etere Automation will start operating in clone-mode. At this point, the replacement of the main server with the spare one is completed.

Replacing back the main server
■Spare Server
In the spare server, set Etere Automation to operate in clone-mode:
This action will make the backup server to operate in master-mode, thus releasing the spare server, which now can be stopped and closed. The backup server will remain as the sole automation while the main server is being reconfigured and activated. -Now it is possible to turn-off the spare server-

■Main Server
In the main Server, launch Etere Configuration, select Settings / Executive Editor / On Air, and set the “main” publishing path of the on-air playlist in the relative “automation path” field. For example, change the \\AUSPARE\ETERE\USER.001 path into \\AUMAIN1\ETERE\USER.001.
Once done, press the “Apply” button and close Etere Configuration, it will be also necessary to restart all the running instances of Executive Editor to load the new parameter. Publish the schedule and check it is present in the new configured path. Launch Etere Automation and open the “On-Air” module. As usual, it will start operating in clone-mode, cueing the next event of the playlist. After clip change, when the automation already started playing, click on the “Clone” button to make it operate in master-mode, this action will make the backup server to operate in clone-mode.

NB: It is important to check that the main server is operating in master-mode (i.e.: currently transmitting the playlist) before passing to the next step, otherwise when the backup server will be restarted to load the new settings, you will have black going on air.
Backup Server
■ In the backup server, stop Etere Automation and close the “On-Air” window, go to the “parameters” section and under the “Main/Backup” tab set the IP address and communication port of the -now active- main server: Once configured, and in order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module, Etere Automation will start operating in clone-mode. Now, the replacement of the spare server with the main one is completed.

Replacing the backup server
In this chapter it will be explained in a step-by-step method, how to reconfigure the spare server to be used for replacing a backup server during its downtime (e.g.: due to hardware maintenance) without altering the redundancy structure of the system, just as shown in the image below:

Backup Server
Check the automation on the main server is running in master-mode and the one on the backup server is running in clone-mode. Stop the Automation clone on the backup server, close the “On-Air” window and close Etere Automation. The main server will remain as the sole automation while the spare server is being configured and activated. Now it is possible to turn-off the backup server.
Spare Server
In the spare server, launch Etere Configuration and set the publishing path of the on-air playlist in the “automation backup path” field. For example, change the \\AUCLONE1\ETERE\USER.001 path into \\AUSPARE\ETERE\USER.001

Once done, press the “Apply” button and close Etere Configuration. NB: It will be also necessary to restart all instances of Executive Editor to load the new parameter. Publish the schedule and check it is present in the just set path.
■ Launch Etere Automation, go to the “parameters” section and under the “Main/Backup” tab set the IP address and communication port of the main server and press the “OK” button:

In order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module, Etere Automation will start operating in clone-mode.
At this point, set Etere Automation to operate in master-mode (it will make the main server to operate in clone-mode).
NB: It is important to check that the spare server is operating in master-mode (i.e.: currently transmitting the playlist) before passing to the next step, otherwise when the main server will be restarted to load the new settings, you will have black going on air.

Main Server
In the main server, stop Etere Automation and close the “On-Air” window, go to the “parameters” section and under the “Main/Backup” tab set the IP address and communication port of the –now active- spare server

Once configured, and in order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module, Etere Automation will start operating in clone-mode; set it to operate in master-mode: At this point, the replacement of the backup server with the spare one is completed.

Replacing back the backup server
Spare Server
■ In the spare server, stop Etere Automation, close the “On-Air” window and close Etere Automation. The main server will remain as the sole automation while the backup server is being configured and activated. Now it is possible to turn-off the spare server.

Backup Server
■ In the backup server, launch Etere Configuration and set the publishing path of the on-air playlist in the “automation backup path” field. For example, change the \\AUSPARE\ETERE\USER.001 path into \\AUCLONE1\ETERE\USER.001

Once done, press the “Apply” button and close Etere Configuration, it will be also necessary to restart all the running instances of Executive Editor to load the new parameter. Publish the schedule and check it is present in the new configured path.
■ Launch Etere Automation and open the “On-Air” module, Etere Automation will start operating in clone-mode. At this point, set Etere Automation to function in master-mode (it will make the main server to operate in clone-mode).

NB: It is important to check that the backup server is operating in master-mode (i.e.: currently transmitting the playlist) before passing to the next step, otherwise when the main server will be restarted to load new settings, a blackout will occur.

Main Server
■ In the main server, stop Etere Automation and close the “On-Air” window, go to the “parameters” section and under the “Main/Backup” tab set the IP address and communication port of the backup server.
Once configured, and in order to make changes take effect, close Etere Automation, launch it again and open the “On-Air” module, Etere Automation will start operating in clone-mode. At this point, the replacement of the spare server with the backup one is completed.

KEY CONSIDERATIONS
The use of a spare server for replacement during routine maintenance is a solution that can help keeping a redundant playout system and help stations to effectively respond to situations that requires immediate answers for a hardware failure on any of the main/backup servers. When using a spare server, Etere recommends taking into account the following considerations:
■ Using a spare server requires maximum accuracy when manually reconfiguring assigned resources, automation parameters, publishing paths, folder permissions, etc,
■ The procedures explained in chapters 3, 4 and 5 regard a generic spare server, those procedures can be applied to the spare server of any of the three on-air playlists,
■ Using only one spare server requires few minutes of downtime of the replaced automation (either master or clone),
■ For a faster replacement, Etere suggests using one cold-standby server per automation; this will result in only seconds of downtime (since reconfiguring is not needed because spare servers are configured only once for replacing an exclusive server)