Etere Guide to Investing in a Digital Archive

With the popularity of digital broadcasting, the need for digital video archive solutions increases. Data rates for video are increasing dramatically as more stations move towards broadcasting in High Definition (HD) and 4K (UHD).

Video tape cartridges are no longer commonly used as before. In fact, studio cameras these days no longer contain tape cartridges, instead they use a direct connection or store recorded video on a hard disk, flash memory or optical media. Many stations also now process video directly from camera to their Non Linear Editing (NLE) systems. While recording to videotape is no longer an option, archives must still be created. Having a digital video archive system that is ready to match the needs of the facility is important.

This white paper will explore topics such as how to identify a system that works well for your needs, not only presently but in the future as well.

Some consideration factors to look out for include:
■ Size of the archive
■ What to do with legacy footage
■ Selecting a format to store video
■ Future growth
■ Adding Media Asset Management
■ Workflow
■ Who will access the archive
■ Webcasting
■ Access from NLEs

First consideration factors include the size of archive you will need now and how much it will grow over time. The first part is easy as you know how many hours of video you are creating right now, but what do you want to do with your existing footage? This is a trade off: do you convert the video content on your old tapes to a new digital format to store them, or do you leave the existing video library alone and only archive your new content? The problem becomes more complex if you add asset management to your new archive, as you now have to decide if you wish to catalogue the legacy footage, as well.

In addition, sometimes, your legacy footage may be deteriorating or you may really need the floor space occupied by the existing video library. Other factors include the availability of older equipment to play the video as well as the maintenance and support costs for that equipment.

Having a Media Asset Management (MAM) package like Etere Budget MAM will bring many advantages. For example, it allows you to search and browse content over the web and to play it out on your computer without making any request for a video tape. Another benefit of using a MAM is that you can easily add metadata about the clip recording to your workflow, and it can be as simple or complex as you want to use. However, do you want to add legacy content to your archive using the same rules?

Converting legacy content for a digital archive requires careful planning. Unlike a digital archive system that runs at computer network speeds, video transfers from tape are one to one.

Another important consideration is the time factor, including the time to
■ Prepare the video
■ View the content
■ Copy it into a digital format
■ Add metadata

From all of these time considerations, you can clearly see that a a 3000 hour library can take 4500 hours to convert. Once converted, however, the ongoing cost for maintaining the archive drops precipitously. Calculating the size and type of the archive equipment needed revolves around several key factors:

Total number of hours of video
■ Additional video added each year
■ Storage format or formats in the case of legacy content and workflow
■ Size of near-line versus off-line (if any) storage

Calculating size of the existing archive is easy if everything is in a single format, which is never the case. You may wish to select a single format going forward and convert all legacy content into this format, or you may use multiple formats to reduce the archive size. A common format currently being used is IMX50, or XDCAM HD , which occupies approx 24 GB of storage for each hour of content. This format lends itself to usage with NLEs as well as producing a good quality image. But what if most or your content is Beta SP tape? A straight MPEG2 conversion of Beta SP would only need 3.6 GB of storage for each hour of content. That’s a 3:1 difference in cost per hour of storage. With digital storage it is simple and easy to mix formats as they are stored as digital files just like a WORD document.

Next, think about your future needs. It is highly likely that you will be recording and playing Ultra High Definition programming in the near future, and that the quality of the video will continue to increase as technology moves forward. For the next 10 years it is unlikely that we will see an increase beyond current compression rates and that 100 Mb/sec and 300Mb/sec will be the standard video compression rates we will use. Fortunately, computer technology develops at a faster rate than video and the next generation storage systems will easily double in capacity every 2-3 years.

So, for example, if we have a library of 4000 hours of legacy content that consists of 3000 hours of Beta SP and 1000 hours of HD, and decide to use IMX50 and XDCAM50 601 format when converting the video, we would need an archive of:

(3000 x 24 GB) + (1000 x 24 GB) or approximately 96 TB

If we add 500 hours of content each year and we use XDCAM50 as the standard, we need 60 TB to archive the additional content over the next 5 years. Combining the old and new content requirements yields a digital video archive of approximately 160TB to keep everything near-line.

Next, we have established our format and current and future archive requirements, but is it necessary to have everything near-line? It is often possible to put considerable content on the shelf instead of keeping it near-line. Using ETERE HSM you can easy export of content on digital data tape that is still tracked even when residing on the shelf. If a request is made for a clip that is no longer near-line, a message is generated identifying the clip and associated tape cartridge via its barcode. Then it is a simple process to reintroduce the tape cartridge to your system and the archive software restores the clip.

Adding a media asset management software package such as Etere MAM can also make life easier as it allows the content to be moved off-line. The archive can now be viewed and searched using low-resolution proxies without having to access the high resolution content until it is needed.

The comparative cost of storing content in a robotic tape library is very low, much lower than your existing analog video library. Since modern digital archive systems use the same data tapes used in the computer industry they are readily available, competitively priced and store vast quantities of data. Table 1 shows the number of hours of video that can be stored on an LTO 9, Disk Archive and Etere DiskLibrary. This compaction of data results in being able to store a 3000 hour archive in XDCAM50 format on just 5 tapes in a small 2U rack mounted video archive system. It is now perfectly feasible and cost effective to keep all of the content near-line in a robotics cabinet as part of the video archive system.

Sometimes we need to access video clips rapidly as is the case in a news environment, or do we? It is usually acceptable to be able to browse and view low-resolution content instantly to determine what is needed for that next project. This reduces time to access and move content over a network and reduces bandwidth requirements dramatically. Using low-resolution proxies is even more important if the content library is being shared with several geographically disperse sister stations. Keeping frequently accessed content on hard disk allow us to optimize the workflow. Putting all of the content on hard disk results in a visit from the chief accountant.

The best solution for most archive systems combines hard disk and automated tape with the ability to seamlessly move content between the two mediums as dictated by workflow and cost efficiencies. The hard disk storage allows instant access for low-resolution proxies, short length clips and those clips that have recently been part of your workflow. The tape storage is for the higher resolution video, longer program content, and any content that is not in frequent or recent use. If you can wait three minutes for a clip to be restored from a tape then you can save a lot of money when purchasing your system by reducing the amount of disk storage needed.

Sizing the archive system disk and tape storage elements centers around your workflow. If you have hours of content that requires frequent or rapid access then more hard disk storage is necessary. If most of your content can be accessed within minutes using a larger near-line robotic tape library is much more cost effective than buying lots of disk.

We have established how to size the archive, but we haven’t established the usage model for it. Analyzing your workflow is critical to selecting your digital video archive; get it wrong and you will be suffering for the next ten years; get it right and life will become much easier. This is where a System Integrator or a consultant can help you. Experience has shown that most broadcast facilities don’t have a full understanding of what can be done with the storage hardware and software systems available today. A TV station will often purchase a digital archive system and use it just as they did their old analog video tape library, failing to take advantage of the system’s capabilities because they applied their old workflow.

Designing the workflow requirements should be the first step in defining a digital video archive system. Then the archive can be designed, sized and configured to work for you. Items to be considered include how you ingest content and where it comes from. Is the content processed first by NLE stations? Does it go directly to air? What are your usage rights? What content do you need to archive and what content do you need to keep readily available and for how long? How does the video move through your facility? How do you want to track the content and who needs access to it?

Let’s take a local television station, for example. Much of their standard programming is downloaded from satellite but the weather and sports programs are all produced locally. They also do some special programming that they want to retain. They are still using Beta SP cameras for their news crews, and have two NLE stations that are primarily used for preparing the news. They have automation software to manage the station and use two parallel video playout servers. They have approximately 3000 hours of mixed format content in their video library.

Currently, they load the camera tapes into a deck connected to the Etere ETX and transfer raw footage into the server and convert it to mpeg format. Then they transfer it from the ETX to the NLE for editing. The finished content is then moved back to the ETX manually so that it can be given an ID and be managed by the automation software, which schedules it for playout. A copy of the finished content is made on a video tape and stored in the video library together with the original raw content video tape from the camera. A listing of the content on the video tapes in the library is entered into an Excel spreadsheet. Once the content has been played out the clip is manually deleted from the ETX.

How can this workflow be changed to improve operations when adding the digital video archive system?

First, the video from the cameras can be routed directly to the NLE stations, easing the burden on the control room staff to keep the ETX free to play out scheduled programming. A simple encoding station can be added to do this that can do double duty. It can convert the Beta SP format into an MPEG format and generate a low-resolution proxy for the MAM software. At this point you can add some basic metadata to the low res images of the raw footage so that you can search on it later, even to the point of splitting the low res proxies into multiple sub clips with metadata attached to each clip. A small amount of work up front will pay dividends later when searching for content.

The edited footage is then sent to the ETX for playout, or picked up by the station automation software and placed into the digital archive until needed. It is important that finished content is moved by the automation software, otherwise there is no record of its location. The raw content can be placed directly into the digital video archive as it can now be searched and retrieved via the MAM software without passing through either the ETX or the station automation software because it will be restored to the NLE before it is re-purposed.

Planning Stage
When planning your video archive, think about the people or teams that needs or wants the content. When you transition to a digital archive access to the video library can be drastically changed for the better. However, now the responsibility for obtaining the correct clip for tomorrow’s news shifts to others in the operation. Typically you will want to limit access to the high-resolution format to those who directly use the high-res content, and expand access for others to the low resolution proxy content.

With Etere's seamless integration with Active Directory breaks down complex user authentication structures and allow you to manage user access easily.

When planning your video archive it is important to determine who needs access to the content versus who wants access. Existing analog archives typically have a librarian who is in effect the gatekeeper. Preventing video tapes from leaving the library to sit on someone’s desk, making copies for editors and producers and issuing tapes for upcoming broadcast programs are some of the tasks that they must manage. When you transition to a digital archive access to the video library can be drastically changed for the better. However, now the responsibility for obtaining the correct clip for tomorrow’s news shifts to others in the operation. Typically you will want to limit access to the high-resolution format to those who directly use the high-res content, and expand access for others to the low resolution proxy content. Most MAM software packages allow controlled access to specific video content and general public content. None of this activity compromises the station automation software accessing the archive. In fact a digital archive system will, when properly designed allow parallel access to video stored in the archive and minimize the amount of video being transferred over your network. This is a great advance over a standard analog tape based video archive.

Future Plans
Think about what are your plans for the next 5-10 years. For example,
■ Are you planning to do webcasting?
■ Are you planning to switch to 4K broadcasting?
■ Is your company likely to be incorporated into a group organization?
■ What are the long-term plans for the organization you belong to?
■ How many stations is your video archive shared amongst?
These are all questions that can affect what you design and install for your video archive system.

Furthermore, adding a MAM simplifies webcasting and allows other television stations to access and browse your video archive safely and securely. Sizing the archive for a switch to HD and UHD is also easier to do up front, without significant additional costs if planned correctly.

Getting Started
■ Get an expert to visit the facility
■ Talk to your staff and gather feedback
■ Develop you workflow and archive needs
■ Plan on using some of you budget to do a study and design before committing to any equipment or software