Broadcasters Eye Cloud As Disaster-Recovery Solution

As broadcasters gradually move their workflows to public cloud technology, day-to-day news production remains challenging because of latency issues. But the cloud is already being used by some large groups as a “business continuity,” or disaster recovery (DR), tool for news operations in case of a failure in their legacy on-premise hardware. And that use case is likely to become more widespread, said a panel of top technology executives at TVNewsCheck’s NewsTECHForum last week, despite concerns over several recent major outages in the Amazon Web Services cloud.

Sinclair Broadcast Group turned to the public cloud for help when it was hit by a ransomware attack in October that encrypted servers and workstations at its stations and rendered them unusable, including key news production functions like Avid editing systems. With projected downtimes ranging from weeks to months, Sinclair needed a quick solution, so the group used Sony Ci, a file-sharing system that runs on the AWS cloud and is popular with movie studios, to get back up and running, said Mike Palmer, senior director, Media Management for Sinclair, during the panel The Cloud & the Future of News Production.

“It creates thumbnails and proxies and gives you some nice tools for downloading,” Palmer said. “It’s generally intended to support about 30 users at most.”

Sinclair had been using Sony Ci in its regional sports networks (RSNs) to trade content and already had an account. He contacted Sony to ask them if they had any concerns about using it enterprise-wide at Sinclair, and they said they didn’t.

Palmer set it up so anyone invited to use Sinclair’s Ci account could invite anyone else, provided they had a Sinclair email address. He started with five email addresses “and threw them into the wind.” Within a few hours, there were 3,000 Ci users across Sinclair.

The Ci system lets anyone in the shared workspace do anything with any of the content, which Palmer called “a complete nightmare” for media managers. And Sinclair did have a major scare when one staffer inadvertently sent 15,000 files to the trash. But Palmer was able to recover them, and the group is still using Ci today as it gradually restores its normal production systems.

While Palmer doesn’t view Ci as a long-term replacement for those systems, the cloud-based solution has probably earned a permanent place in the group’s toolbox.

“It’s a simple tool,” Palmer said. “And I think the lesson for us is that users are looking for very simple tools, and many of the tools that we have right now may have more functionality than we really need that we’re paying for. So, one of the lessons that we’re taking out of this is, let’s look for simple tools that provide people the functionality that they need.”

Broadcasters have traditionally done an impressive job in hardening their systems against natural disasters like tropical storms, said Paul Capizzi, CIO and SVP technology, Fox Corp. But ransomware has changed the way they need to think, said Capizzi, who views the cloud as a “perfect opportunity” to provide backup.

“If you think about ransomware, you have to build your environment from scratch,” Capizzi said. “First, you have to make sure your environment is safe to do so.”

The Fox Owned Stations have already been virtualizing as many of their news production systems as possible to run on on-premise COTS hardware. Fox still wants to run that technology locally, Capizzi said, but then use the cloud as a disaster recovery system to store backups of those systems.

“When we say backups I’m not just talking about content, I’m talking about backing up our virtualized sessions,” Capizzi said. “That includes the operating system, configuration of the operating systems, the application and all of the many components that make up a configuration of an application. So, if I were to backup that VM session to AWS, now in the event of a situation such a ransomware attack, we could pull that content, pull those sessions down as backups and restore to our current state.

“The beauty of that type of strategy is that as your team is constantly making changes to the platform, you’re constantly synching up those config changes to the cloud,” he said, “so you have an opportunity to restore the systems in a quicker manner.”

Of course, the public cloud is not immune to its own business continuity problems, which have tended to be ones of availability instead of security. On Dec. 7, AWS had a major outage in its East Coast region that affected several media customers including Amazon’s own Prime Video service and Disney+, as well as Sinclair. And AWS’s West region had a much smaller outage last Wednesday.

Bitcentral COO Sam Peterson noted that many customers have assumed that if a system is running in public cloud, it will be “highly available,” but the recent outages prove otherwise. But he said there are steps broadcasters can take to protect themselves.

“It’s really how we architect it that makes it secure,” Peterson said. “And certainly, from an application perspective, database perspective and then content, there are multiple different strategies.”

For one thing, broadcasters should make sure they’re not tied to just one zone or region within a public cloud platform, Peterson said. And they should also consider the pros and cons of running different workflows across different cloud platforms, a point echoed by Palmer.

Tegna has already moved much of its linear and OTT playout to the cloud, using both public and private platforms, and the group has a long-term goal of moving news production there as well. But first, Tegna CTO Kurt Rao would like to see broadcast vendors improve the security on their existing products, including patching up longtime vulnerabilities that could make them a target.

“There are some that you can’t patch,” Rao said. “Those are vulnerabilities, where we use the expression, ‘Every day is a gift from God that you don’t get hacked.’ ”

To make the cloud more reliable, Rao would also like see broadcast vendors develop their software to easily run across different platforms.

“When I say multi-cloud, I’m going to make a distinction,” Rao said. “It’s not that I can run my environment on AWS or on [Microsoft] Azure, or on whatever. We’d look for a service that runs across all of those, that’s seamless to us as a customer, and we don’t care where it runs. But if one goes down for operational reasons, it automatically switches over and it’s seamless to us.

“That may be the new reality going forward — that we’re going to ask our vendors to not only be secure, but to really go down this path of providing redundancy to their infrastructure,” Rao said.

For more stories on NewsTECHForum 2021, click here.