Riding The Cloud At KTDO

NBCUniversal Local continues to transform broadcast news with a second Telemundo station, KTDO El Paso, Texas, now using the public cloud to produce its daily live newscasts.

The station group, which comprises 31 Telemundo stations, 12 NBC stations, five NBC Regional Sports Networks and two multicast networks (Cozi TV and TeleXitos), has been working to overhaul its news workflows for several years. It first experimented with a private cloud model for news production back in 2019 at Telemundo station KBLR Las Vegas by using a high-speed network to remotely control ST 2110 hardware located in a Dallas data center.

While that effort was successful, the need for more remote workflows due to the COVID-19 pandemic led NBCUniversal Local to pivot away from the private cloud and accelerate its evaluation of public cloud technology.

Its first big move was at KASA Albuquerque, N.M., a newly acquired Telemundo station that had to quickly start up local news production and did it by launching an end-to-end cloud workflow last year. Then in June, KTDO began producing live news in the cloud from a brand-new 12,000-square foot facility in El Paso.

Like KASA, KTDO is operating a production control room (PCR) in the cloud based on Vizrt’s Viz Vectar live production switcher, which it uses to switch both studio sources and live feeds from the field. HD-SDI studio sources are converted to SRT and sent to the cloud via Cisco IP switches, while LiveU and Dejero bonded feeds are received by a software “appliance” in the cloud using SRT (Secure Reliable Transport) or RTMP (Real-Time Messaging Protocol). However, a few latency-sensitive applications that are tightly tied to the studio, such as the weather presentation, still use on-premise hardware.

A Fresh Start

Jaime Martinez

Moving to a new building was the impetus for KTDO to fully adopt IP and cloud-based technologies, says NBCUniversal Local’s Jaime Martinez, who is director of technology for both KTDO and KASA . Planning for the El Paso project began back in September 2021, which is when Martinez joined NBCUniversal Local from Sinclair, where he had worked for Tennis Channel as well as several stations. While Martinez had experience with REMI workflows at Tennis Channel, moving into cloud-based production was a new venture for him as well as KTDO.

“Everything just fell at the right time, and for the right reasons,” says Martinez.

After a year of planning, integration of the new building started in January 2023. Martinez says it was “a soft integration” due to equipment backorders resulting from the pandemic as well as some construction delays, and KTDO extended the lease in its old building several times. The station operated across both facilities through May before switching over fully in June, with its 4 p.m. newscast on June 21 being the first live newscast from the cloud.

KTDO’s new facility is spacious compared to its old home, with an open-concept newsroom and a large control room that looks traditional despite being used mainly to drive software “instances” in the cloud. There is an immersive 360-degree news set which includes the anchor desk, a weather center surrounded by large LED monitors and a dedicated sports desk area that includes more LED monitors.

“It’s night and day,” Martinez says. “You’re talking about something that was legacy equipment, and a very small studio — that has definitely expanded. Now the talent can walk in front of full screen monitors, do the weather presentation, but also walk over to the keywall.”

While the set is bigger, on-premise hardware is minimal, particularly since KTDO’s playout runs out of “Dry Creek,” NBCU’s master control hub at Comcast Media Center in Colorado (the Harmonic VOS360 system is used to encode its transport streams). There are eight racks in the station’s equipment room, but the hardware fits into only four fully populated racks. That includes Cisco IP switches, a WSI weather system and a partial on-premise HD-SDI router. That router supports the weather system and also populates the monitors in the studio.

“So we have that flexibility,” Martinez says. “We want to make sure we have a safety net, because this was one of the first fully-cloud systems. The next [NBCU Local] build, I’m sure it’s going to be with no router.”

APC Integration

KTDO’s cloud production system supports full MOS integration, including the Ross OverDrive production automation system running with Furio camera robotics and the SmartShell control system.

Other key technology includes Sony box-type studio cameras, Cineo lighting, Ross Video XPression graphics, Harrison’s Mixbus VBM cloud-based audio mixer, Dalet’s Galaxy newsroom system and Adobe Premiere editing for high-end packages.

The KTDO weather desk

The station also benefits from an IPTV system that interconnects all of the NBC and Telemundo stations, allowing them to see each other’s feeds as well as the live camera at 30 Rock in New York. Martinez says that system helps foster the sharing of content across the group.

“We share content regionally and also nationally, depending on the story,” he says. “Especially since we are a border town. We have regional news in the morning that comes out of Dallas, and they do pick up a lot of our stories from here.”

KTDO is a lean operation, with 50 total employees including sales, production and news personnel. Like many other NBCU stations, it is a regular OverDrive user. To run a newscast, the station relies on a producer and two APC operators, the “pilot” and “copilot,” who sit in the control room.

The cloud integration with Ross OverDrive was relatively straightforward, Martinez says, as an automated production control (APC) operator can preset the camera shots in the system. KTDO has successfully tested running a newscast with the APC operators in different locations, with the pilot working from his home and the co-pilot at the station in case of a problem.

“We have done some testing where we run it from the APC’s house, and they are able to do everything there that they’re able to do here onsite,” Martinez says.

Overcoming Latency Challenges

Moving to cloud-based production definitely requires a change in mindset, as operators need to become comfortable dealing with virtual panels instead of physical controls.

“You’re shifting the mentality from manual to in the cloud,” says Martinez. “So, for example, [before] you have your audio board right in front of you where you can make the changes physically. In this case, you’ve got to log onto the cloud and make the changes virtually. The same thing with any kind of troubleshooting issues. Right now, you go back and you reset a cloud-up or a cloud-down instance. Where before you would have to go patch around it, or go trace a cable, or you go look at the drawings, do a route.”

Not surprisingly, dealing with latency was the biggest challenge for KTDO in moving from on-premise switching of its newscast to switching in the cloud. The major obstacle is for the talent in viewing their return monitors.

“That two-second delay in TV is a long time,” Martinez says. “So they’ve got to take on the cue, instead of waiting to see themselves on the preview monitor. That took some training, but we’re getting pretty good at it now. So that was one obstacle that we’ve overcome.”

Dealing with live feeds delivered through the cloud by LiveU or Dejero bonded systems via RTMP or SRT signals adds an additional delay, so the station cues talent five seconds in advance before they hit live.

One place where the latency of the cloud is still problematic is in weather presentation, specifically in providing a return feed for meteorologists doing weather segments in front of a green screen. When chroma keying for a meteorologist, the round trip of processing it in the cloud and then bringing a return feed back for the talent is still too long today. That’s why KTDO still handles that function on-premise.

Given the high-resolution graphics involved, being able to support weather presentation fully from the cloud is “still a ways away,” Martinez says. But the technology is improving quickly enough that he thinks in another year it might go into the testing phase.

Overall, Martinez thinks the biggest benefit of cloud production is in its flexibility, particularly in a difficult labor market where TV stations are having a hard time hiring personnel like APC operators.

“Anybody that has connectivity can run the system,” he says. “So the flexibility is definitely the powerful piece of it. And I can see it being a great benefit for sports, because you get your best technical director on the seat in one location, [who] can be jumping around to different sports at different venues.”