Talking TV: AP Resets The Game On AI-Based Archive Search

What do broadcasters do when they know what they’re looking for in the archives but may not have the right metatags at hand to find it?

The Associated Press has just released a new AI-powered search tool for its AP newsroom platform that may mitigate the problem. This tool uses descriptive language to connect searchers with their targets, bypassing metatags altogether. It may end up being a game-changer in the fast-developing world of making media asset management systems and archives more searchable in the process.

In this week’s Talking TV conversation, Paul Caluori, AP’s VP of global products, and Derl McCrudden, AP’s VP and head of global news production, share what’s underpinning the new AI tool and its wider implications for the industry. They also look at developments in generative AI applications like ChatGPT and how they may problematize content authenticity on the one hand, but could mature into a valuable tool on the other.

Episode transcript below, edited for clarity.

Michael Depp: The Associated Press recently announced the launch of an AI-powered search tool on its AP newsroom platform for multimedia content. Rather than just using a conventional metadata search, the new tool understands descriptive language and offers up search results based on the description a user provides. Just think of the implications for all those broadcasters who spent untold hours with knotted brows searching through inadequately tagged archives.

I’m Michael Depp, editor of TVNewsCheck, and this is Talking TV. Today, a conversation with Paul Caluori, AP’s VP of global products, and Derl McCrudden, AP’s VP and head of global news production. We’ll talk about how AI is enabling a more accommodating kind of search ability across a massive archive and the wider implications for AI’s usage in broadcast media asset management systems. We’ll also talk about the widening use of AI at the AP, which was a vanguard adopter of the technology, and the ethical and operative guidelines it’s adopting around AI’s usage. We’ll be right back with that conversation.

Welcome Paul Caluori and Derl McCrudden, to Talking TV.

Paul, this new tool allows users to search through apps, vast photo and video library without needing very specific meta tags to do it. How were you able to build that?

Paul Caluori: We actually are working with another company called Merlin One, which is based in Boston, and they specialize in AI applications for visual assets. So, what we’ve done is we’ve adopted their engine after working with them over the last year to prepare for this.

And what it does is look at the description that a user puts in and is able to sort of understand concepts in a way that keywords and regular tags, you know, are just sort of very, very blunt. And it is also able to translate those into elements within a visual context so it can look for a moment within a video or it can look for a component within a photo, and it makes it a much more specific kind of search than we typically get.

What’s really great about it is that it can find things that we don’t have tags for. And if you think about our whole archives or, you know, our photos, go back to the 1840s. The AP was founded in 1846. We have photos that go all the way back to the beginnings of photography. Nobody was thinking about metadata back then. And so, a lot of these things are not very well tagged and are difficult to find. This is a way to sort of unlock all of that, and we’re very excited about it.

Well, this is fascinating how this searchability works. So, how accommodating can it be exactly for the user who has only a kind of very abstract or imprecise idea of what exactly they’re looking for?

Paul Caluori: That’s such a great question because it gets right to the heart of one of the things that this is going to change, which is the way people approach search. Right, so typically people approach search with that sort of broad scope of what they’re looking for, because that’s how search works. I’m going to put in one or two keywords, then I’m going to start sifting through a bunch of results to see if anything sort of grabs my fancy.

If you want to search that way with this, you can say, I’m looking for, you know, soccer games with a blue sky or I’m looking for soccer games with… or excuse me, Derl might call it a football game, so, no, I’m looking for a soccer game with people who are wearing yellow uniforms, Right? Because I’m an art director and I’m looking for a particular look. So, you can look for abstract ideas like that. If you were looking for something, you weren’t quite sure what you wanted, it will return a whole lot of results. And then you start sifting through them the same way.

And that seems to be the problem. I suppose the danger here is that the user is then flooded with search results. So, how can you avoid that or winnow that down in a more user-friendly way?

Paul Caluori: I think that while we’re always flooded with search results, anything you search for, you get millions and millions of results, and you get past the first 10 and it gets pretty far away from what you were looking for. So, I think the way around it is for users to start thinking a little bit more specifically about what they want.

You know, it’s hard to find something if you don’t quite know what you’re looking for. So, you know, if you do want something very specific, you can enter that into this search engine, and it gives rather strikingly precise results. I spent some time playing around with it, so I started looking around, show me pictures of Winston Churchill or video of Winston Churchill in a garden. And then somebody said, Well, let’s see if we can find him feeding birds. So, all right. I literally typed in Winston Churchill in a garden feeding birds. That’s pretty specific. And it came up with like five videos, and it went right to the moment where that exact thing was happening. I didn’t know that that was part of our archive. And I guarantee you we don’t have anything tagged like that. But I mean, knowing exactly what you’re looking for, you can find out pretty quickly whether we’ve got it.

So, the more language you throw into this search field, the more it is going to winnow it down.

Paul Caluori: That’s right. That’s right. So, instead of doing it after your search, you do it ahead of your search and find the things you’re looking for. We have research teams here both in the U.K. and in the U.S. working with our different markets. And, you know, our great hope is that this makes them more able to serve their customers effectively.

Broadcasters are going to be quite interested in the underlying technology here because so many of them are wrestling with the archives that are woefully undertagged, as you know. For those who started to try to impose some order on all of that chaos, they’ve been going about it by having AI wending through and adding tags. And it seems like this isn’t the same model now with this new tool, which would just appear to kind of circumvent the tagging process altogether. Do I have that right?

Paul Caluori: I think that’s right, yeah. In fact, I was just talking with one of the people who worked on the engine before this conversation, and I asked that question whether, you know, do you anticipate that your service would add in tags and said, no, actually, this is you know, that I think I really think that the AP would be better served by using our own tags, along with a third-party search engine or any sort of AI engine to find content because we have a specific set of tags that we use to identify things.

And, you know, my colleagues who work with Derl in our news department have specific ways of tagging things. And I’ll bet you that most organizations have their own ways of tagging things. And to the extent that any one organization can be consistent about how they tag things, that’s miraculous in and of itself.

If different organizations can be consistent in the way they tag across multiple organizations, I don’t think that’s a much higher peak to climb. I think philosophically it makes sense not to try and impose that. That said, I think that some level of metadata tagging, and AI search combined will give us the best results.

Because for breaking news, for example, an AI agent that knows how to find particular things within, you know, concepts within a visual is very good, but it won’t know that at 3:02 p.m. yesterday a dam broke and that these and it was in this particular product. You know, it’s not going to know that kind of specific information and that’s where metadata is really, really valuable.

Yeah, I’m just saying about the idea that every company might tag in exactly the same way, if wishes were horses, then beggars would ride. Is this tool public-facing or is it only available to AP members?

Paul Caluori: It’s public facing. It’s both. We have an e-commerce function at AP newsroom that walk-up users can use to find visuals. It’s not our entire archive that’s available through that. The experience that our subscribers have will yield a look at a much larger content site. And also, you know, our subscribers have the benefit of working with our staff — those researchers I talked about and other experts who worked on this. So, it’s there for the walk up public. It’s better for our subscribers.

Got it. Derl, let me bring you into this conversation. As I mentioned at the top, AP has been at the vanguard of AI usage, and I’m thinking about years back when it started using AI to generate earnings reports for many smaller publicly traded companies and minor league sports baseball scores. What’s been the widening use of AI at AP since then?

Derl McCrudden: So fundamentally, the use cases we’re looking at are really about the same thing. It’s about making more impactful journalism. So, the earnings reports that you’re talking about dating back to 2014 was about instead of taking a small group of financial journalists and getting them to do as many earnings reports that they can, which had a finite number, we were able to automate a lot of that using and templated it, and then we could make a lot more of those reports and free up the time of those journalists to do higher volume work.

That’s the common thread of everything that we’re looking at. So, the examples I’d give are a few years ago we started working with a transcription company that takes audio, usually on video strips. It often converts it to text. And for any news producer who has ever had to spend, you know, time looking at a 30-minute interview and rushes from three camera outputs and trying to find the sound bite that they know they had in the moment of the interview. But their transcription themselves and their notebook is not as good as you know, as it should be.

This has been a lifesaver for us. And so, the company we use happens to be Trint. There are others out there, but we create about 27,000 hours of live video — we’re the biggest wholesaler of live video in the world, and we default to transcribing all of that content. And what that’s doing is instead of putting the pressure on a producer or an editor, it’s allowing to focus on the journalism and then let the tool do the heavy lifting so that we can find the business we need or discover the bits we need and to do it at speed.

I’ll just add one more thing. It also allows us to do something slightly different, which is to work in a different way, and that involves a mind shift. So, for instance, in our media asset management system, we’re now in a system where the cloud integrates with that. And instead of having to shuttle up and down or to down the timeline, we’re able to go to print transcription, but we want to highlight it and then it drops down into the edit. And it’s a different routine, if you like, from what we we’d gotten used to.

Beyond the sports and the earnings reports I mentioned before, have you found a wider application of sort of templatizable stories using AI to generate those?

Derl McCrudden: Well, we’re not doing that. But anything that revolves around a verifiable data set is fair game. The one application we are looking at is around localizing our content. So, for instance, when we do stories that are about, I don’t know, the price of gas in different states that we can then localize, we can then give our customers and our members the tools to localize that content so they can drill down into specific datasets around a bigger story.

And that allows a degree of reformatting content and reformatting stories and applying them in a different way than would have been possible otherwise.

AP is a very venerable and storied news organization, going back, as you said earlier, to the 1840s. And so, I’m sure that its forays into AI usage are coupled with some pretty serious ethical and operative principles. Can you describe how that process is playing out at AP, Derl?

Derl McCrudden: That is a real-time conversation. I suspect every newsroom in the world is having this right now. Our standards and principles are what we live by, how we operate. And, you know, not just our journalists, but all of our staff members subscribe to those principles.

But they have to not be set in concrete or in stone. They have to be relevant to the environment in which we operate. And what generative A.I. has created is an ever-changing landscape which for some people has come out of nowhere. But for others, as you know, we’ve seen it developing over a long period of time.

And so, where we’re at now is going back to basics. We expect our journalists not to use those nascent, generative AI tools to create journalism unless it’s something that is like a device within a story — like “we asked ChatGPT to give us a comment on this thing about generative AI.” And, you know, we’ve done stories around that kind of device, but otherwise we’ve actually sent an all-staff memo not that long ago saying, just a reminder, if you’re unclear that, you know, we’re not using this kind of technology today.

What it really leads us to is thinking very carefully about how we do take these tools and how they apply to us. We’ve all seen examples of tools out there that will create scenes based on an archive or a database that it draws on. We’re about eyewitness journalism and about putting journalists into the heart of a story and faithfully telling that story. So, the tools will help us in that work, but not create something out of nothing.

For both of you, where are your own key areas of concern around AI’s usage in news right now?

Derl McCrudden: For me, I would say it’s about understanding how we can spot things that are not real or what they’re not purporting to be. And that really means getting under the hood of generative tools to understand them, working with some of the big players in the market in order to understand how they are doing, tagging metadata, how what’s created out of a camera or a microphone is [real], then how we use that and make that available within the newsroom. That’s the direction in which it’s headed.

Watermarking content when it airs?

Derl McCrudden: Watermarking it and not just adding … I’m a journalist not a not a technologist, so I’ll get out of my lane pretty quickly if I go into detail. We need journalists to do journalism. And that means using a gut check. Is something too good to be true? It probably is. You know, if it seems that way, it probably is.

So, we still do journalism one on one. For us, it’s about trying to spot the fake. So, to answer your question about what keeps me awake at night, it’s the fakes that the generative AI tools can lead to, although it has a lot more positive uses as well.

Paul, what about you?

Paul Caluori: I’m right there with the question of whether this is real or not. So, you know, I’m responsible for products and our customers were already asking us, how are you going to guarantee that what you’re sending to us is authentic?

And that’s that is just central to the way our relationship with AI has to play out. We have to be able to be authentic all the way through, particularly when we are looking at UGC. You know, it’s one thing to work within our own journalism and our own people. It’s another to identify other sources.

And we need to do that. We need to do that on a regular basis, and we need to be able to stand behind it so that our subscribers can feel confident. I mean, that’s always been our goal, what we strive for is for our subscribers to feel confident. This is from the AP. This this is something I can stand behind, right?

So being able to sort that out, and as Derl points out, there may be some great ways that we can deploy tools, and that’s an ongoing conversation. But the thing that I’m most concerned about is authenticity. You know, I just I can imagine multiple scenarios, whether it’s visuals or data or, you know, fill in the blank. We need to be certain that something hasn’t been created by an untrustworthy source.

Right. Well, I mean, this technology seems to be exponentially or almost exponentially more sophisticated and potentially also encroaching more and more on the journalist’s role. Do either of you have a concern about that? And does the whole industry need to come together around this issue to come up with some broader guidelines and principles by which everybody should be operating now?

Derl McCrudden: I think in an ideal world, yes. But you alluded before to metadata and tagging, because if there is one system, wouldn’t that be great? I think getting an industry-wide consensus is difficult when something is developing at pace and so is it going to wreak so much change ahead of it. So, I think, yes, there needs to be a wide-scale industry discussion about this, and I think that’ll be ongoing.

There’s a lot of shortcutting there you that potentially some less ethically oriented organizations might take advantage of in AI, it seems.

Derl McCrudden: Yeah, but could I just add one thing? I do want to just make clear, we are not looking at this as a way of cost cutting. We’re looking at this technology as a way of supercharging our journalism and putting our journalists at the heart of stories, whether they are desk editors, editing text copy or photo editors editing photographs, or people in the field creating amazing video and the amazing storytelling we do every day. And we don’t see this technology replacing it.

We see this technology is taking the heavy lift out of mundane tasks to do that higher value work I talked about before.

Paul, you want the last word on this?

Paul Caluori: There are things that are really exciting about this in the future. I can imagine to the point where I was just making that having a large language model that’s been trained on the right things could be a fantastic resource for a journalist to ask questions. It could be like the sort of colleague who knows everything. If you’re confident in what the thing’s been trained on and it becomes a resource for you, it’s valuable.

That’s not the sort of thing that is, you know, to Derl’s point, that’s aiming to just undercut jobs. It’s aiming to make it easier for journalists to do better work. So, I think there’s a lot of discussion about how these tools can harm journalism. I think it’s helpful for us to think about ways in which we can help as well.

That said, to our earlier point, we need to keep our guard up. Authenticity is the name of the game and being trustworthy is what we’re all about. So, it’s a balance, particularly with generative AI. As far as the other types of AI, like the search and recognition tool that we’ve just launched, I’m just nothing but excited about our ability to find things that we couldn’t find before. It’s important to remember that AI is not simply a matter of generative things that we have to scratch our heads over.

Sure. Well, it’s fascinating tool that you’ve built there, and I’m sure broadcasters are going to look at it with great interest. Paul Caluori and Derl McCrudden, it’s undoubtedly a Pandora’s box that we’ve opened here. Thank you for sharing your thoughts about it.

You can watch and listen to past episodes of Talking TV on TVNewsCheck.com and on our YouTube channel, as well as all the major platforms on which you get your podcasts. We’re back most Fridays with a new episode. Thanks very much for tuning in to this one and see you next time.