« TiVo auto-delete flag seen in the wilds again | Main | HD TiVo coming (back) to DirecTV »

The Daily Show and their TiVos

How does The Daily Show compile all their amazing clips of politicians saying one thing one day, and the opposite thing several months before? From a long New York Times profile of Jon Stewart and his operation of the show:

The day begins with a morning meeting where material harvested from 15 TiVos and even more newspapers, magazines and Web sites is reviewed.

Crazy, huh? I suppose that's what it takes to record every 24hr news network around the clock, every day of the year. I wonder if they could reduce some of those by going to dual tuner TiVoHD models and dropping terabyte drives into them? I'm also curious if they store archival video footage onto file servers, or if they're only tracking the past 24hrs of the news cycle each day.

by Matt Haughey August 16, 2008

Comments

I think they must save all of it. There have been some amazing montages of, for example, Dick Cheney contradicting himself a dozen times over the past 8 - 12 years.

Posted by: Beth at Aug 16, 2008 3:09:34 PM

Or they could just get one of these servers: http://www.snapstream.com/enterprise/

Since this server indexes the videos they can find what they need alot quicker than a Tivo.

Posted by: Percy Lee Bell Jr. at Aug 16, 2008 5:29:56 PM

While the Tivo is a fine product, and MythTV is not for everybody, I gotta say I'm surprised that for the purposes of the Daily Show they would not have been very attracted to it.

With MythTV you can have an arbitrary number of tuners in one machine (either using 2-tuner cards or USB encoder boxes) as well as an arbitrary number of digital TV tuners (using the HDHomeRun or USB tuners or cards) so a single machine could easily record all the news shows for you.

And then serve them all up to any computer in the office, with a vastly greater array of tools to navigate quickly through programs to find stuff. Should even be able to log the close captions to search them.

And then, when you find something, the shows are just sitting on the disks as mpegs, ready to import into your own video editing suite. No fuss no muss. And cheap enough to just have more than one box so everything is recorded twice for redundancy.

Surprised they didn't go that way.

Posted by: Brad Templeton at Aug 16, 2008 6:26:15 PM

Well, to be honest, we're talking about Jon Stewart saying "we have 15 TiVos" and it's not like it's their real IT staff saying it. Stewart is very likely the type of guy that equates any type of DVR hardware/software with being "a tivo".

I would think if they are archiving all their footage, they're most certainly using some sort of software PVR hooked up to some kind of massive storage for safekeeping.

Posted by: Matt Haughey at Aug 16, 2008 7:21:36 PM

Nope, it's literally 15 rack-mounted TiVos of various models, many from the pre-Series 2 era. Some Philips boxes, some Sonys. And because there's a limited number of remote codes, when a staffer operates one, he has to hold the remote directly against that box's IR receiver so that the beam doesn't hit any of the other boxes (i.e., so he's not inadvertently controlling multiple boxes at once). No joke! It's pretty primitive.

Re: "I suppose that's what it takes to record every 24hr news network around the clock, every day of the year." I realize you're exaggerating, but that's not exactly what they do, of course. Many reliable shows are recorded every day. For instance, as you'd imagine, "Hannity & Colmes" has a season pass, as do a number of other high-profile or "controversy"-happy shows. If the staff notes an interesting trend like "'CNN American Morning' plays incongruous '80s rock when they come back from commercial" or a special event like "Obama's going to be on 'The View,'" that program will be added to the rotation for a few days.

When TiVo footage is needed for TDS that day (i.e., every day), the clips are dubbed off to Beta tape and brought to an editing bay. Yup, sneakernet. Sounds like a lot of work, right? It is. I wouldn't be surprised if the show upgrades to a networked PVR system -- especially with an imminent move to HD -- but I don't know what their plans are.

Re: "I would think if they are archiving all their footage" -- They're not. There are already services that do this for them. The show would rather pay for those services than pay for the equipment + staff necessary to reinvent the wheel. The show does have a vast tape library, much of it stock footage provided by the AP et al. -- all the stock footage tapes get saved and logged in a database available to everyone in the office.

But since the only way to save things under the current setup is to dub it off to Beta in real time, there is no way to archive all the footage. A lot of stuff does get saved to Beta, particularly major events that are likely to remain relevant. Yet it's not the News Clip Library of Alexandria that people might think.

Here's are some guidelines to the clips you see on the show and where they probably came from (excluding YouTube clips and the like, whose source is obvious). They're far from absolute rules, but they give you a general idea:

- Clip of TV from the past couple days: Recorded from TiVo and dubbed to Beta.

- Clip of a less recent major news event (e.g., presidential debates): Taken from tape library. If there are no chyrons on the clip, it's likely a tape that we originally got from a stock footage service.

- Clip of Neal Cavuto saying something stupid in 2004 that leaves you wondering oh my God how did they dig that up, that is so obscure and hilarious: The staff searched transcripts and then ordered the appropriate footage from an archive service. (That's not to imply it's easy; digging through transcripts is an acquired skill.)

Re: MythTV -- It's a very cool idea, but the show does not have anywhere near the IT staff that would be required to support a setup like this. They barely have any IT staff to speak of. Plus, dedicating the show's footage-gathering duties to a single machine running homebrew software is just asking for a disaster. The TiVo+Beta combination is rock-solid and extremely simple, and that's what they'll need in any replacement rig.

Just so you know I'm not pulling all this out of my ass, I was a researcher at the show for a couple years and still visit the office from time to time. It's fun to watch people speculate about the magical, super-high-tech inner workings of the show. Truth is, it's less about technology or cutting-edge production and more about the staff being really good at their jobs.

A good rule of thumb when theorizing about cable TV production is that everything is less expensive and less elaborate than you think it is.

Hope this now-overlong comment proved illuminating. Thanks for your continued work on PVRblog, Matt!

Posted by: John Teti at Aug 17, 2008 10:35:18 AM

P.S.: One point of clarification. When I say that the stock-footage tapes are "saved and logged in a database," I mean that the tape is put in the library and a synopsis of the contents is typed up by a staffer for a search database. There's no networked database of actual video content (yet).

Posted by: John Teti at Aug 17, 2008 10:39:13 AM

Holy crap, thanks for the amazing info John!

I was wondering about the storage of shows. I figured instead of having terabytes of storage laying around they would instead use a service like TVeyes.com or something to get the older stuff.

I wonder if their jobs could get any easier with an upgrade of their old TiVo setup to something more robust.

Posted by: Matt Haughey at Aug 17, 2008 10:52:25 AM

I work as an editor and replay operator on live sports broadcasts. Are standard DVR is a machine called an EVS-XT, it is essentially a high end TIVO with often over 30 hours of HD recording and playback. The machines are networkable and a companion device, the X-file, allows you to back up footage to an external hard drive that is searchable by many different tags and variables. It's hard to believe that popular nationally televised show is relying on consumer products and Beta stock.

Posted by: S. Nason at Aug 17, 2008 12:55:35 PM

Re: Matt -- You're quite welcome! Happy to share. As far as whether their jobs would get easier with an upgrade of the setup, I'd say definitely. There is a lot of tedium in the current process, as you might imagine. Plus you've got the Rube Goldberg aspect -- the remote trick I mentioned above, and the "footage tracking system" for the TiVos, which is a notepad in front of the rack.

I suppose the system remains in place for now because it's cheap, it works, and everyone knows how to use it. The grind of putting out a quality show every day tends to push thoughts of sweeping equipment upgrades to the back-burner.

Re: S. Nason -- Interesting. Without knowing much of anything about live sports production, I'd speculate that the need for storage and quick retrieval of footage is even more intense than that of The Daily Show.

Regarding it being hard to believe, yeah, I guess it is! The staff is just used to it. I've never heard of the EVS-XT, but you're unlikely to find anything that could be described as "high-end" in the Daily Show offices (although the graphics rig is pretty nice AFAIK). While TDS looks and acts like a network show, it's still on a basic-cable budget.

As I said above, I have no idea what their equipment plans are, but my guess is that the show would upgrade to a more efficient system when the price and user accessibility are right.

Posted by: John Teti at Aug 17, 2008 2:27:12 PM

If I were a technical manager in an operation like this, I wouldn't use dual-tuner Tivos either. If one box goes down, you've lost two streams. It's the same reason network admins spread services across commodity-level hardware, rather than dumping everything on a few high-CPU machines. Redundancy is key.

That said, if it were my operation, I'd probably virtualize MythTV on a redundant pool of hardware and store on a large RAID 50, then deliver MPEG over the network to the editorial team. But that's just me.

Posted by: Scott at Aug 17, 2008 3:14:12 PM

Why use a Tivo instead of Myth TV? You don't need to be a sysadmin to use a Tivo. You don't need to worry about drivers, libraries, getting rooted, or having the one guy that knows how everything works leaving. Their system is old - but its based on very reliable well understood equipment. There is a lot to be said for that.

Posted by: chris at Aug 17, 2008 4:17:08 PM

You must remember, this is 50+ year-old, live TV production meeting up with yesterday-is-obsolete IT technology. While MythTV or something like it might sound like a much faster system to use, you must have a system that you know will work and be bullet-proof. I would trust a 15-year-old, well-maintained BetaCam deck far, far more than the latest, greatest, multi-CPU machine any day - and do almost every day.

You cannot rely on any of the current crop of affordable (ask anyone who works in Cable production - budgets are miniscule) means of recording SD video to a disc and then pulling it off to a digital file. MPEG won't work with most of the current crop of professional NLE apps without a time-consuming conversion/render (predictive compression schemes don't play well with needing to edit on any frame you want). Besides the time factor is huge for a daily, live show. You simply don't have enough time to wait hours for a conversion process to happen for a 3-second sound bite.

It would be great if this kind of thing could be done with super high-end tech, but either the Cable Overlords need to release some of their earnings for tech purchases or (sadly, more often the cause) the people who are hired to do the job aren't always the most skilled when it comes to the borderland world of computer tech and broadcast TV. In my job as the Director of Post-Production, I see this alomst everyday. Some get the IT stuff, but have no idea about what it takes to make good, technically correct video. Or they get braodcast video and are befuddled by computers. No Executive Producer will accept that their show won't make air because the IT-based solution failed or that the staff can't get usable video out of the system. The Tivo & BetaCam system just works and that's really what is needed to get a show on the air every night.

What they have works and will likely continue to work for some time to come (my prediction).

Posted by: Ben Howard at Aug 17, 2008 6:40:24 PM

Very well put, Ben. I wholeheartedly agree.

Posted by: John Teti at Aug 17, 2008 7:23:34 PM

S,
The EVS-XT would be a terrible idea for this application. First the XT is the most expensive disk recorder on the market (and for a good reason – it’s really good at doing what it does). Plan on paying over $100,000 for an XT2. Second, the XT only comes in 4 or 6 channel configurations. This means that it can only record 6 things at a time. So if TDS needs to record 15 things at one time they would need 3 XT systems. So that would cost over $300,000 plus the cost of the X-file to actually get the files off the unit which is not cheep either. Add to that cost the cost of yearly service contract in the tens of thousands of dollars.
Lastly, the XT to is made for recording baseband HD-SDI video which means that obviously it has no way to tune in the cable channels that they need to record and no way to schedule those recordings.
While EVS makes best in class servers for sports replay, they are not in any way appropriate for this application. The Tivo solution, while not that hi-tech, actually does what they need it to do.

-J

Posted by: J. Stevens at Aug 18, 2008 8:26:25 AM

Yes, I said that MythTV is not for everybody, but it's a bit more than "homebrew" software. The user base is very large. It is now at a quite reliable level, but I won't pretend it's as easy to use as 15 Tivos, but at the same time it seems as though its power was made for this sort of fancier application.

And yes, I wouldn't do it all on one machine, it is just interesting to think that you could do so. In reality I would probably do 3 machines, and record each show on 2 of them so there are always 2 copies for redundancy.

As for the mpegs, I'm surprised the modern suites can't import them. That is what the Tivo is recording inside, after all, and if they are dubbing from Tivo to beta, what they are doing is a poor quality analog-based transcode. The reason to go with mythtv (or snapstream or similar) would be video quality, which is indeed sometimes lacking in the Daily Show's dubs. You would be pure digital all the way. (For the NY broadcast channels, you would record them from off-the-air digital TV, in HD when available, for a truly top quality all-digital dub.)

With MythTV, you can tell it to transcode the recording immediately to some other format, including one without inter-frame compression, if such compression is such a bother. Of course that means much bigger files but faster access. The files are just sitting there on your network, you can play them, access them from any computer in the building.

Where it would get interesting is that I suspect the MythTV coder base would be thrilled to see it used for an application like this for a show many of them love. It's not out of the question to see them even code up some features the Daily Show would like, such as issuing closed-caption transcripts, or perhaps a closed-caption search where you can be watching a program, type in a word and have it seek to where they say that word etc.

Actually, that would be handy in general. I use MythTV with an infrared keyboard, though a lot of people in the Tivo-compete mindset don't like to make features that use a keyboard to not scare away the remote-control folks.

With MythTV, you would not have to go into the rooms to look at the shows. While I have a box connected to my TV, if I wish to I can look over shows from any computer in the house.

Some really useful Myth features for something like this:

a) Many speeds of readily available fast forward, from smooth 3x to 180x. You can tune what speeds you like.

b) Speed up (to 2x) without changing the pitch of the audio. Watch a 10 minute interview in 5 minutes and understand it. Great for going quickly through something.

c) Seek any number of integer minutes instantly, or tune what distances your seek back/forward buttons do.

I can't imagine watching the Olympics without this!

Posted by: Brad Templeton at Aug 18, 2008 11:28:37 AM

Roughly the same deal at VH1's Best Week Ever, with Tivo + Beta rigs spread out around the floor in different offices. Tried to upgrade to some sort of digital setup while I was there, since burning a 5 or 10 minute Beta to get a snip of TV up seemed wasteful, but the rock-solid stability and teachability of the beta/tivo setup was hard to beat. Sony no longer makes those Betas, though, so they're going to have to upgrade at some point. Neuros, anyone?

Posted by: arthur at Aug 19, 2008 6:04:11 AM

I wonder what the setup is like over at The Soup? They certainly don't record as much as TDS, but they do as much as Best Week Ever. Probably really easy for them to get the E! stuff via sneakernet.

Posted by: Gavin at Aug 19, 2008 7:40:56 PM

Gavin, The Soup uses our TV search appliance (http://www.snapstream.com/enterprise/) to do all of their recording.

We haven't published our case study on how they use it, but will be publishing it soon.

Basically, they used to have a ton of TIVOs + a massive Excel spreadsheet that captured the stuff they were supposed to be recording across all those TIVOs. The people that write for the show would watch everything and when they had something they want to get into one of their edit stations, they'd play it out from the TIVO to some broadcast tape format (Digibeta I believe?) and then they'd ingest from that tape at an Avid edit workstation.

With our TV search stuff, the recording schedule is centralized on one of our TV search appliances (one can record 10 shows at once) and their writers can watch TV shows from their desks over the LAN. So in addition to finding stuff by watching, they can also do a search to find stuff to make fun of (looking for people who talked about Tom Cruise? just search on "tom cruise". Or "michael phelps" or...). And when they find something they want to use, they clip directly from within our software, it gets flipped from the MPEG-2 recording made on SnapStream to a digital broadcast file (DV25, in their case, I think) and then they load that file straight into their Avid workstation (ie no ingestion required).

I'd say our search feature is the biggest hit with them. If they want to find every story on the latest Brittney Spears debacle, it's a simple search away. Here's a video demo of how our TV search works: http://www.snapstream.com/enterprise/demos3.asp

Posted by: Rakesh Agrawal at Aug 21, 2008 1:27:31 PM

I work for a large news-media monitoring organization. We literally do record the 24-hour news channels 24 hours a day, forever. And we archive everything. And we have been since 2006.

Something like Myth TV is a complete non-starter, for the reasons everybody else has already mentioned. It has to work all the time, and it has to work without anybody having to be especially educated. But it's also a non-starter because it relies on consumer-grade TV tuner boards, and those just aren't good enough. I'm not just referring to picture quality, although that's a factor. We evaluated a system that used those cards, and we found that the audio levels on the recordings were all over the map, boosted as much as 16 dBFS for no reason anybody, including the manufacturer, could track down.

In an environment where you have to do the same thing every day with professional consistency, those kinds of surprises just aren't okay.

Frankly, the TIVO-dubbed-to-Beta solution sounds pretty terrific to me. It checks off the boxes next to reliability and simplicity, and doesn't sacrifice either of those to check the box next to makes-the-IT-nerd-feel-good.

Posted by: Regrettably Anonymous at Aug 26, 2008 5:30:25 PM

Regrettably Anonymous: I'm curious... how do you guys do recording at your large news-media monitoring organization? Hacked ReplayTVs / TIVOs?

Posted by: Rakesh Agrawal at Aug 27, 2008 4:34:58 PM

To make something like MythTV work in this context would take a few steps I think (I am a professional editor and have setup MythTV a few times).

Basically the MPEG thing is an issue, having things on tape is just easier. This could probably be fairly easily managed with a system that played that MPEG out as baseband SDI or HD-SDI for dubbing to a tape or direct NLE ingest, with a high-end NVidia SDI card, or something from Bluefish444 or AJA. Alternatively a simple conversion process for 'selects' would be an option (FFmpeg supports DV25 and Avid's DNxHD which are good options).

The 'eggs in one basket' thing is an issue too. A handful of low cost machines serving as basic head units would be good, all recording over NFS to a centralised server.

The issue of card quality is another one - we can fix things like inconsistent levels in the edit suite (provided they're not clipped) but would rather not have to.

However MythTV's web-scheduling and network viewability are definitely benefits in a scenario like this. But as there are not likely to be any commercial integrators who could setup or support this sort of thing it's unlikely to happen unless they have a few geeky sorts willing to make it happen, but the old 'hit by a bus' question is raised then - who supports it if they go?

Given those things for an operation like TDS I think the Tivo/VTR combo is likely to be the best for a while, although it looks like Snapsteam might have a more commercially viable option.

Posted by: Dylan at Aug 28, 2008 5:14:22 AM

Post a Comment

Name:

Email Address: (required, not shown)

Remember personal info?

Comments:

(html not allowed)