Personalised TV Guide

This project is something we had been planning to build for a long time. There were so many times when we have missed interesting TV programmes simply because we didn't know they were on TV. This project aimed to fix this by downloading and processing freely available TV programme data, to create a personalised Electronic Programme Guide (EPG). It uses a personal interest profile for each family member to deliver a daily personalised summary of interesting content on TV.

The profile use keywords and phrases to identify content of interest and this analyses programme title, sub-title, presenters/actors and other information available. Once a programme or series recording has been set up, the programme could then be removed from the personal profile and future recommendation but, it is quite useful to leave it to catch new series starting.

EPG delivered by email
We get a personalised EPG delivered by email each morning. It is also accessible via Dropbox.

The set of characters at the end of the description is a code to show which rule caused this programme to be included in the results. In this image the 15-1ti means rule 15, condition 1 and the programme title caused Homeland to be included. This is really useful tool to help in refining our profile.

Design

XML Data Feeds

The best source of freely available EPG data we found was from the Radio Times, so this is what we used for this project. The Radio Times site provides a number of data feeds providing up to 14 days of information, which are freely available for personal use. This is used by projects such as XMLTV to provide programme guides for systems such as MythTV and many others, and it was our original intention to use the same XMLTV format. A quick look at the data from the Radio Times site revealed that it is simply a number of text files with delimiters and a line of text for each TV programme. It just didn't make sense to convert this to XML.

Channels

The list of channels can be downloaded from the following URL: http://xmltv.radiotimes.com/xmltv/channels.dat

This file lists all the the available TV channels. The fields are CHANNEL_ID and CHANNEL_NAME and these are separated by a pipe '|' character, e.g.

22|Adventure One
24|ITV1 Anglia
25|ITV1 Border

We created a much shorter subset of this list in a file called channels.txt. This is a subset of the channels containing only the ones we are interested in (and can actually watch on our YouView STB in the East Anglian region). We remove all of the '+1' channels and all those that are available in HD.

Each time our code runs, it checks the data in channels.dat against the data in channels.txt, to make sure the channel names and numbers match. It reports any differences in the log files.

Channel Listings

The above channel file contains a channel number and this is then used to download the programme listing for each channel. For example the BBC1 download URL is http://xmltv.radiotimes.com/xmltv/92.dat. These channel listing files contain up to 14 days of programme information and they are updated by 08:00 every morning.

Each channel file (identified by it's ID) is called CHANNEL_ID.dat (e.g. 24.dat) contains 14 days of listings for that channel. The fields on each line are separated by a tilda "~" character and end with a new line. The fields that are supplied are:

  1. Programme Title
  2. Sub-Title
  3. Episode
  4. Year
  5. Director
  6. Performers (Cast) - This will be either a string containing the actors names or be a number of character name / actor name pairs which are separated by an asterisk '*' and each pair by a pipe '|' character.
  7. Premiere
  8. Film
  9. Repeat
  10. Subtitles
  11. Widescreen
  12. New series
  13. Deaf signed
  14. Black and White
  15. Film star rating
  16. Film certificate
  17. Genre
  18. Description
  19. Radio Times Choice - selected by the Radio Times editorial team.
  20. Date
  21. Start Time
  22. End Time
  23. Duration (Minutes)

Java Code

To maintain consistency with all of the other code that runs on our Home Control System (HCS), we are writing our application in Java and have created some classes to load and process the channel list and individual channel listing files.

The code will be made available very soon.

channels.txt

The first thing our Java program does is load up our channel list. This is a tab-delimited text file of the initial 412 channels that were returned in the channels.dat file. This has three fields: the channel ID and the channel name. We manually edit this file to make updates.

channels.dat

If the channels.dat file has been updated, it is downloaded and stored locally. We then check to see if any new channels have been added or any channels have been changed or no longer match the ID and names we have in our channel.txt file. Any changes or updates are reported in our logging system. Any line that doesn't include just one '|' character is ignored.

Rules File

This file contains pattern matching rules to include and exclude programmes. By default a programme is ruled out unless one of the rules determines it should be included. Rules can also be set to exclude content.

Each rule has one of more conditions and each condition has a number of elements. This allows us to do really fine filtering such as including Top Gear but not if it is a repeat on Dave and is on an HD channel.

Scope
The scope of the pattern matching can be all programme fields or specific fields such a the programme title, description, director, actor, channel, etc.
Action
This determines if the rule forces inclusion or exclusion.
Variable
This is the search term chosen for this rule.
Case
The case sensitivity can be set of each rule.
Whole Words
This determines if part words or only whole words match.

Our system supports multiple rules files (one for each person in the house) and it is simply a matter of loading the each rules file before invoking the analysis code. This is also passed the profile ID, and thus generates a unique output file per person.

Analysis

The Rules class does the personalised matching of programmes to each users set of rules. Typically the rules file will have about 20-40 entries and this will result in about 4 to 8 programme matches in each day.

Channel Listings

This is the example channel listing file for BBC 1 HD: 2667.dat. It is a text document with a disclaimer and then the programme information, one programme per line. This particular example has 533 lines covering the next 14 days.

Progress

We originally wrote this as a separate Java process, so that we could test it more easily. The software will currently generate a personalised EPG for a person and a supplied date. We took this approach so that we could generate a file for say, the next 10 days (useful if you are about to go on holiday). The output is a simple text file which is created in a Drop Box folder, so that it can be accessed from any device and at any time via the cloud access clients.

An example from 28/09/2012:

Rob's TV Guide for 28/09/2012

[BBC2 East] Helicopter Heroes 28/09/2012 (15:45-16:30) {23}
An air ambulance crew tries to save the life of a man whose leg has been crushed by a seven-ton digger, but bad weather threatens to ground his flight to hospital. The paramedics also attend an accident in one of the UK's biggest caves. Presented by Rav Wilding.

[Channel 4] Alan Carr: Chatty Man 28/09/2012 (22:00-23:10) {26}
Singer-songwriter and The Voice coach Jessie J tells the host about her autobiography Nice to Meet You, boy band One Direction discuss their new single Live While We're Young and forthcoming second album Take Me Home, and designer Kevin McCloud talks Grand Designs. American R'n'B star Alicia Keys provides the music, performing a track from her latest release Girl on Fire.

[Channel 5] Heroes of the Skies 28/09/2012 (19:00-20:00) {23}
Michael Ashcroft recalls the exploits of Canadian fighter pilot George Beurling, whose heroics during the siege of Malta in 1942 helped turn the tide of the Second World War. The programme explores how Beurling's marksmanship and handling of his Spitfire aircraft enabled him to shoot down more enemy aircraft than any other Allied pilot during the campaign - but also reveals why his personality often led to him falling out with senior officers.

[More4] Homeland 28/09/2012 (03:05-04:05) {24}
Carrie has high hopes that Lynne, her spy in the Saudi prince's entourage, can lead her to a specific al-Qaeda operative in America - perhaps even Brody himself. Meanwhile, the marine is preparing for a network TV interview with his wife and kids - but daughter Dana is not interested in playing happy families. Conspiracy thriller, starring Claire Danes, Damian Lewis, Morena Baccarin, Morgan Saylor and Brianna Brown,.

[More4] Grand Designs 28/09/2012 (19:55-21:00) {26}
Kevin McCloud revisits Denise and Bruno Del Tufo, who hoped to transform a concrete water tower designed by Edwin Lutyens into a contemporary home. Kevin was originally far from enthusiastic about their project, believing the original building to be ugly and brutal in its design - but the couple soon had other problems to worry about, as rising steel prices forced them to make drastic changes to their budget.

[More4] Homeland 28/09/2012 (23:05-00:05) {24}
Big plans are made for national hero Brody, but his increasingly erratic behaviour threatens his status as the media's latest darling, prompting the vice president to send an adviser to check him out. Carrie nears the end of her four-week surveillance warrant and grows desperate for evidence linking the marine to Abu Nazir, but Saul instructs her to focus on the terrorist's money trail. Thriller, starring Claire Danes and Damian Lewis.

[ITV4] UEFA Champions League Hat-Trick Heroes 28/09/2012 (06:00-06:05) {23}
Memorable trebles from some of the finest players to have graced European football's elite club competition.

[ITV4] Goodwood Revival 2012 28/09/2012 (19:00-20:00) {22}
Steve Rider and Amanda Stretton present highlights from Goodwood House in West Sussex, as some of motor racing's top names, including Stirling Moss, John Surtees and Martin Brundle, gathered for the world famous event.

[Channel 4 HD] Alan Carr: Chatty Man 28/09/2012 (22:00-23:10) {26}
Singer-songwriter and The Voice coach Jessie J tells the host about her autobiography Nice to Meet You, boy band One Direction discuss their new single Live While We're Young and forthcoming second album Take Me Home, and designer Kevin McCloud talks Grand Designs. American R'n'B star Alicia Keys provides the music, performing a track from her latest release Girl on Fire.

The number after the programme times in {} is currently displayed for debugging purposes only and is ID of the rule that matched and caused this programme to be listed. This is really handy for those times when a programme appears and you don't know why, especially if you don't want it to appear again.

The personalised EPG's are generated automatically at 02:00 and are emailed out each morning at 06:00. They are generated again at 08:30 because the programme data files are updated each day at 08:00. These updates are available via the cloud storage synchronisation.

Summary

The primary method of distribution of EPGs is via an email out to individual family members. This also acts as a notification and reminder.

By default the EPGs are created in a Drop Box folder for remote synchronisation and access but this Drop Box account is not used by all family members. One way around this is to provide an email link to then EPG file as a desktop shortcut. This then provides a direct link to the latest version.

Updates

October 2012

On our daughter's Google Nexus S phone, opening and viewing attachments is a much more painful process than the experience presented on an iPhone, so we've changed the way the email is delivered. It now has the EPG content as part of the email message body and not as an attachment.

November 2014

A lot of this code has been updated and re-written to better integrate into our Home Control System (HCS). It is now a seperate service (Java thread) and takes account of things like our Internet connection being down. The rules files also reside in Dropbox, so that they can be updated from any machine or mobile device. A lot of the improvements are around the error checking and data validation.

Another example file (2014_11_16_EPG.txt).

Share ...
We are on ...
Facebook Twitter
YouTube Flickr Follow us on Pinterest