Prototype - Rytec EPG for Neutrino (DM500)

[hotbaws11]

Prototype - Rytec EPG for Neutrino (DM500) -> Version 1 09-Sep-2010]

Right guys, here goes...

Here's a working prototype (script) which loads the Rytec EPG information onto a DM500 Neutrino image. The image I used is one of the old Russian one.

Just put this little fella into /var/bin, chmod to 755 and run from telnet to see what happens

Comments/Suggestions welcome as its work in progress!!!
I’ll update this thread/post with new versions etc.

No external storage is required, all processing is done in /tmp.

CHANNELREGEXP variable is used to identify what channels you want to extract from Rytec xml file and thus import into Neutrino. At the moment, I've just kept it to 2 channels (ITV2 & ITV2+1). This can be changed to include a lot more channels (refer to script) but the current version of the script has a big performance/space issue (see known issues below) which needs to be resolved first!

Note that you don’t need a local channel mapping file as the script downloads the Rytec version from the server however it’s a good idea to create a one and have few entries in it as I can see that the Rytec version isn’t very good with regional varieties (i.e. BBC/ITV London, Scotland, Ireland). So without a manual mapping for these channels, they will not be loaded into Neutrino. The script will tell you what channels can’t be loaded etc.

Known Issues: ...would appreciate help to solve

Issue 1: Performance
The performance of the script is terrible for parsing the custom xml file after it has been generated from zcat/sed. I have some very crude scripting in there to open/parse the file to extract the channel, title, description, time. Please someone put me out of my misery and suggest an easy/fast way of doing this. I've love to have one long sed line starting from zcat to extract all of the information into pre-formatted channel xml file but have no idea if it’s even possible.

Issue 2: Space Limitation of \tmp
If any performance improvements still require a full cust xml file to be written and then re-parsed to get the information into Neutrino channel xml file format, then this will cause problems as the cust file will be over 5MB when handling over 100 channels. I’d propose splitting the cust xml file into individual channel files and parsing them individually to create the Neutrino channel xml file. This means each individual cust file could be deleted after the channel xml file has been created.

Issue 3: Maximum number of programs/events
In the script there is a counter for the programs/events. The max decimal number that Neutrino can accept is 9999 however if someone tells me how to create a hex counter (without using printf or bc) then the script could cope with up to 65535 events (i.e. FFFF).

Attached:
1) cust_uk_netrino.sh –> the script itself
2) cust_rytecxmltvuk -> custom xml file for channel Five. Basically need a good/fast way of translating this into Neutrino channel xml files instead of using my crude code!!!

#/bin/sh
ZIPFILE="rytecxmltvuk.gz"
CUSTFILE="cust_rytecxmltvuk"
# Regular Expression searching for channel names in channel map file (i.e. BBC*)
#CHANNELREGEXP="BBC1London\|BBC1Scotland\|BBC2\|ITV1London\|STV\|Channel4.uk\|Five.uk"
CHANNELREGEXP="ITV2"
CHANNELMAPZIP="e1_UK_DATfile.tar.gz"
CHANNELMAP="e1_UK.dat"
MANUALCHANNELMAPFILE="/var/etc/channelmap.dat"

# Remove old Neutrino channel xml files
rm /var/tmp/*.xml

if [ ! -f $MANUALCHANNELMAPFILE ]; then
  echo "$MANUALCHANNELMAPFILE not present. Using Rytec channel mapping only."
fi

# get Rytec Channel map file
wget  "http://www.rytec.be/tools/e1_update/"$CHANNELMAPZIP -O /var/tmp/$CHANNELMAPZIP
zcat /var/tmp/$CHANNELMAPZIP > /var/tmp/$CHANNELMAP # extract channel map file to /var/etc
rm /var/tmp/$CHANNELMAPZIP

# get Rytec EPG file
wget  "http://www.xmltvepg.be/"$ZIPFILE -O /var/tmp/$ZIPFILE

# print header of custom epg file
#echo '<?xml version="1.0" encoding="ISO-8859-1"?>
#<!DOCTYPE tv SYSTEM "xmltv.dtd">
#<tv generator-info-name="custom">' > /var/tmp/$CUSTFILE
echo '<tv>' > /var/tmp/$CUSTFILE
# extract all channels and programs from the zip file (based on reg exp) and print into the epg file
zcat /var/tmp/$ZIPFILE | sed -n '/<\(channel\|programme\).*\"\('$CHANNELREGEXP'\)/,/\(channel\|programme\)>/p' | sed 's/\((\|)\|"\|£\)//g' >> /var/tmp/$CUSTFILE
# print bottom of custom epg file
echo '</tv>
' >> /var/tmp/$CUSTFILE
rm /var/tmp/$ZIPFILE

# count number of channels/events in custom epg file
no_of_channels=`sed -n '/<channel/p' /tmp/$CUSTFILE | sed -n '$='`
no_of_events=`sed -n '/<program/p' /tmp/$CUSTFILE | sed -n '$='`
echo 'Channels:'$no_of_channels' Events:'$no_of_events

# print header of index.xml
channel_xml=$onid$tsid$sid.xml
echo '<?xml version="1.0" encoding="UTF-8"?>
<!--
  This file was automatically generated from Rytec UK EPG.
-->
<dvbepgfiles>' > /tmp/index.xml

# go through while loop below to read every line from the custom epg file.
# when time, title, desc and channel map are found, event information can be printed to the channel xml
# very inefficent code here but it works lol!!!!
max_events=9999
event_count=0
channel_count=0
string_count=""
epg_start_time=0
epg_duration=0
cur_channel="empty"
channel_xml=""
epg_title=""
epg_desc=""
chann_match=""
time_found="no"
title_found="no"
desc_found="no"

while read linevar; do
  # long winded way to get the start date, duration and channel map name
  if [ `expr length "$linevar"` -gt 20 ]; then
    sub_str=`expr substr "$linevar" 1 10`
    if [ `expr "$sub_str" = "<programme"` -eq 1 ]; then
      epg_day=`expr substr "$linevar" 24 2`
      epg_month=`expr substr "$linevar" 24 2`
      epg_year=`expr substr "$linevar" 18 4`
      epg_starttime=`expr substr "$linevar" 26 4`
      epg_timestamp=`date -d "$epg_month$epg_day$epg_starttime$epg_year" +%s`
	  #echo $epg_month'-'$epg_day'-'$epg_starttime'-'$epg_year
      
      epg_day2=`expr substr "$linevar" 50 2`
      epg_month2=`expr substr "$linevar" 48 2`
      epg_year2=`expr substr "$linevar" 44 4`
      epg_endtime=`expr substr "$linevar" 52 4`
      epg_timestamp2=`date -d "$epg_month2$epg_day2$epg_endtime$epg_year2" +%s`
	  #echo $epg_month2'-'$epg_day2'-'$epg_endtime'-'$epg_year2
      
      epg_duration=`expr $epg_timestamp2 - $epg_timestamp`
      epg_start_time=$epg_timestamp
      time_found="yes"
      
	  line_len=`expr length "$linevar"`
      line_len=`expr $line_len - 1 - 73`
      epg_channel=`expr substr "$linevar" 73 $line_len`
      
      # if the channel found is not the same as the current channel,
      # then we need to create new channel xml and add name to index xml
      if [ "$cur_channel" != "$epg_channel" ]; then
	    # if this is not the first channel, close the previous channel before proceeding 
        if [ "$cur_channel" != "empty" ] && [ "$chann_match" = "yes" ]; then
          echo '	</service>
</dvbepg>' >> /tmp/$channel_xml
        fi
		
		cur_channel=$epg_channel      
        # for the epg channel, get the onid, tsid and sid out of the channel mapping file
		# check the local mapping file first. If channel isn't found, check Rytec version.
		channel_info=""
		if [ -f $MANUALCHANNELMAPFILE ]; then
		  channel_info=`sed -n '/'$epg_channel'/p' $MANUALCHANNELMAPFILE`
		fi
        if [ "$channel_info" = "" ]; then
		  channel_info=`sed -n '/'$epg_channel'/p' /var/tmp/$CHANNELMAP`
		  if [ "$channel_info" = "" ]; then
            echo "---$epg_channel not found in channel map file."
			echo '---Put channel/sid in '$MANUALCHANNELMAPFILE'.'
			echo '---Example line for '$MANUALCHANNELMAPFILE' is 27e2:2:806=STVCentral.uk'
		    chann_match="no"
		  fi
		fi
        if [ "$channel_info" != "" ]; then
          chann_match="yes"
		  channel_count=`expr $channel_count + 1`
		  
		  no_of_chann_matches=`echo $channel_info | sed -n '/'$epg_channel'/p' | sed -n '$='`
		  if [ $no_of_chann_matches -gt 1 ]; then
		    echo '---'$epg_channel' has more than 1 channel mapping. Last sid will be used.'
			echo '---To use different channel/sid, put one entry in '$MANUALCHANNELMAPFILE'.'
			echo '---Example line for '$MANUALCHANNELMAPFILE' is 27e2:2:806=STVCentral.uk'
		  fi
        
          # crude way to get sid
          index_split=`expr index "$channel_info" :`
          index_end=`expr $index_split - 1`
          index_start=`expr $index_split + 1`
          sid=`expr substr "$channel_info" 1 $index_end`
          # crude way to pad with zeros
          sid=`echo $sid | sed 's/^.$/000&/' | sed 's/^..$/00&/' | sed 's/^...$/0&/'`
        
          # crude way to get onid
          channel_info=`expr substr "$channel_info" $index_start 99`
          index_split=`expr index "$channel_info" :`
          index_end=`expr $index_split - 1`
          index_start=`expr $index_split + 1`
          onid=`expr substr "$channel_info" 1 $index_end`
          # crude way to pad with zeros
          onid=`echo $onid | sed 's/^.$/000&/' | sed 's/^..$/00&/' | sed 's/^...$/0&/'`
        
          # crude way to get tsid
          channel_info=`expr substr "$channel_info" $index_start 99`
          index_split=`expr index "$channel_info" =`
          index_end=`expr $index_split - 1`
          index_start=`expr $index_split + 1`
          tsid=`expr substr "$channel_info" 1 $index_end`
          # crude way to pad with zeros
          tsid=`echo $tsid | sed 's/^.$/000&/' | sed 's/^..$/00&/' | sed 's/^...$/0&/'`
        
          echo 'Channel Map Name:'$epg_channel' sid:'$sid' onid:'$onid' tsid:'$tsid
          channel_xml=$onid$tsid$sid.xml
          echo '	<eventfile name="'$channel_xml'"/>' >> /tmp/index.xml

          # print header of the channel file
          echo '<?xml version="1.0" encoding="UTF-8"?>
<!--
  This file was automatically generated from Rytec UK EPG. 
-->
<dvbepg>
	<service original_network_id="'$onid'" transport_stream_id="'$tsid'" service_id="'$sid'">' > /tmp/$channel_xml
        fi
      fi
    fi
  fi
  # long winded way to get the title and description
  if [ `expr length "$linevar"` -gt 6 ] && [ "$chann_match" = "yes" ]; then
    if [ `expr substr "$linevar" 1 6` = "<title" ]; then
      line_len=`expr length "$linevar"`
      line_len=`expr $line_len - 8 - 16`
      epg_title=`expr substr "$linevar" 16 $line_len`
      title_found="yes"
      read linevar # read the next line as it will be the description (if present)

      if [ `expr substr "$linevar" 1 5` = "<desc" ]; then
        line_len=`expr length "$linevar"`
        line_len=`expr $line_len - 7 - 7`
        epg_desc=`expr substr "$linevar" 7 $line_len`
        desc_found="yes"
      else
        epg_desc="No description available"
        desc_found="yes"
      fi
    fi
  fi
  # when time, title, desc and channel map are found, event information can be printed to the channel xml
  if [ "$time_found" = "yes" ] && [ "$title_found" = "yes" ] && [ "$desc_found" = "yes" ] && [ "$chann_match" = "yes" ]; then
    echo '		<event id="'$event_count'">
			<name lang="OFF" string="'$epg_title'"/>
			<text lang="OFF" string="'$epg_desc'"/>
			<time start_time="'$epg_start_time'" duration="'$epg_duration'"/>
		</event>' >> /tmp/$channel_xml
    event_count=`expr $event_count + 1`
    epg_start_time=0
    epg_duration=0
    epg_title=""
    epg_desc=""
    time_found="no"
    title_found="no"
    desc_found="no"
  fi
  if [ $event_count -gt $max_events ]; then
    echo 'Max events reached:'$max_events
    break
  fi
done < "/tmp/$CUSTFILE"

# close the last channel xml file 
if [ "$chann_match" = "yes" ]; then
  echo '	</service>
</dvbepg>' >> /tmp/$channel_xml
fi

# close the index xml file
echo '</dvbepgfiles>' >> /tmp/index.xml

rm /var/tmp/$CUSTFILE
rm /var/tmp/$CHANNELMAP

echo 'EPG processing finished. '$event_count' events loaded for '$channel_count' channels.'
echo 'Initializing epg reset...'
sleep 5 && killall sectionsd
sleep 5 && sectionsdcontrol --readepg /tmp/

# sleep 5 && rm /var/tmp/*.xml