E1 OpenPLi SQLite XML EPG Plugin (Radio Times)

[PaphosAL]

Hallo! With Team Rytec not supporting E1 any more for the Rytec XML Importer plugin, I know that many peeps as well as myself rely heavily on the Radio Times version for quality EPG and long descriptions in their DM500S OpenPLi image.

But one of the bugbears of using Radio Times UK EPG as the provider in the SQLite XML EPG plugin (1.2) are the strange characters we often see in the EPG Channel Listings and Long Descriptions, as mentioned in this thread on the OpenPLi forum a few weeks back.

For example: The film Brüno will appear as Brüno - while the name of the actress Renée Zellweger will display as Renée

I managed to get LraiZer's interest aroused in finding a fix for these utf-8 corruptions, which he did very quickly. But then we got deeper and deeper into it, to the extent that he is now filtering and fixing 20 different accented vowels: äëïöü áéíóú ÄËÏÖÜ ÁÉÍÓÚ

As well as improved error-trapping, logging and optimizing routines. Hopefully, DAF2000 on Team OpenPLi will use the 3 files in the upgrade rar (where there's a detailed readme) to maybe update the existing 1.2 package in Software Updates. We'll be mighty proud if he does. The 2 files in /var/bin need to be 755, but the one in /var/etc can stay 644, btw.

Also attached is an Options rar, again with a detailed readme, covering different automated methods of actually running the RT EPG load, other than via the plugin GUI (which now has the filecount and progress bar working properly ). Sample files and instructions are included for carrying out this task via the following methods:

1. A cron job at the same time each day
2. In the background, 2 minutes after a reboot
3. When the box is put into Standby Mode for 5 minutes or so

And for RT EPG noobs who can't get their heads around mapping their desired Channels via the plugin GUI, there's also a nice channel map in there for you too! 145 premium channels for 28.2ºE

Finally, just to show you the power of this heavily revised script, I've attached my log from this morning's 09:30 cron EPG load on my testbox. Awesome stuff... And trust me, during the past three weeks this new script has been under daily development, it's been tested to the Nth degree, to the point where it can now be shared in the community.

Enjoy! AL (and megga thanks to LraiZer for all his hard work on this )

ps- if anyone else out there (apart from me) also downloads their RT dat files on a daily basis to their Ubuntu server, then updates their dm500s box(es) locally from server rather than the RT source- please make yourself known.

We have developed a similar routine in parallel for such a setup, and it works really well...

[bobsylas]

Originally Posted by PaphosAL

ps- if anyone else out there (apart from me) also downloads their RT dat files on a daily basis to their Ubuntu server, then updates their dm500s box(es) locally from server rather than the RT source- please make yourself known.

We have developed a similar routine in parallel for such a setup, and it works really well...

Would be interested to know how you do this Al ?
I use a daily cron job on my DM500 to download EPG to a windows based machine set up just to store files I also use this to record to.

Also have a linux machine running "Smoothwall" which is my internet gateway & firewall.

Cheers Bob

[PaphosAL]

Okay, Bob- So in System Settings > EPG - for the SQLite EPG Location you don't see /tmp but another path to a mount on your Win machine, yes?

Tell me what that path is please. Also, purely out of interest, how many channels are in your map? How many days EPG are you grabbing? What time does cron kick in daily? And how big is the epg.db file created on your win system?

Sorry to be nosy, but I just need to ensure I fully understand your EPG setup before customizing the script for you, m8

Cheers- AL

[bobsylas]

Hi AL the mount on my windows machine is called "EPG"
my cron kicks in every day at 06:30 & grabs 4 days worth
at the moment my box has the files you posted above so will be the channels you set it to map 148 I think, this will be similar to what I set up myself
the epg.db file created is 7590 KB

Thanks very much Bob

[PaphosAL]

Thanks, Bob- Okay, the shared folder on your Win macine is \EPG but what is it linked from on the OpenPLi dm500 please? Look in System Settings > EPG > SQLite EPG Location and tell me exactly what path you have set there.

I wouldn't mind seeing your edited rt_uk.sh script and today's /var/log/rt-uk-load.log from your 06:30 cron job, please. You can attach both files direct just by giving them a .txt extension, or pop them both in a rar.

Cheers- AL

[bobsylas]

This is a screen shot of my System Settings > EPG
& the two files but the rt_uk.sh is the one you uploaded in post 1
hope it's the info you want

cheers Bob

[PaphosAL]

Thanks, Bob- but you have old AL well confused now, LOL! You'd better have a look in the \EPG folder on your Win machine, and see when the 7.6MB epg.db file was last updated in there.

Then FTP into your dm500 and look in the /tmp folder. I suspect this is where your epg.db is now being created by the new standard version of the rt_uk.sh script. And that's way larger than the recommended limit of 5mb for epg storage in /tmp

But I see in your screenshot that you have the 'Limit Size' button checked, so you should only see a 5MB epg.db file in there. Please confirm, m8.

As your mapped dat files are downloaded, error-checked, chr filtered, converted into the sqlite database, then deleted from /tmp one by one (in a while/do loop in the script)- this Limiter means that around one third of your mapped chanels will be ignored, I think... Although your log does suggest that the whole lot went hunkydory.

Problem is, I'm not altogether sure how that OpenPLi 5MB cap works in practise. I'll try and test this on a standard image today, and find out...

I need to customize the script for you so that it points epg.db towards your mount folder, and not the default /tmp folder. Then you'll be back to your full-size epg.db on the Win mount. What's the absolute path to /EPG on your dm500? Is it /var/EPG that you mount via?

Cheers- AL

Edit: Test Results- One mystery solved, Bob. The 'Limit Size to 5MB' option does NOT actually work. Upping my epg from 2 days to 4 days gives me a /tmp/epg.db file of 7MB. I'm pretty sure that /tmp on your dm500 is where you'll find a similar sized dBase file.

[bobsylas]

Yes Al the path on the dm500 mount is /var/epg

just had a look in my windows machine & the 7,622KB epg.db was modified 24/6/2011 10:26 ? dont understand the time it got updated ?

and yes there is a epg.db in /tmp on the dm500 but only 80kB

[PaphosAL]

Originally Posted by bobsylas

Yes Al the path on the dm500 mount is /var/epg

just had a look in my windows machine & the 7,622KB epg.db was modified 24/6/2011 10:26 ? dont understand the time it got updated ?

and yes there is a epg.db in /tmp on the dm500 but only 80kB

OIC, thanks, Bob. I'm beginning to understand your setup a bit better now, m8.

Yes, the one in /tmp is the base sqlite dbase file that macho enigma creates from the EIT if it finds /var/bin/db_epg exists. That's the sqlite converter engine / binary.

So the default epg location in the new script is being recognized by enigma, and is creating the small /tmp/epg.db file at boot. But the 'real' location for the working epg.db is in /var/epg (the mount to \EPG on your Win machine).

So, barring a couple of minor tweaks, you're basically all up and running with this new setup, m8. Don't worry about the timestamp on your epg.db file, either. This updates itself dynamically through the day, getting slimmer all the time, as 'old' epg is chucked out.

Can you attach your /var/tuxbox/config/dbepg/rt_uk.cfg file, please? I can then hand you back a bobsylas special script to try.

Also, I forgot to mention. The reason I asked for your cronjob time. Auntie was totally devoid of any data whatsover in her 404 dat files from 07:45 last Sunday until 07:45 last Monday. Did you notice that you had no EPG at all on Monday 20th June, after your 06:30 cron load?

Cheers- AL

[bobsylas]

Originally Posted by PaphosAL

Also, I forgot to mention. The reason I asked for your cronjob time. Auntie was totally devoid of any data whatsover in her 404 dat files from 07:45 last Sunday until 07:45 last Monday. Did you notice that you had no EPG at all on Monday 20th June, after your 06:30 cron load?

No never noticed AL but had a long day at work on Monday lol

Thanks for explaining things, gives a much better understanding of what's going on.

Here's the file
cheers Bob