My TV provider do not push EPG info in the local DVB-T, but i have a xmltv feed that i already use in tvtime. Kaffeine right now only loads EPG from the tv source, it would be a good feature to be able to load the EPG from alternative sources, like a xmltv file (that i already have being updated in a cron)
(In reply to higuita from comment #0) > My TV provider do not push EPG info in the local DVB-T, but i have a xmltv > feed that i already use in tvtime. > Kaffeine right now only loads EPG from the tv source, it would be a good > feature to be able to load the EPG from alternative sources, like a xmltv > file (that i already have being updated in a cron) I suspect it should be easy to add support for it to use a xmltv file. Could you please send me a sample of it (either added as an attachment or sent to me via e-mail? Also, are there any specs for the format used by it?
Created attachment 118676 [details] compressed xmltv attached is a sample of today downloaded xmltv The format and another sample is in this url: http://wiki.xmltv.org/index.php/XMLTVFormat https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd
On a quick look, I guess that the main challenge is how to map the channel ID provided by xmltv into the channel data stored at the mysql tables used by Kaffeine, at epgdata.dvb file. From the file you posted, xmltv stores channels like: <channel id="bloomb.tv.vodafone.pt"> <display-name lang="pt">Bloomberg</display-name> ... <channel id="blazehd.tv.vodafone.pt"> <display-name lang="pt">Blaze HD</display-name> Kaffeine's EPG data is stored using channel name, as it was discovered from MPEG-TS tables. Do you know if the display-name is identical to the channel name that Kaffeine displays? On a quick look, it doesn't sound hard to parse the tables One challenge is how to handle languages. Instead of ISO639 3-letter code, as found at MPEG-TS tables, it uses ISO 639 2-letter codes, but the DTD file says: Many of the details have a 'lang' attribute so that you can store them in multiple languages or have mixed languages in a single listing. This 'lang' should be the two-letter code such as 'en' or 'fr_FR'. Or you can just leave it out and let your reader take a guess. Adding support for 2-letter codes is easy, but converting 'fr_FR', 'pt_BR', ... formats to ISO 639 3-letter codes could be a little tricky. I can't find an easy way of doing that. It is easier to just ignore the Country information and use just the 2 letter code. Could you please record ~60 seconds of a channel and place on GoogleDrive (or something similar) from one channel that has EPG data? I need the record of the entire transponder, in order to have all MPEG tables there, and be able to test some code that would map xmltv data into Kaffeine's EPG internal representation. In order to get it, you should use dvbv5-zap. Instructions about how to use it are at: https://linuxtv.org/wiki/index.php/Dvbv5-zap#Recording_the_full_MPEG_stream_on_a_transponder
Git commit 2380ab25da84c999ea14cd6bb34e0328e9c67059 by Mauro Carvalho Chehab. Committed on 11/03/2019 at 21:38. Pushed by mauroc into branch 'master'. xmltv: add a parser for XMLTV EPG data files This parser handles XMLTV format files as specified at: http://wiki.xmltv.org/index.php/Main_Page The way the parser was written, once a file is added to the parser, it keeps monitoring it for file changes. When a file change is detected, it will reload the file and parse it, adding entries to the EPG guide. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> M +2 -1 src/CMakeLists.txt M +7 -0 src/dvb/dvbchannel.cpp M +1 -0 src/dvb/dvbchannel.h M +44 -15 src/dvb/dvbconfigdialog.cpp M +2 -0 src/dvb/dvbconfigdialog.h M +20 -0 src/dvb/dvbmanager.cpp M +4 -0 src/dvb/dvbmanager.h A +261 -0 src/dvb/xmltv.cpp [License: GPL (v2+)] A +56 -0 src/dvb/xmltv.h [License: GPL (v2+)] https://commits.kde.org/kaffeine/2380ab25da84c999ea14cd6bb34e0328e9c67059
(In reply to Mauro Carvalho Chehab from comment #3) > On a quick look, I guess that the main challenge is how to map the channel > ID provided by xmltv into the channel data stored at the mysql tables used > by Kaffeine, at epgdata.dvb file. > > From the file you posted, xmltv stores channels like: > > <channel id="bloomb.tv.vodafone.pt"> > <display-name lang="pt">Bloomberg</display-name> > ... > <channel id="blazehd.tv.vodafone.pt"> > <display-name lang="pt">Blaze HD</display-name> > > > Kaffeine's EPG data is stored using channel name, as it was discovered from > MPEG-TS tables. Do you know if the display-name is identical to the channel > name that Kaffeine displays? > > On a quick look, it doesn't sound hard to parse the tables > > One challenge is how to handle languages. Instead of ISO639 3-letter code, > as found at MPEG-TS tables, it uses ISO 639 2-letter codes, but the DTD file > says: > > Many of the details have a 'lang' attribute so that you can > store them in multiple languages or have mixed languages in a single > listing. This 'lang' should be the two-letter code such as 'en' or > 'fr_FR'. Or you can just leave it out and let your reader take a > guess. > > Adding support for 2-letter codes is easy, but converting 'fr_FR', 'pt_BR', > ... formats to ISO 639 3-letter codes could be a little tricky. I can't find > an easy way of doing that. It is easier to just ignore the Country > information and use just the 2 letter code. > > Could you please record ~60 seconds of a channel and place on GoogleDrive > (or something similar) from one channel that has EPG data? I need the record > of the entire transponder, in order to have all MPEG tables there, and be > able to test some code that would map xmltv data into Kaffeine's EPG > internal representation. > > In order to get it, you should use dvbv5-zap. Instructions about how to use > it are at: > > https://linuxtv.org/wiki/index.php/Dvbv5- > zap#Recording_the_full_MPEG_stream_on_a_transponder No need anymore. Some of the channels on the xmltv file you provided are identical to the channel names I have here. So, I was able to test it.
Created attachment 118730 [details] kaffeine channel list To start, thank you very much for implementing this! :D > Kaffeine's EPG data is stored using channel name, as it was discovered from > MPEG-TS tables. Do you know if the display-name is identical to the channel > name that Kaffeine displays? many are the same, but most of then have little differences, that are enough to break the parser :) Some have H instead of HD, others drop the SD, others simplify the name (RADICAL -> R or Food Network -> TFN ), others change case (SkyNews -> SKY News) I pulled the git and tested, it worked on the names that match (23 from 62 channels i have available in my local DVB-T) Doing the command below and attaching the list echo 'select name from Channels;' | sqlite3 sqlite.db > /tmp/channels.txt But i think it is a lost battle trying to always automatically match all the feeds and channels, we do never know what is out there. Some automatic rules like the current are good to help the initial setup, but in the end one should be able to map the xmltv name to the channel when needed. So i suggest adding a extra rule to also try to match names without the SD and HD if not matched already, as usually the channel and data is the same, only change the resolution. Also always match names with case insensitive After that, in the channels config, add a extra column (may be optional) for epg that show the EPG/xmltv name (or * if same name). This will allow to quickly find out what channels are without EPG and fix then Finally, editing the channel, we would have a new drop down list with the EPG/xmltv name, so one can manually choose the correct feed source for the channel. Of course, this also requires a new field in the database to store this map
(In reply to higuita from comment #6) > Created attachment 118730 [details] > kaffeine channel list > > To start, thank you very much for implementing this! :D > > > > Kaffeine's EPG data is stored using channel name, as it was discovered from > > MPEG-TS tables. Do you know if the display-name is identical to the channel > > name that Kaffeine displays? > > many are the same, but most of then have little differences, that are enough > to break the parser :) > Some have H instead of HD, others drop the SD, others simplify the name > (RADICAL -> R or Food Network -> TFN ), others change case (SkyNews -> SKY > News) > > I pulled the git and tested, it worked on the names that match (23 from 62 > channels i have available in my local DVB-T) Well, as xmltv format allows multiple display names, IMHO, the right fix would be to address those at the parser. There's one thing feature in Kaffeine's implementation, though: right now, it will associate just one channel to the EPG data. So, on a channel with both "SD" and "HD", it will fill EPG just for the first one. This is easily solvable. I should be writing a patch for it soon. > > Doing the command below and attaching the list > > echo 'select name from Channels;' | sqlite3 sqlite.db > /tmp/channels.txt > > But i think it is a lost battle trying to always automatically match all the > feeds and channels, we do never know what is out there. Some automatic rules > like the current are good to help the initial setup, but in the end one > should be able to map the xmltv name to the channel when needed. > > So i suggest adding a extra rule to also try to match names without the SD > and HD if not matched already, as usually the channel and data is the same, > only change the resolution. Also always match names with case insensitive I don't like the idea of doing partial match. Things can easily go messy. Also, this would increase a lot CPU consumption. Due to performance issues, we use a hash table to associate a channel name to Kaffeine's channel structure. This is very fast - using big-O notation, it is O(1). A partial search will either require a much more complex data structure or it would be O(n). We had already performance issues with EPG parsing time. Also, I'm actually more concerned at the opposite side: what happens if someone has multiple digital TV cards, each connected to different providers? The same channel name could have different programs, depending on the providers. Ideally, we should associate the xmltv file to an specific DVB card. The xmltv.cpp code is ready for this, as it was written in a way that it can monitor multiple files, but some changes would be required at the configuration side, and ad the DVB board configuration file. Also, it should automatically (or via an option?) disable MPEG-TS EPG table handling on cards that are associated with a xmltv file. > > After that, in the channels config, add a extra column (may be optional) for > epg that show the EPG/xmltv name (or * if same name). This will allow to > quickly find out what channels are without EPG and fix then > > Finally, editing the channel, we would have a new drop down list with the > EPG/xmltv name, so one can manually choose the correct feed source for the > channel. Of course, this also requires a new field in the database to store > this map This shouldn't be hard to implement, but some care should be taken about how to handle multiple channels that would map to the same name. That's said, I'm a little busy here. Probably won't have much time for such changes. It would be a way better if you could work on some patches and send me.
Git commit 456f6d07e47fe1b79f4e2f77974d6f8250c60bb5 by Mauro Carvalho Chehab. Committed on 12/03/2019 at 10:15. Pushed by mauroc into branch 'master'. xmltv: associate a xmltv channel to multiple DVB channels It is not uncommon that a single TV channel to have multiple video streams, with different resolutions (SD, HD, 4K, 8K). On those cases, the same EPG data should be added to the different video streams. So, be sure that it will try to use the xmltv data for all possible display-names. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> M +20 -5 src/dvb/xmltv.cpp https://commits.kde.org/kaffeine/456f6d07e47fe1b79f4e2f77974d6f8250c60bb5
Git commit d0cfcc9eaf9cf67d1db7c284f25ac3119710c969 by Mauro Carvalho Chehab. Committed on 12/03/2019 at 10:29. Pushed by mauroc into branch 'master'. xmltv: Fix a bug when adding multiple DVB channels for xmltv channels Last patch was missing the logic that would be retrieving the Kaffeine channel structure. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> M +1 -0 src/dvb/xmltv.cpp https://commits.kde.org/kaffeine/d0cfcc9eaf9cf67d1db7c284f25ac3119710c969
(In reply to Mauro Carvalho Chehab from comment #7) > (In reply to higuita from comment #6) > > many are the same, but most of then have little differences, that are enough > > to break the parser :) > > Some have H instead of HD, others drop the SD, others simplify the name > > (RADICAL -> R or Food Network -> TFN ), others change case (SkyNews -> SKY > > News) > > > > I pulled the git and tested, it worked on the names that match (23 from 62 > > channels i have available in my local DVB-T) > > Well, as xmltv format allows multiple display names, IMHO, the right fix > would be to address those at the parser. There's one thing feature in > Kaffeine's implementation, though: right now, it will associate just one > channel to the EPG data. So, on a channel with both "SD" and "HD", it will > fill EPG just for the first one. Btw, no need to even modify the parser. A simple script using xmlstarlet would be able to add new channel associations. For example, to associate a DVB channel named "DVB_CHANNEL" to a xmltv channel named "xmltv.channel.pt", this would do the job: channel="xmltv.channel.pt" name="DVB_CHANNEL" xmlstarlet ed -s "tv/channel[@id='$channel']" -t elem -n display-name -v "$name" vodafone_pt.xmltv > output.xml
Git commit 211c48d5dd7b0ec86a3203fd454fa9151d18ece6 by Mauro Carvalho Chehab. Committed on 12/03/2019 at 11:52. Pushed by mauroc into branch 'master'. index.docbook: add documentation about XMLTV support Describe the two new configuration settings for xmltv support. Also, add a new section at Kaffeine's handbook describing how to use it with XMLTV. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> M +75 -0 doc/index.docbook https://commits.kde.org/kaffeine/211c48d5dd7b0ec86a3203fd454fa9151d18ece6
Git commit 920f9ba6cb298bb34a1aefb77b06957f2ba9a942 by Mauro Carvalho Chehab. Committed on 12/03/2019 at 20:47. Pushed by mauroc into branch 'master'. README.md: add a quick description about xmltv Make easier for users to work with xmltv files by adding a quick comment at the Kaffeine's readme file. Also, add a small script with an example about how to map XMLTV channels into DVB ones. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> M +47 -0 README.md A +50 -0 tools/map_xmltv_channels.sh https://commits.kde.org/kaffeine/920f9ba6cb298bb34a1aefb77b06957f2ba9a942
> Well, as xmltv format allows multiple display names, IMHO, the right fix would be >to address those at the parser. (...) > Btw, no need to even modify the parser. A simple script using xmlstarlet would >be able to add new channel associations. your script to add to the xmltv file the same display names as DVB-T worked great and i agree that is lot more clean than trying to workaround and map things in kaffeine. This way the source xmltv is already fixed and even doing rescan channels or adding new channels are easy to keep everything working. Again, thanks for the help and this feature > There's one thing feature in Kaffeine's implementation, though: right now, it > will associate just one channel to the EPG data. So, on a channel with both > "SD" and "HD", it will fill EPG just for the first one. At least here in Portugal, the EPG data for SD and HD channels are the same, so using the first one loaded on both versions of the channel is not really a problem... i do not know if there are places where the data is really different between those channels >> Finally, editing the channel, we would have a new drop down list with the with the problem fixed in the xmltv file, i do not see the need to add this now, at least for me, it is easier to maintain the map in the script than in any (one by one) GUI. But a feature would still be useful, a column in the channel config window to show what channels already have epg and those that don't. The alternative is to manually click each channel to find out. in my setup, 62 clicks to find 27 channels with EPG and 35 without Sadly i do not know how to program, so i can't really helping develop this :) all i can do is open a new "feature request", if someone wants later to pick this up (if you agree that this is useful too)
(In reply to higuita from comment #13) > > Well, as xmltv format allows multiple display names, IMHO, the right fix would be >to address those at the parser. > (...) > > Btw, no need to even modify the parser. A simple script using xmlstarlet would >be able to add new channel associations. > > your script to add to the xmltv file the same display names as DVB-T worked > great and i agree that is lot more clean than trying to workaround and map > things in kaffeine. This way the source xmltv is already fixed and even > doing rescan channels or adding new channels are easy to keep everything > working. Good! > > Again, thanks for the help and this feature > > > > There's one thing feature in Kaffeine's implementation, though: right now, it > > will associate just one channel to the EPG data. So, on a channel with both > > "SD" and "HD", it will fill EPG just for the first one. > > At least here in Portugal, the EPG data for SD and HD channels are the same, > so using the first one loaded on both versions of the channel is not really > a problem... i do not know if there are places where the data is really > different between those channels Here in Brazil, the same applies for ISDB-T (1seg, SD and HD) and for the DVB-C and DVB-S providers I'm aware - although here, EPG data is reliable for most channels and there's no xmltv grabber for Brazilian channels - AFAIKT. > > > >> Finally, editing the channel, we would have a new drop down list with the > > with the problem fixed in the xmltv file, i do not see the need to add this > now, at least for me, it is easier to maintain the map in the script than in > any (one by one) GUI. I suspect that this is always the case. Mapping it on GUI has a disadvantage: if a channel changes and a re-scan is needed, one has to re-map everything. > But a feature would still be useful, a column in the channel config window > to show what channels already have epg and those that don't. The alternative > is to manually click each channel to find out. in my setup, 62 clicks to > find 27 channels with EPG and 35 without You don't need to go to the channel's config window. Just open the EPG guide from the TV menu. All channels with EPG are there. > > Sadly i do not know how to program, so i can't really helping develop this :) That's a good opportunity to learn :-) > all i can do is open a new "feature request", if someone wants later to pick > this up (if you agree that this is useful too)
> I'm aware - although here, EPG data is reliable for most channels and there's > no xmltv grabber for Brazilian channels - AFAIKT. Well, here the normal DVB-T do have good EPG, but i have fiber cable, that give all the channels using a operator tv box, also with EPG (and other features, like recording, rent movies, etc). That box (actually the router in my setup) also injecting in the home coax network about 60 analog channels and 60 DVB-T channels (mostly duplicated) for any other TV in the house without using the operator box. Are this "internal distribution" channels that lack the EPG. Using a tv card i can use tvtime to see the analog signal and a dvb-t card for the digital feed. the xmltv grabber for my operator, was i that studied their tv guid website and created a shell script to generate it. Later another guy used my findings and build a proper xmltv grabber > You don't need to go to the channel's config window. Just open the EPG guide > from the TV menu. All channels with EPG are there. yep, i found that, but that show the ones that DO have EPG... finding the ones without still forces one to list all channels and manually remove the ones that are missing. This feature can be in the EPG guide too, by default show only EGP channels (the current way) and pressing some bottom or checkbox, show all channels, even ones without EPG >> Sadly i do not know how to program, so i can't really helping develop this :) > That's a good opportunity to learn :-) I actually want to learn, but something more simple like python or go... c++, i have problems even trying to read it, probably because everything that uses c++ is already way too big for me to understand :D
> This feature can be in the EPG guide too, by default show only EGP channels > (the current way) and pressing some bottom or checkbox, show all channels, > even ones without EPG That's simple to implement. Added. > >> Sadly i do not know how to program, so i can't really helping develop this :) > > That's a good opportunity to learn :-) > > I actually want to learn, but something more simple like python or go... > c++, i have problems even trying to read it, probably because everything > that uses c++ is already way too big for me to understand :D Well, c++ is not more complex than python. What makes harder to understand a code are class inheritance and events handling, as it is harder to check who called a piece of the code. Modern languages have mechanism for those, so, with regards to code complexity, c++ is not more complex than python. Try to understand the src/dvb/xmltv.cpp parser. It doesn't contain too much of those.
>> (the current way) and pressing some bottom or checkbox, show all channels, >> even ones without EPG >That's simple to implement. Added. Would it be too much to put these channels text in grey, so its easier to identify them? I do not know how different themes work with that... alternative is some symbol or marker, like a × in the start or in the end, that may be more neutral to different kde themes. One again, many thanks!
> Would it be too much to put these channels text in grey, so its easier to > identify them? I do not know how different themes work with that... alternative > is some symbol or marker, like a × in the start or in the end, that may be more > neutral to different kde themes. That would be way harder. The code is basically selecting between a list of all channels or a list with EPG channels, each stored on different data structures. It could be possible to add a second column to the table with some symbol, but I'm currently lacking the time for a change like that.
> That would be way harder. The code is basically selecting between a list > of all channels or a list with EPG channels, each stored on different > data structures. ok, thanks, i understand... this isn't critical, it's just a nice to have thing :)