Bug 202366 - [PATCH] LyricWiki API no longer works; lyrics applet needs to be updated
Summary: [PATCH] LyricWiki API no longer works; lyrics applet needs to be updated
Status: RESOLVED FIXED
Alias: None
Product: amarok
Classification: Applications
Component: Tools/Script Manager (show other bugs)
Version: 2.3-GIT
Platform: Compiled Sources Linux
: HI normal
Target Milestone: ---
Assignee: Amarok Developers
URL:
Keywords:
: 203191 206354 (view as bug list)
Depends on:
Blocks:
 
Reported: 2009-08-03 06:05 UTC by Tyler Mandry
Modified: 2009-12-09 11:34 UTC (History)
12 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
makes the lyrics applet parse the lyricwiki website instead of using the API (which is no longer usable to retrieve lyrics) (2.67 KB, patch)
2009-08-03 15:53 UTC, Jakob Kummerow
Details
updated patch (fixes special characters in song title/artist) (3.00 KB, patch)
2009-08-03 17:37 UTC, Jakob Kummerow
Details
updated again (more elegant approach) (2.56 KB, patch)
2009-08-04 02:29 UTC, Jakob Kummerow
Details
screenshot (244.99 KB, image/png)
2009-08-19 22:27 UTC, Edward Karavakis
Details
additional patch (fixes special characters, again) (2.19 KB, patch)
2009-08-31 04:54 UTC, Jakob Kummerow
Details
fix redirection to lyrics.wikia.com (854 bytes, patch)
2009-09-02 04:53 UTC, Jakob Kummerow
Details
add internal functionality to replace the API call (8.47 KB, patch)
2009-09-04 19:31 UTC, Jakob Kummerow
Details
[updated] add internal functionality to replace the API call (9.06 KB, patch)
2009-09-05 00:17 UTC, Jakob Kummerow
Details
[updated again] add internal functionality to replace the API call (9.78 KB, patch)
2009-09-05 05:06 UTC, Jakob Kummerow
Details
[3rd update] add internal functionality to replace the API call (10.06 KB, patch)
2009-09-05 21:08 UTC, Jakob Kummerow
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tyler Mandry 2009-08-03 06:05:28 UTC
Version:            (using KDE 4.2.2)
OS:                Linux
Installed from:    Compiled From Sources

This appears now in the lyrics applet:

Unfortunately, due to licensing restrictions from some of the major music publishers we can no longer return lyrics through the LyricWiki API (where this application gets some or all of its lyrics).

The lyrics for this song can be found at the following URL:
http://lyricwiki.org/The_Rembrandts:The_Other_Side_Of_Night

(Please note: this is not the fault of the developer who created this application, but is a restriction imposed by the music publishers themselves.)


No fault of the developers, but the lyrics applet now needs to find another way to get lyrics.
Comment 1 Constantin Berzan 2009-08-03 14:36:05 UTC
An explanation of the situation: http://groups.google.com/group/lyricwiki-api/browse_thread/thread/733ccd919d654040

This just goes to show you how the music industry will try to screw you in every possible way :-/
Comment 2 Jakob Kummerow 2009-08-03 15:51:49 UTC
I created a quick-n-dirty hack for the lyrics applet script to make it parse the lyricwiki.org website instead of using the API. The patch needs to be applied to the file main.js of the applet, on my system (Gentoo) it's located at /usr/share/apps/amarok/scripts/lyrics_lyricwiki/main.js, I'm not sure about other distributions. Amarok does *not* need to be recompiled after applying the patch, closing and restarting it is enough :-)

I tested it with Amarok-2.1.1, but as far as I can see that particular file has not changed in trunk since then, so it should apply cleanly to current pre-2.2 versions as well. 

I'm not sure about the legal/licensing situation... On the one hand, using my patch Amarok shows lyrics internally just as it used to (which the music industry apparently doesn't approve of), on the other hand, the lyricwiki API is no longer used (at least not for accessing lyrics), but rather the usual, browser-viewable wiki website is parsed.
Comment 3 Jakob Kummerow 2009-08-03 15:53:30 UTC
Created attachment 35811 [details]
makes the lyrics applet parse the lyricwiki website instead of using the API (which is no longer usable to retrieve lyrics)
Comment 4 Myriam Schweingruber 2009-08-03 16:08:55 UTC
Can somebody have a look at this?
Comment 5 Robert Shaw 2009-08-03 16:29:37 UTC
I've played around with the above patch a bit and it works for most simple song titles. There are some songs that it doesn't work for and just displays 'null'.

Examples of songs not working:
Tool - Forty Six & 2
Coheed and Cambria - The Willing Well III: Apollo II: The Telling Truth
Coheed and Cambria - Always & Never
The Mars Volta - Roulette Dares (The Haunt of)

The following track does work:
Coheed and Cambria - Apollo I: The Writing Writer

So it seems it can at least handle one ':' in the title.

I have checked the LyricWiki page and the non-working tracks are present, with the exact same spelling.
Comment 6 Jakob Kummerow 2009-08-03 17:35:43 UTC
OK, thanks for testing (and confirming that the patch basically works).

A tiny bit of background information: the lyricwiki API is (and has always been) used to map from song information (title, artist) to an appropriate URL. I didn't change this. In short, lyricwiki decides itself whether the song is present in their database. 

Now, there was a problem with special characters, because the lyricwiki API returned an escaped URL, and the QUrl constructor used in the script automatically encoded it again. I fixed that by adding a manual decode before passing the value.

In the case of "Coheed and Cambria - The Willing Well III: Apollo II: The Telling Truth", the problem is outside my control. Here, the API call simply returns a wrong URL, namely "...Well Iii: Apollo Ii...", but wiki URLs are case sensitive. There's nothing, at least nothing simple, that Amarok, the lyrics applet or my patch could do against that (one would have to guess which letters of a URL have to be converted to uppercase - very error-prone). The unpatched version of the applet says "not found", so it's not a regression. The lyricwiki guys should fix their code.
Comment 7 Jakob Kummerow 2009-08-03 17:37:22 UTC
Created attachment 35813 [details]
updated patch (fixes special characters in song title/artist)
Comment 8 Tyler Mandry 2009-08-03 22:48:20 UTC
One problem is that the GIT version allows editing of the lyrics so people can contribute/make corrections. One way that might be better would be to open the edit page for these lyrics and parse the wiki code instead of HTML (of course you would have to pull the wiki code out of the HTML first), then if people edit the lyrics it can be saved as it is.
Comment 9 Jakob Kummerow 2009-08-04 02:28:53 UTC
As far as I can see, the editing feature does not update the online version at lyricwiki, but only the locally cached lyrics text. (Although it would surely be cool to be able to correct the online version from within Amarok!)

Using the edit page is a nice idea, though, as it allows to make the patch quite a bit more elegant :-)
Comment 10 Jakob Kummerow 2009-08-04 02:29:48 UTC
Created attachment 35832 [details]
updated again (more elegant approach)
Comment 11 Leo Franchi 2009-08-04 04:56:31 UTC
hokay, patch pushed to master. thanks!
Comment 12 Norberto Moritz Koch 2009-08-08 17:39:39 UTC
Works very fine.
Thank's a lot.
Comment 13 Kuba Serafinowski 2009-08-09 17:42:06 UTC
*** Bug 203191 has been marked as a duplicate of this bug. ***
Comment 14 Alejandro Cubero Mora 2009-08-15 20:33:02 UTC
Excellent work! thanks
Comment 15 Edward Karavakis 2009-08-19 20:55:54 UTC
any chance to backport it to 2.1.1?
Comment 16 Jakob Kummerow 2009-08-19 21:03:06 UTC
yes, you can easily apply the patch to amarok 2.1.1 - see comment #2

take a look at 'man patch' (or google) if you don't know how to apply patches.
Comment 17 Edward Karavakis 2009-08-19 21:57:22 UTC
I thought it was only valid for 2.2 - it works fine now, although sometimes I get the following error: 
"Lyrics were not able to be downloaded. Please check your internet connection: Could not retrieve lyrics: TypeError: not an object"
Cheers, Eddie
Comment 18 Edward Karavakis 2009-08-19 22:27:40 UTC
Created attachment 36286 [details]
screenshot

"Lyrics were not able to be downloaded. Please check your internet connection: Could not retrieve lyrics: TypeError: not an object" instead of "Lyrics not found".

Also, lyrics title doesn't change if there are no lyrics for the new song.
Comment 19 Jakob Kummerow 2009-08-19 22:49:26 UTC
Yeah, well, I know. But personally, I don't particularly care for pretty error messages. 

If it really bugs you, you can change line 51 of the patched file /usr/share/apps/amarok/scripts/lyrics_lyricwiki/main.js, which currently looks like this:

Amarok.Lyrics.showLyricsError( "Could not retrieve lyrics: " + err );

to print any other message that you think sounds sexy ;-)

As for not changing the title when no new lyrics are found, that's standard Amarok behaviour and has nothing to do with this patch. If you want to change that as well, I'd suggest something like the following instead of said line 51:

newxml = newxml.replace( "{lyrics}", "Lyrics not found. Please check your internet connection and make sure that title and artist tags are present and spelled correctly." );
Amarok.Lyrics.showLyrics( newxml );

@Amarok-devs: If you agree with Eddie that the lyrics script deserves a prettier error message, I hereby suggest to incorporate those last two lines into trunk.
Comment 20 Edward Karavakis 2009-08-19 23:24:10 UTC
Cheers! It's up to the devs now.
Comment 21 Bartosz Brachaczek 2009-08-31 01:23:53 UTC
Hey, great job, but one thing: your latest patch (in comment #10) doesn't work with special characters like "&", and your patch in comment #7 works okay. Could this be solved?
Comment 22 Jakob Kummerow 2009-08-31 04:52:36 UTC
OK, special characters should work now. 
I also improved the ugly error messages. 

The new patch applies to the *current trunk version* of main.js (so if you're patching amarok-2.1.1, apply the patch from comment #10 first, then the new one). 

<rant>
Damn, I hate encodings. Could someone please tell those musicians to use their creativity on the music and the lyrics instead of funny file names?
</rant>
Comment 23 Jakob Kummerow 2009-08-31 04:54:07 UTC
Created attachment 36586 [details]
additional patch (fixes special characters, again)
Comment 24 Bartosz Brachaczek 2009-08-31 12:56:22 UTC
Yep, works again. Big thanks! Hope that the new patch will be committed soon.
Comment 25 Mark Kretschmann 2009-08-31 13:33:54 UTC
@jkummerow: I'm confused, is this patch for 2.2-GIT? Then I could commit it.
Comment 26 Jakob Kummerow 2009-08-31 14:29:42 UTC
@Mark: yes, that's what I meant. 

The remark about needing the patch from comment #10 first (which is already committed to GIT, see comment #11) was intended for users of Amarok-2.1.1 who happen to stumble upon this bug.
Comment 27 Mark Kretschmann 2009-08-31 15:01:17 UTC
Thanks, I've pushed the patch from comment #23 to Git:


commit 10e215482ea3c616ea2d24bd25bf3e9581789b44
Author: Mark Kretschmann <kretschmann@kde.org>
Date:   Mon Aug 31 14:52:58 2009 +0200

    Patch for LyricWiki script, supposedly fixes special character handling.

    I can't really tell if the patch is correct, but lyrics still seem to load
    fine, so I guess it's OK. Thanks go to jkummerow@web.de :)
Comment 28 Jakob Kummerow 2009-09-02 04:50:48 UTC
This is driving me crazy... I just noticed that lyrics fetching is broken *again*. Turns out that lyricwiki.org doesn't seem to exist anymore; trying to access one of the (previously correct) lyrics URLs in a browser results in being redirected to lyrics.wikia.com. 

Fortunately, this was easy to fix: the patch is a one-liner that applies to the current GIT master. 

Enjoy.
Comment 29 Jakob Kummerow 2009-09-02 04:53:27 UTC
Created attachment 36620 [details]
fix redirection to lyrics.wikia.com
Comment 30 Lydia Pintscher 2009-09-02 11:16:33 UTC
reopening so someone looks at it.
Thanks for providing those patches :)
Comment 31 Leo Franchi 2009-09-02 15:53:02 UTC
updated, thanks!
Comment 32 Jakob Kummerow 2009-09-04 19:29:47 UTC
Finally, it happened: The API went down, and with it the previous solution to map from artist+title to wiki article URLs. 
The new patch contains a simple replacement for the API call to make the lyrics script work again in the majority of cases. 

The bad news is: The artist/title tags of your music files must match the spellings on lyrics.wikia.com quite closely; the only correction that is performed is upper/lower case, and if no lyrics are found a second attempt is made after prepending "The" to the artist. 
The good news is: uppercase conversion is more intelligent than it used to be, the example above that didn't work before should work now (see comment #6 last paragraph, Roman numerals). 

Once again, the patch applies to the current GIT master (which is to say to a version of main.js that has already had all the previous patches applied to it).
Comment 33 Jakob Kummerow 2009-09-04 19:31:05 UTC
Created attachment 36692 [details]
add internal functionality to replace the API call
Comment 34 Jakob Kummerow 2009-09-05 00:17:29 UTC
Created attachment 36701 [details]
[updated] add internal functionality to replace the API call

Seeing that my previous patch has not been committed to GIT yet, I present an updated version. 

fixed: an infinite loop that occurred when no lyrics were found (shame on me for not seeing this earlier!)
improved: if an artist tag contains "ft.", "feat." or "featuring", everything after that is now stripped to make lyrics retrieval for such songs work
Comment 35 Jakob Kummerow 2009-09-05 05:06:56 UTC
Created attachment 36704 [details]
[updated again] add internal functionality to replace the API call

more testing, more small improvements. we're getting closer to the API's functionality :-)

specifically, I removed the automatic lowercase-conversion: what's spelled with capital letters stays capitalized for the request. Small disadvantage: there might be cases where you have to correct your tags now, if they contain too many capital letters. But the change was necessary because there are some crazy spellings that no automatic script could possibly get right (such as "T.A.T.u.", or "A-ha" [lowercase after -] versus "N-Trance" [uppercase after -]). 
Capitalization of the first letter of each word is still performed.
Comment 36 Mikko C. 2009-09-05 12:47:26 UTC
*** Bug 206354 has been marked as a duplicate of this bug. ***
Comment 37 Mikko C. 2009-09-05 12:47:56 UTC
Reopening, since it broke again.
Comment 38 Myriam Schweingruber 2009-09-05 13:36:55 UTC
I really doubt it makes much sense to reopen that over and over again, we need another reliable lyrics source, and I very much fear this will not happen soon. Blame the RIAA...
Comment 39 Mikko C. 2009-09-05 14:34:07 UTC
The last patch works fine for me, so far.
Comment 40 Jakob Kummerow 2009-09-05 21:03:17 UTC
Myriam: I think we should continue trying to keep the lyrics script up to date. Of course, a website that you don't have control over always is a moving target, and some breakage from time to time is to be expected. Also, since they recently took an important tool (the API) away from us and we have to make up for the lost functionality, a couple more evolutionary steps will probably be necessary until the lyrics script works as reliably as it should. 

But despite this, I do think that it's worth it, because there is no viable alternative, and personally I think that the lyrics display is one of Amarok's greatest features. And I'm willing to keep updating the script from time to time, as needs for fixes or ideas for improvements are found. Also, while it's true that the recent history of lyricwiki.org/lyrics.wikia.com has been quite turbulent, I think (or should I say hope?) that this was a period of transition that will be over soon, so that we will have more stability in the future. 

Nonetheless, if and when someone finds a better long-term solution, we can always switch to that, right? ;)
Comment 41 Jakob Kummerow 2009-09-05 21:08:56 UTC
Created attachment 36724 [details]
[3rd update] add internal functionality to replace the API call

Thanks to Mikko's testing, some more refinements.
- strip HTML comments from lyrics
- switched back to parsing the normal wiki page (instead of the edit page); this has three advantages: (1) it reduces the amount of character conversion work the script has to do, (2) it can cope with wiki entries with "<lyric>" instead of "<lyrics>" tags, (3) it loads considerably faster: while the script felt laggy before, it's quite snappy now!
Comment 42 Claus Appel 2009-09-06 16:57:51 UTC
I have a suggestion: MP3 and OGG files have a "Lyrics" tag where the user can add lyrics to the file. I suggest that the Amarok Lyrics applet use the "Lyrics" tag as a backup source. Meaning:

1. First check LyricWiki or whatever for lyrics.
2. If that fails for any reason, then try to read the lyrics from the file's tags.
3. If that also fails, then display an error message. This error message could advise the user to add lyrics to the file's tags.
Comment 43 Jakob Kummerow 2009-09-06 17:02:51 UTC
Good idea - Amarok already does that. As far as I can tell, it works very reliably in 2.2-Beta1.
Comment 44 Christian Morales Vega 2009-09-06 19:52:55 UTC
Shouldn't lyricwiki be contacted so they can explain their plans? I mean, two days ago http://lyrics.wikia.com/api.php returned something different.

Not sure which would be the best way to contact them though. Looking at it, at http://lyrics.wikia.com/lyrics/User_talk:Sean_Colombo and http://lyrics.wikia.com/lyrics/User_talk:LYRIC-Sean_Colombo some people asks for the source code if the API can't be made available.
Comment 45 Martin Sandsmark 2009-09-10 22:12:52 UTC
I just applied the patch; g9b3dba6.