Bug 256896

Summary: Look up spellchecking dictionaries using full language+country code (was: Lokalize spelling the original strings, not my translations)
Product: [Applications] lokalize Reporter: Mario Santagiuliana <fedora>
Component: generalAssignee: Nick Shaforostoff <shafff>
Status: RESOLVED FIXED    
Severity: normal CC: azamat.hackimov, belegdol, fedora, kevin.kofler, mkyral, rdieter
Priority: NOR    
Version: 1.1   
Target Milestone: ---   
Platform: Fedora RPMs   
OS: Linux   
Latest Commit: Version Fixed In:
Attachments: Screenshot view of the problem

Description Mario Santagiuliana 2010-11-14 16:34:31 UTC
Version:           1.1 (KDE 4.5) (using KDE 4.5.2) 
OS:                Linux

I update my Fedora system to Fedora 14.

Lokalize now check the words only in the original strings view. It just underlines the wrong words in English language in this view.

If I try to spell my translation with tools->spelling it doesn't report my anything.
If I translate something with wrong words in Italian(second view, the translation), lokalize report me anythings, no underline.

Reproducible: Always
Comment 1 Nick Shaforostoff 2010-11-15 11:06:07 UTC
are you sure you have the spell dictionary for your language?
how do other apps behave? (like kate or kwrite)
Comment 2 Mario Santagiuliana 2010-11-15 20:23:01 UTC
Yes. I use frequently kmail and it works without problems. I don't use kwrite or kate in my work...I try them now and with them I can check spell.
Comment 3 Mario Santagiuliana 2010-11-18 15:30:41 UTC
I create new user and I have got the same problem...
I kate and kwrite (kmail too) I can configure spelling selecting the language. In lokalize not. Can this be a problem?
Comment 4 Nick Shaforostoff 2010-11-26 14:51:34 UTC
the problem may be that the target language is set wrong.
Comment 5 Nick Shaforostoff 2010-11-26 14:53:16 UTC
can you send me the file you translate?
and the project file you're using (extension: .lokalize)
Comment 6 Mario Santagiuliana 2010-11-26 18:24:01 UTC
I send you my files.

I report here the log of lokalize when I start it from bash:
$ lokalize 
278 
279 
280 
Enchant dict for "it_IT" 0x2360ca0 
Enchant dict for "it_IT" 0x2360ca0 
lokalize(6776) LokalizeMainWindow::setupActions: finished 30 
lokalize(6776)/lokalize (TM) TM::DBFilesModel::DBFilesModel: initial row count 1 
lokalize(6776) main: showing LokalizeMainWindow 
lokalize(6776)/lokalize (TM) TM::OpenDBJob::run: db "default"  opened  11 "it" 
lokalize(6776)/lokalize (TM) TM::OpenDBJob::run: db "default"  opened  0 "it" 
lokalize(6776)/lokalize (TM) TM::OpenDBJob::run: db "default"  opened  0 "it" 
lokalize(6776) main: LokalizeMainWindow shown 
loadProjectScripts() 1111 "/home/lokalize" 
loadProjectScripts() 333 
lokalize(6776) ProjectScriptingPlugin::~ProjectScriptingPlugin: ("scripts.rc") 
loadProjectScripts() 444 
loadProjectScripts() 555 
loadProjectScripts() 666 
Enchant dict for "it_IT" 0x2360ca0 
Enchant dict for "it_IT" 0x2360ca0 
lokalize(6776) Catalog::loadFromUrl: file opened in 15 
autoSave QObject(0x0)  
Enchant dict for "it_IT" 0x2360ca0 
Enchant dict for "en_US" 0x31389b0 
Enchant dict for "it_IT" 0x2360ca0 
lokalize(6776): No language dictionaries for the language :  "it" 

lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showmsgiddiff_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showmergeview_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showmergeviewsecondary_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showcatalogtreeview_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showmsgctxt_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showtmqueryview_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showglossaryview_action" with KXMLGUIFactory! 
lokalize(6776)/kdeui (kdelibs): Attempt to use QAction "showbinunitsview_action" with KXMLGUIFactory! 
lokalize(6776) ProjectScriptingPlugin::~ProjectScriptingPlugin: ("scripts.rc") 
lokalize(6776)/kdecore (KConfigSkeleton) KCoreConfigSkeleton::writeConfig:
lokalize(6776)/kdecore (KConfigSkeleton) KCoreConfigSkeleton::writeConfig:
QCoreApplication::processEvents()... 
Finishing Project jobs... 
Finishing TM jobs... 

** (process:6776): WARNING **: 1 dictionaries weren't free'd.

klauncher(6221)/kio (KLauncher): SlavePool: No communication with slave. 

I don't undesrtand why I need to set the language in kwrite (for example) to have the correct spelling when I just set the language in my systemsettings.
Comment 7 Marian Kyral 2010-12-27 17:49:14 UTC
Hi,
I have the same issue (gentoo, kde 4.5.90).

lokalize(17177): No language dictionaries for the language :  "cs" 
lokalize(17177): No language dictionaries for the language :  "cs"
lokalize(17177): No language dictionaries for the language :  "cs" 
lokalize(17177): No language dictionaries for the language :  "cs" 
lokalize(17177): No language dictionaries for the language :  "cs" 

After switch of language in systemsettings (there was "Czech", changed to "Czech (Czech republic)"), message changed:

Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 
Enchant dict for "cs_CZ" 0x21a45d0 

But only dialog about "End of file reached, continue from beginning?" and another dialog with "Lokalize finished the spell check" messages are shown. And nothing is underlined, even there are misspells in text.

Lokalize should complain about missing spell dictionary or about issues with spellcheck.
Comment 8 Mario Santagiuliana 2011-03-01 14:18:29 UTC
The problem still present on my fedora. Any other news?
I open a new bug in Fedora bugzilla system here:
https://bugzilla.redhat.com/show_bug.cgi?id=681208
Comment 9 Kevin Kofler 2011-03-01 15:47:29 UTC
*** Bug 266200 has been marked as a duplicate of this bug. ***
Comment 10 Kevin Kofler 2011-03-01 15:49:30 UTC
As I explained in the duplicate, the hunspell dictionaries are called cs_CZ,
it_IT, pl_PL etc., not just cz, it, pl etc.

See also bug 171432, which is about Parley being broken the same way.
Comment 11 Nick Shaforostoff 2011-04-14 23:46:41 UTC
wouldn't the nicer solution for the problem be creating symlinks for the dictionaries?

like pl -> pl_PL
Comment 12 Nick Shaforostoff 2011-04-14 23:49:59 UTC
or patching enchant/hunspell ot try more options, i.e. add country code to the langcode if it consists of two letters and no dict was found
Comment 13 Kevin Kofler 2011-04-14 23:57:41 UTC
Well, e.g. de_AT and de_CH are not the same dictionary as de_DE, just "de" is not specific enough.

Parley already got fixed for this very issue (see bug 171432).
Comment 14 Mario Santagiuliana 2011-06-08 16:31:57 UTC
The problem  still persist on my fedora. I update kde packages but no solution.
It is possible to check translated text area using short menu after right click mouse button. But I don't see the red line under the wrong text. I see this thing in the original language text area.

How can I help you in the lokalize code?
Comment 15 Azamat H. Hackimov 2011-07-06 13:00:42 UTC
I have same problem, but not only with lokalize. kwrite/kate affects to this bug too.
My workaround - since enchant cannot use hunspell xx_YY dicts, I switched enchant to aspell.
Comment 16 Mario Santagiuliana 2012-01-08 12:23:32 UTC
Bug still present on Fedora 16:
$ rpm -q kdesdk-lokalize
kdesdk-lokalize-4.7.4-1.fc16.x86_64

I will try to create a patch...hoping to resolve the problems...can anybody help me?
Comment 17 Nick Shaforostoff 2012-01-08 16:55:37 UTC
what is displayed in the tooltip of translation window for you?
and, if possible, please also give me list of dicts for your language installed,
and the speller you're using.
Comment 18 Mario Santagiuliana 2012-01-08 17:13:11 UTC
Created attachment 67586 [details]
Screenshot view of the problem
Comment 19 Mario Santagiuliana 2012-01-08 17:14:05 UTC
Thank you for your answer.
In attachment I create a screenshot of the problem. Hope this help to understand better.

Dictionary should be these:
$ rpm -qa|grep spell|sort
hunspell-1.3.2-1.fc16.x86_64
hunspell-en-0.20110318-1.fc16.noarch
hunspell-it-2.4-0.6.20070901.fc15.noarch

And the speller should be hunspell.
Comment 20 Nick Shaforostoff 2012-01-10 09:10:14 UTC
i cannot see a tooltip in the screenshot. to get a tooltip you move your mouse cursor over the translation text edit area and wait for few seconds without doing anything with mouse nor keyboard.
Comment 21 Mario Santagiuliana 2012-01-10 09:31:53 UTC
Sorry Nick...the tooltip in the text edit area report just "()". No other.
In the text area of original string report to me "Inglese Americano (en_US)".
Comment 22 Mario Santagiuliana 2012-01-14 09:27:35 UTC
For me is really hard to understand lokalize code...I don't know very well c++ and the code is not much commented...is there any news about this bug?
Comment 23 Mario Santagiuliana 2012-01-14 11:36:42 UTC
I try somethings...If I open spellchecker from right click menu I can select a different language, for Italian I have got three languages:
- Italian
- Italian (Italy)
- Italian (Switzerland)

From default I have got "Italian". If I change to "Italian (Italy)" the spellcecker and highlight work correctly and tooltip report (it_IT).

If I close lokalize and reopen it the tooltip report to me "()" and I cannot open the ortographic checker from right click menu. I need to move to other string translation, I can open the ortographic checker and I see that language is set up to "Italian (Italy)". I change language to another and than change language again to "Italian (Italy)", now highlight work correctly.

I look a little bit into the lokalize code, I see in src/common/stemming.cpp:

 SpellerAndCodec::SpellerAndCodec(const QString& langCode)
 : speller(0)
 {
         QString dic=QString("/usr/share/myspell/dicts/%1.dic").arg(langCode);
         if (QFileInfo(dic).exists())
         {
             speller = new Hunspell(QString("/usr/share/myspell/dicts/%1.aff").arg(langCode).toUtf8().constData(),dic.toUtf8().constData());
             codec=QTextCodec::codecForName(speller->get_dic_encoding());
             if (!codec)
                 codec=QTextCodec::codecForLocale();


The directory /usr/share/myspell/dicts/ doesn't exist, all files are in /usr/share/myspell/. I try a patch here but the problems seems to not change.
All the time I reopen lokalize the tooltip report "()" I must reselect my language to see in tooltip (it_IT).

And also, there is another bug too, I don't know if related with this, I change my language in spell checker from right menu and the text in that edit box is copied and pasted twice...I can see the string copy only when I change string and when I return to the first.

These are really annoying, I can use lokalize to review translation, I can't use lokalize to make translation, I must, all the time recheck my po file with other application to be sure all is fine...
Comment 24 Kevin Kofler 2012-01-14 16:42:02 UTC
> and the speller you're using.

He's using hunspell, the default spell checker in Fedora.

Please test your code using (only) hunspell! Aspell is:
* obsolescent,
* not supported by Firefox nor LibreOffice (they only support hunspell),
* not the default in upstream Enchant (which in turn is the default Sonnet backend), it prefers hunspell if available,
* technically inferior to hunspell: aspell's support for morphology is much more limited, hunspell can even handle Hungarian (which is where the name comes from),
so we cannot possibly be the only distribution shipping hunspell as the default. And hunspell is also used directly by Lokalize itself for stemming!

I recommend uninstalling aspell completely to make sure aspell dictionaries won't get picked up by Enchant, probably masking this bug.

See also: https://fedoraproject.org/wiki/Releases/FeatureDictionary


Mario, re comment #23, I think that code only does stemming, the spellchecking goes through Sonnet.
Comment 25 Kevin Kofler 2012-01-14 16:44:27 UTC
(or at the very least, ensure you do NOT have aspell dictionaries installed for the language(s) you're testing)
Comment 26 Mario Santagiuliana 2012-01-14 17:23:30 UTC
Thank you Kevin for your answer.
The code I report should do only stemming...I don't know exactly how it works...
Changing the code using this directory: /usr/share/myspell/ , doesn't change my problem. I resume it here the step to reproduce it:
- launching lokalize and open a po file;
- lokalize doesn't underling the wrong words (tooltip report "()");
- run spellchecker (should be Sonnet, I think) from quick menu with right click in edit text area, it works but first I must all the times select my language otherwise not;
- exit from spellchecker window and lokalize underling the wrong words (tooltip report "(it_IT)");
- move to the next string to translate and return to previous, lokalize copy and paste at the end the string just translated;
- close lokalize;
- reopen it, the problem remain;
- fix underlining with previous step;
- open a second po file same problem.

I just set in my project the destination language to "Italian (it)", there is not "Italian (it_IT)".

Running lokalize from terminal:
$ lokalize 
282 
283 
284 
Enchant dict for "it_IT" 0x296b000 
Enchant dict for "it_IT" 0x296b000 
lokalize(26075) LokalizeMainWindow::setupActions: finished 14 
lokalize(26075)/lokalize (TM) TM::DBFilesModel::DBFilesModel: initial row count 0 
accessing tm db config true use cache: true 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "default"  opened  4 "it" 
accessing tm db config true use cache: true 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "fedora-trans-it"  opened  1 "it" 
lokalize(26075) main: showing LokalizeMainWindow 
lokalize(26075) main: LokalizeMainWindow shown 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "default"  opened  0 "it" 
accessing tm db config true use cache: true 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "kde-it"  opened  1 "it" 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "fedora-trans-it"  opened  0 "it" 
accessing tm db config true use cache: true 
lokalize(26075)/lokalize (TM) TM::OpenDBJob::run: db "scripts"  opened  1 "it" 
loadProjectScripts() 1111 "/home/mario/traduzioneFedora" 
loadProjectScripts() 333 
lokalize(26075) ProjectScriptingPlugin::~ProjectScriptingPlugin: ("scripts.rc") 
loadProjectScripts() 444 
loadProjectScripts() 555 
loadProjectScripts() 666

opening a po file from the project add other lines, interesting this (I report the last 3 lines from previous output too):

loadProjectScripts() 444 
loadProjectScripts() 555 
loadProjectScripts() 666 
Enchant dict for "it_IT" 0x296b000 
Enchant dict for "it_IT" 0x296b000 
lokalize(26075) Catalog::loadFromUrl: file opened in 15 
autoSave QObject(0x0)  
Enchant dict for "it_IT" 0x296b000 
Enchant dict for "en_US" 0x3af6fb0 
Enchant dict for "it_IT" 0x296b000 
lokalize(26075): No language dictionaries for the language :  "it" 

lokalize(26075)/lokalize (TM) TM::ScanJob::run: started  [.....file opened and other lines.....]

P.S. sorry for my English in reporting issues...
Comment 27 Mario Santagiuliana 2012-01-15 17:47:59 UTC
Installing aspell packages seems not resolve the problem...I remove them now.
Comment 28 Nick Shaforostoff 2012-01-18 14:20:48 UTC
i failed to reproduce the issue.

what i did:
svn up trunk/l10n-kde4/it
sudo apt-get install iitalian aspell-lt myspell-it

then i created a project in Lokalize.
$ cat trunk/l10n-kde4/it/index.lokalize 
[General]
LangCode=it
ProjectID=kde-it
TargetLangCode=it

then i opene this project in Lokalize and a file in it.
The spellchecking worked, and the tooltip displayed 'Italian (it)'
Comment 29 Mario Santagiuliana 2012-01-18 15:02:15 UTC
Thank you for your test. Please see comment #24. I don't use aspell, it is obsolete. Don't install aspell package, use hunspell instead.
Also, installing aspell package in Fedora I don't resolve my problem.
Comment 30 Nick Shaforostoff 2012-01-18 15:21:22 UTC
ok, so this way  i'm able to reproduce the problem, What is interesting is that running spellcheck via Tools->Spellcheck does work, and the dictionary is autoselected (i don't have to choose it from combobox).

note that running spellcheck from context menu is not supported, please use the Tools menu instead. It will do the spellcheck in all translation units starting from the current one.
Comment 31 Mario Santagiuliana 2012-01-18 15:28:39 UTC
(In reply to comment #30)
> ok, so this way  i'm able to reproduce the problem, What is interesting is that
> running spellcheck via Tools->Spellcheck does work, and the dictionary is
> autoselected (i don't have to choose it from combobox).
I have got the same problem in comment #7. If I insert wrong worlds likalize doesn't show them...

> note that running spellcheck from context menu is not supported, please use the
> Tools menu instead. It will do the spellcheck in all translation units starting from the current one.

I can do a spellcheck only from context menu...in comment #26 you can see the step to reproduce exactly the bug...
Comment 32 Nick Shaforostoff 2012-01-18 15:38:25 UTC
> I can do a spellcheck only from context menu
why?
Comment 33 Mario Santagiuliana 2012-01-18 15:44:01 UTC
(In reply to comment #32)
> > I can do a spellcheck only from context menu
> why?

I don't know...I try to understand lokalize code but I am not a C/C++ developer :)

The spellcheck is only inside the text area where I start spelchecking...not in all po file but then tooltip report it_IT view comment #26...
Comment 34 Nick Shaforostoff 2012-01-18 15:51:00 UTC
i didn't understand me.

please do spellcheck by selecting Tools->Spellcheck from the app menu, and not from the context menu. Spellcheck from context menu is broken in Lokalize.
Comment 35 Nick Shaforostoff 2012-01-18 15:54:07 UTC
i could get on-the-fly spellcheck working by installing hunspell-en-us (or myspell-en-us) and choosing it in system settings -> locale -> spellcheck as default language for spellcheck.

it is a bug in kdelibs/sonnet that stops on-the-fly spellcheck from working if no dictionary for the default language can be found
Comment 36 Mario Santagiuliana 2012-01-18 16:11:37 UTC
(In reply to comment #34)
> i didn't understand me.
> 
> please do spellcheck by selecting Tools->Spellcheck from the app menu, and not
> from the context menu. Spellcheck from context menu is broken in Lokalize.

??? No...for me it is the contrary...from app menu I never have a working spellchecker...similar to comment #7

In comment #19 you can find my hunspell library. I have installed Italian and English library...

Nick I don't understand if you don't believe me and my bug report...I have got this problem from one year (more or less), it is frustrating , I try to understand lokalize code to search a bugfix but I am not a C/C++ developer...
Comment 37 Kevin Kofler 2012-01-18 16:17:54 UTC
> Installing aspell packages seems not resolve the problem...I remove them now.

In Fedora, aspell will not get picked up by default by Sonnet, you also have to install enchant-aspell. (We don't build Sonnet against aspell-devel and we don't install Enchant's aspell backend by default.)
Comment 38 Kevin Kofler 2012-01-18 16:18:48 UTC
Oh, and since I've seen this personally too, I'm setting this bug to confirmed status.
Comment 39 Nick Shaforostoff 2012-01-18 16:39:10 UTC
SVN commit 1274522 by shaforo:


request spell dictionaries with their full code (language_COUNTRY),
because this is the format hunspell/myspell have their dicts named in



 M  +11 -0     common/stemming.cpp  
 M  +1 -1      common/stemming.h  
 M  +2 -1      editortab_findreplace.cpp  
 M  +3 -2      xlifftextedit.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1274522
Comment 40 Nick Shaforostoff 2012-01-18 16:40:21 UTC
SVN commit 1274523 by shaforo:

backport r1274522


 M  +11 -0     common/stemming.cpp  
 M  +1 -1      common/stemming.h  
 M  +2 -1      editortab_findreplace.cpp  
 M  +3 -2      xlifftextedit.cpp  


WebSVN link: http://websvn.kde.org/?view=rev&revision=1274523
Comment 41 Kevin Kofler 2012-01-18 16:48:12 UTC
Changing the subject to the resolution which was actually implemented.

Thanks for fixing this longstanding bug!
Comment 42 Mario Santagiuliana 2012-01-18 20:26:32 UTC
Bug is fixed on upstream! Thank you Nick for your work!
Comment 43 Nick Shaforostoff 2012-01-18 21:06:30 UTC
and thank you all for continuosly bringing my attention to the issue. with reallife going on it's not always easy to prioritize and remember things ;)