Bug 66411

KDE 3.1.5 from Aurox distro (Redhat 9)

German and Polish characters appear as squares when using sftp:// protocol.

When I create a file with Polish and German characters :
       sftp://localhost      - shows squares and names are shortened
       file:/                - both ok
       fish://localhost      - only german are ok, but since utf8 is NOT selected, this is understood

Polish characters were created using iso8859-2.

Using QString::latin1() and QString::fromLatin() doesn't preserve these characters, so maybe these calls in kio/sftp could be the problem ?

To understand,

here is a single snapshot with

KDE 3.2.2

Umlauts are shown als "nul"

ftp://ftp.eui.fh-koblenz.de/pub/studenten/Duck.Donald/55.png

PaddyX

Here is a screenshot from Debian unstable (KDE 3.2.2-1)

http://users.utu.fi/mijutu/bugreports/sftpbug.png

I'm using fedora core 1 with kde 3.2.2. Using sftp://<host>/<some_directory> in konqueror to display directory with filenames with Polish characters with utf8 encoding works correctly, but files can not be opened. There is message being displayed, i.e.:
"File or directory sftp://localhost/home/pawel/łąka doesn't exist". 
The file exists of course, I created it for test purposes.

*** Bug 83410 has been marked as a duplicate of this bug. ***

*** Bug 71258 has been marked as a duplicate of this bug. ***

*** Bug 85929 has been marked as a duplicate of this bug. ***

As sftp also depends on the server: as anybody tried to use the sftp program directly? Does it works?

Have a nice day!

Yes, it does:

unix.utu.fi:~> echo foo >ööö.txt

s:~$ sftp mijutu@unix.utu.fi
Connecting to unix.utu.fi...
mijutu@unix.utu.fi's password: 
sftp> get ööö.txt
Fetching /home/fysop/96/mijutu/ööö.txt to ööö.txt
sftp> quit
s:~$ cat ööö.txt
foo

The API for this is present in KDE 3.3. 

Maintainer: please make kio_sftp use KRemoteEncoding as kio_ftp does.

sftp works for me, but fish isn't.

fish works fine for me. If it doesn't on your machine, check what it shows using ssh. 
I also get the squares with sftp in KDE 3.3, so I tried different settings for "Remote Charset", but that had no effect. 
However, when I create a file with non-ascii characters, it shows up correctly under fish (and ssh console). KRemoteEncoding seems to be partially implemented, as I understand it.

Pure ssh works fine, I can use and handle UTF-8 names just fine. When using fish I get some iso-8859-1, I think. I can't find any options in Konqueror to change encoding. It isn't just that the names aren't displayed properly. When I try to upload/download, it's unable to actually find a file with that name. So it's practically impossible to do anything with a file like that using fish, I have to utilize ssh.

CVS commit by adawit: 

- Support for remoteEncoding(). Need feedback from people that connect 
  to non-latin1 sftp servers to verify if this works properly...

- Use KUser instead of own concaction.

- Avoid duplicate error reporting.

CCMAIL:66411@bugs.kde.org


  M +165 -173  kio_sftp.cpp   1.59
  M +8 -25     kio_sftp.h   1.14

*** Bug 94159 has been marked as a duplicate of this bug. ***

Hello Dawit,

with SuSE 9.2 (KDE 3.3.0), I can display directories with german Umlauts correctly, but I cannot download or rename such a file!
a) should your patch be in 3.3.0?
b) of not, could you provide a patch for kde 3.3.0, so I can test this?

I really would like to get this work!

KDE 3.3.2, under Gentoo.
The bug still exists.
With sftp, non-ascii caracters will be displayed, but files with names containing such caracters won't be opened.
(with fish or ftp, KDE assumes the remote encoding is ISO-latin1 and shows rubbish, but the files can be opened ...)

The fix for this issue did not make it into the 3.3.0 release which I believe 
is what comes with SUSE 9.2. The fix should be in the upcoming 3.4 release 
which is currently in Release Candidate 1. 

On Tuesday 08 March 2005 15:44, you wrote:
> Hi Dawit, im confused about using unicode with kde and ftp services, i
> dont know if im leading with a bug or something, hope dont bother you
> can help me.
>
> I speak spanish, so i use latin characters, i have a suse 9.2 with
> default lang=en_US.UTF-8 , so im using UTF enconding for my filenames (i
> attached my language file at the end of the mail). The problem is when i
> use konqueror or kbear to connect to another site with sftp.
>
> If connect to another UTF site i can see the filenames correctly, but
> when i try to get a file (named with special chars )  i receive a
> message saying that the file doesnt exist, i looks like there is a
> problem with UTF, maybe related or similar with this bug
> http://bugs.kde.org/show_bug.cgi?id=66411
> But its pretty weird that it was opened on 2003 and still remains as
> "new", i mean, if its a bug it should be so common because all the
> people using different languages around the world.
> Anyway i saw your post and for that reason im writting to you.
>
> But i have another problem, i have an old solaris 8 machine with
> lang=en_US.ISO8859-1, when i connect i see the files with latin names in
> a very weird manner, i suppose the application "think" is unicode but is
> not, this case i can transfer the files but with the funny name.
>
> I also did another test changing sftp by ftp, and it seems that the
> application assumes a default ISO8859-1 always, so when i connect to the
> UTF site the chars are ugly and when i do the same to my solaris the
> files are ok :(.
>
> As you can see im really confused, i dont know if its possible to have
> my own encoding and work fine with whatever site i connect or if i
> should stardarize all the locales in my net, or what im doing wrong or
> if it is a bug.
>
> thanx a lot for your time
>
> Carlos

I've just retested with 3.4 rc1 (debian packages from http://pkg-kde.alioth.debian.org/kde-3.4.0/) and the bug is still there

Steph

Does anyone have a publicly available sftp where I can test this at ?

Yes, here it is:
sftp://testsftp@lispc45.epfl.ch
pwd: testsftp

lispc45.epfl.ch's locales are UTF8. My laptop's locales are UTF8. Konqueror's SFTP uses latin1.

Steph

Well I tested against that server and I cannot duplicate the problem. Here is a screenshot of what I get with the current CVS version of KDE which has the same sftp io-slave as the 3.4 RC1:

Result of listing the directory in konqueror:
http://members.cox.net/kdelists/screenshots/konq-sftp-list.png

Result of clicking on the file with non-ascii text:
http://members.cox.net/kdelists/screenshots/konq-sftp-saveas-dlg.png

Result of clicking on Open with 'KWrite':
http://members.cox.net/kdelists/screenshots/kate-opening-non-ascii-file.png

The only difference with my setup is that my locale is not UTF8...

Dawit: sorry, but all your screenshots show you *could* duplicate the problem, not that you couldn't.

The file whose name you saw as "testaccent_éèà" should have been seen as "testaccent_éèà".

On Friday 11 March 2005 09:17, Thiago Macieira wrote:
[bugs.kde.org quoted mail]

Well then I do not understand what is going on. The sftp io-slave user the 
remoteEncoding() function from SlaveBase to encode all pathes just like the 
ftp io-slave does. Also people reported that they are unable to open the file 
which is not the case at all for me. Maybe I need to change my locale to UTF8  
and see if that changes things...

Hi,

The test server has changed name and is now:
sftp://testsftp@lissrv3.epfl.ch

Steph

> Well then I do not understand what is going on. The sftp io-slave user the 
> remoteEncoding() function from SlaveBase to encode all pathes just like the 
> ftp io-slave does.

Indeed, but it's not working. I'll debug the code this weekend to see where the real problem is.

Here is the example of this bug:
http://www.iitis.gliwice.pl/~gawron/ja/sftpkdebug.png
Both windows represent same directory. Upper one is oppened by sftp,
bottom localy.
Local and remote systems are standard Mandrake 10.1 boxes set to pl_PL
locale.

Piotr Gawron

CVS commit by thiago: 

Found the bug and fixed it. It was quite simple: the file attribute struct
was erasing the encoding before it was even first used.

To be backported.
BUG:66411


  M +0 -1      sftpfileattr.cpp   1.15


--- kdebase/kioslave/sftp/sftpfileattr.cpp  #1.14:1.15
@@ -295,5 +295,4 @@ void sftpFileAttr::clear(){
     mLongname = "\0";
     mLinkType = 0;
-    mEncoding = "\0";
 }
 

CVS commit by thiago: 

(Backport 1.14:1.15) Backporting the bug fix to KDE 3.4.x.

CCBUG:66411


  M +0 -1      sftpfileattr.cpp   1.14.2.1


--- kdebase/kioslave/sftp/sftpfileattr.cpp  #1.14:1.14.2.1
@@ -295,5 +295,4 @@ void sftpFileAttr::clear(){
     mLongname = "\0";
     mLinkType = 0;
-    mEncoding = "\0";
 }
 

Hi, im currently using this package kdebase3-3.4.0-33,  and the problem is still present; i also reviewed the cvs source tree and it seems the fix was removed in recent revisions.

On the other hand, im concerned that maybe the problem is not in the clear method but in here:

-------------------------------------------------
/** Use to read a file attribute from a sftp packet */
QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
    // XXX set all member variable to defaults

    // XXX Add some error checking in here in case
    //     we get a bad sftp packet.
    fa.clear();

    QByteArray fn;
    Q_ULLONG size;
    if( fa.mDirAttrs ) {
        s >> fn;

        KRemoteEncoding encoder ( fa.mEncoding.data() );

-------------------------------------------------
this is correct ? call clear and then ask about fa.mEncoding ??

Anyway i hope you guys can fix it, and even better fix it permanently ;)

Carlos

I have fixed this locally now. Unfortunately, I am unable to commit the fix or generate a patch because of the recent transition to SVN has made it impossible to access CVS right now. As soon as I am able to access the machine, I will commit the fix and let you test it...

SVN commit 409911 by adawit:

-Save the encoding data before clearing the file attribute object...

CCMAIL:66411@bugs.kde.org



 M  +8 -4      trunk/KDE/kdebase/kioslave/sftp/sftpfileattr.cpp  


--- trunk/KDE/kdebase/kioslave/sftp/sftpfileattr.cpp #409910:409911
@@ -149,18 +149,22 @@
 
 /** Use to read a file attribute from a sftp packet */
 QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
-    // XXX set all member variable to defaults
 
     // XXX Add some error checking in here in case
     //     we get a bad sftp packet.
+
+    // Save the encoding info before clearing fa...
+    QCString encoding ( fa.mEncoding );
+
     fa.clear();
 
     if( fa.mDirAttrs ) {
-        QByteArray fn;
+        QCString fn;
         s >> fn;
+        fn.truncate( fn.size() );
 
-        KRemoteEncoding encoder ( fa.mEncoding.data() );
-        fa.mFilename = encoder.decode( QCString(fn.data(), fn.size()+1) );
+        KRemoteEncoding encoder ( encoding.data() );
+        fa.mFilename = encoder.decode( fn );
 
         s >> fa.mLongname;
         fa.mLongname.truncate( fa.mLongname.size() );

*** Bug 105383 has been marked as a duplicate of this bug. ***

SVN commit 412078 by adawit:

- Fix regression of BR# 66411.

BUGS:66411



 M  +8 -4      branches/KDE/3.4/kdebase/kioslave/sftp/sftpfileattr.cpp  


--- branches/KDE/3.4/kdebase/kioslave/sftp/sftpfileattr.cpp #412077:412078
@@ -149,18 +149,22 @@
 
 /** Use to read a file attribute from a sftp packet */
 QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
-    // XXX set all member variable to defaults
 
     // XXX Add some error checking in here in case
     //     we get a bad sftp packet.
+
+    // Save the encoding info before clearing fa...
+    QCString encoding ( fa.mEncoding );
+
     fa.clear();
 
     if( fa.mDirAttrs ) {
-        QByteArray fn;
+        QCString fn;
         s >> fn;
+        fn.truncate( fn.size() );
 
-        KRemoteEncoding encoder ( fa.mEncoding.data() );
-        fa.mFilename = encoder.decode( QCString(fn.data(), fn.size()+1) );
+        KRemoteEncoding encoder ( encoding.data() );
+        fa.mFilename = encoder.decode( fn );
 
         s >> fa.mLongname;
         fa.mLongname.truncate( fa.mLongname.size() );

Is this supposed to be fixed in KDE 3.4.1 ?

I have tried the last patch but it seems that doesnt work as expected, because files again appear with the ugly chars and for that reason i cannot access the file; but if a write the right name in the address bar then the file/dir are retrieved correctly, so the problem is with the "naming" thing.

 I have downloaded the source code and i modified this line (sftpfileattr.cpp)

QCString encoding ( fa.mEncoding ); 
to
QCString encoding ( "UTF-8" ); 

now, everything works fine. im confused about the encoding contained in fa, is the file encoding or the remote encoding or which one ?. My remote enconding is UTF-8 for that reason the change works, so i suggest that you must review the remote encoding in this case. What do you think, im right ??

As debuging purpose, can i print the fa.mEncoding value to the console ?, the kde suite overrides stdout so i cant check which value comes in fa.

Carlos

Dawit: I can still reproduce it, latest trunk.

Working on it.

SVN commit 428892 by thiago:

Fixing again the encoding problem with kio_sftp. I hope this time for good.
To be backported in two weeks.

BUG:66411


 M  +14 -14    kio_sftp.cpp  
 M  +4 -10     sftpfileattr.cpp  
 M  +5 -3      sftpfileattr.h  

SVN commit 433353 by thiago:

Backporting the fix for kio_sftp. No one complained of it being broken
in two weeks, so I hope it's safe.

BACKPORT:428891:428892
CCBUG:66411


 M  +14 -14    kio_sftp.cpp  
 M  +4 -10     sftpfileattr.cpp  
 M  +5 -3      sftpfileattr.h  

Using KDE 3.5.0 at this moment. Currently neither sftp nor fish works for me with Umlauts, also tried the test server. My locale is as follows:

$ locale
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

Any ideas what I have to change?

This bug still exists for me, using fish and both (file) systems using UTF-8. Just this time it seems the umlauts are over-escaped:
http://screenshots.cweiske.de/2007-01-13%20kde%20fish%20utf8.png

On Saturday 13 January 2007 20:56, Christian Weiske wrote:
[bugs.kde.org quoted mail]

Cant confirm. (since KDE 3.5 or even earlier) 

Try Konqueror menu  Services  -> Network Disk Encoding 
(names may vary in different localizations)  -- works for me with KOI8-R 
and UTF-8 in various combinations.