Bug 66411 - sftp won't work with file names containing non-us-ascii characters
Summary: sftp won't work with file names containing non-us-ascii characters
Status: RESOLVED FIXED
Alias: None
Product: kio
Classification: Frameworks and Libraries
Component: sftp (show other bugs)
Version: unspecified
Platform: Debian stable Linux
: NOR normal
Target Milestone: ---
Assignee: Thiago Macieira
URL:
Keywords:
: 71258 83410 85929 94159 105383 (view as bug list)
Depends on:
Blocks:
 
Reported: 2003-10-22 19:51 UTC by Mikko Tuumanen
Modified: 2007-01-19 13:20 UTC (History)
8 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mikko Tuumanen 2003-10-22 19:51:38 UTC
Version:            (using KDE KDE 3.1.4)
Installed from:    Debian stable Packages
OS:          Linux

I created a file named töttöröö.txt.
Then I clicked the icon of the file in konqueror using sftp connection.

The error message was:

The file or directory
sftp://mijutu@unix.utu.fi/home/fysop/96/mijutu/tttr.txt does not
exist.

The non-us-ascii characters get lost somewhere.
Comment 1 cybernet 2004-03-25 00:03:52 UTC
KDE 3.1.5 from Aurox distro (Redhat 9)

German and Polish characters appear as squares when using sftp:// protocol.

When I create a file with Polish and German characters :
       sftp://localhost      - shows squares and names are shortened
       file:/                - both ok
       fish://localhost      - only german are ok, but since utf8 is NOT selected, this is understood

Polish characters were created using iso8859-2.

Using QString::latin1() and QString::fromLatin() doesn't preserve these characters, so maybe these calls in kio/sftp could be the problem ?
Comment 2 PaddyX 2004-04-24 02:49:35 UTC
To understand,

here is a single snapshot with

KDE 3.2.2

Umlauts are shown als "nul"

ftp://ftp.eui.fh-koblenz.de/pub/studenten/Duck.Donald/55.png

PaddyX
Comment 3 Mikko Tuumanen 2004-05-11 12:01:53 UTC
Here is a screenshot from Debian unstable (KDE 3.2.2-1)

http://users.utu.fi/mijutu/bugreports/sftpbug.png
Comment 4 Pawel Orzechowski 2004-05-11 22:44:47 UTC
I'm using fedora core 1 with kde 3.2.2. Using sftp://<host>/<some_directory> in konqueror to display directory with filenames with Polish characters with utf8 encoding works correctly, but files can not be opened. There is message being displayed, i.e.:
"File or directory sftp://localhost/home/pawel/łąka doesn't exist". 
The file exists of course, I created it for test purposes.
Comment 5 Tommi Tervo 2004-06-27 17:17:18 UTC
*** Bug 83410 has been marked as a duplicate of this bug. ***
Comment 6 Tommi Tervo 2004-06-27 17:17:38 UTC
*** Bug 71258 has been marked as a duplicate of this bug. ***
Comment 7 Nicolas Goutte 2004-07-26 11:18:16 UTC
*** Bug 85929 has been marked as a duplicate of this bug. ***
Comment 8 Nicolas Goutte 2004-08-06 21:50:41 UTC
As sftp also depends on the server: as anybody tried to use the sftp program directly? Does it works?

Have a nice day!
Comment 9 Mikko Tuumanen 2004-08-07 10:47:01 UTC
Yes, it does:

unix.utu.fi:~> echo foo >ööö.txt

s:~$ sftp mijutu@unix.utu.fi
Connecting to unix.utu.fi...
mijutu@unix.utu.fi's password: 
sftp> get ööö.txt
Fetching /home/fysop/96/mijutu/ööö.txt to ööö.txt
sftp> quit
s:~$ cat ööö.txt
foo
Comment 10 Thiago Macieira 2004-08-11 00:26:14 UTC
The API for this is present in KDE 3.3. 

Maintainer: please make kio_sftp use KRemoteEncoding as kio_ftp does.
Comment 11 Adrian Dziubek 2004-08-28 09:57:25 UTC
sftp works for me, but fish isn't.
Comment 12 Martin Rieder 2004-09-10 03:21:15 UTC
fish works fine for me. If it doesn't on your machine, check what it shows using ssh. 
I also get the squares with sftp in KDE 3.3, so I tried different settings for "Remote Charset", but that had no effect. 
However, when I create a file with non-ascii characters, it shows up correctly under fish (and ssh console). KRemoteEncoding seems to be partially implemented, as I understand it.
Comment 13 Adrian Dziubek 2004-09-10 18:13:52 UTC
Pure ssh works fine, I can use and handle UTF-8 names just fine. When using fish I get some iso-8859-1, I think. I can't find any options in Konqueror to change encoding. It isn't just that the names aren't displayed properly. When I try to upload/download, it's unable to actually find a file with that name. So it's practically impossible to do anything with a file like that using fish, I have to utilize ssh.
Comment 14 Dawit Alemayehu 2004-10-02 07:02:09 UTC
CVS commit by adawit: 

- Support for remoteEncoding(). Need feedback from people that connect 
  to non-latin1 sftp servers to verify if this works properly...

- Use KUser instead of own concaction.

- Avoid duplicate error reporting.

CCMAIL:66411@bugs.kde.org


  M +165 -173  kio_sftp.cpp   1.59
  M +8 -25     kio_sftp.h   1.14



Comment 15 Tommi Tervo 2004-11-30 11:53:29 UTC
*** Bug 94159 has been marked as a duplicate of this bug. ***
Comment 16 Rainer Lay 2005-01-05 11:39:09 UTC
Hello Dawit,

with SuSE 9.2 (KDE 3.3.0), I can display directories with german Umlauts correctly, but I cannot download or rename such a file!
a) should your patch be in 3.3.0?
b) of not, could you provide a patch for kde 3.3.0, so I can test this?

I really would like to get this work!
Comment 17 Aldoo 2005-01-24 19:52:31 UTC
KDE 3.3.2, under Gentoo.
The bug still exists.
With sftp, non-ascii caracters will be displayed, but files with names containing such caracters won't be opened.
(with fish or ftp, KDE assumes the remote encoding is ISO-latin1 and shows rubbish, but the files can be opened ...)
Comment 18 Dawit Alemayehu 2005-03-09 02:33:07 UTC
The fix for this issue did not make it into the 3.3.0 release which I believe 
is what comes with SUSE 9.2. The fix should be in the upcoming 3.4 release 
which is currently in Release Candidate 1. 

On Tuesday 08 March 2005 15:44, you wrote:
> Hi Dawit, im confused about using unicode with kde and ftp services, i
> dont know if im leading with a bug or something, hope dont bother you
> can help me.
>
> I speak spanish, so i use latin characters, i have a suse 9.2 with
> default lang=en_US.UTF-8 , so im using UTF enconding for my filenames (i
> attached my language file at the end of the mail). The problem is when i
> use konqueror or kbear to connect to another site with sftp.
>
> If connect to another UTF site i can see the filenames correctly, but
> when i try to get a file (named with special chars )  i receive a
> message saying that the file doesnt exist, i looks like there is a
> problem with UTF, maybe related or similar with this bug
> http://bugs.kde.org/show_bug.cgi?id=66411
> But its pretty weird that it was opened on 2003 and still remains as
> "new", i mean, if its a bug it should be so common because all the
> people using different languages around the world.
> Anyway i saw your post and for that reason im writting to you.
>
> But i have another problem, i have an old solaris 8 machine with
> lang=en_US.ISO8859-1, when i connect i see the files with latin names in
> a very weird manner, i suppose the application "think" is unicode but is
> not, this case i can transfer the files but with the funny name.
>
> I also did another test changing sftp by ftp, and it seems that the
> application assumes a default ISO8859-1 always, so when i connect to the
> UTF site the chars are ugly and when i do the same to my solaris the
> files are ok :(.
>
> As you can see im really confused, i dont know if its possible to have
> my own encoding and work fine with whatever site i connect or if i
> should stardarize all the locales in my net, or what im doing wrong or
> if it is a bug.
>
> thanx a lot for your time
>
> Carlos


Comment 19 Stéphane Magnenat 2005-03-09 09:11:54 UTC
I've just retested with 3.4 rc1 (debian packages from http://pkg-kde.alioth.debian.org/kde-3.4.0/) and the bug is still there

Steph
Comment 20 Dawit Alemayehu 2005-03-11 03:06:54 UTC
Does anyone have a publicly available sftp where I can test this at ?
Comment 21 Stéphane Magnenat 2005-03-11 11:36:54 UTC
Yes, here it is:
sftp://testsftp@lispc45.epfl.ch
pwd: testsftp

lispc45.epfl.ch's locales are UTF8. My laptop's locales are UTF8. Konqueror's SFTP uses latin1.

Steph
Comment 22 Dawit Alemayehu 2005-03-11 14:49:16 UTC
Well I tested against that server and I cannot duplicate the problem. Here is a screenshot of what I get with the current CVS version of KDE which has the same sftp io-slave as the 3.4 RC1:

Result of listing the directory in konqueror:
http://members.cox.net/kdelists/screenshots/konq-sftp-list.png

Result of clicking on the file with non-ascii text:
http://members.cox.net/kdelists/screenshots/konq-sftp-saveas-dlg.png

Result of clicking on Open with 'KWrite':
http://members.cox.net/kdelists/screenshots/kate-opening-non-ascii-file.png

The only difference with my setup is that my locale is not UTF8...
Comment 23 Thiago Macieira 2005-03-11 15:17:00 UTC
Dawit: sorry, but all your screenshots show you *could* duplicate the problem, not that you couldn't.

The file whose name you saw as "testaccent_éèà" should have been seen as "testaccent_éèà".
Comment 24 Dawit Alemayehu 2005-03-11 15:29:20 UTC
On Friday 11 March 2005 09:17, Thiago Macieira wrote:
[bugs.kde.org quoted mail]

Well then I do not understand what is going on. The sftp io-slave user the 
remoteEncoding() function from SlaveBase to encode all pathes just like the 
ftp io-slave does. Also people reported that they are unable to open the file 
which is not the case at all for me. Maybe I need to change my locale to UTF8  
and see if that changes things...
Comment 25 Stéphane Magnenat 2005-03-11 15:40:00 UTC
Hi,

The test server has changed name and is now:
sftp://testsftp@lissrv3.epfl.ch

Steph
Comment 26 Thiago Macieira 2005-03-11 16:40:21 UTC
> Well then I do not understand what is going on. The sftp io-slave user the 
> remoteEncoding() function from SlaveBase to encode all pathes just like the 
> ftp io-slave does.

Indeed, but it's not working. I'll debug the code this weekend to see where the real problem is.
Comment 27 Piotr Gawron 2005-03-11 19:26:36 UTC
Here is the example of this bug:
http://www.iitis.gliwice.pl/~gawron/ja/sftpkdebug.png
Both windows represent same directory. Upper one is oppened by sftp,
bottom localy.
Local and remote systems are standard Mandrake 10.1 boxes set to pl_PL
locale.

Piotr Gawron
Comment 28 Thiago Macieira 2005-03-13 20:54:15 UTC
CVS commit by thiago: 

Found the bug and fixed it. It was quite simple: the file attribute struct
was erasing the encoding before it was even first used.

To be backported.
BUG:66411


  M +0 -1      sftpfileattr.cpp   1.15


--- kdebase/kioslave/sftp/sftpfileattr.cpp  #1.14:1.15
@@ -295,5 +295,4 @@ void sftpFileAttr::clear(){
     mLongname = "\0";
     mLinkType = 0;
-    mEncoding = "\0";
 }
 
Comment 29 Thiago Macieira 2005-03-13 20:55:59 UTC
CVS commit by thiago: 

(Backport 1.14:1.15) Backporting the bug fix to KDE 3.4.x.

CCBUG:66411


  M +0 -1      sftpfileattr.cpp   1.14.2.1


--- kdebase/kioslave/sftp/sftpfileattr.cpp  #1.14:1.14.2.1
@@ -295,5 +295,4 @@ void sftpFileAttr::clear(){
     mLongname = "\0";
     mLinkType = 0;
-    mEncoding = "\0";
 }
 
Comment 30 Carlos Proal 2005-05-05 07:44:37 UTC
Hi, im currently using this package kdebase3-3.4.0-33,  and the problem is still present; i also reviewed the cvs source tree and it seems the fix was removed in recent revisions.

On the other hand, im concerned that maybe the problem is not in the clear method but in here:

-------------------------------------------------
/** Use to read a file attribute from a sftp packet */
QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
    // XXX set all member variable to defaults

    // XXX Add some error checking in here in case
    //     we get a bad sftp packet.
    fa.clear();

    QByteArray fn;
    Q_ULLONG size;
    if( fa.mDirAttrs ) {
        s >> fn;

        KRemoteEncoding encoder ( fa.mEncoding.data() );

-------------------------------------------------
this is correct ? call clear and then ask about fa.mEncoding ??

Anyway i hope you guys can fix it, and even better fix it permanently ;)

Carlos
Comment 31 Dawit Alemayehu 2005-05-05 12:13:51 UTC
I have fixed this locally now. Unfortunately, I am unable to commit the fix or generate a patch because of the recent transition to SVN has made it impossible to access CVS right now. As soon as I am able to access the machine, I will commit the fix and let you test it...
Comment 32 Dawit Alemayehu 2005-05-06 02:06:59 UTC
SVN commit 409911 by adawit:

-Save the encoding data before clearing the file attribute object...

CCMAIL:66411@bugs.kde.org



 M  +8 -4      trunk/KDE/kdebase/kioslave/sftp/sftpfileattr.cpp  


--- trunk/KDE/kdebase/kioslave/sftp/sftpfileattr.cpp #409910:409911
@@ -149,18 +149,22 @@
 
 /** Use to read a file attribute from a sftp packet */
 QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
-    // XXX set all member variable to defaults
 
     // XXX Add some error checking in here in case
     //     we get a bad sftp packet.
+
+    // Save the encoding info before clearing fa...
+    QCString encoding ( fa.mEncoding );
+
     fa.clear();
 
     if( fa.mDirAttrs ) {
-        QByteArray fn;
+        QCString fn;
         s >> fn;
+        fn.truncate( fn.size() );
 
-        KRemoteEncoding encoder ( fa.mEncoding.data() );
-        fa.mFilename = encoder.decode( QCString(fn.data(), fn.size()+1) );
+        KRemoteEncoding encoder ( encoding.data() );
+        fa.mFilename = encoder.decode( fn );
 
         s >> fa.mLongname;
         fa.mLongname.truncate( fa.mLongname.size() );
Comment 33 Tommi Tervo 2005-05-10 09:48:16 UTC
*** Bug 105383 has been marked as a duplicate of this bug. ***
Comment 34 Dawit Alemayehu 2005-05-10 20:09:11 UTC
SVN commit 412078 by adawit:

- Fix regression of BR# 66411.

BUGS:66411



 M  +8 -4      branches/KDE/3.4/kdebase/kioslave/sftp/sftpfileattr.cpp  


--- branches/KDE/3.4/kdebase/kioslave/sftp/sftpfileattr.cpp #412077:412078
@@ -149,18 +149,22 @@
 
 /** Use to read a file attribute from a sftp packet */
 QDataStream& operator>> (QDataStream& s, sftpFileAttr& fa) {
-    // XXX set all member variable to defaults
 
     // XXX Add some error checking in here in case
     //     we get a bad sftp packet.
+
+    // Save the encoding info before clearing fa...
+    QCString encoding ( fa.mEncoding );
+
     fa.clear();
 
     if( fa.mDirAttrs ) {
-        QByteArray fn;
+        QCString fn;
         s >> fn;
+        fn.truncate( fn.size() );
 
-        KRemoteEncoding encoder ( fa.mEncoding.data() );
-        fa.mFilename = encoder.decode( QCString(fn.data(), fn.size()+1) );
+        KRemoteEncoding encoder ( encoding.data() );
+        fa.mFilename = encoder.decode( fn );
 
         s >> fa.mLongname;
         fa.mLongname.truncate( fa.mLongname.size() );
Comment 35 Helge Hielscher 2005-06-03 02:20:17 UTC
Is this supposed to be fixed in KDE 3.4.1 ?
Comment 36 Carlos Proal 2005-06-18 11:38:56 UTC
I have tried the last patch but it seems that doesnt work as expected, because files again appear with the ugly chars and for that reason i cannot access the file; but if a write the right name in the address bar then the file/dir are retrieved correctly, so the problem is with the "naming" thing.

 I have downloaded the source code and i modified this line (sftpfileattr.cpp)

QCString encoding ( fa.mEncoding ); 
to
QCString encoding ( "UTF-8" ); 

now, everything works fine. im confused about the encoding contained in fa, is the file encoding or the remote encoding or which one ?. My remote enconding is UTF-8 for that reason the change works, so i suggest that you must review the remote encoding in this case. What do you think, im right ??

As debuging purpose, can i print the fa.mEncoding value to the console ?, the kde suite overrides stdout so i cant check which value comes in fa.

Carlos
Comment 37 Thiago Macieira 2005-06-25 16:30:38 UTC
Dawit: I can still reproduce it, latest trunk.
Comment 38 Thiago Macieira 2005-06-25 19:37:26 UTC
Working on it.
Comment 39 Thiago Macieira 2005-06-25 20:17:18 UTC
SVN commit 428892 by thiago:

Fixing again the encoding problem with kio_sftp. I hope this time for good.
To be backported in two weeks.

BUG:66411


 M  +14 -14    kio_sftp.cpp  
 M  +4 -10     sftpfileattr.cpp  
 M  +5 -3      sftpfileattr.h  
Comment 40 Thiago Macieira 2005-07-10 16:49:09 UTC
SVN commit 433353 by thiago:

Backporting the fix for kio_sftp. No one complained of it being broken
in two weeks, so I hope it's safe.

BACKPORT:428891:428892
CCBUG:66411


 M  +14 -14    kio_sftp.cpp  
 M  +4 -10     sftpfileattr.cpp  
 M  +5 -3      sftpfileattr.h  
Comment 41 Arne Schmitz 2006-01-07 15:04:42 UTC
Using KDE 3.5.0 at this moment. Currently neither sftp nor fish works for me with Umlauts, also tried the test server. My locale is as follows:

$ locale
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

Any ideas what I have to change?
Comment 42 Christian Weiske 2007-01-13 18:56:38 UTC
This bug still exists for me, using fish and both (file) systems using UTF-8. Just this time it seems the umlauts are over-escaped:
http://screenshots.cweiske.de/2007-01-13%20kde%20fish%20utf8.png
Comment 43 shift 2007-01-19 13:20:50 UTC
On Saturday 13 January 2007 20:56, Christian Weiske wrote:
[bugs.kde.org quoted mail]

Cant confirm. (since KDE 3.5 or even earlier) 

Try Konqueror menu  Services  -> Network Disk Encoding 
(names may vary in different localizations)  -- works for me with KOI8-R 
and UTF-8 in various combinations.