Bug 339289

Would you be able to install the debug packages for Kubuntu and get debug messages from your attempts to copy those files? If so, then please follow these instructions to generate debug output:

1.) To install the debug packages, follow the instructions at
https://techbase.kde.org/Development/Tutorials/Debugging/How_to_create_useful_crash_reports#Kubuntu

2.) Follow the direction in the link below to generate debug output for debug areas 7103 and 7113 and post the output here:
https://techbase.kde.org/Development/Tutorials/Debugging/Debugging_IOSlaves#GUI

Please be sure to remove any personal information from the debug output before posting here, e.g. any line that contains "Authorization:"

So, there is no dolphin-dbg. I installed kdebase-dbg and kde-baseapps-dbg.

Running `kdebugdialog --fullmode` doesn't seem to do anything. I can select stuff but that's about it. Running just `kdebugdialog` seemed to make more sense. Hopefully the output is the same? I don't know if there's something about --fullmode that is different.

Anyway, kdebugdialog, select 7103 and 7113. Nothing outputs to the konsole. If I select everything that has "kio" in the same then too much stuff outputs. KIOJob seems to have info about the file I am copy-pasting from webdav...but it is 7007. Anyway, what to do?

(In reply to Mauricio from comment #2)
> So, there is no dolphin-dbg. I installed kdebase-dbg and kde-baseapps-dbg.

Sorry I guess my instructions were not very clear. You need to install kdelibs-dbg. kio_http is part of kdelibs. 
 
> Running `kdebugdialog --fullmode` doesn't seem to do anything. I can select
> stuff but that's about it. Running just `kdebugdialog` seemed to make more
> sense. Hopefully the output is the same? I don't know if there's something
> about --fullmode that is different.

It is only when you launch kdebugdialog in this mode can you select to send the debug output to a specific file. And just as the example shows in the instrurction, you should send the output from both 7103 and 7113 to the same output file, /tmp/kio_http.log (or whatever filename suites you).

> Anyway, kdebugdialog, select 7103 and 7113. Nothing outputs to the konsole.
> If I select everything that has "kio" in the same then too much stuff
> outputs. KIOJob seems to have info about the file I am copy-pasting from
> webdav...but it is 7007. Anyway, what to do?

Both 7113 and 7103 debug areas are part of the kio_http ioslave which is in kdelibs that is why you did not get any output. You should not enable the other debug areas for the reason you stated above (too much unnecessary noise for debugging this issue).

Created attachment 88861 [details]
screenshot of kdebugdialog in full mode

Ok, so I managed to get a file output from kdebugdialog in full mode. Uhm, between 7103 and 7113 I get about 2400 lines... Am I to post all of that? The error is only for getting files, not the connection or browsing.

Virtually all start with `kio_http(14710)/kio_http_debug` or `kio_http(14490)`

(In reply to Mauricio from comment #5)
> Ok, so I managed to get a file output from kdebugdialog in full mode. Uhm,
> between 7103 and 7113 I get about 2400 lines... Am I to post all of that?
> The error is only for getting files, not the connection or browsing.
> 
> Virtually all start with `kio_http(14710)/kio_http_debug` or
> `kio_http(14490)`

Yes. Please post the file as an attachement if it has 2400 lines.

Created attachment 88875 [details]
Output from kdebugdialog --fullmode for 7103

Created attachment 88876 [details]
Output from kdebugdialog --fullmode for 7113

I have no idea what anything in the file means. Hence I do not know how to sanitize. I did some ctrl+F stuff but found nothing obvious. I've already changed my login password to the webdav server and my user password. Am I to change any others?

Anyway, I think I figured out how to output to a single file. I created a new .dbg before changing my password. I am reluctant to post because I really don't know the extent to which sensitive info is in there.

Created attachment 88877 [details]
kio 7103 and 7113 `kdebugdialog --fullmode` output

Is this better?

(In reply to Mauricio from comment #9)
> I have no idea what anything in the file means. Hence I do not know how to
> sanitize. I did some ctrl+F stuff but found nothing obvious. I've already

In comment #1, I said 

"Please be sure to remove any personal information from the debug output before posting here, e.g. any line that contains "Authorization:"

> changed my login password to the webdav server and my user password. Am I to
> change any others?

Not unless you used the same login/password combination as this server for everything else.

> Anyway, I think I figured out how to output to a single file. I created a
> new .dbg before changing my password. I am reluctant to post because I
> really don't know the extent to which sensitive info is in there.

What you posted now is should be fine since you seem to have properly sensored the "Authorization" and "Cookie" headers.

(In reply to Mauricio from comment #10)
> Created attachment 88877 [details]
> kio 7103 and 7113 `kdebugdialog --fullmode` output
> 
> Is this better?

Yep. I will look at it when I get a chance. Thanks.

No, thank you. You've been rather patient through this whole thing. Bugs are frustrating.

I appreciate the help (:

When looking at the log you sent starting from the point where you clicked on the PDF document, the following is what I see:

1.)  You attempted to get
webdavs://[classes].[university].edu/dav/[CLASS9999]_001_2014_3/Class Notes/slides_intro.pdf

2.) You got redirected by the server to 
webdavs://[classes].[university].edu/access/content/group/[CLASS9999]_001_2014_3/Class Notes/slides_intro.pdf

3.) You got redirected by the server again to
https://[classes].[university].edu/sakai-login-tool/container

4.) You got redirected for the third time by the server to
https://cas.[university].edu/cas/login?service=https%3A%2F%2F[classes].[university].edu%2Fsakai-login-tool%2Fcontainer

5.) To which the server responded by sending you back a HTML document:

HTTP/1.1 200 OK
Date: Sun, 28 Sep 2014 23:17:44 GMT
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Cache-Control: no-store
Set-Cookie: JSESSIONID=[deleted]; Path=/cas/; Secure; HttpOnly
Content-Type: text/html;charset=UTF-8
Content-Length: 7487
Vary: User-Agent,Accept-Encoding
Keep-Alive: timeout=15, max=33
Connection: Keep-Alive
Set-Cookie: BIGipServer~UIT~cas.[university].edu-443-pool=[deleted]; expires=Sun, 28-Sep-2014 23:22:44 GMT; path=/

I have literally no idea why the server redirected you 3X before returing a HTML document. What does the HTML document it returns say?

Here's the HTMl file. It's a mock-up of the login page.

Created attachment 88894 [details]
File I get instead of file I try to get

(In reply to Mauricio from comment #15)
> Here's the HTMl file. It's a mock-up of the login page.

Sorry, I guess you already mentioned that up top which I did not see. To be honest, it is really hard to tell why this is happening. It could potentially be related to cookies. Let us do the following test if you do not mind:

1.) Remove the old file where you were outputing kio_http log data.

2.) Launch the cookie management dialog: 
    ALT+F2, type "cookies" and press Enter.

3.) Go to the management tab and remove any old cookies for the server you are accessing. Just enter the server name in the search box, highlight the filtered item and click delete.

4.) Launch konsole: 
   ALT+F2, type "konsole" and press Enter.

5.) Directly download one of those files you were attempting to download through Dolphin. For example,

kioclient download "webdavs://[classes].[university].edu/dav/[CLASS9999]_001_2014_3/Class Notes/slides_intro.pdf"

See if the downloaded file is valid. If this still gets you the wrong file, then you have to try to login into the site with either Firefox or Chrome and see if there is a difference in the cookies they get and what Dolphin/Konqueror gets.

I have downloaded the file through kioclient but I get the same html file.

So, the firefox cookies, I'm told, are in cookies.sqlite. From the kio.dbg file I have something like

kio_http(15699) HTTPProtocol::sendQuery: "Cookie: __utma=123456789.9999999999.9999999999.9999999999.9999999999.1; __utmz=123456789.9999999999.9.9.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _ga=GA1.9.9999999999.9999999999; JSESSIONID=[deleted].acorn-ci; BIGipServer~UIT~[classes].[university].edu-443-pool=999999999.99999.0000"

The formats match, the 123456789 parts match, but the rest doesn't match.  Then there's this line

kio_http(15699) HTTPProtocol::sendQuery: "Cookie: JSESSIONID=[deleted]; _ga=GA1.9.9999999999.9999999999; BIGipServer~UIT~cas.[university].edu-443-pool=1234567890.98765.0000"

Again, formats match, and BIGipServer~UIT~cas.[university].edu-443-pool matches exactly (1234567890.98765.0000 matches BIGipServer~UIT~cas.[university].edu-443-pool in firefox's cookies.sqlite); but _ga do not match. That's all I am seeing.

(In reply to Mauricio from comment #18)
> I have downloaded the file through kioclient but I get the same html file.
> 
> So, the firefox cookies, I'm told, are in cookies.sqlite. From the kio.dbg
> file I have something like
>
> kio_http(15699) HTTPProtocol::sendQuery: "Cookie:
> __utma=123456789.9999999999.9999999999.9999999999.9999999999.1;
> __utmz=123456789.9999999999.9.9.
> utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
> _ga=GA1.9.9999999999.9999999999; JSESSIONID=[deleted].acorn-ci;
> BIGipServer~UIT~[classes].[university].edu-443-pool=999999999.99999.0000"
> 
> The formats match, the 123456789 parts match, but the rest doesn't match. 
> Then there's this line
> 
> kio_http(15699) HTTPProtocol::sendQuery: "Cookie: JSESSIONID=[deleted];
> _ga=GA1.9.9999999999.9999999999;
> BIGipServer~UIT~cas.[university].edu-443-pool=1234567890.98765.0000"
> 
> Again, formats match, and BIGipServer~UIT~cas.[university].edu-443-pool
> matches exactly (1234567890.98765.0000 matches
> BIGipServer~UIT~cas.[university].edu-443-pool in firefox's cookies.sqlite);
> but _ga do not match. That's all I am seeing.

I do not see anything in those cookie differences that would warrant a different outcome. At this point I have no clue as to why you are getting redirected to a login page when you attempt to download a file. The only thing you have not tried is to spoof the user-agent string in case the server is sending you to different destinations based on that. I will be really surprised if this resolves the issue though. Anyhow you can try sending a different user agent by adding the following to your ~/.kde/share/config/kio_httprc file:

[example.com]
UserAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36

Change "example.com" to "courseworks.columbia.edu". Make sure you restart Konqueror/Dolphin for the change to take effect. Or press CTRL+ESC and kill all kio_http processes that might be running.

No, that didn't do it. Is there a way to compare kio to how Nautilus/Nemo handles the protocol? I can see the files fine through there.

(In reply to Mauricio from comment #20)
> No, that didn't do it. Is there a way to compare kio to how Nautilus/Nemo
> handles the protocol? I can see the files fine through there.

No because those file managers probably use gvfs which mounts the remote folders and let you browse it as if it is a local filesystem. That is basically the biggest difference between KIO and what gnome uses as a vfs. KIO does not do mounts.

Can you please post the output you get when you try to download the file directly using kioclient? Please delete an existing cookies before you do that again though.

Created attachment 88975 [details]
This is what I get from kioclient. It's pretty much the same.

(In reply to Mauricio from comment #22)
> Created attachment 88975 [details]
> This is what I get from kioclient. It's pretty much the same.

My fault. I meant the kio_http log output and not the response from the webserver. I wanted to see if there is something else we can see when you attempt to download the file directly. BTW, can you also please set the KDE version you are currently running?

By the way I changed the User-Agent in ~/.kde/share/config/kio_httprc to

[courseworks.columbia.edu]
UserAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36

but in the dbg file it seems it hasn't changed. I have turned my computer off, on since yesterday. I also deleted the cookies via Alt + F2 > Cookies and restarted kdeinit4.

I am using kde 4.13.3 in Kubuntu 14.04.

Created attachment 88977 [details]
I took out the cookies also. Do you need them?

(In reply to Mauricio from comment #25)
> Created attachment 88977 [details]
> I took out the cookies also. Do you need them?

Yes, I currently suspect the login problem has something to do with cookies because I do not see anything else that could cause such problem. I want to follow the trail of cookies from the server to the client and back to the server. If you do not feel like posting such information to the bug report, simply send it to me directly (click on my name in any of the comments to get my email address). However, you should remove/sensor the "Authorization" information.

Created attachment 88978 [details]
The file with the cookies and JSESSIONID

(In reply to Mauricio from comment #27)
> Created attachment 88978 [details]
> The file with the cookies and JSESSIONID

Great and the value sent through the Authorization header is always the same, correct?

Yes.

I just noticed this response...

(In reply to Mauricio from comment #24)
> By the way I changed the User-Agent in ~/.kde/share/config/kio_httprc to
> 
> [courseworks.columbia.edu]
> UserAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
> Gecko) Chrome/37.0.2062.120 Safari/537.36
>
> but in the dbg file it seems it hasn't changed. I have turned my computer
> off, on since yesterday. I also deleted the cookies via Alt + F2 > Cookies
> and restarted kdeinit4.

Yeah. This is my mistake! I should have told you to put that configuration entry into ~/.kde/share/config/kio_webdavrc and not kio_httprc. Can you retry it with that instead? Thanks.

Created attachment 88980 [details]
dbg from changing user agent

Is it odd that the last UserAgent string goes back to "User-Agent: Mozilla/5.0 (X11; Linux x86_64) KHTML/4.13.3 (like Gecko) Konqueror/4.13" while the others are the ones you specified?

(In reply to Mauricio from comment #31)
> Created attachment 88980 [details]
> dbg from changing user agent
> 
> Is it odd that the last UserAgent string goes back to "User-Agent:
> Mozilla/5.0 (X11; Linux x86_64) KHTML/4.13.3 (like Gecko) Konqueror/4.13"
> while the others are the ones you specified?

Not really. That happened because the Chrome user-agent string was set for the domain "courseworks.columbia.edu" and not "cas.columbia.edu" which is where the last redirect sent you. You can set the user-agent string for the entire "columbia.edu" domain in that file and see if that helps. I doubt it will since I do not think that is the problem. The fact that it redirects you away from the PDF document you were attempting to access seems to be the issue to me. Unfortunately I cannot tell what is causing that based on the log you have provided. Everything looks OK to me.

The one last thing you can try is to see what response you get from the server if you attempt to directly download the PDF document from the webdav server using Firefox for example. You can see the request/response HTTP headers in Firefox by launching the developer tools for network (CTRL+SHIFT+Q) and clicking the little button to start performance analysis.

Mauricio, any progress with comment #32? I hope you did not gave up :)

I had, and have been using nemo. I tried through firefox. The network changed whenever I loaded a new page. The last page gave showed some info, but I don't really know whether the cookies there are importantly different. They are not the same numbers/letters, but if I use chromium I also get different cookies (and I still get the file). User-Agent are also different.

By the way, I just realized I never thanked you for dedicating so much effort to helping me out.

Thank you. Honestly. I generally get good help from KDE and I really appreciate it. I apologize that I gave up on debugging but it seems the root of the problem is not obvious and the nemo workaround is not terrible.

I have the same problem. If someone is still interested in solving it I may help him in debugging it.

(In reply to Simone Gaiarin from comment #36)
> I have the same problem. If someone is still interested in solving it I may
> help him in debugging it.

I am. Can you please follow the steps outlined in comment #1 and post the santized (e.g. remove any authorization headers) debug output here?

So I'm using Manjaro linux an Arch derivate distribution. From what I've just learned the KDE Arch packages do not provide debug symbols (and the same for Manjaro), so I should recompile kdelibs manually with the debug symbols on. That's annoying. So since it's not a straightforward operation, I have to find the time to do it, but I'll do it.
I'll keep you updated.

References:
https://wiki.archlinux.org/index.php/Debug_-_Getting_Traces#KDE_applications
https://techbase.kde.org/Development/Tutorials/Debugging/How_to_create_useful_crash_reports#Arch

Thank you for reporting this issue in KDE software. As it has been a while since this issue was reported, can we please ask you to see if you can reproduce the issue with a recent software version?

If you can reproduce the issue, please change the status to "REPORTED" when replying. Thank you!

@Justin Zobel While I still use KDE I haven't used Kubuntu in a while, and I haven't used the webdav protocol since 2015. I'm happy to close or mark as resolved if nobody else can reproduce. LMK.

Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!

This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!