Bug 416647 - KBibTex unable to save files on a remote server via sftp
Summary: KBibTex unable to save files on a remote server via sftp
Status: RESOLVED WAITINGFORINFO
Alias: None
Product: KBibTeX
Classification: Applications
Component: Loading/saving files (show other bugs)
Version: 0.9
Platform: Neon Linux
: NOR normal
Target Milestone: ---
Assignee: Thomas Fischer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-23 14:10 UTC by Joan
Modified: 2020-04-25 08:46 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:


Attachments
Console output during kbibtex execution (2.17 KB, text/plain)
2020-02-07 13:28 UTC, Joan
Details
Compiler output (clog?) except warnings (37.12 KB, text/plain)
2020-02-07 13:30 UTC, Joan
Details
Console output + compiler warnings (64.36 KB, text/plain)
2020-02-07 13:32 UTC, Joan
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Joan 2020-01-23 14:10:53 UTC
SUMMARY

Remote access to a server via sftp. Remote .bib files are loaded and, apparently, can be modified and then saved. However, the remote file is not modified. Checked that the remote file has read-write permissions for all users enabled.

STEPS TO REPRODUCE
1. Open .bib file on remote server
2. Modify the contents of the file
3. Save the file, either click to the icon or use Ctrl + S shortcut
4. Apparently the modified file has been saved.

 

OBSERVED RESULT

The time stamp of the file in the server directory has been updated.
However, a check on the contents of the remote file (via terminal+vim editor) shows that the file was not modified, it keeps the original contents.


EXPECTED RESULT

A modified file with the contents as visualized in KBibTeX


SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: KDE Neon 5.17 User Edition
(available in About System)
KDE Plasma Version: 5.17.5
KDE Frameworks Version: 5.66.0
Qt Version: 5.13.2

ADDITIONAL INFORMATION

When KBibTeX is started from terminal, as a result of the Save action (on the remote file) the following message appears in the terminal:

kbibtex.parts: watchableFilename is Empty
kbibtex.io: Don't know how to encode Unicode char "0x0301"
kbibtex.io: Don't know how to encode Unicode char "0x2010"
kbibtex.parts: watchableFilename is Empty
Comment 1 Thomas Fischer 2020-01-28 20:22:56 UTC
I can confirm that something is broken with SFTP uploads, despite that the "copy to remote location" functions invoked by KBibTeX do not report any error. This requires further investigation.
Comment 2 Thomas Fischer 2020-02-01 22:52:11 UTC
I think I was able to fix this problem. Please have a look at the following patch and report back if it fixed your problem:
https://commits.kde.org/clones/kbibtex/thomasfischer/kbibtex/5040708c6486e075c7048ebd371ab6ccdd9262ed

You can test this code without interfering with your default KBibTeX installation by following the "quick start" instructions:
https://userbase.kde.org/KBibTeX/Development#Quick_Start_to_Run_KBibTeX_from_Git

To test the Git version containing the fix for your bug, please run the following command:
bash run-kbibtex.sh https://anongit.kde.org/clones/kbibtex/thomasfischer/kbibtex.git bugs/kde416647
Comment 3 Joan 2020-02-03 17:38:16 UTC
(In reply to Thomas Fischer from comment #2)
> I think I was able to fix this problem. Please have a look at the following
> patch and report back if it fixed your problem:
> https://commits.kde.org/clones/kbibtex/thomasfischer/kbibtex/
> 5040708c6486e075c7048ebd371ab6ccdd9262ed
> 
> You can test this code without interfering with your default KBibTeX
> installation by following the "quick start" instructions:
> https://userbase.kde.org/KBibTeX/
> Development#Quick_Start_to_Run_KBibTeX_from_Git
> 
> To test the Git version containing the fix for your bug, please run the
> following command:
> bash run-kbibtex.sh
> https://anongit.kde.org/clones/kbibtex/thomasfischer/kbibtex.git

> bugs/kde416647

Downloaded and compiled the new version as indicated. The patch did not work here, same behaviour as before. Let me add some additional information (tests):

1- To discard that I made something stupid with file permissions or sftp connection settings, I opened the remote (.bib) file with Kate and it worked as supposed, modifications were incorporated into the remote file after saving.

2- In kbibtex, the "save as" (new file) operation works well: I can succesfully save the modified contents into a new file in the remote directory.

3- After opening the remote file in kbibtex, I removed (externally, via cli) the remote file. After modifying the file (in kbibtex) and saving it, the OLD (unmodified) version 'reappeared' in the remote directory.
Comment 4 Thomas Fischer 2020-02-06 22:41:04 UTC
(In reply to Joan from comment #3)
> (In reply to Thomas Fischer from comment #2)
> > I think I was able to fix this problem. Please have a look at the following
> > patch and report back if it fixed your problem:
> > https://commits.kde.org/clones/kbibtex/thomasfischer/kbibtex/
> > 5040708c6486e075c7048ebd371ab6ccdd9262ed
> > 

> Downloaded and compiled the new version as indicated. The patch did not work
> here, same behaviour as before. Let me add some additional information
> (tests):
> 
> 1- To discard that I made something stupid with file permissions or sftp
> connection settings, I opened the remote (.bib) file with Kate and it worked
> as supposed, modifications were incorporated into the remote file after
> saving.
> 
> 2- In kbibtex, the "save as" (new file) operation works well: I can
> succesfully save the modified contents into a new file in the remote
> directory.
> 
> 3- After opening the remote file in kbibtex, I removed (externally, via cli)
> the remote file. After modifying the file (in kbibtex) and saving it, the
> OLD (unmodified) version 'reappeared' in the remote directory.

Either I did not understand your way to reproduce the problem or it is working file on my end. I added some debug output to learn more.
Please clean all directories created in /tmp by run-kbibtex.sh and run this bash script like before.
In your answer, please include the the complete output from the console, but feel free to replace sensitive filenames or hostnames with 'xxxx'.
Comment 5 Joan 2020-02-07 13:28:39 UTC
Created attachment 125734 [details]
Console output during kbibtex execution
Comment 6 Joan 2020-02-07 13:30:19 UTC
Created attachment 125735 [details]
Compiler output (clog?) except warnings
Comment 7 Joan 2020-02-07 13:32:08 UTC
Created attachment 125736 [details]
Console output + compiler warnings
Comment 8 Joan 2020-02-07 13:58:43 UTC
I am attaching 3 text files; the total output from console is splitted between the second and third files; in the first file there is the part of console output which I guess might be more relevant, after kbibtex execution started. 

Let me explain how the last seven output lines appeared:

kbibtex.parts: watchableFilename is Empty
kbibtex.io: Don't know how to encode Unicode char "0x0301"
kbibtex.io: Don't know how to encode Unicode char "0x2010"
kbibtex.parts: watchableFilename is Empty
qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 14661, resource id: 55074827, major code: 40 (TranslateCoords), minor code: 0
kbibtex.io: Don't know how to encode Unicode char "0x0301"
kbibtex.io: Don't know how to encode Unicode char "0x2010"

Each time I try to "Save", I get (almost immediately) these four lines on output:

kbibtex.parts: watchableFilename is Empty
kbibtex.io: Don't know how to encode Unicode char "0x0301"
kbibtex.io: Don't know how to encode Unicode char "0x2010"
kbibtex.parts: watchableFilename is Empty

and after a few seconds the floppy icon vanishes, meaning that the modified file should have been rewritten in the remote directory (but it is not, even though its time stamp is updated). The last 3 lines,

qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 14661, resource id: 55074827, major code: 40 (TranslateCoords), minor code: 0
kbibtex.io: Don't know how to encode Unicode char "0x0301"
kbibtex.io: Don't know how to encode Unicode char "0x2010"

appear instead during the "Save as" alternative, i.e, when (successfully) saving the modified file into the remote directory with a different name. 
I guess the "XCB error" is unrelated to the current issue, it seems has something to do with a small window shown during the "Save as" process. Thus I would say that the relevant difference in ouptut between a failed and a successful remote save operation is,

kbibtex.parts: watchableFilename is Empty

showing twice per failed try. 

Let me also explain the results from an additional little test: this time I modified the remote file, while keeping kbibtex open (I know, a sensible user should never do this), not 'directly' (using a konsole on the server) but locally (using Kate); this is the summary (scheme) of the file update history:

#1 original remote file opened locally both in kbibtex and kate
#2 version modified internally in kbibtex, but cannot be updated into remote directory
#3, #4, #5 subsequently modified versions in Kate, successfully updated each time in the remote directory

At this point, the original (#1) file only 'lives' as a temporary file (somewhere). If now I try a "Save" in bibtex, #1 reappears again in the remote directory (it overwrites the newer version #5, previosly saved by Kate).
Comment 9 Thomas Fischer 2020-02-07 21:45:43 UTC
Some comments on your comments

> kbibtex.parts: watchableFilename is Empty
Debug output from the monitor looking for other programs changing files currently open in KBibTeX. The message is irrelevant for remote files (no monitoring there).

> kbibtex.io: Don't know how to encode Unicode char "0x0301"
> kbibtex.io: Don't know how to encode Unicode char "0x2010"
There are thousands of Unicode characters for which no mapping to a LaTeX command is known to KBibTeX; you just encountered two of those. If you provide me with an example file containing BibTeX data with those characters, I can look for a fix, but it is not urgent and not relevant for this bug here.

> Each time I try to "Save", I get (almost immediately) these four lines on
> output:
> 
> kbibtex.parts: watchableFilename is Empty
> kbibtex.io: Don't know how to encode Unicode char "0x0301"
> kbibtex.io: Don't know how to encode Unicode char "0x2010"
> kbibtex.parts: watchableFilename is Empty
None of the additional debug output I added in my recent changes got printed. It looks like the run-kbibtex.sh script didn't have all debug settings enabled. I updated this script now, so please fetch the most recent version of this script kbibtex-related.git and run everything again. Sorry for the hassle ...

> qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 14661,
> resource id: 55074827, major code: 40 (TranslateCoords), minor code: 0
That is due to X being a complex, broken protocol (but still alive).

> Let me also explain the results from an additional little test: this time I
> modified the remote file, while keeping kbibtex open (I know, a sensible
> user should never do this), not 'directly' (using a konsole on the server)
> but locally (using Kate); this is the summary (scheme) of the file update
> history:
> 
> #1 original remote file opened locally both in kbibtex and kate
> #2 version modified internally in kbibtex, but cannot be updated into remote
> directory
> #3, #4, #5 subsequently modified versions in Kate, successfully updated each
> time in the remote directory
> 
> At this point, the original (#1) file only 'lives' as a temporary file
> (somewhere). If now I try a "Save" in bibtex, #1 reappears again in the
> remote directory (it overwrites the newer version #5, previosly saved by
> Kate).
If I understand it correctly, if you in KBibTeX:
1. Open a remote file
2. Modify the file
3. Save it to the same remote location under same name ...
... the file saved in step three is not containing the modifications made in step 2 but is still the old one as it was in step 1?
And by concurrently modifying the remote file in Kate, you know that KBibTeX actually writes something to the remote location in step 3?

Interesting behaviour ...
Comment 10 Thomas Fischer 2020-03-31 20:38:23 UTC
Can you please test the lastest code? Also, is my summary from comment 9 correct?

(In reply to Thomas Fischer from comment #9)
> If I understand it correctly, if you in KBibTeX:
> 1. Open a remote file
> 2. Modify the file
> 3. Save it to the same remote location under same name ...
> ... the file saved in step three is not containing the modifications made in
> step 2 but is still the old one as it was in step 1?
> And by concurrently modifying the remote file in Kate, you know that KBibTeX
> actually writes something to the remote location in step 3?
Comment 11 Joan 2020-04-01 08:49:28 UTC
(In reply to Thomas Fischer from comment #10)
> Can you please test the lastest code? Also, is my summary from comment 9
> correct?
> 
> (In reply to Thomas Fischer from comment #9)
> > If I understand it correctly, if you in KBibTeX:
> > 1. Open a remote file
> > 2. Modify the file
> > 3. Save it to the same remote location under same name ...
> > ... the file saved in step three is not containing the modifications made in
> > step 2 but is still the old one as it was in step 1?
> > And by concurrently modifying the remote file in Kate, you know that KBibTeX
> > actually writes something to the remote location in step 3?

I can confirm your summary is correct.

I'm working now in a different hardware, same system+DEnv (KDE/Plasma 5.18.3). I have downloaded, compiled and executed kbibtex from git as explained in #2. I get the same previous anomalous behavior when trying to save the modified .bib file to a remote system via sftp. I have however spotted two small differences with respect to the previous tests:

a) Now both the "Save" and the "Save As" commands fail to replace the original (remote) file by the modified one. However, the "Save Copy As" command works, it saves the modified version of the file into the remote system (as a new file, different name).

b) When performing either the "Save" or the "Save As" operations described above, I get now a system notification of: "Moving (Finished)", followed by the corresponding sftp command (in correct syntax, I would say). However, when I perform the "Save Copy As", I get no such a notification. That is, somehow the two first commands (that fail to replace the file) trigger a file-moving operation in the KDE environment whereas the latter command (that is working as supposed) does not.
Comment 12 Thomas Fischer 2020-04-24 21:21:12 UTC
I think I found and fixed the problem this time. Please test the latest incarnation of the code in the 'bugs/kde416647' branch, the commit hash should be 46e34341514442da9208f367055c135bb9e6d5d9.
Comment 13 Joan 2020-04-25 08:46:17 UTC
(In reply to Thomas Fischer from comment #12)
> I think I found and fixed the problem this time. Please test the latest
> incarnation of the code in the 'bugs/kde416647' branch, the commit hash
> should be 46e34341514442da9208f367055c135bb9e6d5d9.

Sorry, same behavior as before:

1- Save: I get a "Moving file" notification from KDE environment and the ORIGINAL remote file is kept

2- Save As: I get the notification again and a new remote file is created (with a different name) and the contents of the ORIGINAL file

3- Save Copy As: No notification from KDE and a new remote file is created with the contents of the (locally) MODIFIED (in kbibtex) file 

Please also note that at the end of compilation I got an error (which didn't happen last time):

/tmp/root-kbibtex-usr/bin/kbibtex: error while loading shared libraries: libkbibtexprocessing.so.0: cannot open shared object file: No such file or directory

I fixed it by adding 
/tmp/root-kbibtex-build-46be55653125cd1d93901531a3ff131d/bin/

to the dynamic library default path.

In doing so, I realized that the commit hash might be (?) 46be55653125cd1d93901531a3ff131d, different from what it should be (46e34341514442da9208f367055c135bb9e6d5d9) according to your message.

I executed :
bash run-kbibtex.sh https://anongit.kde.org/clones/kbibtex/thomasfischer/kbibtex.git bugs/kde416647

both as normal user and root (same result)