Bug 143744 - fetches dir listing after every upload attempt
Summary: fetches dir listing after every upload attempt
Status: RESOLVED FIXED
Alias: None
Product: kftpgrabber
Classification: Unclassified
Component: general (show other bugs)
Version: unspecified
Platform: Ubuntu Packages Linux
: NOR normal (vote)
Target Milestone: ---
Assignee: Jernej Kos
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-04-02 18:27 UTC by IgnorantGuru
Modified: 2007-04-02 20:32 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description IgnorantGuru 2007-04-02 18:27:02 UTC
Version:           0.8.0 (using KDE KDE 3.5.5)
Installed from:    Ubuntu Packages
Compiler:          http://karl.glatz.biz/oss/kftpgrabber_0.8.0-0ubuntu1~edgy_i386.deb 
OS:                Linux

> ------- Additional Comment #2 From Jernej Kos 2007-04-02 13:15 -------
> Oh and please open another bug report for the second problem
http://bugs.kde.org/show_bug.cgi?id=143617

KFTPGrabber seems to fetch the directory listing after every single upload attempt, causing it to take a VERY long time to upload multiple files.

To reproduce...  I have a local webpage in the left pane, and am connected to my FTP site in the right pane.  I drag and drop a folder (in this example "2007-04-01") from the left pane to the right pane.  In this example, the folder "2007-04-01" already exists on the site and is the same as the local, so KFTPGrabber has nothing to do.  I choose "Skip Always" at the first prompt, and KFTPGrabber attempts to upload every file in the folder and subfolders.  (It doesn't upload any as expected.)  Passive mode and other settings are at their default.

Here is an excerpt from the middle of the log - actual log was much longer.  It did a CWD and LIST for each of the 142 files in the folder.

[10:07:17] *** Uploading file 'thumb-P1000752-1X.JPG'...
[10:07:17] SIZE /2007-04-01/pics/thumb-P1000752-1X.JPG
[10:07:17] 213 8593
[10:07:17] *** Fetching directory listing...
[10:07:17] CWD /2007-04-01/pics
[10:07:17] 250 CWD command successful.
[10:07:17] PWD
[10:07:17] 257 "/2007-04-01/pics" is current directory.
[10:07:17] TYPE A
[10:07:17] 200 Type set to A.
[10:07:17] PASV
[10:07:17] 227 Entering Passive Mode (204,127,198,23,12,172).
[10:07:17] *** Establishing data connection with 204.127.198.23:3244...
[10:07:17] *** Data connection established.
[10:07:17] LIST -a
[10:07:17] 150 Opening ASCII mode data connection for file list
[10:07:17] 226-Transfer complete.
[10:07:17] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:18] *** Transfer completed.
[10:07:18] *** Uploading file 'thumb-P1000753.JPG'...
[10:07:18] SIZE /2007-04-01/pics/thumb-P1000753.JPG
[10:07:18] 213 7723
[10:07:18] *** Fetching directory listing...
[10:07:18] CWD /2007-04-01/pics
[10:07:18] 250 CWD command successful.
[10:07:18] PWD
[10:07:18] 257 "/2007-04-01/pics" is current directory.
[10:07:18] TYPE A
[10:07:18] 200 Type set to A.
[10:07:18] PASV
[10:07:18] 227 Entering Passive Mode (204,127,198,23,14,175).
[10:07:18] *** Establishing data connection with 204.127.198.23:3759...
[10:07:18] *** Data connection established.
[10:07:18] LIST -a
[10:07:18] 150 Opening ASCII mode data connection for file list
[10:07:18] 226-Transfer complete.
[10:07:18] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:19] *** Transfer completed.
[10:07:19] *** Uploading file 'thumb-P1000754-10X.JPG'...
[10:07:19] SIZE /2007-04-01/pics/thumb-P1000754-10X.JPG
[10:07:19] 213 8962
[10:07:19] *** Fetching directory listing...
[10:07:19] CWD /2007-04-01/pics
[10:07:19] 250 CWD command successful.
[10:07:19] PWD
[10:07:19] 257 "/2007-04-01/pics" is current directory.
[10:07:19] TYPE A
[10:07:19] 200 Type set to A.
[10:07:19] PASV
[10:07:19] 227 Entering Passive Mode (204,127,198,23,13,7).
[10:07:19] *** Establishing data connection with 204.127.198.23:3335...
[10:07:19] *** Data connection established.
[10:07:19] LIST -a
[10:07:19] 150 Opening ASCII mode data connection for file list
[10:07:19] 226-Transfer complete.
[10:07:19] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:19] *** Transfer completed.
[10:07:20] *** Uploading file 'thumb-P1000755.JPG'...
[10:07:20] SIZE /2007-04-01/pics/thumb-P1000755.JPG
[10:07:20] 213 9855
[10:07:20] *** Fetching directory listing...
[10:07:20] CWD /2007-04-01/pics
[10:07:20] 250 CWD command successful.
[10:07:20] PWD
[10:07:20] 257 "/2007-04-01/pics" is current directory.
[10:07:20] TYPE A
[10:07:20] 200 Type set to A.
[10:07:20] PASV
[10:07:20] 227 Entering Passive Mode (204,127,198,23,12,147).
[10:07:20] *** Establishing data connection with 204.127.198.23:3219...
[10:07:20] *** Data connection established.
[10:07:20] LIST -a
[10:07:20] 150 Opening ASCII mode data connection for file list
[10:07:20] 226-Transfer complete.
[10:07:20] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:20] *** Transfer completed.
[10:07:20] *** Uploading file 'thumb-P1000757.JPG'...
[10:07:20] SIZE /2007-04-01/pics/thumb-P1000757.JPG
[10:07:20] 213 8062
[10:07:20] *** Fetching directory listing...
[10:07:20] CWD /2007-04-01/pics
[10:07:21] 250 CWD command successful.
[10:07:21] PWD
[10:07:21] 257 "/2007-04-01/pics" is current directory.
[10:07:21] TYPE A
[10:07:21] 200 Type set to A.
[10:07:21] PASV
[10:07:21] 227 Entering Passive Mode (204,127,198,23,11,216).
[10:07:21] *** Establishing data connection with 204.127.198.23:3032...
[10:07:21] *** Data connection established.
[10:07:21] LIST -a
[10:07:21] 150 Opening ASCII mode data connection for file list
[10:07:21] 226-Transfer complete.
[10:07:21] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:21] *** Transfer completed.
[10:07:21] *** Uploading file 'thumb-P1000760.JPG'...
[10:07:21] SIZE /2007-04-01/pics/thumb-P1000760.JPG
[10:07:21] 213 9802
[10:07:21] *** Fetching directory listing...
[10:07:21] CWD /2007-04-01/pics
[10:07:21] 250 CWD command successful.
[10:07:21] PWD
[10:07:21] 257 "/2007-04-01/pics" is current directory.
[10:07:21] TYPE A
[10:07:22] 200 Type set to A.
[10:07:22] PASV
[10:07:22] 227 Entering Passive Mode (204,127,198,23,12,142).
[10:07:22] *** Establishing data connection with 204.127.198.23:3214...
[10:07:22] *** Data connection established.
[10:07:22] LIST -a
[10:07:22] 150 Opening ASCII mode data connection for file list
[10:07:22] 226-Transfer complete.
[10:07:22] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:22] *** Transfer completed.
[10:07:22] *** Uploading file 'thumb-P1000763.JPG'...
[10:07:22] SIZE /2007-04-01/pics/thumb-P1000763.JPG
[10:07:22] 213 8176
[10:07:22] *** Fetching directory listing...
[10:07:22] CWD /2007-04-01/pics
[10:07:22] 250 CWD command successful.
[10:07:22] PWD
[10:07:22] 257 "/2007-04-01/pics" is current directory.
[10:07:22] TYPE A
[10:07:22] 200 Type set to A.
[10:07:22] PASV
[10:07:22] 227 Entering Passive Mode (204,127,198,23,12,83).
[10:07:22] *** Establishing data connection with 204.127.198.23:3155...
[10:07:22] *** Data connection established.
[10:07:22] LIST -a
[10:07:22] 150 Opening ASCII mode data connection for file list
[10:07:23] 226-Transfer complete.
[10:07:23] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:23] *** Transfer completed.
[10:07:23] *** Uploading file 'picview.html'...
[10:07:23] CWD /2007-04-01
[10:07:23] 250 CWD command successful.
[10:07:23] SIZE /2007-04-01/picview.html
[10:07:23] 213 12427
[10:07:23] *** Fetching directory listing...
[10:07:23] CWD /2007-04-01
[10:07:23] 250 CWD command successful.
[10:07:23] PWD
[10:07:23] 257 "/2007-04-01" is current directory.
[10:07:23] TYPE A
[10:07:23] 200 Type set to A.
[10:07:23] PASV
[10:07:23] 227 Entering Passive Mode (204,127,198,23,13,70).
[10:07:23] *** Establishing data connection with 204.127.198.23:3398...
[10:07:23] *** Data connection established.
[10:07:23] LIST -a
[10:07:26] 150 Opening ASCII mode data connection for file list
[10:07:26] 226-Transfer complete.
[10:07:26] 226 Quotas on: using 16835379 of 26214400 bytes
[10:07:26] *** Transfer completed.
Comment 1 Jernej Kos 2007-04-02 20:32:32 UTC
SVN commit 649480 by kostko:

Fixed cache handling for file uploads.

BUG: 143744

 M  +35 -7     ftpsocket.cpp  


--- trunk/extragear/network/kftpgrabber/src/engine/ftpsocket.cpp #649479:649480
@@ -1668,12 +1668,22 @@
     KURL sourceFile;
     KURL destinationFile;
     
+    bool fetchedSize;
+    filesize_t destinationSize;
+    
+    void cleanup()
+    {
+      // Unclean upload termination, be sure to erase the cached stat infos
+      Cache::self()->invalidateEntry(socket(), destinationFile.directory());
+    }
+    
     void process()
     {
       switch (currentState) {
         case None: {
           sourceFile.setPath(socket()->getConfig("params.get.source"));
           destinationFile.setPath(socket()->getConfig("params.get.destination"));
+          fetchedSize = false;
           
           // Check if the local file exists
           if (!QDir::root().exists(sourceFile.path())) {
@@ -1695,8 +1705,6 @@
             socket()->sendCommand("SIZE " + destinationFile.path());
           } else {
             // SIZE is not available, try stat directly
-            Cache::self()->invalidateEntry(socket(), destinationFile.directory());
-            
             currentState = StatDone;
             socket()->protoStat(destinationFile);
           }
@@ -1704,18 +1712,16 @@
         }
         case SentSize: {
           if (socket()->isResponse("213")) {
+            destinationSize = socket()->getResponse().mid(4).toULongLong();
+            fetchedSize = true;
+            
             // File exists, we have to stat to get more data
-            Cache::self()->invalidateEntry(socket(), destinationFile.directory());
-            
             currentState = StatDone;
             socket()->protoStat(destinationFile);
           } else if (socket()->isResponse("500") || socket()->getResponse().contains("Operation not permitted", false)) {
             // Yes, some servers don't support the SIZE command :/
             socket()->setConfig("feat.size", 0);
             
-            // Try stat instead
-            Cache::self()->invalidateEntry(socket(), destinationFile.directory());
-            
             currentState = StatDone;
             socket()->protoStat(destinationFile);
           } else {
@@ -1726,6 +1732,17 @@
         }
         case StatDone: {
           if (!socket()->getStatResponse().filename().isEmpty()) {
+            if (fetchedSize) {
+              if (socket()->getStatResponse().size() != destinationSize) {
+                // It would seem that the size has changed, cached data is invalid
+                Cache::self()->invalidateEntry(socket(), destinationFile.directory());
+                
+                currentState = StatDone;
+                socket()->protoStat(destinationFile);
+                return;
+              }
+            }
+            
             // Remote file exists, emit a request for action
             DirectoryListing list;
             list.addEntry(socket()->getStatResponse());
@@ -1776,6 +1793,8 @@
               }
               case FileExistsWakeupEvent::Skip: {
                 // Transfer should be aborted
+                markClean();
+                
                 socket()->resetCommandClass(UserAbort);
                 socket()->emitEvent(Event::EventTransferComplete);
                 return;
@@ -1806,6 +1825,7 @@
         case WaitTransfer: {
           // Transfer has been completed
           socket()->getTransferFile()->close();
+          markClean();
           
           socket()->emitEvent(Event::EventTransferComplete);
           socket()->emitEvent(Event::EventReloadNeeded);
@@ -2175,6 +2195,10 @@
         companion->setConfig("params.fxp.abort", 1);
         companion->protoAbort();
       }
+      
+      // Unclean upload termination, be sure to erase the cached stat infos
+      if (!socket()->getConfigInt("params.fxp.keep_cache"))
+        Cache::self()->invalidateEntry(socket(), destinationFile.directory());
     }
     
     void process()
@@ -2183,6 +2207,7 @@
         case None: {
           sourceFile.setPath(socket()->getConfig("params.fxp.source"));
           destinationFile.setPath(socket()->getConfig("params.fxp.destination"));
+          socket()->setConfig("params.fxp.keep_cache", 0);
           
           // Who are we ? Where shall we begin ?
           if (socket()->getConfigInt("params.fxp.companion")) {
@@ -2263,6 +2288,9 @@
               }
               case FileExistsWakeupEvent::Skip: {
                 // Transfer should be aborted
+                companion->setConfig("params.fxp.keep_cache", 1);
+                socket()->setConfig("params.fxp.keep_cache", 1);
+                
                 socket()->resetCommandClass(UserAbort);
                 socket()->emitEvent(Event::EventTransferComplete);
                 return;