Bug 389273

Also by copying/moving some images selected by a group, any images will be copied not the selected once. This behavior is unpredictically and very strange.

Reproducible with 5.9.0 pre release AppImage ?

https://files.kde.org/digikam/

Did you use Mysql or sqlite database ?

Gilles Caulier

I am using sqlite. The total amount of images is approx. 16.000. One album is connected as a network drive, mount via smbfs. The other album is connected to local drive.
I didn't use 5.9.0. So I have to check later.
Manfred Doerner

Another point : run the AppImage from a console and look the debug trace while you perform the operation in DK. This can be instructive.

Gilles Caulier

The problem is also reproducible with version 5.9.0. 
Debug logs are only "[New Thread...]" and "[Thread....exited]", not more.
I selected 156 items and this was showed but round about 3000 items where copied.

How I reproduced: 
1. Went in my my Album with 1896 images.
2. Showed all images also which are grouped. I am using grouping.
3. Selected 4+ rating. -> 156 images showed.
3. Selected all 156 images. It signed at status bar that 156 images are selected.
4. Copied to a new album. Digikam noticed that 156 images will be copied. 
5. But all 1896 images where copied. That's the problem.

One notice more:
After copying images by digikam the RAM buff/cache is always using 5.6GB. It is freezed and isn't decreasing. So I had to reboot my computer.

Copying all images of the group is a very old behavior of digiKam. We have recently created the possibility that this behavior can already be configured for many actions. For file operations via drag & drop or copy / move it is not yet implemented. You have to open all groups and select only the images you want to copy. That the memory overflows when copying, I can not reproduce here. I can imagine no reason at the moment which could be triggered. A test here with many more images shows no abnormalities.

Maik

I investigated one more time and I can explain my mistake.
I am using grouping of images. When the group is opened, the "master" image, which hold the information, looks very similar to others. It differs only by a "number" which indicates the amount of the group. So when only the "master" image is selected digikam shows that one image is selected, ok. Now when I copy/move only the one selected "master" image digikam will be copying the hole group without any information. This was a little bit confusion for me. But now I know this issue and it's ok for me.
In fact the memory overflows isn't a problem of digikam.
Now this topic can be closed because it isn't a bug.
Many thanks for the support.
Finally, digikam is a great software:-) Thanks a lot to the whole digikam team!
Manfred

Created attachment 110071 [details]
DIOProgressManager.patch

This is the first version for a patch to provide a progress bar and the ability to cancel file operations (copy, move and delete items).

Maik

Comment on attachment 110071 [details]
DIOProgressManager.patch

>diff --git a/libs/database/utils/dio.cpp b/libs/database/utils/dio.cpp
>index 82813bae60..952c9e8336 100644
>--- a/libs/database/utils/dio.cpp
>+++ b/libs/database/utils/dio.cpp
>@@ -48,6 +48,7 @@
> #include "collectionmanager.h"
> #include "dnotificationwrapper.h"
> #include "digikamapp.h"
>+#include "progressmanager.h"
> 
> namespace Digikam
> {
>@@ -188,7 +189,7 @@ void DIO::Private::processJob(int operation, const QList<QUrl>& srcList, const Q
>     {
>         emit jobToCreate(operation, finder.remoteFiles, dest);
>         // stat'ing is unreliable; just try to copy and suppress error message
>-        emit jobToCreate(operation | SourceStatusUnknown, finder.possibleRemoteSidecars, dest);
>+        emit jobToCreate(operation | IOJobsThread::SourceStatusUnknown, finder.possibleRemoteSidecars, dest);
>     }
> }
> 
>@@ -200,7 +201,7 @@ void DIO::Private::processRename(const QUrl& src, const QUrl& dest)
>     {
>         for (int i = 0 ; i < finder.localFiles.length() ; ++i)
>         {
>-            emit jobToCreate(Rename, QList<QUrl>() << finder.localFiles.at(i),
>+            emit jobToCreate(IOJobsThread::Rename, QList<QUrl>() << finder.localFiles.at(i),
>                              QUrl::fromLocalFile(dest.toLocalFile() + finder.localFileSuffixes.at(i)));
>         }
> 
>@@ -209,11 +210,11 @@ void DIO::Private::processRename(const QUrl& src, const QUrl& dest)
> 
>     for (int i = 0 ; i < finder.remoteFileSuffixes.length() ; ++i)
>     {
>-        emit jobToCreate(Rename | SourceStatusUnknown,
>+        emit jobToCreate(IOJobsThread::Rename | IOJobsThread::SourceStatusUnknown,
>                          QList<QUrl>() << finder.possibleRemoteSidecars.at(i),
>                          QUrl::fromLocalFile(dest.toLocalFile() + finder.possibleRemoteSidecarSuffixes.at(i)));
>     }
>-    emit jobToCreate(Rename, QList<QUrl>() << src, dest);
>+    emit jobToCreate(IOJobsThread::Rename, QList<QUrl>() << src, dest);
> }
> 
> void DIO::Private::albumToAlbum(int operation, const PAlbum* const src, const PAlbum* const dest)
>@@ -231,7 +232,7 @@ void DIO::Private::imagesToAlbum(int operation, const QList<ImageInfo>& infos, c
>     QList<qlonglong> ids;
>     QList<QUrl>      urls;
> 
>-    if (operation == Move)
>+    if (operation == IOJobsThread::IOJobsThread::Move)
>     {
>         // update the image infos
>         CoreDbAccess access;
>@@ -290,7 +291,7 @@ void DIO::Private::deleteFiles(const QList<ImageInfo>& infos, bool useTrash)
> 
>     qCDebug(DIGIKAM_DATABASE_LOG) << "Deleting files:" << urls;
> 
>-    emit jobToProcess(useTrash ? Trash : Delete, urls, QUrl());
>+    emit jobToProcess(useTrash ? IOJobsThread::Trash : IOJobsThread::Delete, urls, QUrl());
> }
> 
> // ------------------------------------------------------------------------------------------------
>@@ -335,19 +336,38 @@ void DIO::createJob(int operation, const QList<QUrl>& src, const QUrl& dest)
>         return;
>     }
> 
>+    ProgressItem* item      = 0;
>     IOJobsThread* jobThread = 0;
>-    int flags               = operation & FlagMask;
>-    operation              &= OperationMask;
>+    int flags               = operation & IOJobsThread::FlagMask;
>+    operation              &= IOJobsThread::OperationMask;
> 
>-    if (operation == Copy)
>+    if (operation == IOJobsThread::Copy)
>     {
>+        item = getProgressItem(IOJobsThread::Copy);
>+
>+        if (!item || item->totalCompleted())
>+        {
>+            item = ProgressManager::instance()->createProgressItem(QLatin1String("DIOCopy"),
>+                                                                   i18n("Copy"), QString(), true, false);
>+        }
>+
>+        item->setTotalItems(item->totalItems() + src.count());
>         jobThread = IOJobsManager::instance()->startCopy(src, dest);
>     }
>-    else if (operation == Move)
>+    else if (operation == IOJobsThread::Move)
>     {
>+        item = getProgressItem(IOJobsThread::Move);
>+
>+        if (!item || item->totalCompleted())
>+        {
>+            item = ProgressManager::instance()->createProgressItem(QLatin1String("DIOMove"),
>+                                                                   i18n("Move"), QString(), true, false);
>+        }
>+
>+        item->setTotalItems(item->totalItems() + src.count());
>         jobThread = IOJobsManager::instance()->startMove(src, dest);
>     }
>-    else if (operation == Rename)
>+    else if (operation == IOJobsThread::Rename)
>     {
>         if (src.size() != 1)
>         {
>@@ -363,23 +383,53 @@ void DIO::createJob(int operation, const QList<QUrl>& src, const QUrl& dest)
>         connect(jobThread, SIGNAL(renameFailed(QUrl)),
>                 this, SIGNAL(imageRenameFailed(QUrl)));
>     }
>-    else if (operation == Trash)
>+    else if (operation == IOJobsThread::Trash)
>     {
>+        item = getProgressItem(IOJobsThread::Trash);
>+
>+        if (!item || item->totalCompleted())
>+        {
>+            item = ProgressManager::instance()->createProgressItem(QLatin1String("DIOTrash"),
>+                                                                   i18n("Trash"), QString(), true, false);
>+        }
>+
>+        item->setTotalItems(item->totalItems() + src.count());
>         jobThread = IOJobsManager::instance()->startDelete(src);
>     }
>     else // operation == Del
>     {
>+        item = getProgressItem(IOJobsThread::Delete);
>+
>+        if (!item || item->totalCompleted())
>+        {
>+            item = ProgressManager::instance()->createProgressItem(QLatin1String("DIODelete"),
>+                                                                   i18n("Delete"), QString(), true, false);
>+        }
>+
>+        item->setTotalItems(item->totalItems() + src.count());
>         qCDebug(DIGIKAM_DATABASE_LOG) << "SRCS " << src;
>         jobThread = IOJobsManager::instance()->startDelete(src, false);
>     }
> 
>-    if (flags & SourceStatusUnknown)
>+    if (flags & IOJobsThread::SourceStatusUnknown)
>     {
>         jobThread->setKeepErrors(false);
>     }
> 
>+    connect(jobThread, SIGNAL(oneProccessed(int)),
>+            this, SLOT(slotOneProccessed(int)));
>+
>     connect(jobThread, SIGNAL(finished()),
>             this, SLOT(slotResult()));
>+
>+    if (item)
>+    {
>+        connect(item, SIGNAL(progressItemCanceled(ProgressItem*)),
>+                jobThread, SLOT(cancel()));
>+
>+        connect(item, SIGNAL(progressItemCanceled(ProgressItem*)),
>+                this, SLOT(slotCancel(ProgressItem*)));
>+    }
> }
> 
> void DIO::slotResult()
>@@ -398,6 +448,27 @@ void DIO::slotResult()
>         DNotificationWrapper(QString(), errors, DigikamApp::instance(),
>                              DigikamApp::instance()->windowTitle());
>     }
>+
>+    ProgressItem* const item = getProgressItem(jobThread->operation());
>+    slotCancel(item);
>+}
>+
>+void DIO::slotCancel(ProgressItem* item)
>+{
>+    if (item)
>+    {
>+        item->setComplete();
>+    }
>+}
>+
>+void DIO::slotOneProccessed(int operation)
>+{
>+    ProgressItem* const item = getProgressItem(operation);
>+
>+    if (item)
>+    {
>+        item->advance(1);
>+    }
> }
> 
> void DIO::slotRenamed(const QUrl& oldUrl, const QUrl& newUrl)
>@@ -410,6 +481,31 @@ void DIO::slotRenamed(const QUrl& oldUrl, const QUrl& newUrl)
>     emit imageRenameSucceeded(oldUrl);
> }
> 
>+ProgressItem* DIO::getProgressItem(int operation)
>+{
>+    ProgressItem* item = 0;
>+
>+    switch ((IOJobsThread::Operation)operation)
>+    {
>+        case IOJobsThread::Copy:
>+            item = ProgressManager::instance()->findItembyId(QLatin1String("DIOCopy"));
>+            break;
>+        case IOJobsThread::Move:
>+            item = ProgressManager::instance()->findItembyId(QLatin1String("DIOMove"));
>+            break;
>+        case IOJobsThread::Trash:
>+            item = ProgressManager::instance()->findItembyId(QLatin1String("DIOTrash"));
>+            break;
>+        case IOJobsThread::Delete:
>+            item = ProgressManager::instance()->findItembyId(QLatin1String("DIODelete"));
>+            break;
>+        default:
>+            break;
>+    }
>+
>+    return item;
>+}
>+
> // Album -> Album -----------------------------------------------------
> 
> void DIO::copy(const PAlbum* const src, const PAlbum* const dest)
>@@ -419,7 +515,7 @@ void DIO::copy(const PAlbum* const src, const PAlbum* const dest)
>         return;
>     }
> 
>-    instance()->d->albumToAlbum(Copy, src, dest);
>+    instance()->d->albumToAlbum(IOJobsThread::Copy, src, dest);
> }
> 
> void DIO::move(const PAlbum* src, const PAlbum* const dest)
>@@ -433,7 +529,7 @@ void DIO::move(const PAlbum* src, const PAlbum* const dest)
>     AlbumManager::instance()->removeWatchedPAlbums(src);
> #endif
> 
>-    instance()->d->albumToAlbum(Move, src, dest);
>+    instance()->d->albumToAlbum(IOJobsThread::Move, src, dest);
> }
> 
> // Images -> Album ----------------------------------------------------
>@@ -445,7 +541,7 @@ void DIO::copy(const QList<ImageInfo>& infos, const PAlbum* const dest)
>         return;
>     }
> 
>-    instance()->d->imagesToAlbum(Copy, infos, dest);
>+    instance()->d->imagesToAlbum(IOJobsThread::Copy, infos, dest);
> }
> 
> void DIO::move(const QList<ImageInfo>& infos, const PAlbum* const dest)
>@@ -455,7 +551,7 @@ void DIO::move(const QList<ImageInfo>& infos, const PAlbum* const dest)
>         return;
>     }
> 
>-    instance()->d->imagesToAlbum(Move, infos, dest);
>+    instance()->d->imagesToAlbum(IOJobsThread::Move, infos, dest);
> }
> 
> // External files -> album --------------------------------------------
>@@ -472,7 +568,7 @@ void DIO::copy(const QList<QUrl>& srcList, const PAlbum* const dest)
>         return;
>     }
> 
>-    instance()->d->filesToAlbum(Copy, srcList, dest);
>+    instance()->d->filesToAlbum(IOJobsThread::Copy, srcList, dest);
> }
> 
> void DIO::move(const QUrl& src, const PAlbum* const dest)
>@@ -487,7 +583,7 @@ void DIO::move(const QList<QUrl>& srcList, const PAlbum* const dest)
>         return;
>     }
> 
>-    instance()->d->filesToAlbum(Move, srcList, dest);
>+    instance()->d->filesToAlbum(IOJobsThread::Move, srcList, dest);
> }
> 
> // Rename --------------------------------------------------------------
>@@ -520,7 +616,7 @@ void DIO::del(const PAlbum* const album, bool useTrash)
>     AlbumManager::instance()->removeWatchedPAlbums(album);
> #endif
> 
>-    instance()->createJob(useTrash ? Trash : Delete, QList<QUrl>() << album->fileUrl(), QUrl());
>+    instance()->createJob(useTrash ? IOJobsThread::Trash : IOJobsThread::Delete, QList<QUrl>() << album->fileUrl(), QUrl());
> }
> 
> } // namespace Digikam
>diff --git a/libs/database/utils/dio.h b/libs/database/utils/dio.h
>index acc88c80c7..b4d3822ef9 100644
>--- a/libs/database/utils/dio.h
>+++ b/libs/database/utils/dio.h
>@@ -40,6 +40,7 @@ namespace Digikam
> 
> class PAlbum;
> class ImageInfo;
>+class ProgressItem;
> 
> class DIGIKAM_EXPORT DIO : public QObject
> {
>@@ -91,9 +92,15 @@ Q_SIGNALS:
>     void imageRenameSucceeded(const QUrl&);
>     void imageRenameFailed(const QUrl&);
> 
>+protected:
>+
>+    ProgressItem* getProgressItem(int operation);
>+
> protected Q_SLOTS:
> 
>     void slotResult();
>+    void slotCancel(ProgressItem* item);
>+    void slotOneProccessed(int operation);
>     void slotRenamed(const QUrl& oldUrl, const QUrl& newUrl);
>     void createJob(int operation, const QList<QUrl>& src, const QUrl& dest);
> 
>diff --git a/libs/database/utils/dio_p.h b/libs/database/utils/dio_p.h
>index 4e61874c9c..5768de1ced 100644
>--- a/libs/database/utils/dio_p.h
>+++ b/libs/database/utils/dio_p.h
>@@ -109,19 +109,6 @@ private:
>     void process(const QList<ImageInfo>& source);
> };
> 
>-enum Operation
>-{
>-    Copy                = 1 << 0,
>-    Move                = 1 << 1,
>-    Rename              = 1 << 2,
>-    Trash               = 1 << 3,
>-    Delete              = 1 << 4,
>-    SourceStatusUnknown = 1 << 20,
>-
>-    OperationMask       = 0xffff,
>-    FlagMask            = 0xffff0000
>-};
>-
> } // anonymous namespace
> 
> } // namespace Digikam
>diff --git a/libs/iojobs/iojob.cpp b/libs/iojobs/iojob.cpp
>index f5f1dc2ece..fe8a5dced1 100644
>--- a/libs/iojobs/iojob.cpp
>+++ b/libs/iojobs/iojob.cpp
>@@ -61,6 +61,11 @@ CopyJob::CopyJob(const QUrl& src, const QUrl& dest, bool isMove)
> 
> void CopyJob::run()
> {
>+    if (m_cancel)
>+    {
>+        return;
>+    }
>+
>     QFileInfo srcInfo(m_src.toLocalFile());
>     QDir dstDir(m_dest.toLocalFile());
> 
>@@ -166,6 +171,11 @@ DeleteJob::DeleteJob(const QUrl& srcToDelete, bool useTrash, bool markAsObsolete
> 
> void DeleteJob::run()
> {
>+    if (m_cancel)
>+    {
>+        return;
>+    }
>+
>     QFileInfo fileInfo(m_srcToDelete.toLocalFile());
>     qCDebug(DIGIKAM_IOJOB_LOG) << "Deleting:   " << fileInfo.filePath();
>     qCDebug(DIGIKAM_IOJOB_LOG) << "File exists?" << fileInfo.exists();
>diff --git a/libs/iojobs/iojobsthread.cpp b/libs/iojobs/iojobsthread.cpp
>index 5014ebc7da..d315eca7cb 100644
>--- a/libs/iojobs/iojobsthread.cpp
>+++ b/libs/iojobs/iojobsthread.cpp
>@@ -44,12 +44,14 @@ public:
> 
>     Private()
>         : jobsCount(0),
>+          operation(SourceStatusUnknown),
>           isCanceled(false),
>           keepErrors(true)
>     {
>     }
> 
>     int            jobsCount;
>+    int            operation;
>     bool           isCanceled;
> 
>     bool           keepErrors;
>@@ -81,6 +83,7 @@ void IOJobsThread::copy(const QList<QUrl>& srcFiles, const QUrl destAlbum)
>         d->jobsCount++;
>     }
> 
>+    d->operation = Copy;
>     appendJobs(collection);
> }
> 
>@@ -98,6 +101,7 @@ void IOJobsThread::move(const QList<QUrl>& srcFiles, const QUrl destAlbum)
>         d->jobsCount++;
>     }
> 
>+    d->operation = Move;
>     appendJobs(collection);
> }
> 
>@@ -107,7 +111,7 @@ void IOJobsThread::deleteFiles(const QList<QUrl>& srcsToDelete, bool useTrash)
> 
>     foreach (const QUrl& url, srcsToDelete)
>     {
>-        DeleteJob* const j = new DeleteJob(url, useTrash,true);
>+        DeleteJob* const j = new DeleteJob(url, useTrash, true);
> 
>         connectOneJob(j);
> 
>@@ -115,6 +119,15 @@ void IOJobsThread::deleteFiles(const QList<QUrl>& srcsToDelete, bool useTrash)
>         d->jobsCount++;
>     }
> 
>+    if (useTrash)
>+    {
>+        d->operation = Trash;
>+    }
>+    else
>+    {
>+        d->operation = Delete;
>+    }
>+
>     appendJobs(collection);
> }
> 
>@@ -194,6 +207,7 @@ void IOJobsThread::renameFile(const QUrl& srcToRename, const QUrl& newName)
>     collection.insert(j, 0);
>     d->jobsCount++;
> 
>+    d->operation = Rename;
>     appendJobs(collection);
> }
> 
>@@ -266,6 +280,8 @@ void IOJobsThread::oneJobFinished()
> {
>     d->jobsCount--;
> 
>+    emit oneProccessed(d->operation);
>+
>     if (d->jobsCount == 0)
>     {
>         emit finished();
>@@ -278,4 +294,9 @@ void IOJobsThread::error(const QString& errString)
>     d->errorsList.append(errString);
> }
> 
>+int IOJobsThread::operation()
>+{
>+    return d->operation;
>+}
>+
> } // namespace Digikam
>diff --git a/libs/iojobs/iojobsthread.h b/libs/iojobs/iojobsthread.h
>index d7dd979e9b..477f2de588 100644
>--- a/libs/iojobs/iojobsthread.h
>+++ b/libs/iojobs/iojobsthread.h
>@@ -41,6 +41,19 @@ class DIGIKAM_EXPORT IOJobsThread : public ActionThreadBase
> 
> public:
> 
>+enum Operation
>+{
>+    Copy                = 1 << 0,
>+    Move                = 1 << 1,
>+    Rename              = 1 << 2,
>+    Trash               = 1 << 3,
>+    Delete              = 1 << 4,
>+    SourceStatusUnknown = 1 << 20,
>+
>+    OperationMask       = 0xffff,
>+    FlagMask            = 0xffff0000
>+};
>+
>     IOJobsThread(QObject* const parent);
>     ~IOJobsThread();
> 
>@@ -90,11 +103,6 @@ public:
>      */
>     void renameFile(const QUrl& srcToRename, const QUrl& newName);
> 
>-    /**
>-     * @brief cancels thread execution
>-     */
>-    void cancel();
>-
>     /**
>      * @brief isCanceled
>      * @return true if the thread was inturrupted
>@@ -125,6 +133,12 @@ public:
>      */
>     QList<QString>& errorsList();
> 
>+   /**
>+     * @brief operation
>+     * @return
>+     */
>+    int operation();
>+
> public Q_SLOTS:
> 
>     /**
>@@ -139,12 +153,18 @@ public Q_SLOTS:
>      */
>     void error(const QString& errString);
> 
>+    /**
>+     * @brief cancels thread execution
>+     */
>+    void cancel();
>+
> Q_SIGNALS:
> 
>     void finished();
> 
>     void renamed(const QUrl& oldUrl, const QUrl& newURl);
>     void renameFailed(const QUrl& oldUrl);
>+    void oneProccessed(int operation);
> 
>     void collectionTrashItemInfo(const DTrashItemInfo& trashItemInfo);
>

I close this bug as a duplicate, because the desire for a progress bar and cancel function is also described here.

Maik

*** This bug has been marked as a duplicate of bug 233063 ***

Git commit 1803bd2a555890b91095d92b0321e1357db0fa59 by Maik Qualmann.
Committed on 25/02/2018 at 19:11.
Pushed by mqualmann into branch 'development/6.0.0'.

add progress indicator and cancel function to file operations
FIXED-IN: 6.0.0

M  +2    -1    NEWS.6.0.0
M  +101  -5    libs/database/utils/dio.cpp
M  +5    -0    libs/database/utils/dio.h
M  +10   -0    libs/iojobs/iojob.cpp
M  +8    -8    libs/iojobs/iojobsmanager.cpp
M  +8    -4    libs/iojobs/iojobsmanager.h
M  +21   -7    libs/iojobs/iojobsthread.cpp
M  +20   -9    libs/iojobs/iojobsthread.h

https://commits.kde.org/digikam/1803bd2a555890b91095d92b0321e1357db0fa59