Bug 466671 - Discover is very slow to fetch updates
Summary: Discover is very slow to fetch updates
Status: RESOLVED UPSTREAM
Alias: None
Product: Discover
Classification: Applications
Component: PackageKit (show other bugs)
Version: 5.27.1
Platform: Fedora RPMs Linux
: NOR normal
Target Milestone: ---
Assignee: Plasma Bugs List
URL:
Keywords:
: 472175 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-03-01 18:47 UTC by Jonathan Wakely
Modified: 2023-11-06 15:11 UTC (History)
9 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Wakely 2023-03-01 18:47:57 UTC
SUMMARY

When opening discover from the systray widget when it shows updates are available, it takes 30+ seconds to show the available updates. The previous cause of slowness was supposed to have been fixed (Bug 457408) but I see no improvement in the default use case of "click on the systray widget when it shows updates". Maybe something got fixed in the KNewStuff backend, but there's another cause of slowness, and it has the same symptom: the "Fetching updates..." progress bar goes to about 97% immediately, then stops for 30 seconds.

The same slowness happens when clicking the "Refresh" button.


STEPS TO REPRODUCE
1. plasma-discover --mode Update
2. click Refresh
3. waaaaaaaiiiiiiiiiiiiiit

OBSERVED RESULT

Slow

EXPECTED RESULT

More fastish!

SOFTWARE/OS VERSIONS
Operating System: Fedora Linux 37
KDE Plasma Version: 5.27.1
KDE Frameworks Version: 5.103.0
Qt Version: 5.15.8
Kernel Version: 6.1.11-200.fc37.x86_64 (64-bit)
Graphics Platform: X11
Processors: 12 × Intel® Core™ i7-9850H CPU @ 2.60GHz
Memory: 31.1 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics 630

ADDITIONAL INFORMATION

$ plasma-discover  --listbackends
Available backends:
 * fwupd-backend
 * kns-backend
 * flatpak-backend
 * packagekit-backend

By a process of elimination I found that it's the packagekit backend that's slow. Clicking Refresh after running it as 'plasma-discover --mode Update  --backends fwupd-backend,flatpak-backend,kns-backend' finished almost immediately, and any combination of backends that doesn't involve packagekit-backend is fast. Any combination that involves packagekit-backend (including just that one, i.e. --backends packagekit-backend) takes 30+ seconds.


DNF and PackageKit can both refresh their package lists in far less time than Discover can:

$ time pkcon refresh 
Refreshing cache              [=========================]         
Loading cache                 [=========================]         
Finished                      [=========================]         

real    0m0.352s
user    0m0.007s
sys     0m0.007s
$ time sudo dnf check-update --refresh
Copr repo for centpkg owned by james                                                                                        7.2 kB/s | 3.3 kB     00:00    
Copr repo for elektroid owned by jwakely                                                                                    7.3 kB/s | 3.3 kB     00:00    
Fedora 37 - x86_64                                                                                                           35 kB/s |  18 kB     00:00    
Fedora 37 openh264 (From Cisco) - x86_64                                                                                    1.7 kB/s | 989  B     00:00    
Fedora Modular 37 - x86_64                                                                                                   31 kB/s |  18 kB     00:00    
Fedora 37 - x86_64 - Updates                                                                                                 29 kB/s |  14 kB     00:00    
Fedora Modular 37 - x86_64 - Updates                                                                                         37 kB/s |  17 kB     00:00    
google-chrome                                                                                                               5.7 kB/s | 1.3 kB     00:00    
RCM Tools for Fedora 37 (RPMs)                                                                                              5.5 kB/s | 3.8 kB     00:00    
RPM Fusion for Fedora 37 - Free                                                                                              38 kB/s | 6.8 kB     00:00    
RPM Fusion for Fedora 37 - Free - Updates                                                                                    18 kB/s | 6.5 kB     00:00    
RPM Fusion for Fedora 37 - Nonfree                                                                                           40 kB/s | 6.9 kB     00:00    
RPM Fusion for Fedora 37 - Nonfree - Updates                                                                                 39 kB/s | 6.6 kB     00:00    
RHEL8 CSB packages                                                                                                          6.2 kB/s | 3.0 kB     00:00    

real    0m7.659s
user    0m2.361s
sys     0m0.227s

In fact I can usually check *and install* all updates using dnf before Discover has even managed to show me the available updates.
Comment 1 Aleix Pol 2023-03-30 23:08:18 UTC
That's interesting. Would you be able to run "pkmon" in parallel to one of these tests? It would help us pinpoint which is the transaction that is taking londer. It should take something similar to what pkcon does (~8s).
Comment 2 Jonathan Wakely 2023-05-03 12:11:37 UTC
Here's the output of pkmon when I click "Refresh" in the plasma-discover GUI:

$ pkmon
Transactions:
 [none]
daemon connected=1
network status=online
Transactions:
 1      /5271_ecabaeab
/5271_ecabaeab  allow_cancel 1
/5271_ecabaeab  percentage   -1
/5271_ecabaeab  role         refresh-cache
/5271_ecabaeab  sender       /usr/bin/plasma-discover
/5271_ecabaeab  status       setup
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   1
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   6
/5271_ecabaeab  percentage   7
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   13
/5271_ecabaeab  percentage   14
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   20
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   27
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  percentage   33
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   34
/5271_ecabaeab  percentage   48
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   54
/5271_ecabaeab  percentage   55
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   60
/5271_ecabaeab  percentage   61
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   68
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   75
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   81
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   88
/5271_ecabaeab  percentage   89
/5271_ecabaeab  status       download-repository
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   95
/5271_ecabaeab  percentage   96
/5271_ecabaeab  status       query
/5271_ecabaeab  status       loading-cache
/5271_ecabaeab  percentage   100
/5271_ecabaeab  status       finished
/5271_ecabaeab  exit code: success
Transactions:
 [none]
Transactions:
 1      /5272_abaddace
/5272_abaddace  allow_cancel 1
/5272_abaddace  percentage   -1
/5272_abaddace  role         get-updates
/5272_abaddace  sender       /usr/bin/plasma-discover
/5272_abaddace  status       setup
/5272_abaddace  status       query
updates-changed
/5272_abaddace  percentage   3
/5272_abaddace  status       loading-cache
Transactions:
 1      /5272_abaddace
 2      /5273_cbccceeb
/5273_cbccceeb  allow_cancel 1
/5273_cbccceeb  percentage   -1
/5273_cbccceeb  role         get-updates
/5273_cbccceeb  sender       /usr/libexec/DiscoverNotifier
/5273_cbccceeb  status       wait
/5272_abaddace  percentage   6
/5272_abaddace  percentage   8
/5272_abaddace  percentage   39
/5272_abaddace  percentage   91
/5272_abaddace  percentage   100
/5272_abaddace  status       finished
Transactions:
 1      /5273_cbccceeb
/5272_abaddace  exit code: success
/5273_cbccceeb  status       setup
/5273_cbccceeb  percentage   91
/5273_cbccceeb  percentage   100
/5273_cbccceeb  status       finished
Transactions:
 [none]
/5273_cbccceeb  exit code: success
Comment 3 Jonathan Wakely 2023-05-03 12:15:22 UTC
(In reply to Jonathan Wakely from comment #0)
> SUMMARY
> 
> When opening discover from the systray widget when it shows updates are
> available, it takes 30+ seconds to show the available updates. The previous
> cause of slowness was supposed to have been fixed (Bug 457408) but I see no
> improvement in the default use case of "click on the systray widget when it
> shows updates". Maybe something got fixed in the KNewStuff backend, but
> there's another cause of slowness, and it has the same symptom: the
> "Fetching updates..." progress bar goes to about 97% immediately, then stops
> for 30 seconds.
> 
> The same slowness happens when clicking the "Refresh" button.
> 
> 
> STEPS TO REPRODUCE
> 1. plasma-discover --mode Update
> 2. click Refresh
> 3. waaaaaaaiiiiiiiiiiiiiit
> 
> OBSERVED RESULT
> 
> Slow
> 
> EXPECTED RESULT
> 
> More fastish!
> 
> SOFTWARE/OS VERSIONS
> Operating System: Fedora Linux 37
> KDE Plasma Version: 5.27.1
> KDE Frameworks Version: 5.103.0
> Qt Version: 5.15.8
> Kernel Version: 6.1.11-200.fc37.x86_64 (64-bit)
> Graphics Platform: X11
> Processors: 12 × Intel® Core™ i7-9850H CPU @ 2.60GHz
> Memory: 31.1 GiB of RAM
> Graphics Processor: Mesa Intel® UHD Graphics 630
> 
> ADDITIONAL INFORMATION
> 
> $ plasma-discover  --listbackends
> Available backends:
>  * fwupd-backend
>  * kns-backend
>  * flatpak-backend
>  * packagekit-backend
> 
> By a process of elimination I found that it's the packagekit backend that's
> slow. Clicking Refresh after running it as 'plasma-discover --mode Update 
> --backends fwupd-backend,flatpak-backend,kns-backend' finished almost
> immediately, and any combination of backends that doesn't involve
> packagekit-backend is fast. Any combination that involves packagekit-backend
> (including just that one, i.e. --backends packagekit-backend) takes 30+
> seconds.
> 
> 
> DNF and PackageKit can both refresh their package lists in far less time
> than Discover can:
> 
> $ time pkcon refresh 
> Refreshing cache              [=========================]         
> Loading cache                 [=========================]         
> Finished                      [=========================]         
> 
> real    0m0.352s
> user    0m0.007s
> sys     0m0.007s


Although `pkcon refresh` takes less than a second, `pkcon refresh force` is much slower:

$ time pkcon refresh force
Refreshing cache              [=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Querying                      [=========================]         
Loading cache                 [=========================]         
Finished                      [=========================]         

real    0m33.664s
user    0m0.024s
sys     0m0.013s


It seems that discover is doing the equivalent of `refresh force` when you click on the systray widget, even if it's recently done a refresh. That's why it's always so slow.
Comment 4 svlmrc 2023-09-02 10:10:25 UTC
I can confirm it's still the same for 5.27.7, Discover "Update" takes the same time as "pkcon refresh force".
Comment 5 Alessandro Astone 2023-10-01 16:01:48 UTC
`pkcon refresh` takes less than a second because it does nothing in Fedora. That is, unfortunately, intended.
We have to use of `force` (or `--cache-age 1`) to actually download the new metadata: https://src.fedoraproject.org/rpms/plasma-discover/blob/rawhide/f/discover-5.21.4-pk_refresh_force.patch

But improvements are coming:
 - https://github.com/rpm-software-management/libdnf/pull/1626
 - https://github.com/PackageKit/PackageKit/pull/678
 - https://invent.kde.org/plasma/discover/-/merge_requests/640 -- We need to use `--cache-age 1` instead of `force` to make use of the other upstream improvements

With that it'll only be a bit slower than `dnf makecache`
Comment 6 Nate Graham 2023-10-11 18:00:43 UTC
*** Bug 472175 has been marked as a duplicate of this bug. ***
Comment 7 Alessandro Astone 2023-11-06 15:11:25 UTC
This is fixed in Fedora 39 as a day-1 update