Bug 41345 - kdat memory allocation problem
Summary: kdat memory allocation problem
Status: RESOLVED WORKSFORME
Alias: None
Product: kdat
Classification: Miscellaneous
Component: general (show other bugs)
Version: 2.0.1
Platform: Compiled Sources Linux
: NOR normal
Target Milestone: ---
Assignee: Larry Widman
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2002-04-20 09:48 UTC by Andrew Schulman
Modified: 2008-02-15 03:46 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gerald Sternagl 2002-04-20 09:35:12 UTC
(*** This bug was imported into bugs.kde.org ***)

Package:           kdat
Version:           2.0.1 (using KDE 3.0.0 )
Severity:          normal
Installed from:    compiled sources
Compiler:          gcc version 2.95.3 20010315 (release)
OS:                Linux (i686) release 2.4.16
OS/Compiler notes: 

kdat keeps the complete list of files which have been backed up in an open window during the backup process. As more files you back up as more memory gets allocated for the file display window which in my case (128MB RAM in system appr. 200.000 files 2.8GB) leads to a situation where the system is constanly swapping and can't get enough memory and CPU to finish the backup process. So the backup runs forever (>24h). A handfired tar command (without kdat) for the same files only takes appr. 1 hour. I recommend to write the list of files which have been backed up directly into a file and only display the current file in the GUI.

(Submitted via bugs.kde.org)
(Called from KBugReport dialog)
Comment 1 Lawrence E. Widman 2002-04-21 15:15:38 UTC
> kdat keeps the complete list of files which have been backed
> up in an open window ... I recommend to write the list of
> files which have been backed up directly into a file and only
> display the current file in the GUI.

Good idea.  I've had the same problem myself.

I'm not sure the best way to handle this but yours is
certainly a reasonable option.  The problem arises because Qt
allocates about 2000 bytes for an 80 byte string.  I'm guessing
that this is due to internationalization and am hoping it'll
be fixed in the future.

In the meantime I was thinking of providing the user with a
switch that selects between the current behavior and the
behavior you suggest.  Does this sound reasonable?
Comment 2 Gerald Sternagl 2002-04-21 15:25:16 UTC
Thanks for the quick reply. I believe your suggested fix sounds reasonable.=
=20
Even if QT gets fixed in the way you indicated it would allocate some amoun=
t=20
of memory for each file entry and therefor wouldn't scale well for systems=
=20
with lots of files. Also it is not redundant. So if your machine crashes=20
during backup you end up having no catalogue and will have to recreate one =
by=20
reading the whole tape again. So I believe writing the filenames into a fil=
e=20
and just displaying a reasonable but limited amount of last filenames into =
a=20
scroll window would probably be a safer solution.

Gerald

On Sunday 21 April 2002 17:15 Lawrence E. Widman wrote:
> > kdat keeps the complete list of files which have been backed
> > up in an open window ... I recommend to write the list of
> > files which have been backed up directly into a file and only
> > display the current file in the GUI.
>
> Good idea.  I've had the same problem myself.
>
> I'm not sure the best way to handle this but yours is
> certainly a reasonable option.  The problem arises because Qt
> allocates about 2000 bytes for an 80 byte string.  I'm guessing
> that this is due to internationalization and am hoping it'll
> be fixed in the future.
>
> In the meantime I was thinking of providing the user with a
> switch that selects between the current behavior and the
> behavior you suggest.  Does this sound reasonable?
Comment 3 Andrew Schulman 2003-02-04 19:56:15 UTC
I also have this problem.  It makes kdat unbearably slow after a while.  The
workaround that I found was to go to the top of the listview.  Once the list of
files isn't scrolling by any longer, the speed picks up sharply-- although it's
still quite a bit slower than just tar.

I like kdat, but it's bothersome to me that it's so slow.  When I run tar, text
output comes by pretty fast-- much faster than in kdat.  So the slowness has to
come from either the GUI, or whatever processing kdat is doing to create its
index.  Not to be naive, but is it possible to have one thread running tar, and
another processing tar's text output?  Might speed things up.  Not that I know
anything about it of course... just a suggestion.

(Sorry but I'm not sure which version of kdat I'm using-- it's whatever is the
latest at download.us.kde.org.)
Comment 4 Andrew Schulman 2003-08-03 07:43:59 UTC
The author of KDirStat has recently gotten enormous speed improvements by
limiting cloning of the internal tree to the QListView widget.  KDirStat 2.3.6
would traverse my root directory tree in 152 seconds; version 2.3.7 does it in 9
seconds.  This is not a misprint, it's a better-than-order-of-magnitude
improvement in speed.  Please see http://kdirstat.sourceforge.net/, news from
2003-05-25.

Could this approach apply to kdat too?

Comment 5 Stephan Kulow 2004-05-25 09:18:00 UTC
Replaced gerald.sternagl@t-online.de with andrex@radsoft.zzn.com due to bounces by reporter
Comment 6 George Goldberg 2007-12-18 04:39:47 UTC
Is this bug still present in a recent version of KDE, such as 3.5.8 or KDE 4 RC2?
Comment 7 George Goldberg 2008-02-15 03:46:13 UTC
Closing due to no response. Please reopen if this bug can be reproduced with KDE >= 4.0.1