Bug 379739

Summary:	insane baloo_file (and plasmashell) 256 GB virtual memory size
Product:	[Frameworks and Libraries] frameworks-baloo	Reporter:	skierpage <info>
Component:	Baloo File Daemon	Assignee:	Pinak Ahuja <pinak.ahuja>
Status:	RESOLVED NOT A BUG
Severity:	normal	CC:	kde, opensource, rdieter
Priority:	NOR
Version First Reported In:	5.33.0
Target Milestone:	---
Platform:	Other
OS:	Linux
Latest Commit:		Version Fixed/Implemented In:
Sentry Crash Report:

Description skierpage 2017-05-12 05:50:26 UTC

I'm running Fedora 25 KDE spin on an 8GB Thinkpad T460 laptop with SSD. Its fan got loud, so I ran htop, found Firefox at 99% CPU, and closed it. But I noticed some insane VIRT(ual memory sizes) for plasmashell and baloo_file, 261G and 256G! The output of `top` is below (.255 terabytes of virtual memory!), you can see the laptop has plenty of RAM free and isn't even swapping. baloosearch works fine.

I futzed around in /proc/<pid_of_baloo_file> and in /proc/1454/smaps I found a possible smoking gun:

7f6150000000-7fa150000000 r--s 00000000 fd:03 133792                     /home/spage/.local/share/baloo/index
Size:           268435456 kB
Rss:              667868 kB

That 268 GB size for baloo/index is obviously wrong (it's bigger than my entire SSD), its actual size is 684468K. The plasmashell process's smaps file has a similar entry for baloo_index, same huge size. balooctl reports the right size:

% balooctl status
Baloo File Indexer is running
Indexer state: Indexing file content
Indexed 290638 / 290638 files
Current size of index is 668.43 MiB

Bug 378749 seems similar, but doesn't mention the baloo_file process.
Maybe this is related to bug 371878 . It might suggest a bug in the way baloo_file and plasmashell handle the baloo_index file is handled. Or it could be a meaningless glitch.  I've never created a .255 terabyte process before :-)

- - - -

`top` output:
Tasks: 205 total,   1 running, 200 sleeping,   3 stopped,   1 zombie
%Cpu(s):  4.3 us,  1.7 sy,  0.6 ni, 92.5 id,  0.2 wa,  0.3 hi,  0.4 si,  0.0 st
KiB Mem :  8038468 total,   443528 free,  2164324 used,  5430616 buff/cache
KiB Swap:  6402044 total,  6402044 free,        0 used.  5172720 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND             
 1459 spage     20   0  0.255t 285740 150876 S   5.9  3.6  26:20.73 plasmashell         
 1454 spage     39  19  0.251t 811336 686360 S   0.0 10.1  28:25.88 baloo_file          
 1448 spage     20   0 3375068 116400  68672 S   0.0  1.4  37:56.80 kwin_x11            
16487 spage     20   0 3080924 446576 135028 S   0.0  5.6   9:06.60 firefox

Comment 1 Rex Dieter 2017-05-13 19:03:04 UTC

fyi, the VIRT table of memory you reference is not a good metric.  RES+SHR is a better measure of real usage.

Comment 2 skierpage 2017-05-14 20:13:27 UTC

(In reply to Rex Dieter from comment #1)
> fyi, the VIRT table of memory you reference is not a good metric.  RES+SHR
> is a better measure of real usage.

Sure. But it's not free to maintain mapping for 1/4 Terabyte of VM. And if the cause of the enormous virtual size is the wrong huge size for baloo/index, something seems wrong in how Baloo accesses this file.

Comment 3 skierpage 2017-05-26 23:00:18 UTC

It seems this is intentional.
In https://code.woboq.org/qt5/kf5/baloo/src/engine/database.cpp.html there's the comment
     * size limit for database == size limit of mmap
     * use 1 GB on 32-bit, use 256 GB on 64-bit
and 256 GB is what baloo passes to OpenLDAP Lightning Memory-Mapped Database Manager's  mdb_env_set_mapsize() function.
 
That seems odd to me but I can't immediately see why this is the cause of baloo_file consuming 99% CPU. I guess it's more plausible my baloo/index file is corrupt.

Comment 4 skierpage 2017-05-27 07:07:55 UTC

(In reply to skierpage from comment #3)
> ... I can't immediately see why this is the cause of
> baloo_file consuming 99% CPU. I guess it's more plausible my baloo/index
> file is corrupt.

I blew away my ~/.local/share/baloo/index and changed folders[$e] in ~/.config/baloofilerc to index less of my Windows partition, and after `balooctl start` I didn't have problems with baloo_file_index, and still had 256 GB baloo_file process and 261 GB plasmashell processes. So maybe this bug is INVALID, just a design choice to freak semi-knowledgeable users out :-)

Comment 5 David Edmundson 2017-06-11 07:25:57 UTC

Virtual memory is just that, virtual. 

You don't need that size mapping in a VM. You don't need any more actual memory. It's not a relevant number of anything.