Bug 28321

On Sat 07 Jul 2001 you wrote:
> Package: khexedit
> Version: KDE 2.2.0 CVS/CVSup/Snapshot
> Severity: normal
> Installed from:    Compiled sources
> Compiler:          Not Specified
> OS:                Not Specified
> OS/Compiler notes: Not Specified
> 
> This seems to be actually a Qt bug (I have 2.3.0) but I report this in the hope that you can work around it.

No this is not really a Qt bug. khexedit will always (try to) allocate a buffer
that is large enough for the entire file. The only scalable solution is to load 
a part of the file (using memory mapping perhaps). I have postponed this 
capability way too long. 


> 
> When I open a large (500 MB) file in khexedit it aborts with a qFatal:
> 
> In file tools/qgarray.cpp line 229: Out of memory 
> 
> [New Thread 1024 (runnable)]
> 0x40db8ad9 in wait4 () from /lib/libc.so.6
> #0  0x40db8ad9 in wait4 () from /lib/libc.so.6
> #1  0x40e109b4 in h_nerr () from /lib/libc.so.6
> #2  0x40eced69 in waitpid (pid=4987 stat_loc=0x0 options=0)
>     at wrapsyscall.c:134
> #3  0x405d61b5 in KCrash::defaultCrashHandler ()
>    from /opt/kde22/lib/libkdecore.so.3
> #4  0x40d4a3d8 in sigaction () from /lib/libc.so.6
> #5  0x40d4b6ff in abort () from /lib/libc.so.6
> #6  0x407ccaad in qFatal () from /opt/qt/lib/libqt.so.2
> #7  0x407ccb73 in qt_check_pointer () from /opt/qt/lib/libqt.so.2
> #8  0x407c5ca5 in QGArray::resize () from /opt/qt/lib/libqt.so.2
> #9  0x808ac9f in CHexBuffer::readFile ()
> #10 0x807f1a2 in CHexViewWidget::readFile ()
> #11 0x8076253 in CHexEditorWidget::readFile ()
> #12 0x80753bd in CHexEditorWidget::readURL ()
> #13 0x8071d67 in CHexEditorWidget::open ()
> #14 0x8064558 in KHexEdit::open ()
> #15 0x80640c3 in KHexEdit::delayedStartupOpen ()
> #16 0x4088143e in QObject::activate_signal () from /opt/qt/lib/libqt.so.2
> #17 0x408b26d2 in QSignal::activate () from /opt/qt/lib/libqt.so.2
> #18 0x408b82ef in QSingleShotTimer::event () from /opt/qt/lib/libqt.so.2
> #19 0x4082fbcb in QApplication::notify () from /opt/qt/lib/libqt.so.2
> #20 0x40546534 in KApplication::notify () from /opt/kde22/lib/libkdecore.so.3
> #21 0x407feedd in qt_activate_timers () from /opt/qt/lib/libqt.so.2
> 
> #22 0x407fce16 in QApplication::processNextEvent () from /opt/qt/lib/libqt.so.2
> 
> #23 0x40831934 in QApplication::enter_loop () from /opt/qt/lib/libqt.so.2
> #24 0x407fc73e in QApplication::exec () from /opt/qt/lib/libqt.so.2
> #25 0x805ec00 in main ()
> #26 0x40d44577 in __libc_start_main () from /lib/libc.so.6
> 
> 
> (Submitted via bugs.kde.org)


-- 
Espen Sand - Linux developer
espen@opera.com

Is this still happening, possibly with newer version(s)?

I hope to help get bugs that have been left uncommented looked at. 
P.s. I am not a KDE developer, just someone trying to help, like you ;)

Subject: Re:  Out of memory abort when opening large file

On Wednesday January 22, 2003 02:28, thebeast wrote:
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>
> http://bugs.kde.org/show_bug.cgi?id=28321
>
>
>
>
> ------- Additional Comments From thebeastwitheyesthatstared@yahoo.dk 
> 2003-01-22 23:28 ------- Is this still happening, possibly with newer
> version(s)?
>
> I hope to help get bugs that have been left uncommented looked at.
> P.s. I am not a KDE developer, just someone trying to help, like you ;)

Hello,

It doesn't abort anymore, but it still fails to load large files.

In KDE 3.1.1 it still tries to load the entire file into ram at once. I just verified 
this on my system. 

khexedit 0.8.5 (released 1999-12-14) is still the current version in KDE 3.2.1. Perhaps the handling of big files is a design error of khexedit. I think 28321 should be move to the khexedit wishlist, because it isn't a bug.

How does kate/kedit/kwrite handle this?

This is a real design error, AND a bug. It's quite strange for an hex editor to try to copy the whole file to memory. Some people (like me) sometimes want to tweak on files potentially bigger than the system memory (like databases or CD/DVD images), which render khexedit unusable.

I just rebuilt my home machine and hadn't time to install all my usual software yet. I had to view some bytes of a 100MB-big file and decided to try khexedit since it came installed along with KDE... It sent my machine onto its knees trying to load the whole file to the memory, swapping for more than 3 minutes.

Mmapping the file is a reasonably good fast solution, but it is far from the best. Doing this way still limits the filesize to the maximum VM size (around 3GB on most archs). For example, if I need to reconstruct a damaged harddisk partition table by hand, I usually open /dev/hdx or /dev/sdx directly into an hex editor, just to toy with the first 512 bytes of the boot sector. But today we have harddisks 200GB big, its just insane.

Suggestion (from someone who already made an hex editor way back on the old MSDOS days): make it work with blocks of (possibly configurable) size. Load into memory only the ones you are displaying on the grid, replace them once the user scrolls or jumps to another offset. If the user changes something, mark that block and store it along with its offset for further dump into the disk file. When loading a block from disk to memory, the program should first check if the offset is already on the "modified blocks" list, and get the block from there if it is. It's easier if you abstract the "read byte from memory for displaying it into the grid" operation.

IMHO, this is not just a wish. If you are not planning to do this yet, you should at least consider adding a warning the user BEFORE opening a file bigger than 1/4 of total system memory, telling him/her that khexedit is not suitable for large files and allowing the operation to be cancelled.

The bug is still present in KDE 3.3.2. It's really annoying. I frequently need to open large video files to fiddle with some bytes in the header, those are usually in the 1-2 GB range.

Of course khexedit most of the time opens them all right, but it takes _really_ long to load a 1 GB file and save it afterwards.

It might be a good idea to work with a sliding, mapped memory window instead, this avoids reading the whole file into memory first. I admit though that inserting and deleting data will become more difficult.

KhexEdit is now flagged as unmaintained. Okteta replaces it.

Closing this bug report.