Bug 136300 - valgrind doesn't work on ppc{,64} with 64K pages
Summary: valgrind doesn't work on ppc{,64} with 64K pages
Status: RESOLVED FIXED
Alias: None
Product: valgrind
Classification: Developer tools
Component: general (show other bugs)
Version: 3.2.1
Platform: Compiled Sources Linux
: NOR major
Target Milestone: ---
Assignee: Julian Seward
URL:
Keywords:
: 139124 (view as bug list)
Depends on:
Blocks:
 
Reported: 2006-10-25 17:42 UTC by Jakub Jelinek
Modified: 2007-01-10 00:51 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
valgrind-3.2.1-ppc-pagesize.patch (8.69 KB, patch)
2006-10-25 17:45 UTC, Jakub Jelinek
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jakub Jelinek 2006-10-25 17:42:38 UTC
Fedora Core 6/ppc as well as RHEL5 and AFAIK some SUSE version too ships
with 64K pages in 64-bit kernels (32-bit kernels still use 4K pages).
Unfortunately valgrind has a fixed page size of 4K compiled in in many places
and dies very early without it.
I'll attach a quick&dirty patch which fixed this, of course feel free to fix
it in a cleaner way.
What's also important is that mmap2 syscall doesn't have last argument in
page size units, but in 4K units.
Comment 1 Jakub Jelinek 2006-10-25 17:45:32 UTC
Created attachment 18262 [details]
valgrind-3.2.1-ppc-pagesize.patch

As ppc32 valgrind can be run on both 32-bit and 64-bit kernels which have
different page sizes, I'm afraid the page size can't be a configure time
determined constant, so the patch uses a variable.
But, in the vg*preload*.so libraries I can't get to that variable, so I
use getpagesize () instead (could perhaps better be sysconf (_SC_PAGE_SIZE)).
Comment 2 Julian Seward 2006-12-25 09:41:53 UTC
*** Bug 139124 has been marked as a duplicate of this bug. ***
Comment 3 Julian Seward 2006-12-29 17:34:24 UTC
Evaluating this patch for inclusion into 3.2.2.  A question: is it really
necessary to make VKI_PAGE_SHIFT/VKI_PAGE_SIZE into variables on ppc-linux?

- do you need to support 64k pages in 32-bit mode?
- do you need to support 4k pages in 64-bit mode?

fedoraproject.org says

  The PowerPC64 kernel now runs with a page size of 64 KiB, instead of 
  4 KiB as used in Fedora Core 5. The installer should reformat any 
  swap partitions automatically during an upgrade.

This leaves me with the impression that (1) in 32 bit mode only 4k pages
are supported, and (2) in 64-bit mode only 64k pages are supported.  If 
that's so, a configure-time test for the pagesize in 64-bit mode would 
suffice.

Can you clarify?  Is there anywhere any documentation of page size 
issues from the userspace side of the fence?
Comment 4 Jakub Jelinek 2006-12-29 17:49:06 UTC
In Fedora Core 6+ (and RHEL5+) 32bit kernels really use just 4k pages and 64bit
kernels just 64k pages.  But 32bit binaries can and often are executed on
64bit kernels, so at least 32bit valgrind needs to support both 4k and 64k pages.
Additionally, in ppc64 kernels the page size is kernel compile time configurable,
so if people boot their own compiled ppc64 kernel with say 4k pages and ppc64
valgrind only supports 64k pages, it will not work properly.
Comment 5 Julian Seward 2006-12-29 20:35:43 UTC
Ok, I'll commit a modified version of your patch and it would be good if you
could verify I didn't break anything.

I still would like to know if these page-size variants are documented for
application developers anywhere.  If toolchain developers like me are having
difficulty figuring out what's going on then I imagine app developers will
have the same problem.
Comment 6 Julian Seward 2006-12-30 04:05:42 UTC
This doesn't seem right, for ppc32-linux and ppc64-linux:

+/* PAGE_SHIFT determines the page size, unfortunately
+   page size might vary between 32-bit and 64-bit ppc kernels */
+extern unsigned long VKI_PAGE_SHIFT, VKI_PAGE_SIZE;
+#define VKI_MAX_PAGE_SHIFT	16
+#define VKI_MAX_PAGE_SIZE	(1UL << VKI_PAGE_SHIFT)

Shouldn't VKI_MAX_PAGE_SIZE be (1UL << VKI_MAX_PAGE_SHIFT),
not (1UL << VKI_PAGE_SHIFT) ?

J
Comment 7 Julian Seward 2006-12-30 19:03:03 UTC
I have committed fixes both on the 3.2 branch and on the trunk.
It seems to work OK for 4K pages on 32/64-bit ppc-linux, but I
have no way to check 64k pages are working, so it would be good
if you could check.  To get working copies:

  svn co svn://svn.valgrind.org/valgrind/trunk  and
  svn co svn://svn.valgrind.org/valgrind/branches/VALGRIND_3_2_BRANCH

respectively.  You need to run ./autogen.sh and then configure etc
as usual.
Comment 8 Jakub Jelinek 2007-01-03 15:08:45 UTC
Worked fine, except the usual compilation problem:
--- valgrind/VEX/Makefile.jj    2007-01-03 14:22:13.000000000 +0100
+++ valgrind/VEX/Makefile       2007-01-03 14:35:25.000000000 +0100
@@ -64,10 +64,10 @@ PUB_INCLUDES = -Ipub
 PRIV_INCLUDES = -Ipriv
 
 
-ifndef $(CC)
+ifndef CC
    CC = gcc 
 endif 
-ifndef $(AR)
+ifndef AR
    AR = ar 
 endif
 
(ifndef's argument is the name of the variable, see info make;
when CC is empty or more than one word (say CC='gcc -m64' ./configure)
the above fails and certainly doesn't do what it should do).
Comment 9 Julian Seward 2007-01-03 15:27:02 UTC
Thanks.  Was that the trunk you tested, or 3_2_BRANCH ?
Comment 10 Jakub Jelinek 2007-01-03 15:42:04 UTC
Both.
trunk configured with CC='gcc -m64' CXX='g++ -m64' ./configure
(i.e. ppc64-linux targetted valgrind), valgrind-3.2 branch
configured with ppc32 ./configure (i.e. ppc32-linux targetted
valgrind), both running on 64-bit kernel with 64KB pages.
Comment 11 Julian Seward 2007-01-04 17:30:25 UTC
> trunk configured with CC='gcc -m64' CXX='g++ -m64' ./configure
> (i.e. ppc64-linux targetted valgrind), valgrind-3.2 branch
> configured with ppc32 ./configure (i.e. ppc32-linux targetted
> valgrind), both running on 64-bit kernel with 64KB pages.


Thanks for verification and makefile fix.  Note ppc64-linux support
should work just as well in the 3.2 branch as it does in the trunk
(I mention this because you mention a different ./configure line for
the two of them).
Comment 12 Julian Seward 2007-01-04 19:30:19 UTC
Fixed.  Jakub, thanks for your help with this.
Comment 13 David Woodhouse 2007-01-09 07:51:39 UTC
On Tue, 2007-01-09 at 15:40 +1100, Paul Mackerras wrote:
> Julian Seward writes:
> 
> > Evaluating this patch for inclusion into 3.2.2.  A question: is it really
> > necessary to make VKI_PAGE_SHIFT/VKI_PAGE_SIZE into variables on ppc-linux?
> 
> [Catching up on email backlog after vacation...]
> 
> > - do you need to support 64k pages in 32-bit mode?
> > - do you need to support 4k pages in 64-bit mode?


Yes to both.

> 32-bit *kernels* only support 4k pages, and 64-bit *kernels* support
> either 4k or 64k pages (chosen at kernel config time, and FC chooses
> 64k).  However, a 64-bit kernel can run both 32-bit and 64-bit
> processes, and in fact most of the userspace programs installed as
> part of a distro would be 32-bit, even on a 64-bit machine (since
> 32-bit binaries are smaller and faster for most things than 64-bit).
> 
> Thus the critical question is - does FC have completely separate sets
> of packages for ppc32 and ppc64?  Or does it have one set of packages
> for both, and just install a 32-bit or 64-bit kernel depending on the
> machine?  I don't know, but David Woodhouse (cc'd) would.


Obviously we have separate _64-bit_ packages, but we only have one set
of 32-bit packages; we don't have separate "32-bit packages for 32-bit
host" and "32-bit packages for 64-bit host". So the 32-bit userspace
packages have to cope with page size being either 4KiB or 64KiB.

In fact, the 64-bit userspace packages had better handle both cases too.
We've already switched back to 4KiB pages on PPC64 in the development
tree. It's not really much of a win except on certain benchmarks and
then only on machines where you have _plenty_ of RAM to waste. And
machines like the PS3 with its paltry 256MiB really don't have RAM to
waste.
Comment 14 Julian Seward 2007-01-09 17:59:18 UTC
Thanks Paul, David for the feedback.  In the end I merged a patch from
Jacob Jelinek which provides both page sizes in both 32- and 64-bit modes,
and he seemed to be happy with that.

64k pages seems a lot of extra verification hassle / sw engineering effort
for minimal gain.  Does anybody actually use them?

J

On Tuesday 09 January 2007 06:52, David Woodhouse wrote:
> On Tue, 2007-01-09 at 15:40 +1100, Paul Mackerras wrote:
> > Julian Seward writes:
> > > Evaluating this patch for inclusion into 3.2.2.  A question: is it
> > > really necessary to make VKI_PAGE_SHIFT/VKI_PAGE_SIZE into variables on
> > > ppc-linux?
> >
> > [Catching up on email backlog after vacation...]
> >
> > > - do you need to support 64k pages in 32-bit mode?
> > > - do you need to support 4k pages in 64-bit mode?
>
> Yes to both.
>
> > 32-bit *kernels* only support 4k pages, and 64-bit *kernels* support
> > either 4k or 64k pages (chosen at kernel config time, and FC chooses
> > 64k).  However, a 64-bit kernel can run both 32-bit and 64-bit
> > processes, and in fact most of the userspace programs installed as
> > part of a distro would be 32-bit, even on a 64-bit machine (since
> > 32-bit binaries are smaller and faster for most things than 64-bit).
> >
> > Thus the critical question is - does FC have completely separate sets
> > of packages for ppc32 and ppc64?  Or does it have one set of packages
> > for both, and just install a 32-bit or 64-bit kernel depending on the
> > machine?  I don't know, but David Woodhouse (cc'd) would.
>
> Obviously we have separate _64-bit_ packages, but we only have one set
> of 32-bit packages; we don't have separate "32-bit packages for 32-bit
> host" and "32-bit packages for 64-bit host". So the 32-bit userspace
> packages have to cope with page size being either 4KiB or 64KiB.
>
> In fact, the 64-bit userspace packages had better handle both cases too.
> We've already switched back to 4KiB pages on PPC64 in the development
> tree. It's not really much of a win except on certain benchmarks and
> then only on machines where you have _plenty_ of RAM to waste. And
> machines like the PS3 with its paltry 256MiB really don't have RAM to
> waste.

Comment 15 David Woodhouse 2007-01-10 00:51:17 UTC
On Tue, 2007-01-09 at 17:10 +0000, Julian Seward wrote:
> 
> Thanks Paul, David for the feedback.  In the end I merged a patch from
> Jacob Jelinek which provides both page sizes in both 32- and 64-bit modes,
> and he seemed to be happy with that.
> 
> 64k pages seems a lot of extra verification hassle / sw engineering effort
> for minimal gain.  Does anybody actually use them? 


IBM insisted that we did it for RHEL5. It ended up in the Fedora 6
kernel too by mistake. If it didn't mean that we had to reformat swap,
I'd probably turn it off again in an erratum kernel for FC6.

I can give you accounts on afflicted machines if you want.