Version: (using KDE 4.3.2) Compiler: g++-4.2 older compiler used for project, system compiler is 4.4.1 OS: Linux Installed from: Ubuntu Packages I have a repeatable crash with callgrind on a big C++ application, both with valgrind-3.5.0-Debian (Ubuntu 9.10) and valgrind-3.4.1-Debian on another machine (Ubuntu 9.04) Looking at the trace, the symbols are long, so it might be running out of stack? But it does not look like demangle bug. Trace with -v -d: --19620-- Client Request: instrumentation switched ON --19620:1:mallocfr newSuperblock at 0x41190D000 (pszB 65504) owner VALGRIND/demangle --19620-- Symbol match: found runtime_resolve: ld-2.10.1.so +0x145a0=0x40145a0 --19620-- REDIR: 0xffffffffff600000 (???) redirected to 0x3802e723 (vgPlain_amd64_linux_REDIR_FOR_vgettimeofday) --19620:1:mallocfr newSuperblock at 0x41191D000 (pszB 4194272) owner VALGRIND/tool --19620-- REDIR: 0xffffffffff600400 (???) redirected to 0x3802e72d (vgPlain_amd64_linux_REDIR_FOR_vtime) --19620:1:mallocfr newSuperblock at 0x411D1D000 (pszB 4194272) owner VALGRIND/tool --19620:1:syswrap- run_a_thread_NORETURN(tid=12): pre-thread_wrapper --19620:1:syswrap- thread_wrapper(tid=12): entry --19620:1:mallocfr newSuperblock at 0x41211D000 (pszB 65504) owner VALGRIND/ttaux Segmentation fault (core dumped) Just in case it is useful, below are few last lines of a trace with -v -d --ct-verbose=2 Cost full [0x40200F350]: Ir 19 1a251812/T 9088:void std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >*, std::vector<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >, std::allocator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > > >, std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >, (anonymous namespace)::entity_i::value_map::less>(__gnu_cxx::__normal_iterator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recurs data-context.so.12.0.2 /usr/include/c++/4.2/bits/stl_algo.h Boring 1a251812 -> 1a251851, SP 7fefdb620 1a251851/T 9088:void std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >*, std::vector<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >, std::allocator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > > >, std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> >, (anonymous namespace)::entity_i::value_map::less>(__gnu_cxx::__normal_iterator<std::pair<FK::type::string_key<FK::support::basic_symbol<std::string>, char>, boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recurs data-context.so.12.0.2 /usr/include/c++/4.2/bits/stl_algo.h Call 1a25185a -> 1a275300, SP 7fefdb618 + 23 0x1a25185a => 0x1a275300 [1/0,0,0], SP 0x7fefdb618, RA 0x1a25185f 1a275300/T 9098:boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::variant_assign(boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) data-context.so.12.0.2 /home/harri/fk/72/nirvana/module/service/data-context/.././.././../tools/boost/boost/variant/variant.hpp:1548 Conditional 1a27530b -> 1a2754ff, SP 7fefdb5f0 1a2754ff/T 9098:boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>::variant_assign(boost::variant<boost::detail::variant::recursive_flag<FK::type::null>, FK::type::non_existing, bool, FK::numeric::integer<38, FK::numeric::round_down, FK::numeric::throw_error>, FK::numeric::decimal<38, 8, FK::numeric::round_nearest_up, FK::numeric::throw_error>, FK::numeric::money<38, 4, FK::numeric::round_nearest_up, FK::numeric::throw_error>, double, FK::date::date, FK::date::date_time, std::string, FK::type::binary, std::vector<boost::recursive_variant_, std::allocator<boost::recursive_variant_> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) data-context.so.12.0.2 /home/harri/fk/72/nirvana/module/service/data-context/.././.././../tools/boost/boost/variant/variant.hpp:1139 Segmentation fault (core dumped)
> Looking at the trace, the symbols are long, so it might be running out of > stack? But it does not look like demangle bug. It still looks suspiciously similar to bug 197988. Can you check if increasing VG_STACK_ACTIVE_SZB to 2 Mb fixes your problem?
It does not build with VG_STACK_ACTIVE_SZB set to 2Mb (2147483648), 3.5.0 compilation fails with make[3]: Entering directory `/home/harri/src/valgrind-3.5.0/coregrind' if gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I../include -I../VEX/pub -DVGA_amd64=1 -DVGO_linux=1 -DVGP_amd64_linux=1 -I../coregrind -DVG_LIBDIR="\"/usr/local/lib/valgrind"\" -DVG_PLATFORM="\"amd64-linux\"" -m64 -fomit-frame-pointer -O2 -g -Wall -Wmissing-prototypes -Wshadow -Wpointer-arith -Wstrict-prototypes -Wmissing-declarations -Wno-format-zero-length -fno-strict-aliasing -Wno-long-long -Wno-pointer-sign -Wdeclaration-after-statement -fno-stack-protector -MT libcoregrind_amd64_linux_a-m_main.o -MD -MP -MF ".deps/libcoregrind_amd64_linux_a-m_main.Tpo" -c -o libcoregrind_amd64_linux_a-m_main.o `test -f 'm_main.c' || echo './'`m_main.c; \ then mv -f ".deps/libcoregrind_amd64_linux_a-m_main.Tpo" ".deps/libcoregrind_amd64_linux_a-m_main.Po"; else rm -f ".deps/libcoregrind_amd64_linux_a-m_main.Tpo"; exit 1; fi /tmp/ccKVPsyr.s: Assembler messages: /tmp/ccKVPsyr.s:18: Error: suffix or operands invalid for `add' make[3]: *** [libcoregrind_amd64_linux_a-m_main.o] Error 1 However, with 1 Mb stack (0x40000000) it compiles, but when using that my process promptly runs out of memory. 0x10000000, or 256Kb worked for me, so it does seem it is a stack size issue. But I guess memory usage can still be an issue when lots of threads are running.
> VG_STACK_ACTIVE_SZB set to 2Mb (2147483648) Oh, you set it to 2Gb. I wouldn't expect this to work ;-) 2Mb are 2097152. > 0x10000000, or 256Kb worked for me That is 256MB not 256Kb. Nice to hear that Valgrind works with such a huge tool stack. So it really seems to be a duplicate of bug 197988. > But I guess memory usage can still be an issue when lots of threads are > running. This is about the tool's stack. Hmm. I do not think that there will be multiple VG tool stacks allocated if the client code has multiple threads (valgrind serializes execution of threads). So this should not be an issue.
Oops, I was thinking big there... (or rather not really thinking at all). I'll run a few tests to see what still works. The 64Kb is probably too small default for heavily templated C++ code and my app might be a good guinea pig to provide a real world 'worst case' behaviour.
My app seems to work fine with 256 Kb stack as well. But running with the accidentally put 256 Mb stack the my process eats up ~3Gb of virtual memory. With 256 Kb stack it is only ~1.2Gb. So the additional memory consumption seems to be a multiple of VG_STACK_ACTIVE_SZB, even if not in relation to number of client threads...
*** This bug has been marked as a duplicate of bug 197988 ***