Hello, I'm trying to use valgrind for a router (arch ar71xx). However, it never passes the vgPlain_machine_get_hwcaps. This is my cpuinfo: root@router:/tmp# cat /proc/cpuinfo system type : Atheros AR7242 rev 1 machine : TP-LINK TL-WR2543N/ND processor : 0 cpu model : MIPS 24Kc V7.4 BogoMIPS : 265.42 wait instruction : yes microsecond timers : yes tlb_entries : 16 extra interrupt vector : yes hardware watchpoint : yes, count: 4, address/irw mask: [0x0000, 0x08c0, 0x0f50, 0x0f00] isa : mips1 mips2 mips32r1 mips32r2 ASEs implemented : mips16 shadow register sets : 1 kscratch registers : 0 core : 0 VCED exceptions : not available VCEI exceptions : not available And running with gdb: root@router:/tmp# gdb --args /usr/bin/valgrind --help run Program received signal SIGILL, Illegal instruction. #0 0x3807ea0c in vgPlain_machine_get_hwcaps () at m_machine.c:1543 #1 0x3807f9bc in valgrind_main (argc=argc@entry=2, argv=argv@entry=0x7fff6d84, envp=envp@entry=0x7fff6d90) at m_main.c:1735 #2 0x38084ddc in _start_in_C_linux (pArgc=0x7fff6d80) at m_main.c:3202 #3 0x3807ec8c in __start () ASM: 0x3807ea0c <vgPlain_machine_get_hwcaps+660> precr.qb.ph t2,t0,t1 Which, I guess, might be catch by "if (VG_MINIMAL_SETJMP(env_unsup_insn))". I continue using cont. The same sig at: #0 0x3807ea50 in vgPlain_machine_get_hwcaps () at m_machine.c:1554 #1 0x3807f9bc in valgrind_main (argc=argc@entry=2, argv=argv@entry=0x7fff6d84, envp=envp@entry=0x7fff6d90) at m_main.c:1735 #2 0x38084ddc in _start_in_C_linux (pArgc=0x7fff6d80) at m_main.c:3202 #3 0x3807ec8c in __start () ASM: 0x3807ea50 <vgPlain_machine_get_hwcaps+728> rddsp t0,0x3f Now I continued instruction by instruction with stepi. It completely freezes at cfc1: 0x3807ea5c <vgPlain_machine_get_hwcaps+740> beqz v0,0x3807ea70 <vgPlain_machine_get_hwcaps+760> 0x3807ea60 <vgPlain_machine_get_hwcaps+744> lui v0,0x3841 0x3807ea64 <vgPlain_machine_get_hwcaps+748> lw v1,31264(v0) 0x3807ea68 <vgPlain_machine_get_hwcaps+752> ori v1,v1,0x9500 0x3807ea6c <vgPlain_machine_get_hwcaps+756> sw v1,31264(v0) 0x3807ea70 <vgPlain_machine_get_hwcaps+760> cfc1 v0,$0 If I jump over it, it, at least, showed the --help. :-) jump * 0x3807ea74 All those cases are directly __asm__ inside ./coregrind/m_machine.c. The first two seems to be ok. The last one should be avoided. Reproducible: Always Steps to Reproduce: 1.compile valgrind for ar71xx (using openwrt SDK) 2.install it 3.run it Actual Results: 4.it never returns Expected Results: 4.works :-) I just removed all the code related to cfc1 and now valgrind runs at my system (the --help). However, I still does not have a working valgrind. It failed at: vex: priv/guest_mips_toIR.c:1210 (putIReg): Assertion `typeOfIRExpr(irsb->tyenv, e) == ty' failed. vex storage: T total 17864392 bytes allocated vex storage: P total 0 bytes allocated valgrind: the 'impossible' happened: LibVEX called failure_exit(). I'm no mips expert but I can help testing patches on my system if needed.
Can you share what particular OpenWRT SDK you were using? What configure line did you use for Valgrind? What exactly is the behaviour when you use Valgrind only? (i.e. not via gdb)
Created attachment 89742 [details] valgrind -d if invalid instruction is commented
I used: https://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/OpenWrt-SDK-ar71xx-for-linux-x86_64-gcc-4.8-linaro_uClibc-0.9.33.2.tar.bz2 and https://downloads.openwrt.org/snapshots/trunk/ar71xx/OpenWrt-SDK-ar71xx-for-linux-x86_64-gcc-4.8-linaro_uClibc-0.9.33.2.tar.bz2 However, the valgrind-3.10 provided by openwrt snapshots/trunk also shares the problem for ar71xx. The config line is generated with a combination of OpenWRT own options and valgrind package ones. This is the result: AR=mips-openwrt-linux-uclibc-ar AS="mips-openwrt-linux-uclibc-gcc -c -Os -pipe -mno-branch-likely -mips32r2 -mtune=34kc -g3 -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -msoft-float" LD=mips-openwrt-linux-uclibc-ld NM=mips-openwrt-linux-uclibc-nm CC="mips-openwrt-linux-uclibc-gcc" GCC="mips-openwrt-linux-uclibc-gcc" CXX="mips-openwrt-linux-uclibc-g++" RANLIB=mips-openwrt-linux-uclibc-ranlib STRIP=mips-openwrt-linux-uclibc-strip OBJCOPY=mips-openwrt-linux-uclibc-objcopy OBJDUMP=mips-openwrt-linux-uclibc-objdump SIZE=mips-openwrt-linux-uclibc-size CFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=34kc -g3 -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -msoft-float " CXXFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=34kc -g3 -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -msoft-float " CPPFLAGS="-I/home/luizluca/prog-local/openwrt/bb/staging_dir/target-mips_34kc_uClibc-0.9.33.2/usr/include -I/home/luizluca/prog-local/openwrt/bb/staging_dir/target-mips_34kc_uClibc-0.9.33.2/include -I/home/luizluca/prog-local/openwrt/bb/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/usr/include -I/home/luizluca/prog-local/openwrt/bb/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/include " LDFLAGS="-L/home/luizluca/prog-local/openwrt/bb/staging_dir/target-mips_34kc_uClibc-0.9.33.2/usr/lib -L/home/luizluca/prog-local/openwrt/bb/staging_dir/target-mips_34kc_uClibc-0.9.33.2/lib -L/home/luizluca/prog-local/openwrt/bb/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/usr/lib -L/home/luizluca/prog-local/openwrt/bb/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/lib " UNAME_R=3.10.49 ./configure --target=mips-openwrt-linux --host=mips-openwrt-linux --build=x86_64-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --enable-only32bit --enable-tls --without-x --without-uiout --disable-valgrindmi --disable-tui --disable-valgrindtk --without-included-gettext The exactly behavior is that valgrind stops after the "Get hardware capabilities ...". On gcc, I discovered that it stopped at the cfc1 instruction. This is my output: root@router:/# valgrind -d --6318:1:debuglog DebugLog system started by Stage 1, level 1 logging requested --6318:1:launcher no tool requested, defaulting to 'memcheck' --6318:1:launcher no client specified, defaulting platform to 'mips32-linux' --6318:1:launcher launching /usr/lib/valgrind/memcheck-mips32-linux --6318:1:debuglog DebugLog system started by Stage 2 (main), level 1 logging requested --6318:1:main Welcome to Valgrind version 3.10.0 debug logging --6318:1:main Checking current stack is plausible --6318:1:main Checking initial stack was noted --6318:1:main Starting the address space manager --6318:1:main Address space manager is running --6318:1:main Starting the dynamic memory manager --6318:1:mallocfr newSuperblock at 0x41FFC000 (pszB 4194288) owner VALGRIND/core --6318:1:mallocfr deferred_reclaimSuperblock at 0x41FFC000 (pszB 4194288) (prev 0x0) owner VALGRIND/core --6318:1:main Dynamic memory manager is running --6318:1:main Initialise m_debuginfo --6318:1:main VG_(libdir) = /usr/lib/valgrind --6318:1:main Getting launcher's name ... --6318:1:main ... /usr/bin/valgrind --6318:1:main Get hardware capabilities ... I skipped this problem by commenting the code that used the instruction. Maybe kernel can have a better way to check for this instruction. However, valgrind is still unusable as some assert always fails: vex: priv/guest_mips_toIR.c:1210 (putIReg): Assertion `typeOfIRExpr(irsb->tyenv, e) == ty' failed. vex storage: T total 17919632 bytes allocated vex storage: P total 0 bytes allocated I attached the full log for the assert issue (but i guess it might belong to a new bug)
Is this issue still valid? Are you seeing the issue with the latest Valgrind SVN code?
Is it really necessary to test SVN? At least with 3.12.0, the problem remains: # valgrind ping 8.8.8.8 ==24067== Memcheck, a memory error detector ==24067== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==24067== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==24067== Command: ping 8.8.8.8 ==24067== ==24067== Conditional jump or move depends on uninitialised value(s) ==24067== at 0x40731E4: ??? (in /lib/libc.so) ==24067== by 0x4083164: ??? (in /lib/libc.so) ==24067== ==24067== Conditional jump or move depends on uninitialised value(s) ==24067== at 0x4072924: ??? (in /lib/libc.so) ==24067== by 0x4072D98: ??? (in /lib/libc.so) ==24067== vex mips->IR: unhandled instruction bytes: 0x41 0x67 0x25 0x22 ==24067== valgrind: Unrecognised instruction at address 0x4077b5. ==24067== at 0x4077B5: ??? (in /bin/busybox) ==24067== by 0x401E6A0: ??? (in /lib/libc.so) ==24067== Your program just tried to execute an instruction that Valgrind ==24067== did not recognise. There are two possible reasons for this. ==24067== 1. Your program has a bug and erroneously jumped to a non-code ==24067== location. If you are running Memcheck and you just saw a ==24067== warning about a bad jump, it's probably your program's fault. ==24067== 2. The instruction is legitimate but Valgrind doesn't handle it, ==24067== i.e. it's Valgrind's fault. If you think this is the case or ==24067== you are not sure, please let us know and we'll try to fix it. ==24067== Either way, Valgrind will now raise a SIGILL signal which will ==24067== probably kill your program. ==24067== ==24067== Process terminating with default action of signal 4 (SIGILL) ==24067== Illegal opcode at address 0x4077B5 ==24067== at 0x4077B5: ??? (in /bin/busybox) ==24067== by 0x401E6A0: ??? (in /lib/libc.so) ==24067== ==24067== HEAP SUMMARY: ==24067== in use at exit: 0 bytes in 0 blocks ==24067== total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==24067== ==24067== All heap blocks were freed -- no leaks are possible ==24067== ==24067== For counts of detected and suppressed errors, rerun with: -v ==24067== Use --track-origins=yes to see where uninitialised values come from ==24067== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0) Illegal instruction If needed, I can recompile source from SVN with debuginfo
(In reply to Luiz Angelo De Luca from comment #5) >==24067== Illegal opcode at address 0x4077B5 >==24067== at 0x4077B5: ??? (in /bin/busybox) This looks suspicious. Is that busybox that has been compiled with -mips16 option? Note that Valgrind does not support mips16 for the time being. Can you do 'readelf -h /bin/busybox' and show us the output?
Most of binaries are mips16 on OpenWRT or LEDE. $ ./staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-readelf -h staging_dir/target-mips_24kc_musl-1.1.16/root-ar71xx/bin/busybox ELF Header: Magic: 7f 45 4c 46 01 02 01 00 01 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, big endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 1 Type: EXEC (Executable file) Machine: MIPS R3000 Version: 0x1 Entry point address: 0x4037d0 Start of program headers: 52 (bytes into file) Start of section headers: 366924 (bytes into file) Flags: 0x74001005, noreorder, cpic, o32, mips16, mips32r2 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 8 Size of section headers: 40 (bytes) Number of section headers: 39 Section header string table index: 36 And on those that are not, valgrind still fails (different bug?) root@router.lan3:~# valgrind ssh ==13507== Memcheck, a memory error detector ==13507== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==13507== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info ==13507== Command: ssh ==13507== ==13507== Conditional jump or move depends on uninitialised value(s) ==13507== at 0x40731E4: ??? (in /lib/libc.so) ==13507== by 0x4083164: ??? (in /lib/libc.so) ==13507== ==13507== Conditional jump or move depends on uninitialised value(s) ==13507== at 0x4072924: ??? (in /lib/libc.so) ==13507== by 0x4072D98: ??? (in /lib/libc.so) ==13507== Segmentation fault $ ./staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-readelf -h staging_dir/target-mips_24kc_musl-1.1.16/root-ar71xx/usr/bin/ssh ELF Header: Magic: 7f 45 4c 46 01 02 01 00 01 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, big endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 1 Type: EXEC (Executable file) Machine: MIPS R3000 Version: 0x1 Entry point address: 0x402260 Start of program headers: 52 (bytes into file) Start of section headers: 208740 (bytes into file) Flags: 0x70001005, noreorder, cpic, o32, mips32r2 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 9 Size of section headers: 40 (bytes) Number of section headers: 39 Section header string table index: 36 I don't know if something it uses is compiled with mips16 root@router.lan3:~# ldd /usr/bin/ssh /lib/ld-musl-mips-sf.so.1 (0x55bf4000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x770c4000) libc.so => /lib/ld-musl-mips-sf.so.1 (0x55bf4000) $ ./staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-readelf -h staging_dir/target-mips_24kc_musl-1.1.16/root-ar71xx/lib/ld-musl-mips-sf.so.1 ELF Header: Magic: 7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, big endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: MIPS R3000 Version: 0x1 Entry point address: 0xf0d0 Start of program headers: 52 (bytes into file) Start of section headers: 2784856 (bytes into file) Flags: 0x70001007, noreorder, pic, cpic, o32, mips32r2 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 8 Size of section headers: 40 (bytes) Number of section headers: 30 Section header string table index: 27 $ ./staging_dir/toolchain-mips_24kc_gcc-5.4.0_musl-1.1.16/bin/mips-openwrt-linux-readelf -h staging_dir/target-mips_24kc_musl-1.1.16/root-ar71xx/lib/libgcc_s.so.1 ELF Header: Magic: 7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, big endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: MIPS R3000 Version: 0x1 Entry point address: 0x2810 Start of program headers: 52 (bytes into file) Start of section headers: 361672 (bytes into file) Flags: 0x70001005, noreorder, cpic, o32, mips32r2 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 8 Size of section headers: 40 (bytes) Number of section headers: 37 Section header string table index: 34 I always get either segfault or illegal instruction
Btw, at least the --help do works now (first failing case)
I believe the new (non-mips16) issues are not MIPS-related, but rather MUSL-related. Can you try to apply the patch in the second comment Bug 359202 and see if it sorts out your issues with binaries that do not have or use mips16 code? https://bugs.kde.org/show_bug.cgi?id=359202#c2
The patch was already in use. See: https://github.com/lede-project/source/blob/lede-17.01/package/devel/valgrind/patches/200-musl_fix.patch I managed to compile a SVN version https://github.com/lede-project/source/compare/lede-project:lede-17.01...luizluca:cc/valgrind-svn?expand=1 But it changed nothing. At least, it would be easier to test SVN changes in the future. I also compiled valgrind with debug and used gdb. Inside gdb, I still see SIGILL (Illegal instruction), but running the program outside gdb seems to miss it. Maybe it is catched. Program received signal SIGILL, Illegal instruction. 0x38065bf0 in vgPlain_machine_get_hwcaps () at m_machine.c:1719 1719 have_DSPr2 = False; (gdb) cont Continuando. Program received signal SIGILL, Illegal instruction. 0x38065c34 in vgPlain_machine_get_hwcaps () at m_machine.c:1730 1730 have_DSP = False; (gdb) cont Continuando. warning: GDB can't find the start of the function at 0x42ae1d6c. GDB is unable to find the start of the function at 0x42ae1d6c and thus can't determine the size of that function's stack frame. This means that GDB may be unable to access that stack frame, or the frames below it. This problem is most likely caused by an invalid program counter or stack pointer. However, if you think GDB should simply search farther back from 0x42ae1d6c for code which looks like the beginning of a function, you can increase the range of the search using the `set heuristic-fence-post' command. Program received signal SIGSEGV, Segmentation fault. 0x42ae1d6c in ?? () Here is where I get the SIGSEGV. Maybe stack got corrupted. Is there any tips from here?
(In reply to Luiz Angelo De Luca from comment #10) > Here is where I get the SIGSEGV. Maybe stack got corrupted. Is there any > tips from here? Receiving some of these signals in GDB is an expected behaviour. Can you pass the following lines to gbd and rerun the executable? set heuristic-fence-post 999999 handle SIGSEGV noprint nostop pass Keep running (w/ "continue") GDB until it fails.
> Can you pass the following lines to gbd and rerun the executable? > > set heuristic-fence-post 999999 > handle SIGSEGV noprint nostop pass > > Keep running (w/ "continue") GDB until it fails. If I use both, gdb goes until valgrind quits with 'Child terminated with signal = 0xb (SIGSEGV)' If I use only heuristic-fence-post (before first cont), nothing changes. I also tried to use it after sigsegv hit, but then, gdb never returned.
We are not able to reproduce this issue on our board with OpenWRT. Can you confirm that you have tried running the latest SVN version of Valgrind WITH the patch for MUSL (https://bugs.kde.org/show_bug.cgi?id=359202#c2) applied on the top of it?
Luiz, any update?
root@router.lan3:/usr/bin# valgrind --version valgrind-3.13.0.SVN The test was using commit svn://svn.valgrind.org/valgrind/trunk@16225 a5019735-40e9-0310-863c-91ae7b9d1cf9 That I got from https://github.com/liquid-mirror/valgrind. Is it recent enough? I'm using the patch. All patches at https://github.com/luizluca/source/tree/cc/valgrind-svn/package/devel/valgrind/patches are applied before compiling. The behavior changes a lot depending on the checked program. scp is just a symlink to dropbear (as ssh is). However, valgrind does not break with it (for showing help). root@router.lan3:/usr/bin# valgrind scp ==14632== Memcheck, a memory error detector ==14632== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==14632== Using Valgrind-3.13.0.SVN and LibVEX; rerun with -h for copyright info ==14632== Command: scp ==14632== ==14632== Conditional jump or move depends on uninitialised value(s) ==14632== at 0x40731E4: ??? (in /lib/libc.so) ==14632== by 0x4083164: ??? (in /lib/libc.so) ==14632== ==14632== Conditional jump or move depends on uninitialised value(s) ==14632== at 0x4072924: ??? (in /lib/libc.so) ==14632== by 0x4072D98: ??? (in /lib/libc.so) ==14632== usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-P port] [-S program] [[user@]host1:]file1 [...] [[user@]host2:]file2 ==14632== ==14632== HEAP SUMMARY: ==14632== in use at exit: 0 bytes in 0 blocks ==14632== total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==14632== ==14632== All heap blocks were freed -- no leaks are possible ==14632== ==14632== For counts of detected and suppressed errors, rerun with: -v ==14632== Use --track-origins=yes to see where uninitialised values come from ==14632== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0) But does break when it does something else: root@router.lan3:/usr/bin# valgrind scp /etc/passwd localhost:/tmp ==14715== Memcheck, a memory error detector ==14715== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==14715== Using Valgrind-3.13.0.SVN and LibVEX; rerun with -h for copyright info ==14715== Command: scp /etc/passwd localhost:/tmp ==14715== ==14715== Conditional jump or move depends on uninitialised value(s) ==14715== at 0x40731E4: ??? (in /lib/libc.so) ==14715== by 0x4083164: ??? (in /lib/libc.so) ==14715== ==14715== Conditional jump or move depends on uninitialised value(s) ==14715== at 0x4072924: ??? (in /lib/libc.so) ==14715== by 0x4072D98: ??? (in /lib/libc.so) ==14715== Host 'localhost' is not in the trusted hosts file. (ssh-rsa fingerprint md5 33:1c:ed:be:01:8d:58:97:bc:8f:45:c7:db:63:24:e9) Do you want to continue connecting? (y/n) y root@localhost's password: Segmentation fault My system is a tplink tl-wr2543nd router. It's a very popular wifi router.
(In reply to Luiz Angelo De Luca from comment #15) > root@router.lan3:/usr/bin# valgrind --version > valgrind-3.13.0.SVN > > The test was using commit svn://svn.valgrind.org/valgrind/trunk@16225 > a5019735-40e9-0310-863c-91ae7b9d1cf9 That I got from > https://github.com/liquid-mirror/valgrind. Is it recent enough? > No, it is not recent enough. Please take the latest SVN (better option) or cherry pick the change r16261. Let us know the behaviour of Valgrind after that.
I switched to a new git mirror source. valgrind works like a charm with trunk@16292. Thank you very much. For the record, this patch updates valgrind from 17.01 to the reference SVN version https://github.com/luizluca/source/commit/926d33d26c968f4661a59625426cab10a82d998e Would it take too long to get a new valgrind released? If not, I'll not bother to backport to previous valgrind package.
(In reply to Luiz Angelo De Luca from comment #17) > I switched to a new git mirror source. > > valgrind works like a charm with trunk@16292. Thank you very much. > Glad to hear so. > Would it take too long to get a new valgrind released? If not, I'll not > bother to backport to previous valgrind package. There is a discussion about new release, possibly in May: https://sourceforge.net/p/valgrind/mailman/message/35756720/