Bug 426148 - Valgrind crash with "impossible happened" when running BPF CO-RE programs
Summary: Valgrind crash with "impossible happened" when running BPF CO-RE programs
Status: RESOLVED FIXED
Alias: None
Product: valgrind
Classification: Developer tools
Component: general (show other bugs)
Version: 3.15 SVN
Platform: unspecified Linux
: NOR crash
Target Milestone: ---
Assignee: Mark Wielaard
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-03 15:10 UTC by Luka Oreskovic
Modified: 2021-10-12 21:21 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments
The complete output generated when running "opensnoop" example with valgrind -v flag (10.15 KB, text/plain)
2020-09-03 15:10 UTC, Luka Oreskovic
Details
Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE (2.80 KB, text/plain)
2021-10-10 21:40 UTC, Mark Wielaard
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Luka Oreskovic 2020-09-03 15:10:52 UTC
Created attachment 131394 [details]
The complete output generated when running "opensnoop" example with valgrind -v flag

SUMMARY
Valgrind fails with error: 

FATAL: unhandled eBPF command 22

valgrind: the 'impossible' happened:
   ... bye!

when trying to run some of the BPF CO-RE program examples found on the iovisor BCC github repository (https://github.com/iovisor/bcc/tree/master/libbpf-tools). 
The error appears on "tcpconnect", "opensnoop", "syscount" and others.

Libbpf is required as a dependency but it can be cloned as a submodule in src/cc/libbpf

STEPS TO REPRODUCE
1. compile kernel to enable BTF support (explained in the repository readme)
2. compile and run example program with valgrind

OBSERVED RESULT

FATAL: unhandled eBPF command 22

valgrind: the 'impossible' happened:
   ... bye!

The full output is in the attached file

EXPECTED RESULT

Valgrind doesn't crash

SOFTWARE/OS VERSIONS

Linux sheepofawesome 5.7.6-gentoo-x86_64 #9 SMP Mon Aug 17 13:26:04 CEST 2020 x86_64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz GenuineIntel GNU/Linux

Also reproduced on:

Linux pc 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64 GNU/Linux
Comment 1 Mark Wielaard 2021-02-20 22:47:36 UTC
eBPF command 22 is BPF_MAP_FREEZE.

valgrind currently handle eBPF commands up to 20 (BPF_TASK_FD_QUERY), it doesn't yet handle some newer commands:

        BPF_MAP_LOOKUP_AND_DELETE_ELEM, /* 21 */
        BPF_MAP_FREEZE,
        BPF_BTF_GET_NEXT_ID,
        BPF_MAP_LOOKUP_BATCH,
        BPF_MAP_LOOKUP_AND_DELETE_BATCH,
        BPF_MAP_UPDATE_BATCH,
        BPF_MAP_DELETE_BATCH,
        BPF_LINK_CREATE,
        BPF_LINK_UPDATE,
        BPF_LINK_GET_FD_BY_ID,
        BPF_LINK_GET_NEXT_ID,
        BPF_ENABLE_STATS,
        BPF_ITER_CREATE,
        BPF_LINK_DETACH,
Comment 2 Andreas Gerstmayr 2021-10-04 17:17:30 UTC
How to reproduce on Fedora 34:

$ dnf install -y git clang llvm elfutils-libelf-devel zlib-devel valgrind
$ git clone --recursive https://github.com/iovisor/bcc.git
$ cd bcc/libbpf-tools && make
$ valgrind ./opensnoop
Comment 3 Mark Wielaard 2021-10-10 21:40:51 UTC
Created attachment 142314 [details]
Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE

Implements BPF_MAP_LOOKUP_AND_DELETE_ELEM (command 20) and BPF_MAP_FREEZE (command 21) and produces a WARNING instead of a fatal error for unrecognized BPF commands.

This does not fully solve the issue with opensnoop, that will still report WARNINGS, but will no longer crash under valgrind:

==3618029== Command: ./opensnoop
==3618029== 
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 28
PID    COMM              FD ERR PATH
Comment 4 Mark Wielaard 2021-10-12 21:21:17 UTC
I am closing this now because the reported eBPF command has been implemented. But please feel free to open new bugs for other missing eBPF commands.

commit ef95220ddae1af65c85d8d59a8f0dcbb9d7af90f
Author: Mark Wielaard <mark@klomp.org>
Date:   Tue Oct 12 23:15:41 2021 +0200

    Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE
    
    Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM (command 21) and
    BPF_MAP_FREEZE (command 22) and produce a WARNING instead of a fatal
    error for unrecognized BPF commands.
    
    https://bugs.kde.org/show_bug.cgi?id=426148