Bug 426148

Summary:	Valgrind crash with "impossible happened" when running BPF CO-RE programs
Product:	[Developer tools] valgrind	Reporter:	Luka Oreskovic <luka.oreskovic>
Component:	general	Assignee:	Mark Wielaard <mark>
Status:	RESOLVED FIXED
Severity:	crash	CC:	andreas, mark, pjfloyd
Priority:	NOR
Version First Reported In:	3.15 SVN
Target Milestone:	---
Platform:	unspecified
OS:	Linux
Latest Commit:		Version Fixed/Implemented In:
Sentry Crash Report:
Attachments:	The complete output generated when running "opensnoop" example with valgrind -v flag Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE

Description Luka Oreskovic 2020-09-03 15:10:52 UTC

Created attachment 131394 [details]
The complete output generated when running "opensnoop" example with valgrind -v flag

SUMMARY
Valgrind fails with error: 

FATAL: unhandled eBPF command 22

valgrind: the 'impossible' happened:
   ... bye!

when trying to run some of the BPF CO-RE program examples found on the iovisor BCC github repository (https://github.com/iovisor/bcc/tree/master/libbpf-tools). 
The error appears on "tcpconnect", "opensnoop", "syscount" and others.

Libbpf is required as a dependency but it can be cloned as a submodule in src/cc/libbpf

STEPS TO REPRODUCE
1. compile kernel to enable BTF support (explained in the repository readme)
2. compile and run example program with valgrind

OBSERVED RESULT

FATAL: unhandled eBPF command 22

valgrind: the 'impossible' happened:
   ... bye!

The full output is in the attached file

EXPECTED RESULT

Valgrind doesn't crash

SOFTWARE/OS VERSIONS

Linux sheepofawesome 5.7.6-gentoo-x86_64 #9 SMP Mon Aug 17 13:26:04 CEST 2020 x86_64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz GenuineIntel GNU/Linux

Also reproduced on:

Linux pc 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64 GNU/Linux

Comment 1 Mark Wielaard 2021-02-20 22:47:36 UTC

eBPF command 22 is BPF_MAP_FREEZE.

valgrind currently handle eBPF commands up to 20 (BPF_TASK_FD_QUERY), it doesn't yet handle some newer commands:

        BPF_MAP_LOOKUP_AND_DELETE_ELEM, /* 21 */
        BPF_MAP_FREEZE,
        BPF_BTF_GET_NEXT_ID,
        BPF_MAP_LOOKUP_BATCH,
        BPF_MAP_LOOKUP_AND_DELETE_BATCH,
        BPF_MAP_UPDATE_BATCH,
        BPF_MAP_DELETE_BATCH,
        BPF_LINK_CREATE,
        BPF_LINK_UPDATE,
        BPF_LINK_GET_FD_BY_ID,
        BPF_LINK_GET_NEXT_ID,
        BPF_ENABLE_STATS,
        BPF_ITER_CREATE,
        BPF_LINK_DETACH,

Comment 2 Andreas Gerstmayr 2021-10-04 17:17:30 UTC

How to reproduce on Fedora 34:

$ dnf install -y git clang llvm elfutils-libelf-devel zlib-devel valgrind
$ git clone --recursive https://github.com/iovisor/bcc.git
$ cd bcc/libbpf-tools && make
$ valgrind ./opensnoop

Comment 3 Mark Wielaard 2021-10-10 21:40:51 UTC

Created attachment 142314 [details]
Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE

Implements BPF_MAP_LOOKUP_AND_DELETE_ELEM (command 20) and BPF_MAP_FREEZE (command 21) and produces a WARNING instead of a fatal error for unrecognized BPF commands.

This does not fully solve the issue with opensnoop, that will still report WARNINGS, but will no longer crash under valgrind:

==3618029== Command: ./opensnoop
==3618029== 
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 35
--3618029-- WARNING: unhandled eBPF command 28
PID    COMM              FD ERR PATH

Comment 4 Mark Wielaard 2021-10-12 21:21:17 UTC

I am closing this now because the reported eBPF command has been implemented. But please feel free to open new bugs for other missing eBPF commands.

commit ef95220ddae1af65c85d8d59a8f0dcbb9d7af90f
Author: Mark Wielaard <mark@klomp.org>
Date:   Tue Oct 12 23:15:41 2021 +0200

    Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM and BPF_MAP_FREEZE
    
    Implement BPF_MAP_LOOKUP_AND_DELETE_ELEM (command 21) and
    BPF_MAP_FREEZE (command 22) and produce a WARNING instead of a fatal
    error for unrecognized BPF commands.
    
    https://bugs.kde.org/show_bug.cgi?id=426148