Can be reproducible by calling `calloc()` with values that would result in overflow. All the dynamic memory allocation system calls(malloc, calloc, realloc) set `errno` to ENOMEM when they return NULL. This is POSIX compliant.

Granted, no one checks errno. We all assume that allocation failed when the functions return NULL. But there are use cases where errno is used for diagnostic purposes. That's how I found this bug anyway.

This bug is 11 years old. Would someone please at least give this bug a verdict? I'd take WONTFIX as an answer.

(In reply to Ashe David Goulding from comment #1)
> Can be reproducible by calling `calloc()` with values that would result in
> overflow. All the dynamic memory allocation system calls(malloc, calloc,
> realloc) set `errno` to ENOMEM when they return NULL. This is POSIX
> compliant.
> 
> Granted, no one checks errno. We all assume that allocation failed when the
> functions return NULL. But there are use cases where errno is used for
> diagnostic purposes. That's how I found this bug anyway.
> 
> This bug is 11 years old. Would someone please at least give this bug a
> verdict? I'd take WONTFIX as an answer.

In theory this (setting errno to ENOMEM on an allocation function returning NULL) could be supported. But it is not that easy to do correct without knowing how the libc the program is running against handles errno. In practice this is always glibc, so maybe we can try.

To show why this is non-trivial from errno(3):

       errno is defined by the ISO C standard to be a modifiable lvalue
       of type int, and must not be explicitly declared; errno may be a
       macro.

And glibc defines it as:

extern int *__errno_location (void) __THROW __attribute__ ((__const__));
#define errno (*__errno_location ())

Now the malloc intercepts actually run on the simulated cpu (see coregrind/m_replacemalloc/vg_replace_malloc.c) and could in theory use the above definition to set errno to ENOMEM in ALLOC_or_NULL (when NULL).

But that would make that vg_replace_malloc rely on a libc that has the magic __errno_location function.

Created attachment 135649 [details]
Set (glibc) errno to ENOMEM when malloc/calloc/realloc/memalign fail

This turned out to be less hard than I assumed. We already have a trick for calling functions which might only be defined in glibc. The idea is to define the symbol as weak in out preload library. It will then be NULL unless some other library defines it for real. Then we can simply call it when it is non-NULL (and only when one of the allocation functions returns NULL).

12 years! Thank you.

commit 1c9a0bf58a47e855e6e5bf78a30bcee0af835804
Author: Mark Wielaard <mark@klomp.org>
Date:   Fri Feb 12 23:29:34 2021 +0100

    PR217695 malloc/calloc/realloc/memalign failure doesn't set errno to ENOMEM
    
    When one of the allocation functions in vg_replace_malloc failed
    they return NULL, but didn't set errno. This is slightly tricky since
    errno is implementation defined and might be a macro. In the case of
    glibc ernno is defined as:
    
      extern int *__errno_location (void) __THROW __attribute__ ((__const__));
      #define errno (*__errno_location ())
    
    We can use the same trick as we use for __libc_freeres in
    coregrind/vg_preloaded.c. Define the function as "weak". This means
    it will only be defined if another library (glibc in this case)
    actually provides a definition. Otherwise it will be NULL.
    So we will only call it if it is defined and one of the allocation
    functions failed, returned NULL.
    
    Include a new linux only memcheck testcase, enomem.vgtest.
    
    https://bugs.kde.org/show_bug.cgi?id=217695