Do you have any Valgrind logs? The Java logs are meaningless to me, other than SIGILL.

Created attachment 169752 [details]
sample valgrind file (snipped for size)

had to truncate the file (actual size ~350MB) -- hope this helps

Created attachment 169753 [details]
associated hs_err_pid file

This looks like
66 48 0f 73 f1 0c rex.W psllq xmm1,0xc

Some more from the log

vex amd64->IR: unhandled instruction bytes: 0x66 0x48 0xF 0x73 0xF1 0xC 0x66 0xF 0x70 0xF5
vex amd64->IR:   REX=1 REX.W=1 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR:   PFX.66=1 PFX.F2=0 PFX.F3=0

"psllq" should be handled, but it looks like it's the REX prefix that is causing the problem.

The odd thing is that the SIGILL is only triggered on the X5670 CPU -- the i7-4770R does not trigger the SIGILL (exact same code, OS, etc.)

That's not unusual. There may be some kind of runtime instruction set selection based on hwcaps/cpuid.

(In reply to Paul Floyd from comment #5)
> This looks like
> 66 48 0f 73 f1 0c rex.W psllq xmm1,0xc
> 
> Some more from the log
> 
> vex amd64->IR: unhandled instruction bytes: 0x66 0x48 0xF 0x73 0xF1 0xC 0x66
> 0xF 0x70 0xF5
> vex amd64->IR:   REX=1 REX.W=1 REX.R=0 REX.X=0 REX.B=0
> vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
> vex amd64->IR:   PFX.66=1 PFX.F2=0 PFX.F3=0
> 
> "psllq" should be handled, but it looks like it's the REX prefix that is
> causing the problem.

I think we can just ignore the REX prefix. The intel  64 and IA-32 Architectures Software Developer’s Manual section 10.2.1 "Intel SSE in 64-Bit Mode and Compatibility Mode" says:

"In 64-bit mode, eight additional XMM registers are accessible. Registers XMM8-XMM15 are accessed by using REX prefixes. Memory operands are specified using the ModR/M, SIB encoding described in Section 3.7.5. Some Intel SSE instructions may be used to operate on general-purpose registers. Use the REX.W prefix to access 64-bit general-purpose registers. Note that if a REX prefix is used when it has no meaning, the prefix is ignored."

Bill Torpey, could you build Valgrind to test a fix?

This change seems to fix your problem:

diff --git a/VEX/priv/guest_amd64_toIR.c b/VEX/priv/guest_amd64_toIR.c
index f0b1c5516..28c37f092 100644
--- a/VEX/priv/guest_amd64_toIR.c
+++ b/VEX/priv/guest_amd64_toIR.c
@@ -14138,7 +14138,7 @@ Long dis_ESC_0F__SSE2 ( Bool* decode_OK,
          goto decode_success;
       }
       /* 66 0F 73 /6 ib = PSLLQ by immediate */
-      if (have66noF2noF3(pfx) && sz == 2 
+      if (have66noF2noF3(pfx) && (sz == 2 || /* ignore redundant REX.W */ sz == 8)
           && epartIsReg(getUChar(delta))
           && gregLO3ofRM(getUChar(delta)) == 6) {
          delta = dis_SSE_shiftE_imm( pfx, delta, "psllq", Iop_ShlN64x2 );

Mark, do you know how to force using rex.W in assembler?

This testcase also seems to work

/* with redundant rex.W */
static void psllq_4(void)
{
   reg128_t arg1 = { .uq = { 0x0123456789abcdefULL, 0x0123456789abcdefULL } };
   reg128_t result0;
   char state[108];

   if (sigsetjmp(catchpoint, 1) == 0)
   {
      asm(
         "ffree %%st(7)\n"
         "ffree %%st(6)\n"
         "ffree %%st(5)\n"
         "ffree %%st(4)\n"
         "movlps %2, %%xmm1\n"
         "movhps %3, %%xmm1\n"
         //".rex.W psllq $12, %%xmm1\n"
         ".byte 0x66,0x48,0x0f,0x73,0xf1,0x0c\n"
         "movlps %%xmm1, %0\n"
         "movhps %%xmm1, %1\n"
         "cld\n"
         : "=m" (result0.uq[0]), "=m" (result0.uq[1])
         : "m" (arg1.uq[0]), "m" (arg1.uq[1]), "m" (state[0])
         : "xmm1"
      );

      if (result0.uq[0] == 0x3456789abcdef000ULL && result0.uq[1] == 0x3456789abcdef000ULL )
      {
         printf("psllq_4 ... ok\n");
      }
      else
      {
         printf("psllq_4 ... not ok\n");
         printf("  result0.uq[0] = %llu (expected %llu)\n", result0.uq[0], 0x3456789abcdef000ULL);
         printf("  result0.uq[1] = %llu (expected %llu)\n", result0.uq[1], 0x3456789abcdef000ULL);
      }
   }
   else
   {
      printf("psllq_4 ... failed\n");
   }

   return;
}

but it doesn't fit in with the gen_insn_test.pl script.

(In reply to Paul Floyd from comment #9)
> diff --git a/VEX/priv/guest_amd64_toIR.c b/VEX/priv/guest_amd64_toIR.c
> index f0b1c5516..28c37f092 100644
> --- a/VEX/priv/guest_amd64_toIR.c
> +++ b/VEX/priv/guest_amd64_toIR.c
> @@ -14138,7 +14138,7 @@ Long dis_ESC_0F__SSE2 ( Bool* decode_OK,
>           goto decode_success;
>        }
>        /* 66 0F 73 /6 ib = PSLLQ by immediate */
> -      if (have66noF2noF3(pfx) && sz == 2 
> +      if (have66noF2noF3(pfx) && (sz == 2 || /* ignore redundant REX.W */
> sz == 8)
>            && epartIsReg(getUChar(delta))
>            && gregLO3ofRM(getUChar(delta)) == 6) {
>           delta = dis_SSE_shiftE_imm( pfx, delta, "psllq", Iop_ShlN64x2 );
> 
> Mark, do you know how to force using rex.W in assembler?

It looks like binutils gas allows you to use rex.W as prefix, so:
rex.W psllq $12, %xmm1

The documentation is a little confusing though (I am not sure whether W should be seen as X, Y and Z extension bit or not):
https://sourceware.org/binutils/docs/as/i386_002dPrefixes.html

The ‘rex’ family of prefixes is used by x86-64 to encode extensions to i386 instruction set. The ‘rex’ prefix has four bits — an operand size overwrite (64) used to change operand size from 32-bit to 64-bit and X, Y and Z extensions bits used to extend the register set.

You may write the ‘rex’ prefixes directly. The ‘rex64xyz’ instruction emits ‘rex’ prefix with all the bits set. By omitting the 64, x, y or z you may write other prefixes as well. Normally, there is no need to write the prefixes explicitly, since gas will automatically generate them based on the instruction operands.

That doesn't work with clang unfortunately and I have to resort to the byte array.

Hi all -- back from the long holiday here in US.  Will try suggestions and post back with results.  Thanks for all the help!

Been testing for the past couple days, and it appears that the patch from https://bugs.kde.org/show_bug.cgi?id=487439#c10 is working fine for us -- thanks!

(In reply to Bill Torpey from comment #13)
> Been testing for the past couple days, and it appears that the patch from
> https://bugs.kde.org/show_bug.cgi?id=487439#c10 is working fine for us --
> thanks!

Good. It might take me a while to finalize the change - I'd like to see if this applies to other opcodes and write a testcase.

Do you have any way of telling which compiler was used to produce the binary that is generating sigill?

The offending instructions are not compiled - they are defined as inline asm in OpenJDK, specifically in src/hotspot/cpu/x86/stubGenerator_x86_64_log.cpp (attached).  

The only thing I can tell from the executable file (/usr/lib/jvm/java-11-openjdk-11.0.4.11-0.el7_6.x86_64/bin/java)  is that it has a dependency on GLIBC_2.2.5.

FWIW, we get similar results with OpenJDK 17, as well as with Azul build of OpenJDK 11.

Created attachment 170348 [details]
offending instructions

commit c19d19d34a6dadaf4a9d590f516f813e9cbacdd0
Author: Paul Floyd <pjfloyd@wanadoo.fr>
Date:   Sun Jun 16 09:25:51 2024 +0200

    Bug 487439 - SIGILL in JDK11, JDK17

See commit in comment #18