Open Radar

 
Some illegal instructions cause the process to hang instead of generating an illegal instruction signal (SIGILL).

Compile and run the following program:

main.c:

void foo();
int main(int argc, char *argv[]) {
  foo();
}

main.s:

	.globl _foo
_foo:
	vpopcntw	%zmm1, %zmm0
	ret

Compile with:

    gcc main.c main.s

Run with

    ./a.out

It should generate the text "Illegal instruction" and exit. Instead, it hangs forever.

You need to run this on a mac for which vpopcntw is an unsupported instruction, which may be all macs at the moment.

Disassembling the binary shows the correct instruction, so I don't think the instruction is getting compiled incorrectly (to a jump with 0 offset, say).

The same program behaves correctly on Linux (remove the underscores on in the assembly file to port it to Linux).

The program also behaves correctly if you replace vpopcntw with ud2. So it isn't all instructions, just some of them (maybe some subset of avx512?).

When run under a debugger, there's no signal being generated. Any interrupt from the debugger end shows the PC at the vpopcntw instruction.

First observed in the Go project, https://github.com/golang/go/issues/42649