Open Radar

 
Summary:
The attached program (mem-leaker.c (EDIT: for openradar: https://gist.github.com/weissi/50930f3898bdc18c63e0 )) which does the following:

- shm_open a shared memory segment
- shm_unlink it immediately
- ftruncate the file descriptor referencing the shm segment to 100MiB
- mmap the shared memory
- fork
- parent: munmap & close shared mem immediately
- child: sleep 10, munmap & close shared mem (which should release all the shared memory as the last reference to it did now vanish)

will leak 100MiB per invocation of the program (compile eg as `clang -Wall -Werror -o mem-leaker mem-leaker.c`). The memory is not reclaimable in any way.

The bug is in the shm_truncate function in bsd/kern/posix_shm.c [1]

it sets the pshm_flags to PSHM_ALLOCATED (pinfo->pshm_flags = PSHM_ALLOCATED;) overriding any other flags that have possibly been set. For example PSHM_REMOVED (which is set by shm_unlink(2)). In the pshm_close function however the (shared) memory is only freed iff the PSHM_REMOVED flag is set [2].

This bug is extremely easy to use as a denial-of-service (DOS) exploit since it creates an arbitrary amount of wired memory from userspace (for an unprivileged user). It therefore is a major security issue. After enough invocations of the program, the machine will freeze and not recover. The bug affects at least OS X 10.10 and 10.9.5 but I suspect many more versions are affected.

The easiest way to reproduce is by switching off the dynamic pager and the memory compressor. Then the host machine will freeze after a maximum of (physical memory size in GiB)*10 invocations of the program even with an arbitrarily long pause between the invocations (even though the program should release all memory after 10s).

However. if you take the attached program and swap ftruncate and shm_unlink, it doesn't leak memory at all.

[1]: https://github.com/opensource-apple/xnu/blob/10.10/bsd/kern/posix_shm.c#L745
[2]: https://github.com/opensource-apple/xnu/blob/10.10/bsd/kern/posix_shm.c#L1175

Steps to Reproduce:
1. switch off the dynamic pager (sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist)
2. switch off the memory compressor (sudo nvram boot-args=vm_compressor)
3. reboot
4. Download the attached program mem-leaker.c
5. Compile it using clang -Wall -Werror -o mem-leaker mem-leaker.c
6. Run the program a lot of times, eg for f in $(seq $(( $(sysctl hw.memsize | cut -d: -f2) / 1024/1024/1024 * 10 )) ); do echo "Run $f"; ./mem-leaker; sleep 5; done 

Expected Results:
Nothing should happen

Actual Results:
Machine will freeze

Version:
The MacBook Pros running 10.9.5 -- 13F34 and the Mac mini running 10.10


Notes:


Configuration:
MacBook Pro Retina 13" early 2013 and MacBook Pro Retina 15" late 2013 and new Mac mini.

Attachments:
'mem-leaker.c' was successfully uploaded.