Open Radar

 
Background: One component of Chrome, Widevine (used for encrypted media playback), is not currently available for arm64. For the time being, we’re shipping an x86_64 version of Widevine to arm64 users, and hosting it in its own x86_64 process under Rosetta translation, communicating with an otherwise fully arm64 running version of Chrome.

In the case where Rosetta is not installed on the system but we detect that Widevine might be necessary, we offer the user the ability to install Rosetta. We can’t rely on the OS to do this for us, because the OS only offers to install Rosetta when launching an x86_64 .app via Launch Services and Rosetta is not already installed. In our case, the x86_64 process is a child of the browser’s, started with posix_spawn (https://chromium.googlesource.com/chromium/src/+/dcfbb59cc22614ae318618824fccc5d446adcb12/base/process/launch_mac.cc#262 base::LaunchProcess), and cannot be started via Launch Services. As such, we must manage this ourselves.

We’re currently using the (private, we know, but we’ve got a job to do here and app store distribution is not relevant to this case) OAHSoftwareUpdate.framework to perform the installation of Rosetta for us (https://chromium.googlesource.com/chromium/src/+/c780b01e35977685e78c948c6badc46ccd0e6333/base/mac/rosetta.mm#55 base::mac::RequestRosettaInstallation).

Aside from leaks and multiple calls to the completion callback block, we’re satisfied with how OAHSoftwareUpdate is working in “happy-path” cases where installation completes successfully. Unfortunately, “sad-path” cases, where the user clicks “Not Now” during download or installation, frequently result in crashes. These crashes are documented at https://crbug.com/1150538.

Steps to reproduce:

I’ve prepared a reduced test case for these crashes, in the attached InstallRosetta.tar.gz. This test case functions equally well whether or not Rosetta is already installed. Build and run the Xcode project, and try the following tests:

A. Cancel during download.
1. Click “Install”.
2. While Rosetta is downloading, QUICKLY click “Not Now”. If you can’t have your mouse positioned JUST right, hit the Return key as soon as possible after choosing “Install” in step 1.
3. Even though you clicked “Not Now”, an authentication dialog will appear. Authenticate as an administrator and click “Install Software”.

B. Cancel during install.
1. Click “Install”.
2. An authentication dialog will appear. Authenticate as an administrator and click “Install Software”.
3. While Rosetta is installing, QUICKLY click “Not Now”. If you can’t have your mouse positioned JUST right, hit the Return key as soon as possible after choosing “Install Software” in step 2.

Expected behavior:

The installation should be cancelled as soon as “Not Now” is clicked. The window should dismiss immediately, and no authentication dialogs should appear. The callback block should receive an indication that installation was cancelled, and all objects related to the installation should be released. The callback block should only be called once.

Observed behavior:

The installation window does not disappear when “Not Now” is clicked. If “Not Now” was clicked during download, the authentication dialog appears anyway, and after successfully authenticating, the application is likely to crash. If “Not Now” was clicked during installation, the application may crash. Even following a successful installation, the OAHSoftwareUpdateController object is leaked. The callback block may be called more than once (observe 2b below, where it’s called twice in succession, once with success = 0, and then again with success = 1).

System information:

mark@arm-and-hammer zsh% sw_vers
ProductName:	macOS
ProductVersion:	11.0.1
BuildVersion:	20B29
mark@arm-and-hammer zsh% xcodebuild -version
Xcode 12.2
Build version 12B45b
mark@arm-and-hammer zsh% uname -m
arm64
mark@arm-and-hammer zsh% system_profiler SPHardwareDataType | grep 'Model Identifier'
      Model Identifier: ADP3,2

Additional information:

Per https://crbug.com/1150538, here are two different characteristic stack backtraces reports that we observe when “Not Now” is clicked, along with equivalent log messages and backtraces from the attached test program.

1a. objc_fatal abort, Chrome crash report.

List Annotations objc_removeExceptionHandler() called with unknown alt handler; this is probably a bug in multithreaded AppKit use. 
Thread 0 (id: 0x000bbf2a) CRASHED [0x00000000 / 0x00000000 @ 0x000000019a168130 ] MAGIC SIGNATURE THREAD
Stack Quality84%Show frame trust levels
0x000000019a168130	(libsystem_kernel.dylib + 0x00029130)		__abort_with_payload
0x000000019a16aa1c	(libsystem_kernel.dylib + 0x0002ba1c)		abort_with_payload_wrapper_internal
0x000000019a16a9b4	(libsystem_kernel.dylib + 0x0002b9b4)		abort_with_reason
0x000000019a0351d8	(libobjc.A.dylib + 0x000291d8)		_objc_fatalv(unsigned long long, unsigned long long, char const*, char*)
0x000000019a035160	(libobjc.A.dylib + 0x00029160)		_objc_fatal(char const*, ...)
0x000000019a022634	(libobjc.A.dylib + 0x00016634)		alt_handler_error(unsigned long)
0x000000019a01d55c	(libobjc.A.dylib + 0x0001155c)		objc_removeExceptionHandler
0x000000019a239314	(CoreFoundation + 0x00049314)		_CFDoExceptionOperation
0x000000019cd6c4b8	(AppKit + 0x003674b8)		-[NSApplication endModalSession:]
0x00000001be9ddf50	(OAHSoftwareUpdate + 0x00010f50)		-[OAHSoftwareUpdateWindowController dismissWindow]
0x00000001be9dd99c	(OAHSoftwareUpdate + 0x0001099c)		__58-[OAHSoftwareUpdateWindowController installButtonClicked:]_block_invoke_2.203
0x0000000199fcc41c	(libdispatch.dylib + 0x0000441c)		_dispatch_client_callout
0x0000000199fcf798	(libdispatch.dylib + 0x00007798)		_dispatch_continuation_pop
0x0000000199fe1294	(libdispatch.dylib + 0x00019294)		_dispatch_source_invoke
0x0000000199fd9d00	(libdispatch.dylib + 0x00011d00)		_dispatch_main_queue_callback_4CF
0x000000019a2b39d8	(CoreFoundation + 0x000c39d8)		__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__
0x000000019a272864	(CoreFoundation + 0x00082864)		__CFRunLoopRun

1b. objc_fatal abort, log and lldb output from the attached test program.

2020-11-23 14:37:32.537661-0500 InstallRosetta[2808:52674] Metal API Validation Enabled
2020-11-23 14:37:34.369166-0500 InstallRosetta[2808:53065] Package Authoring Error: 001-79711: Package reference com.apple.pkg.RosettaUpdateAuto is missing installKBytes attribute
2020-11-23 14:37:34.952145-0500 InstallRosetta[2808:52674] completion, success = 0, error = Error Domain=NSCocoaErrorDomain Code=3072 "The operation was cancelled."
objc[2808]: objc_removeExceptionHandler() called with unknown alt handler; this is probably a bug in multithreaded AppKit use. Set environment variable OBJC_DEBUG_ALT_HANDLERS=YES or break in objc_alt_handler_error() to debug.
objc[2808]: objc_removeExceptionHandler() called with unknown alt handler; this is probably a bug in multithreaded AppKit use. 
objc_removeExceptionHandler() called with unknown alt handler; this is probably a bug in multithreaded AppKit use.
(lldb) thread backtrace
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x000000018ff64130 libsystem_kernel.dylib`__abort_with_payload + 8
    frame #1: 0x000000018ff66a20 libsystem_kernel.dylib`abort_with_payload_wrapper_internal + 104
    frame #2: 0x000000018ff669b8 libsystem_kernel.dylib`abort_with_reason + 32
    frame #3: 0x000000018fe311dc libobjc.A.dylib`_objc_fatalv(unsigned long long, unsigned long long, char const*, char*) + 120
    frame #4: 0x000000018fe31164 libobjc.A.dylib`_objc_fatal(char const*, ...) + 44
    frame #5: 0x000000018fe1e638 libobjc.A.dylib`alt_handler_error(unsigned long) + 440
    frame #6: 0x000000018fe19560 libobjc.A.dylib`objc_removeExceptionHandler + 364
    frame #7: 0x0000000190035318 CoreFoundation`_CFDoExceptionOperation + 316
    frame #8: 0x0000000192b684bc AppKit`-[NSApplication endModalSession:] + 40
    frame #9: 0x00000001b47d9f54 OAHSoftwareUpdate`-[OAHSoftwareUpdateWindowController dismissWindow] + 196
    frame #10: 0x00000001b47d99a0 OAHSoftwareUpdate`__58-[OAHSoftwareUpdateWindowController installButtonClicked:]_block_invoke_2.203 + 44
    frame #11: 0x0000000104bbdd20 libdispatch.dylib`_dispatch_client_callout + 20
    frame #12: 0x0000000104bc13cc libdispatch.dylib`_dispatch_continuation_pop + 612
    frame #13: 0x0000000104bda3a4 libdispatch.dylib`_dispatch_source_invoke + 1380
    frame #14: 0x0000000104bcfd50 libdispatch.dylib`_dispatch_main_queue_callback_4CF + 560
    frame #15: 0x00000001900af9dc CoreFoundation`__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
    frame #16: 0x000000019006e868 CoreFoundation`__CFRunLoopRun + 2508
    frame #17: 0x000000019006d730 CoreFoundation`CFRunLoopRunSpecific + 600
    frame #18: 0x0000000197b76f0c HIToolbox`RunCurrentEventLoopInMode + 292
    frame #19: 0x0000000197b76d3c HIToolbox`ReceiveNextEventCommon + 688
    frame #20: 0x0000000197b76a6c HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 76
    frame #21: 0x0000000192842db0 AppKit`_DPSNextEvent + 868
    frame #22: 0x0000000192841730 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1312
    frame #23: 0x000000019283358c AppKit`-[NSApplication run] + 600
    frame #24: 0x0000000192804d74 AppKit`NSApplicationMain + 1064
  * frame #25: 0x00000001049a736c InstallRosetta`main(argc=3, argv=0x000000016b45b438) at main.m:34:10
    frame #26: 0x000000018ff90f54 libdyld.dylib`start + 4

2a. malloc_report abort, Chrome crash report.

List Annotations abort() called
Google Chrome(36255,0x10056bd40) malloc: *** error for object 0x123c17720: pointer being freed was not allocated
Thread 0 (id: 0x000c3f70) CRASHED [0x00000000 / 0x00000000 @ 0x000000019a147cec ] MAGIC SIGNATURE THREAD
Stack Quality84%Show frame trust levels
0x000000019a147cec	(libsystem_kernel.dylib + 0x00008cec)		__pthread_kill
0x000000019a178c20	(libsystem_pthread.dylib + 0x00006c20)		pthread_kill
0x000000019a0c0860	(libsystem_c.dylib + 0x00078860)		abort
0x0000000199fabd04	(libsystem_malloc.dylib + 0x0000fd04)		malloc_vreport
0x0000000199faf454	(libsystem_malloc.dylib + 0x00013454)		malloc_report
0x0000000199f9e0fc	(libsystem_malloc.dylib + 0x000020fc)		free
0x000000019d34e29c	(AppKit + 0x0094929c)		-[_NSBlockDeallocationHook dealloc]
0x000000019a03490c	(libobjc.A.dylib + 0x0002890c)		AutoreleasePoolPage::releaseUntil(objc_object**)
0x000000019a013cfc	(libobjc.A.dylib + 0x00007cfc)		objc_autoreleasePoolPop
0x000000019a23114c	(CoreFoundation + 0x0004114c)		_CFAutoreleasePoolPop
0x000000019a347adc	(CoreFoundation + 0x00157adc)		__CFRunLoopPerCalloutARPEnd
0x000000019a2728a8	(CoreFoundation + 0x000828a8)		__CFRunLoopRun

2b. malloc_report abort, log and lldb output from the attached test program.

2020-11-23 14:40:24.578787-0500 InstallRosetta[2884:55316] Metal API Validation Enabled
2020-11-23 14:40:26.133875-0500 InstallRosetta[2884:55346] Package Authoring Error: 001-79711: Package reference com.apple.pkg.RosettaUpdateAuto is missing installKBytes attribute
2020-11-23 14:40:26.286167-0500 InstallRosetta[2884:55316] completion, success = 0, error = Error Domain=NSCocoaErrorDomain Code=3072 "The operation was cancelled."
2020-11-23 14:40:32.522308-0500 InstallRosetta[2884:55316] completion, success = 1, error = (null)
InstallRosetta(2884,0x102f4bd40) malloc: *** error for object 0x600001232e20: pointer being freed was not allocated
InstallRosetta(2884,0x102f4bd40) malloc: *** set a breakpoint in malloc_error_break to debug
InstallRosetta(2884,0x102f4bd40) malloc: *** error for object 0x600001232e20: pointer being freed was not allocated
(lldb) thread backtrace
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x000000018ff43cec libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000102df7384 libsystem_pthread.dylib`pthread_kill + 292
    frame #2: 0x000000018febc864 libsystem_c.dylib`abort + 104
    frame #3: 0x000000018fda7d08 libsystem_malloc.dylib`malloc_vreport + 560
    frame #4: 0x000000018fdab458 libsystem_malloc.dylib`malloc_report + 64
    frame #5: 0x000000018fd9a100 libsystem_malloc.dylib`free + 532
    frame #6: 0x000000019314a2a0 AppKit`-[_NSBlockDeallocationHook dealloc] + 40
    frame #7: 0x000000018fe30910 libobjc.A.dylib`AutoreleasePoolPage::releaseUntil(objc_object**) + 204
    frame #8: 0x000000018fe0fd00 libobjc.A.dylib`objc_autoreleasePoolPop + 212
    frame #9: 0x000000019002d150 CoreFoundation`_CFAutoreleasePoolPop + 32
    frame #10: 0x0000000190143ae0 CoreFoundation`__CFRunLoopPerCalloutARPEnd + 48
    frame #11: 0x000000019006e8ac CoreFoundation`__CFRunLoopRun + 2576
    frame #12: 0x000000019006d730 CoreFoundation`CFRunLoopRunSpecific + 600
    frame #13: 0x0000000197b76f0c HIToolbox`RunCurrentEventLoopInMode + 292
    frame #14: 0x0000000197b76d3c HIToolbox`ReceiveNextEventCommon + 688
    frame #15: 0x0000000197b76a6c HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 76
    frame #16: 0x0000000192842db0 AppKit`_DPSNextEvent + 868
    frame #17: 0x0000000192841730 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1312
    frame #18: 0x000000019283358c AppKit`-[NSApplication run] + 600
    frame #19: 0x0000000192804d74 AppKit`NSApplicationMain + 1064
  * frame #20: 0x0000000102b1336c InstallRosetta`main(argc=3, argv=0x000000016d2ef438) at main.m:34:10
    frame #21: 0x000000018ff90f54 libdyld.dylib`start + 4