Crash: Double-Free when using Core Image (deep in CoreImage /libFosl_dynamic)

Originator:steipete
Number:rdar://28252672 Date Originated:12-Sep-2016 02:54 AM
Status:Closed Resolved:
Product:CoreImage Product Version:
Classification:Crash Reproducible:Sometimes
 
Peter Steinberger12-Sep-2016 02:54 AM

Area:
Something not on this list

Summary:
We've been seeing random crashes in our CI. So far I haven't been able to reproduce this locally. We're using CoreImage to apply filters to images (dark mode for PDF).

Steps to Reproduce:
So far I've been unable to reduce this issue to a sample project - it seems to be a race condition somewhere in libFosl/libLLVMContainer). However, I enabled ASAN for our tests and after running them often enough, I managed to get a good stack trace of the issue:



==45917==ERROR: AddressSanitizer: attempting double-free on 0x644f0a80 in thread T292:
    #0 0x238dafa in wrap_free (libclang_rt.asan_iossim_dynamic.dylib+0x50afa)
    #1 0x6e851245 in llvm::StringMapImpl::RehashTable(unsigned int) (libFosl_dynamic.dylib+0x173245)
    #2 0x6e8817bf in llvm::StringMap<fosl::builtin::ID, llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul>&>::insert(std::__1::pair<llvm::StringRef, fosl::builtin::ID>) (libFosl_dynamic.dylib+0x1a37bf)
    #3 0x6e88143a in fosl::BuiltinTableLookup::getOrCreateBuiltinMap() (libFosl_dynamic.dylib+0x1a343a)
    #4 0x6e6e1f17 in fosl::Compiler::initFromOptions(fosl::ASTContext&, fosl::FrontendOptions const&, clang::DiagnosticsEngine&) (libFosl_dynamic.dylib+0x3f17)
    #5 0x6e6e2438 in fosl::Compiler::createFromMemBufferOptions(fosl::ASTContext&, fosl::FrontendOptions const&, clang::DiagnosticsEngine&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer> >) (libFosl_dynamic.dylib+0x4438)
    #6 0x6e6e26d1 in fosl::Compiler::createForFilterKernels(fosl::ASTContext&, clang::DiagnosticsEngine&, bool) (libFosl_dynamic.dylib+0x46d1)
    #7 0x6e7bc236 in fosl::filter::DAG::DAG(fosl::ASTContext&, clang::DiagnosticsEngine&) (libFosl_dynamic.dylib+0xde236)
    #8 0x6e7c0d1f in fosl_filter_createGraph (libFosl_dynamic.dylib+0xe2d1f)
    #9 0x3dd7326 in ___ZN2CIL19prepare_render_treeEPNS_7ContextEPKcPNS_4NodeE_block_invoke32 (CoreImage+0x76326)
    #10 0x3daf0f7 in CI::traverse_nodes(CI::Node*, CI::Node*, int, void (CI::Node*, CI::Node*, int) block_pointer) (CoreImage+0x4e0f7)
    #11 0x3dd463f in CI::prepare_render_tree(CI::Context*, char const*, CI::Node*) (CoreImage+0x7363f)
    #12 0x3dd60b2 in CI::image_get_cgimage(CI::Context*, CI::Image*, CGRect, CGColorSpace*, CI::PixelFormat) (CoreImage+0x750b2)
    #13 0x3d93197 in -[CIContext createCGImage:fromRect:format:colorSpace:] (CoreImage+0x32197)
    #14 0x3d92e57 in -[CIContext createCGImage:fromRect:] (CoreImage+0x31e57)
2016-09-11 17:21:18.727 PSPDFTestHost[45917:26584283] Main Thread doesn't answer...
==45917==T262 TSDDtor
==45917==T262 exited
==45917==T170 TSDDtor
==45917==T170 exited
2016-09-11 17:21:23.726 PSPDFTestHost[45917:26584283] Main Thread doesn't answer...
2016-09-11 17:21:28.727 PSPDFTestHost[45917:26584283] Main Thread doesn't answer...
2016-09-11 17:21:33.730 PSPDFTestHost[45917:26584283] Main Thread doesn't answer...
    #15 0xee24a80 in __74-[PSPDFDefaultRenderManager applyFilters:toContext:clipRect:pageRotation:]_block_invoke_2 PSPDFRenderManager.mm:477
    #16 0xee22958 in -[PSPDFDefaultRenderManager performOperationInContext:] PSPDFRenderManager.mm:410
    #17 0xee23c94 in -[PSPDFDefaultRenderManager applyFilters:toContext:clipRect:pageRotation:] PSPDFRenderManager.mm:476
    #18 0xee219fe in -[PSPDFDefaultRenderManager renderAdditionalOptionsInContext:pageInfo:clipRect:options:] PSPDFRenderManager.mm:364
    #19 0xee1ff8d in -[PSPDFDefaultRenderManager renderPageAtIndex:documentProvider:inContext:atPoint:withZoom:pageInfo:annotations:options:error:] PSPDFRenderManager.mm:305
    #20 0xee1d861 in -[PSPDFDefaultRenderManager renderPageAtIndex:documentProvider:inContext:rectangle:pageInfo:annotations:options:error:] PSPDFRenderManager.mm:207
    #21 0xef79ad3 in __88-[PSPDFDocument renderPageAtIndex:context:size:clippedToRect:annotations:options:error:]_block_invoke_2 PSPDFDocument.mm:1709
    #22 0x10eb2c2c in pspdf_performAndTrackTime PSPDFDispatch.m:107
    #23 0xef78781 in -[PSPDFDocument renderPageAtIndex:context:size:clippedToRect:annotations:options:error:] PSPDFDocument.mm:1708
    #24 0xef76af4 in -[PSPDFDocument imageForPageAtIndex:size:clippedToRect:annotations:options:error:] PSPDFDocument.mm:1651
    #25 0xefb06a5 in -[PSPDFRenderJob start] PSPDFRenderJob.m:109
    #26 0xf2d4782 in __30-[PSPDFRenderQueue dequeueJob]_block_invoke.135 PSPDFRenderQueue.m:283
    #27 0x2e616fc in _dispatch_client_callout (libdispatch.dylib+0x1f6fc)
    #28 0x2e46227 in _dispatch_block_invoke (libdispatch.dylib+0x4227)
    #29 0x2e5ef17 in ___dispatch_block_create_block_invoke (libdispatch.dylib+0x1cf17)
    #30 0x238c92c in __wrap_dispatch_async_block_invoke (libclang_rt.asan_iossim_dynamic.dylib+0x4f92c)
    #31 0x2e439f2 in _dispatch_call_block_and_release (libdispatch.dylib+0x19f2)
    #32 0x2e616fc in _dispatch_client_callout (libdispatch.dylib+0x1f6fc)
    #33 0x2e47020 in _dispatch_async_redirect_invoke (libdispatch.dylib+0x5020)
    #34 0x2e616fc in _dispatch_client_callout (libdispatch.dylib+0x1f6fc)
    #35 0x2e4a85f in _dispatch_root_queue_drain (libdispatch.dylib+0x885f)
    #36 0x2e4a570 in _dispatch_worker_thread3 (libdispatch.dylib+0x8570)
    #37 0x317c25b in _pthread_wqthread (libsystem_pthread.dylib+0x325b)
    #38 0x3179f55 in start_wqthread (libsystem_pthread.dylib+0xf55)

0x644f0a80 is located 0 bytes inside of 2296-byte region [0x644f0a80,0x644f1378)
freed by thread T18 here:
    #0 0x238dafa in wrap_free (libclang_rt.asan_iossim_dynamic.dylib+0x50afa)
    #1 0x843e769 in llvm::InstCombiner::visitAllocaInst(llvm::AllocaInst&) (libLLVMContainer.dylib+0x6f2769)
    #2 0x83e6144 in llvm::InstCombiner::DoOneIteration(llvm::Function&, unsigned int) (libLLVMContainer.dylib+0x69a144)
    #3 0x83e6432 in llvm::InstCombiner::runOnFunction(llvm::Function&) (libLLVMContainer.dylib+0x69a432)
    #4 0x87b5e72 in llvm::FPPassManager::runOnFunction(llvm::Function&) (libLLVMContainer.dylib+0xa69e72)
    #5 0x87b57e9 in llvm::legacy::FunctionPassManagerImpl::run(llvm::Function&) (libLLVMContainer.dylib+0xa697e9)
    #6 0x87b56ef in llvm::legacy::FunctionPassManager::run(llvm::Function&) (libLLVMContainer.dylib+0xa696ef)
    #7 0x6b022499 in cvmOptimizeModularFunction (libGLVMPlugin.dylib+0x2499)
    #8 0x6b061945 in glvmBuildFPTransformFunction (libGLVMPlugin.dylib+0x41945)
    #9 0x6b021a61 in cvmsPluginElementBuild (libGLVMPlugin.dylib+0x1a61)
    #10 0x7d3d1c3 in cvmsCompBuildElement (libCoreVMClient.dylib+0x11c3)
    #11 0x7d42012 in cvmsServerElementBuild (libCoreVMClient.dylib+0x6012)
    #12 0x7d3daea in cvms_element_build_from_source (libCoreVMClient.dylib+0x1aea)
    #13 0x7d36871 in cvm_deferred_build_modular(void*) (libCVMSPluginSupport.dylib+0x2871)
    #14 0x238bb23 in asan_dispatch_call_block_and_release (libclang_rt.asan_iossim_dynamic.dylib+0x4eb23)
    #15 0x2e616fc in _dispatch_client_callout (libdispatch.dylib+0x1f6fc)
    #16 0x2e4933d in _dispatch_queue_drain (libdispatch.dylib+0x733d)
    #17 0x2e48c88 in _dispatch_queue_invoke (libdispatch.dylib+0x6c88)
    #18 0x2e4a731 in _dispatch_root_queue_drain (libdispatch.dylib+0x8731)
    #19 0x2e4a570 in _dispatch_worker_thread3 (libdispatch.dylib+0x8570)
    #20 0x317c25b in _pthread_wqthread (libsystem_pthread.dylib+0x325b)
    #21 0x3179f55 in start_wqthread (libsystem_pthread.dylib+0xf55)

previously allocated by thread T18 here:
    #0 0x238dd00 in wrap_realloc (libclang_rt.asan_iossim_dynamic.dylib+0x50d00)
    #1 0x882ef57 in llvm::SmallVectorBase::grow_pod(void*, unsigned long, unsigned long) (libLLVMContainer.dylib+0xae2f57)
    #2 0x843e331 in llvm::InstCombiner::visitAllocaInst(llvm::AllocaInst&) (libLLVMContainer.dylib+0x6f2331)
    #3 0x83e6144 in llvm::InstCombiner::DoOneIteration(llvm::Function&, unsigned int) (libLLVMContainer.dylib+0x69a144)
    #4 0x83e6432 in llvm::InstCombiner::runOnFunction(llvm::Function&) (libLLVMContainer.dylib+0x69a432)
    #5 0x87b5e72 in llvm::FPPassManager::runOnFunction(llvm::Function&) (libLLVMContainer.dylib+0xa69e72)
    #6 0x87b57e9 in llvm::legacy::FunctionPassManagerImpl::run(llvm::Function&) (libLLVMContainer.dylib+0xa697e9)
    #7 0x87b56ef in llvm::legacy::FunctionPassManager::run(llvm::Function&) (libLLVMContainer.dylib+0xa696ef)
    #8 0x6b022499 in cvmOptimizeModularFunction (libGLVMPlugin.dylib+0x2499)
    #9 0x6b061945 in glvmBuildFPTransformFunction (libGLVMPlugin.dylib+0x41945)
    #10 0x6b021a61 in cvmsPluginElementBuild (libGLVMPlugin.dylib+0x1a61)
    #11 0x7d3d1c3 in cvmsCompBuildElement (libCoreVMClient.dylib+0x11c3)
    #12 0x7d42012 in cvmsServerElementBuild (libCoreVMClient.dylib+0x6012)
    #13 0x7d3daea in cvms_element_build_from_source (libCoreVMClient.dylib+0x1aea)
    #14 0x7d36871 in cvm_deferred_build_modular(void*) (libCVMSPluginSupport.dylib+0x2871)
    #15 0x238bb23 in asan_dispatch_call_block_and_release (libclang_rt.asan_iossim_dynamic.dylib+0x4eb23)
    #16 0x2e616fc in _dispatch_client_callout (libdispatch.dylib+0x1f6fc)
    #17 0x2e4933d in _dispatch_queue_drain (libdispatch.dylib+0x733d)
    #18 0x2e48c88 in _dispatch_queue_invoke (libdispatch.dylib+0x6c88)
    #19 0x2e4a731 in _dispatch_root_queue_drain (libdispatch.dylib+0x8731)
    #20 0x2e4a570 in _dispatch_worker_thread3 (libdispatch.dylib+0x8570)
    #21 0x317c25b in _pthread_wqthread (libsystem_pthread.dylib+0x325b)
    #22 0x3179f55 in start_wqthread (libsystem_pthread.dylib+0xf55)

Thread T292 created by T18 here:
    <empty stack>

Thread T18 created by T0 here:
    <empty stack>

SUMMARY: AddressSanitizer: double-free (libclang_rt.asan_iossim_dynamic.dylib+0x50afa) in wrap_free
==45917==T354: stack [0xb0833000,0xb08b3000) size 0x80000; local=0xb08b2f20
==45917==T355: stack [0xb021b000,0xb029b000) size 0x80000; local=0xb029af20
==45917==T356: stack [0xb0e4b000,0xb0ecb000) size 0x80000; local=0xb0ecaf20
==45917==T357: stack [0xb0ecd000,0xb0f4d000) size 0x80000; local=0xb0f4cf20
==45917==T358: stack [0xb0f4f000,0xb0fcf000) size 0x80000; local=0xb0fcef20
==45917==T359: stack [0xb1053000,0xb10d3000) size 0x80000; local=0xb10d2f20
==45917==T360: stack [0xb0937000,0xb09b7000) size 0x80000; local=0xb09b6a90
==45917==T361: stack [0xb0cc5000,0xb0d45000) size 0x80000; local=0xb0d44950
==45917==T362: stack [0xb10d5000,0xb1155000) size 0x80000; local=0xb11549f0
==45917==T363: stack [0xb1157000,0xb11d7000) size 0x80000; local=0xb11d6f20
==45917==ABORTING


Here's the code that we use:

[Omitted from open radar]

Expected Results:
This should not crash.

Actual Results:
Sporadic crashes.

Version:
iOS 9.1/9.3/10

Notes:
Has this been fixed in iOS 10? Can we somehow work around this?

I haven't check closely on which nodes this happened so I cannot report with 100% accuracy if this only affects iOS 9 or also iOS 10. Will keep an eye on it and update this radar as I find more; but I wanted to document my findings right away. If this is an internal known bug, please let us know. Worst-case we disable this feature for iOS 9 if it is fixed in iOS 10 and if we cannot work around it in any way.

Configuration:
iPhone 5 Simulator

Attachments:

Comments

Fixed in iOS 10.

Ignore the " Main Thread doesn't answer..." parts - since this crashed on a background thread other tests were already running on the main thread (that made it way more fun to track down...). I removed the unrelated log but forget to clean these parts up - they are not relevant. (It's just a variation of https://gist.github.com/steipete/3933090)


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!