VideoToolbox throws kVTVideoDecoderBadDataErr when encountering an SPS change

Originator:Nitz.Carola
Number:rdar://45418397 Date Originated:10/20/2018
Status:closed Resolved:
Product:macOS + SDK Product Version:
Classification:bug Reproducible:always
 
Area:
VideoToolbox

Summary:
On some platforms (observed on macOS Sierra with 12" macbook), the VT decoder will start returning errors when encountering an SPS change in the h264 bitstream. To work around this bug we have to catch the kVTVideoDecoderBadDataErr response from the decoder and the decompression session is recreated. The "bad data" is then fed into the new decompression session so that it can be decoded correctly. 
More details can be found here:

https://patchwork.ffmpeg.org/patch/2581/

Steps to Reproduce:
Try to play http://tmm1.s3.amazonaws.com/videotoolbox/spschange.ts (attached)
with VLC or ffmpeg

Expected Results:
no kVTVideoDecoderBadDataErr

Actual Results:

Version/Build:
Xcode 10O23u, iOS 12 macOS 10.14

reply by Apple:
This issue behaves as intended based on the following:  When the SPS changes, and the ‘avcC’ is updated, are you calling VTDecompressionSessionCanAcceptFormatDescription to see if the decoder is ok with the updated CMVideoFormatDescription?   If VTDecompressionSessionCanAcceptFormatDescription returns “false”, you should preempt this failure and go ahead and rebuild the VTDecompressionSession before trying to decode with the new CMVideoFormatDescription.  If VTDecompressionSessionCanAcceptFormatDescription returns “true”, but the VTDecompressionSession with the new CMVideoFormatDescription returns an error complaining about the SPS change, we would need to investigate.  Please update your bug report to let us know if this is still an issue for you.

Comments

This issue behaves as intended based on the following: When the SPS changes, and the ‘avcC’ is updated, are you calling VTDecompressionSessionCanAcceptFormatDescription to see if the decoder is ok with the updated CMVideoFormatDescription? If VTDecompressionSessionCanAcceptFormatDescription returns “false”, you should preempt this failure and go ahead and rebuild the VTDecompressionSession before trying to decode with the new CMVideoFormatDescription. If VTDecompressionSessionCanAcceptFormatDescription returns “true”, but the VTDecompressionSession with the new CMVideoFormatDescription returns an error complaining about the SPS change, we would need to investigate. Please update your bug report to let us know if this is still an issue for you.

By Nitz.Carola at Oct. 24, 2018, 9:58 p.m. (reply...)

Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!