Open Radar

 
Area:
ReplayKit

Summary:
If you do end up shipping iOS 11 where the com.apple.broadcast-services-setupui extension is _not_ used for global screen recordings with 3rd-party upload extensions, then this bug report merely suggests that there doesn't seem to be a way for a broadcast upload extension to determine video orientation on its own. That orientation information (whether it's video orientation or device orientation) should be made available somewhere.

Details:
I am able to receive sample buffers in my broadcast upload extension (NSExtensionPointIdentifier: com.apple.broadcast-services-upload). The video sample buffers appear to be aligned to the camera sensor. For three of the four device orientations, I need to rotate the display of the video frame to show it correctly.

I know how to apply this kind of rotation to a video file or a video broadcast. My problem is determining what the rotation should be. The sample buffer itself doesn't appear to provide details about orientation or a preferred transform. The complicating factor is that queries to both UIDevice and UIScreen _from within a broadcast upload extension_ appear to lack correct device orientation as well. Both of them either report device orientation to be unknown or they incorrectly report the device to be unrotated.

I am guessing that the the broadcast setup UI (NSExtensionPointIdentifier: com.apple.broadcast-services-setupui) will solve this problem. It's a view controller that can resolve orientation. You already provide a way for a setupui extension to send this kind of data as setupInfo to an upload extension.

However, as of beta 5, the setupui extension is not used when initiating a global screen recording from the Screen Recording button. (I'm hoping that will be fixed before iOS 11 is released.)

Steps to Reproduce:
- Run the attached VideoOrientationTest project to install the VideoOrientationTest app and associated "Broadcast" extension on an iOS 11 device.
- Leave your device attached to Xcode on your Mac.
- Install the Screen Recording button into your Control Center if it's not already installed.
- Bring up the Control Center.
- 3D touch the Screen Recording button.
- Choose the "Broadcast" extension.
- Tap "Start Broadcast". Wait for the count down and let the recording continue running.

- In Xcode, set a breakpoint the processSampleBuffer() call in SampleHandler.swift (line 35).
- In Xcode select Debug -> Attach to Process.
- Wait for the process list to resolve.
- Select Broadcast at the top of the list.
- Wait for the break point to trap.

Expected Results:
At the breakpoint, in that call stack, in that process, there should be a way to query or determine which way the video sampleBuffer is oriented.

Lot's of possible solutions. (You guys would know better than me.) Some possibilities:

- Run the setupui extension before running a 3rd-party broadcast upload extension for a global Screen Recording. Give the setupui extension the opportunity to send device orientation to the upload extension.
- Or, add orientation to the setupInfo of the broadcastStarted() call to the upload extension.
- Or, embed the orientation somewhere into each video sample buffer.
- Or, get UIDevice or UIScreen capable of communicating correct device orientation into a broadcast upload extension.

Observed Results:
At the breakpoint, in that call stack, in that process, there doesn't appear to be a way to query or determine which way the video sampleBuffer is oriented.

Version:
- iOS 11 beta 5 (15A5341f)
- Xcode 9 beta 5 (9M202q)

Notes:
The SampleHandler can determine the size and aspect ratio of a video sample buffer but that doesn't tell us which way the content inside the video frame is oriented.

Without a rotation, three of the four device orientations will render content incorrectly. It's not clear how code inside a SampleHandler could query or determine what that rotation should be.

Configuration:
This has been the case for all the iOS 11 betas so far (up to and including beta 5).