Updating application renders ODR unusable

Originator:borissotillos
Number:rdar://29218750 Date Originated:11/11/2016
Status:open Resolved:no
Product:tvos Product Version:10.0
Classification:bug Reproducible:100%
 
Boris Sotillos
11-Nov-2016 04:05 PM

Summary:
Application downloads ODR just fine if fresh installed. But ODR does not download if application is updated from its previous version.
The issue only happens when the update is done via TestFligh or the Store. With Xcode, the application downloads its resources just fine.
As long as you do not update the application, ODR works as expected.

Steps to Reproduce:
0) Uninstall our application from the device.
1) Install previous version from store.
2) Run it.
   ODR is downloaded, while user gets to see a loading screen with a progress bar.
3) Application works fine using the ODR resources.

4) Install current version from store (or TestFlight).
5) Run it.
   In the loading screen there is no progress bar -there would be one if resources were being downloaded-.
   Resources are not downloaded. Instead, they are instantly reported as available.
6) Application tries to access the resources and fails because they are not there.
   To the user, it looks like the application got stucked in the loading screen.

7) If you let the application run. Resources are never downloaded.

8) If you restart the application { GOTO step (5)}, will follow the same steps.

Expected Results:
We would expect that, at step (5), the new ODR was downloaded.
It that was the case, we would get to see a loading screen w/ a progress bar, while downloading the new content. And on step (5), application would be able to access the content.


Actual Results:
At Step (5), the conditionallyBeginAccessingResources method invokes its completion handler, with resourcesAvailable = YES.
This is despite the fact that:
- Assets from current version have different names that assets from previous version.
- Tags from current version are different than tags from previous version.
(Differences being forced in order to investigate the issue).

From the device console, we see one significative log trace, appearing once per asset pack, with the following message:
Apple-TV XXXXXXX(CoreUI)[XXX] <Notice>: CoreUI: can't locate asset pack '<private>' for bundle '<private>' contents of asset pack will be ignored

We also know that resources are acknowledged as available from console logs added to investigate this issue.

Also, if fresh installed, the TestFlight version works fine.


Version:
tvOS 10.0 (14T330) with wifi connection. model A1625 (32 GB)

Notes:
This issue does not happen on tvOS 9.2 ( anything < 10.0, we presume): ODR downloads fine after update with the repro steps above in a tvOS 9.2.

This bug reminds us to the bug https://openradar.appspot.com/radar?id=4957459995164672 , "Updating an application that uses ODR from Xcode or 'ad hoc' webserver renders ODR cached and non-updatable", that we opened 10 months ago and has not received an answer.

Configuration:
tvOS 10.0 (14T330). Xcode 8.0 -- cannot reproduce the issue on tvOS 9.2.

=====================================================================================================================================
Boris Sotillos 29-Nov-2016 10:00 AM

I am attaching the files: - [filename redacted] : the device console output from the Xcode Devices window. - [filename redacted] : the sys diagnose report.

I couldn't follow the link [redacted link] as instructions were meant for iOS devices. Instead, I followed [redacted link].

Regarding profile installations, I installed the advised iOS profiles from [redacted link] , "App Store/iTunes Store for iOS" and "sysdiagnose for iOS". The only way I could install them was with the "Apple Configurator 2" tool.

The timings were: (match this against the repro steps) 17.20 install previous version (4.1.0) from store. 17.20 run it, allow time for initial ODR download. 17.22 install current TestFlight version (version displayed as "4.2.0 (4.2.30)"). 17.23 run it, allow time to download while we see the intro movie. 17.25 application crashes once intro movie finishes, as it tries to access the files that are not there.

Sorry that I couldn't get more accurate times.

note that: Our current TestFlight version has been modified in order to investigate this issue. In the current TF build, files are assumed to be downloaded when the ODR system tells us so, therefore it crashes after the intro movie. In the the original bug description, if files were not present, the application would retry. This is of litle importance, it's just to clarify the crash.

=====================================================================================================================================

Apple Developer Relations 26-Nov-2016 02:45 PM

Engineering has requested the following information in order to further investigate this issue:

A sysdiagnose taken after reproducing the issue is definitely needed here. For store-specific issues, you can install the App Store/iTunes Store debug profile, reproduce the issue, and retrieve the additional log info with a sysdiagnose.

sysdiagnose Instructions:
[redacted link]

iOS App Store Logging Instructions: [redacted link]

iOS App Store Logging Profile: [redacted link]

For a complete list of logging instructions visit: [redacted link]

Comments

ACTUAL FIX

Well after 7 days of pain I have found an actual solution to this issue. In my situation the issue was resolved by adding a single "Initial Install" tagged ODR resource to the app.

  1. Add a small file you won't use to the app
  2. Add an ODR tag to that file
  3. Open the project settings, enter the Resource Tags section, and select Prefetch
  4. Add the tag that you just created to the "Initial Install Tags" section.

In my case I didn't have any pre-install tagged resources, so I added a dummy image with an ODR tag and put it in the pre-install bucket. After that, problem solved. Can't say if that will do it for everyone, but I was having this problem 100% of the time, and for me this was a 100% always works solution. tvOS 10, tvOS 11.2.

I tried this and it didn't work for me. But I managed to get it to work by first calling "request.conditionallyBeginAccessingResources", then from within the completion handler "beginAccessingResources" (if available = false). This is not intuitive and according to the docs not even necessary but seems to work fine so far.

We went one step further. Forget about the "application". We did an updated version (changing version number, asset catalog files and their tags) of our "fake application" and we uploaded it to TestFlight. Let's call the two versions fake1 and fake2.

In short, We did this test:

1 Fresh install fake1.

2 Run fake1. ODR downloaded fine.

3 Upload fake2 to TestFlight(*).

4 Update fake1 with fake2.

5 Run fake2.

... and we got the same behavior. Now our application should NOT be in the picture any more. Except for the BundleId.

Questions: - Is it possible that in the Apple Store's guts is there some bad, broken "metadata" regarding our application's Bundle (possibly specific to tvOS10)? - Maybe something was broken in our application submission process? With a failed submission? - Maybe Apple had a bug when adding tvOS10 support? Maybe bug is fixed, but it left some corrupted (cached?) data.

(*) BTW TestFlight allowing only one version per application, didn't facilitate this process.

By borissotillos at Dec. 15, 2016, 10:09 a.m. (reply...)

Hi Borissotillos,

We have recently run into this issue as well but on iOS instead of tvOS. Same thing, when update our app via TestFlight the ODRs are reported as available but when we try to access them they fail because they are not there. We are crossing our fingers you were able to find a solution with Apple? I'd really appreciate any info you could provide! colin@sagomini.com.

iOS (10.3) xcode: 8.0 Unity3d: 5.4.5f1

Our build pipeline is still using xcode 8.0 and we are working on doing a test build with 8.3.2 to see if that helps.

Update: So we have created a default tvOS "game application" using the Xcode wizard -we were using a 3rd party tool to generate projects-. This app has an ODR asset catalog (different files, tagged with different names from our application's). It starts, downloads the ODR, reads and dumps its files in the console -they are small text files- as proof that the download really worked. From now on, I'm calling this the "fake app", in contrast to the "application" that is suffering the issue.

Now, "application" and "fake app" are different, disjoint, project-wise, code-wise and data-wise. They only have in common the Bundle Identifier. Since we need to be able to update the application.

Updating the application with the fake app, the issue reproduced exactly. (initial application was freshly installed on a device that had been factory-reseted).

Fresh instalations of the fake application did download the ODR.

By borissotillos at Dec. 15, 2016, 10:09 a.m. (reply...)

Just letting you know that there has been some development on our end.

We have downloaded our source from Dec 2015, rebuilt it with the same Unity version and re-uploaded it to TestFlight with Xcode v7.2 and this new build, with the exact same On-Demand resources, overwrote our AppStore build and loaded properly. Now, we still do not know the root of the problem but we at least know that, with the same configuration as before, a TestFlight build can successfully overwrite an AppStore build without having the ODR loading problem.

By sim.lavigne at Nov. 30, 2016, 10:15 p.m. (reply...)

I have been investigating this exact issue for days now and I'm glad to see it has been brought to Apple's attention.

I can't add much to this thread but we are using tvOS 10.0.1 (14U71) and can only reproduce this issue when overwriting an AppStore build with TestFlight. I also found it interesting that you also experience this issue when performing these different steps: 1. Download AppStore version (Do not launch) 2. Immediately overwrite with TestFlight version 3. Download ODR 4. ODR not available

I have also tried using completely different tab names from out AppStore and those are still unable to load when overwriting an AppStore build. They can be downloaded though, which is particular.

Note: I am using Unity and other users are experiencing this issue on the Unity forum. I was suspecting it could be a bug in Unity but seeing this bug report, I'm starting to suspect it being a bug in the ODR feature.

By sim.lavigne at Nov. 29, 2016, 8:33 p.m. (reply...)

Hi there.

For us, this is an issue even on a "normal" update. We are already experiencing this in production code, when users try to update the application from the store.

OTOH, we have another app that uses ODR and works well. We tried putting that app's ODR-management code (pretty different from ours, specially regarding threading) in our failing application and we got the same exact behaviour.

We think there may be an issue in the build settings, but we haven't been able to find nothing worng. It's that, or an Apple bug.

By borissotillos at Nov. 30, 2016, 8:40 a.m. (reply...)

Hello Boris,

Thanks for the update. Would there be any chance that your 'broken' app has been released close to the release of the tvOS SDK? Our app was one of the first apps on the store (Dec 2015) so I'm suspecting that maybe older built ODRs might be incompatible with later SDKs. Is there a chance your 'working' app is newer and doesn't have this problem because of this?

By sim.lavigne at Nov. 30, 2016, 2:16 p.m. (reply...)

Yes. This is something that we are contemplating. How to fix users that already have the downloaded "broken" app, we don't know.

By borissotillos at Dec. 1, 2016, 8:04 a.m. (reply...)

Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!