assetcache fails to serve client requests with increasing incidents of "Failed with status code 503, reason code 36" in the Debug.log
Originator: | j.a.grigutis | ||
Number: | rdar://34956416 | Date Originated: | October 12 2017, 12:16 PM |
Status: | Closed | Resolved: | January 24 2018, 2:16 PM |
Product: | Server | Product Version: | 5.3.1 (16S4128) |
Classification: | Serious Bug | Reproducible: | Sometimes |
Summary: After an undefined period, the Caching Service will stop servicing client requests (see attached screenshot). A typical log entry: 2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Request from xxx.xxx.xxx.xx:xxxxx [cloudd/651.15 CFNetwork/811.5.4 Darwin/16.6.0] to store http://xxx.xx.xx.xxx:xxxxx/apple-mas-00001/A2mCX7gBXxEet-QCEwHn?x-client-request-id=F57395FD-3A55-44A5-B369-521A62DA3840&bin=403000019®ion=amereast&Expires=1507824106&source=uschi-edge.icloud-content.com%3A443&sourceScheme=https denied because the server is too busy; see MaxConcurrentClients setting 2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Failed with status code 503, reason code 36 Increasing the MaxConcurrentClients setting appears to have no effect. These entries seems to increase over time. For example, there were 9,927 instances of "Failed with status code 503, reason code 36" in Debug-2017.10.10-21/43.log when the service still appeared to be servicing clients and 17,602 instances in Debug-2017.10.12-10/11.log when it had stopped completely. Steps to Reproduce: Run the caching service with a large number of clients. Expected Results: If the server is too busy, it should deny service to some clients, but should eventually recover when demand goes down. Actual Results: If the server is too busy, it'll soon stop servicing all clients and never recovers without restarting the service. Version/Build: macOS Sierra 10.12.6 (16G29) Server.app 5.3.1 (16S4128) Configuration: Mac Mini (Late 2014) 3 GHz Intel Core i7 16 GB RAM 1 TB SSD serverperfmode=1 caching:MaxConcurrentClients = 50000 Sonnet xMac mini Server Sonnet Presto 10GbE 2-Port 10 Gigabit Ethernet PCI Express Card Note that there are no alerts issued when the service stops and I am only aware of it because I see no traffic on the Stats page.
Comments
Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!