assetcache fails to serve client requests with increasing incidents of "Failed with status code 503, reason code 36" in the Debug.log

Originator:j.a.grigutis
Number:rdar://34956416 Date Originated:October 12 2017, 12:16 PM
Status:Closed Resolved:January 24 2018, 2:16 PM
Product:Server Product Version:5.3.1 (16S4128)
Classification:Serious Bug Reproducible:Sometimes
 
Summary:
After an undefined period, the Caching Service will stop servicing client requests (see attached screenshot). A typical log entry:

2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Request from xxx.xxx.xxx.xx:xxxxx [cloudd/651.15 CFNetwork/811.5.4 Darwin/16.6.0] to store http://xxx.xx.xx.xxx:xxxxx/apple-mas-00001/A2mCX7gBXxEet-QCEwHn?x-client-request-id=F57395FD-3A55-44A5-B369-521A62DA3840&bin=403000019®ion=amereast&Expires=1507824106&source=uschi-edge.icloud-content.com%3A443&sourceScheme=https denied because the server is too busy; see MaxConcurrentClients setting
2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Failed with status code 503, reason code 36

Increasing the MaxConcurrentClients setting appears to have no effect.

These entries seems to increase over time. For example, there were 9,927 instances of "Failed with status code 503, reason code 36" in Debug-2017.10.10-21/43.log when the service still appeared to be servicing clients and 17,602 instances in Debug-2017.10.12-10/11.log when it had stopped completely.

Steps to Reproduce:
Run the caching service with a large number of clients.

Expected Results:
If the server is too busy, it should deny service to some clients, but should eventually recover when demand goes down.

Actual Results:
If the server is too busy, it'll soon stop servicing all clients and never recovers without restarting the service.

Version/Build:
macOS Sierra 10.12.6 (16G29)
Server.app 5.3.1 (16S4128)

Configuration:
Mac Mini (Late 2014)
3 GHz Intel Core i7
16 GB RAM
1 TB SSD
serverperfmode=1
caching:MaxConcurrentClients = 50000
Sonnet xMac mini Server
Sonnet Presto 10GbE 2-Port 10 Gigabit Ethernet PCI Express Card

Note that there are no alerts issued when the service stops and I am only aware of it because I see no traffic on the Stats page.

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!