Major regession in PF firewall under Monterey

Number:rdar://FB9658819 Date Originated:9/27/2021
Status:Open Resolved:
Product:pf firewall Product Version:macOS 12
Classification:Incorrect/Unexpected Behavior Reproducible:Always
On Monterey, the pf `route-to` directive fails to work correctly.  Re-routing  a TCP stream results in retransmission errors which prevents a connection being established. This is a major regression from behaviour under Big Sur and Catalina - where `route-to` works as expected. 

This regression breaks a major feature of our app .

Reproduction steps:

We use `route-to` to redirect traffic from a virtual utun interface (which has the default route) to a physical WIFI interface (`en0`). 

A minimal reproduction requires us to setup a `utun` interface as the default route before defining a `route-to` rule using `pfctl`

Set a utun interface as the default route. We do this by creating the following two routes:
    - (Set an ip on the utun device first: `sudo ifconfig utun0`)
    - `sudo route add -inet -interface utun0`
    - `sudo route add -inet -interface utun0`

Turn on packet forwarding (necessary for `route-to` to work)
    - `sudo sysctl -w net.inet.ip.forwarding=1`

Setup the pf `route-to` rule to reroute all packets heading to port 8888 to the physical interface (`en0`) instead of the default `utun0` interface:
    - `echo "nat on en0 inet all -> (en0)\npass out inet proto {udp,tcp} from any to any port 8888 no state tag BYPASS4\npass out route-to (en0 inet all no state tagged BYPASS4" | sudo pfctl -f -`
        - Note that `` should be the address of the router.
        - Also note the `nat` rule is required to ensure the correct source ip.
- Create the TCP stream on port 8888:
    - `echo testing | nc -4 <remote TCP server> 8888`
    - Best to use a raw ip for the remote server, as setting the `utun` interface as the default route may break DNS (as well as other non-rerouted packets). This is only the case for this minimal reproduction where we use a non-functional `utun` device.

Expected behaviour:

We expect that the TCP stream to port `8888` will be redirected through the `en0` interface and  a TCP connection to the remote server will be established. 

The expected behaviour is observed on Big Sur and Catalina.

Observed behaviour:

On Monterey, the TCP stream fails to establish. WireShark indicates there are TCP retransmission errors. This breaks any connection we attempt to re-route. 

Further notes:

The correct/expected behaviour has been seen on some hardware even with Monterey installed. A 2017 Macbook pro behaved as expected, but a 2018 Macbook pro did not. It is not clear the link between certain hardware and the presence of this issue - but in about 90% of cases the incorrect behaviour on Monterey was observed. It's suspected this issue may primarily be on newer hardware.

Serial number of a Macbook pro where issue occurs: C02X42NMJG5K
Serial number of a Macbook pro where the issue does NOT occur: C02VD002HTD9

We always observed CORRECT behaviour on Catalina and Big Sur, irrespective of the hardware.


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!