I’ve been running pfSense in Dual WAN mode for more than a decade. Unfortunately, some sites lately are quite sensitive per user session originating from multiple public IP addresses. The best description of the problem is from the official pfSense documentation:
Some websites store session information including the client IP address, and if a subsequent connection to that site is routed out a different WAN interface using a different public IP address, the website will not function properly. This is becoming more common with banks and other security-minded sites. The suggested means of working around this is to create a failover group and direct traffic destined to these sites to the failover group rather than a load balancing group. Alternately, perform failover for all HTTPS traffic.
The sticky connections feature of pf is intended to resolve this problem, but it has historically been problematic. It is safe to use, and should alleviate this, but there is also a downside to using the sticky option. When using sticky connections, an association is held between the client IP address and a given gateway, it is not based off of the destination. When the sticky connections option is enabled, any given client would not load balance its connections between multiple WANs, but it would be associated with whichever gateway it happened to use for its first connection. Once all of the client states have expired, the client may exit a different WAN for its next connection, resulting in a new gateway pairing.
After some testing and consideration let’s leave the sticky connections unchecked. As mentioned above they are problematic.
Other description of the problem here:
Some websites do not work properly if requests from the LAN are initiated from multiple public IP addresses. Hence load balancing is incompatible with these sites. Common examples are sites that maintain login sessions, most frequently online banking. This is most commonly observed with HTTPS sites so usually HTTPS should not be load balanced. Occasionally it is a problem with HTTP sites that maintain session, but this is rare.
For sites that do not function with load balancing, add firewall rules to not load balance traffic to these destinations or protocols.
Web site incompatibility with changing IP addresses
To alleviate this issue, you can do the following:
Here are my two Gateways
Make two Gateway Groups
One for Load Balancing
Set for both Gateways Tier 1
One for Failover
Set Tire1 for the one and Tier 2 for the second
Go to the LAN Rules
Set the default LAN rule to use the Load Balancing Gateway Group.
Add new rule that will be valid only for HTTPS connection and set the Gateway to the Fail-over Gateway Group.
This way all HTTPS connections will pass through the First WAN until it goes down and failover to the Second. The alternative is to make separate rule for each and every HTTPS site with issues. The rule will be very similar to the one for HTTPS. The difference will be that Destination address will be single Public IP. Doing so will load balance all other HTTPS connection that don’t have this problem.