iptable rule for allowing IPsec traffic

First, we need to open the firewall to the ESP protocol and open the IKE ports (UDP 500 and 4500). Assuming eth1 is the external interface of the firewall:

iptables --append INPUT --protocol ESP --in-interface eth1 --jump ACCEPT
iptables --append INPUT --protocol UDP --source-port 500 --destination-port 500 --in-interface eth1 --jump ACCEPT
iptables --append INPUT --protocol UDP --source-port 4500 --destination-port 4500 --in-interface eth1 --jump ACCEPT

A way to identify traffic originating from a valid IPsec session (presumably to allow it inside your network) is to use the policy module, which matches the policy used by IPsec for handling a packet.
A good catchall rule is:

iptables \
         --append INPUT \
         --in-interface eth1 \
         --match policy \
         --pol ipsec \
         --dir in \
         --jump LOG \
         --log-level debug \
         --log-prefix "IPSec "

This rule will log all packets coming from a IPsec connected peer with the message "IPSec ".