This blog post discusses the procedure we followed when troubleshooting a connectivity issue with a firewall (it was not responding to pings even though it should have been). Specifically, we used tcpdump (a packet sniffer) and pflog (a firewall logger specific to the BSD-based pf firewall).
This blog post is of interest to users who have an IPv6 firewall configured with pf, though the troubleshooting techniques could be generalized to, say, an iptables-based linux firewall.
Problem Description
My home firewall did not respond to IPv6 pings to its outside interface (interestingly, it responded to IPv6 pings to its inside interface). In this example, we’ve sent 662 ICMP packets from a third-party machine but received no replies:
$ ping6 2001:558:6045:109:15c7:4a76:5824:6c51
PING6(56=40+8+8 bytes) 2a01:4f8:d12:148e::2 --> 2001:558:6045:109:15c7:4a76:5824:6c51
^C
--- 2001:558:6045:109:15c7:4a76:5824:6c51 ping6 statistics ---
662 packets transmitted, 0 packets received, 100.0% packet loss
First, the basics: Let’s make sure we’re pinging the correct IP address. Let’s run ifconfig against the external interface (em3) of the firewall to confirm:
$ ifconfig em3
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:00:24:ce:7b:fb
inet6 fe80::200:24ff:fece:7bfb%em3 prefixlen 64 scopeid 0xc
inet 24.23.190.188 netmask 0xfffffe00 broadcast 255.255.255.255
inet6 2001:558:6045:109:15c7:4a76:5824:6c51 prefixlen 128
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT )
status: active
Yes, the IPv6 address (2001:558:6045:109:15c7:4a76:5824:6c51) is the correct one (ignore the fe80:… address; it is a link-local address and not routable).
Is the problem with the echo-requests or with the echo-replies? (i.e. is the firewall not receiving the echo-requests, or is the third-party machine not receiving the echo-replies?). We use a non-intuitive invocation of tcpdump on the firewall to trace IPv6 ICMP echo packets while our third-party machine continues to attempt to ping us:
$ sudo tcpdump -ni em3 'icmp6 and (ip6[40] == 128)'
The lack of output indicates that the ICMP echo-request packets are not arriving. Or, if they are arriving at the machine, they are being blocked by the firewall rules. We now turn our attention to the firewall rules. Let us make sure that we are logging blocked packets (notice the “log” keyword in /etc/pf.conf):
# default deny
block in log
We now run tcpdump, but instead of directing tcpdump to listen to a network interface, we tell it to listen to a special interface, pflog0, which is the device to which pf directs its logged (blocked) packets:
sudo tcpdump -ni pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
15:41:16.226899 IP6 fe80::201:5cff:fe63:ec46 > ff02::1: ICMP6, router advertisement, length 80
15:41:16.603519 IP6 fe80::201:5cff:fe63:ec46 > ff02::1:ff24:6c51: ICMP6, neighbor solicitation, who has 2001:558:6045:109:15c7:4a76:5824:6c51, length 32
15:41:19.457168 IP6 fe80::201:5cff:fe63:ec46 > ff02::1: ICMP6, router advertisement, length 80
15:41:20.603631 IP6 fe80::201:5cff:fe63:ec46 > ff02::1:ff24:6c51: ICMP6, neighbor solicitation, who has 2001:558:6045:109:15c7:4a76:5824:6c51, length 32
We don’t see any ping packets (IPv6 ICMP echo request (type 128)) being filtered.
But we notice a much graver problem: the firewall is blocking IPv6 ICMP router advertisements (type 134) and neighbor solicitation (type 135). The firewall should not blocking those packets—those ICMP types are explicitly allowed in the rulesets (from /etc/pf.conf):
pass in quick on em3 inet6 proto ipv6-icmp from any to
( em3 ) icmp6-type { 128, 133, 134, 135, 136, 137 } keep state
So why is the firewall blocking those packets? The problem is our destination address: we’re only accepting ICMP packets that are directed specifically to our IPv6 addresses (e.g. 2001:558:6045:109:15c7:4a76:5824:6c51 and the local-only fe80::200:24ff:fece:7bfb), and we’re [mistakenly] blocking the ICMP packets that are directed to the multicast address (ff02::1).
These router advertisement and neighbor solicitation packets are crucial; they are components of the Neighbor Discovery Protocol (NDP) (RFC 2461) and are the IPv6 replacement of IPv4’s ARP protocol; they allow us to communicate with the machines on our local network. By blocking those packets, we have inadvertently made it so that no one can communicate with our firewall’s external IP address. That’s why the pings aren’t succeeding.
The fix is simple: we modify the firewall rule to also allow ICMP packets with the multicast destination; we change to ( em3 ) to to { ( em3 ), ff02::/16 }; ff02::/16 is the IPv6 Multicast subnet for “All nodes on the local network segment”.
pass in quick on em3 inet6 proto ipv6-icmp from any to
{ ( em3 ), ff02::/16 } icmp6-type { 128, 133, 134, 135, 136, 137 } keep state
We use pfctl to put the new rules into effect:
sudo pfctl -f /etc/pf.conf
And now we are able to ping the firewall’s external IPv6 address!
Bibliography
- Mary Gray Baker of HP has the proper incantation for tcpdump
- Earl Carter of Cisco has a great post about IPv6, ICMP, Security, and Neighbor Discovery Protocol
- An earlier blog post describes the creation of IPv6 firewall rules
