Fortigate Debug Flow, really amazing ninja command.
Well, last week I was in Prague, what is the site where Fortinet support team is located, so my next post shoould be about Fortinet.
I have chosen to talk about one of my favorite “ninja” commands which is debug flow. It is one of the most amazing command that let me troubleshoot lots of issues throughout my career, but just landed from my travel, I faced a new issue where debug flow did not help me enough.
Here you are the details of traffic flow and configuration related which failed at the beginning:
Traffic Flow: from 172.17.5.221 to 172.17.8.254
Route-table entry:
Fortigate # get router info routing-table detail 172.17.8.254
Routing entry for 172.17.8.0/21
Known via "static", distance 10, metric 0, best
* 172.26.254.254, via DWDM
Policy (should match):
Executing a traffic capture with sniffer packet command we only saw first sync packet, but no more so, at the first time, I disabled the Hardware Acceleration but we were still seeing only the first sync packet. That is, there was no incoming traffic from destination.
At that point, we execute a debug flow in order to understand what steps are the traffic flow following through our Fortigate:
Debug flow settings:
#diag debug reset
#diag debug flow filter clear
#diag debug flow filter saddr 172.17.5.221
#diag debug flow filter daddr 172.17.8.254
#diag debug flow show console enable
#diag debug flow trace start 100
#diag debug enable
Debug flow result:
id=20085 trace_id=416 func=init_ip_session_common line=4944 msg="allocate a new session-002dd571"
id=20085 trace_id=416 func=vf_ip_route_input_common line=2586 msg="find a route: flag=84000000 gw-172.17.8.254 via root"
id=20085 trace_id=416 func=fw_local_in_handler line=390 msg="iprope_in_check() check failed on policy 0, drop"
As you can see, Fortigate allocate a new sessión and then find a route to destination “gw-172.17.8.254”, but finally there is an implicit deny (policy id 0).
Briefing, seems to be that debug flow output told us that we have route to destination according to the route table but it does not match with any accept rule (but it should match with the rule above).
Well, that is wrong, finally, further troubleshooting let us realized that there was a disabled vlan interface with IP 172.17.8.254 (the same IP that destination) here you can see:
Because of this, the route found showed in the debug flow was wrong, because it uses the disabled vlan interface direct connected route (in debug flow output you can see “vía root”) rather than route table entry through interface DWDM. As for this, traffic flow output interface was the disabled vlan interface which has no policy accept rule so it matched implicit deny rule.
To solve it, we just changed the IP address for the disabled vlan interface for another IP and it worked fine (taking the properly route of the route table and matching the properly policy accept rule).
id=20085 trace_id=216 func=init_ip_session_common line=4624 msg="allocate a new session-000c5c02"
id=20085 trace_id=216 func=vf_ip4_route_input line=1596 msg="find a route: flags=00000000 gw-172.17.8.254 via DWDM "
id=20085 trace_id=216 func=fw_forward_handler line=686 msg="Allowed by Policy-3456:"
As a conclusion, assuming that debug flow is an amazing “ninja” command, it could be clearer still, at least, regarding route findings between route table and disabled vlan interfaces, but now you know that when you see route finding known "via root" something could be wrong or not regarding interfaces IP addressing.
I really do not know why it happen, I do not know why Fortigate take a rule direct connected as valid when interface is disabled, but as a personal tip, please, check your interface IP addressing, including disabled interfaces (and secondary IP addresses of course) in order to be sure of the route selection in a traffic flow, because maybe debug flow show it not too much clear.
In case someone of Fortipeople read this post and would like to take a look or test in your lab environment, here are the symptoms:
Route to source IP direct connected or properly configured (to avoid antispoofing).
Static route to destination properly configured.
Policy accept rule properly configured.
Interface vlan disabled with the same IP address that the destination (physical interface enabled and up).
Debug flow settings (you can view above).
Easy peasy!!
