DX Unified Infrastructure Management

  • 1.  Harvester Dropping Flows

    Posted Feb 11, 2015 02:39 PM

    Hello Community,

     

    I have set up NFA 9.2.1 in a VM lab environment.

    I have a standalone installation (Console + Harvester) on Windows 2008, and also a separate Linux Harvester.

     

    I've configured a couple of real routers to send NetFlow v5 data to the standalone system, and I've also experimented with various software-based NetFlow simulation tools to generate v5 or v9 flows to either the standalone NFA instance, or the Linux Harvester.

    Flows are sent to UDP port 9995 and Wireshark can see and decode the NetFlow packets on the standalone system.

    All NFA services are running.

     

    My issue is that I only ever as far as getting '.nfa' files generated in the HarvesterArchive folder, and '.flt' files in the NFMInput folder, for both Harvesters.

    None of the other folders under the datafiles folder have any files created, and no interfaces appear in the NFA Console.

     

    Looking at the log file 'harvester-wrapper.log' reveals entries like this:

    INFO   | jvm 1| 2015/02/11 15:22:28 | 3:22:28 PM - [INFO] - Dropped Flows      : 18240
    INFO   | jvm 1| 2015/02/11 15:22:28 | 3:22:28 PM - [INFO] -     Map Failure    : 18240
    INFO   | jvm 1| 2015/02/11 15:22:28 | 3:22:28 PM - [INFO] -     Bin Failure    : 0

     

    I've run the NASTv13.exe tool and the report that it generates shows flow statistics and no 'red line' errors.

     

    Has anyone run across this situation?

     

    Thanks!



  • 2.  Re: Harvester Dropping Flows
    Best Answer

    Broadcom Employee
    Posted Feb 11, 2015 03:02 PM

    It could be due to a defect in 9.2.1 which is resolved by the patch in this tech tip doc Tech Tip: Harvester not showing data after clean NFA 9.2.1 install or upgrade to NFA 9.2.1

     

    If that doesn't fix it, are All CA NFA* services running on the Harvester?

     

    Are these routers able to be SNMP Polled by the Harvester?

     

    If you pick one router IP from the NAST report and use it in the query below, do you see negative numbers in the pstID column?

         mysql -P3308 -D harvester -t -e "select from interfaces where router=inet_aton('10.1.1.1');

     

    Also if you run the command below, what does the "State" column say?

         mysql -P3308 -D poller -t -e "select * from routers where address='192.168.117.49';"

     

     

    If the pstID's are -1 we will drop flows, this will occur if the device is not able to be SNMP Polled, and the device gets put into a RebootRefresh state in the poller.routers table.



  • 3.  Re: Harvester Dropping Flows

    Posted Feb 11, 2015 05:24 PM

    Christopher,

     

    I applied the patch, but this did not have an effect.

     

    I ran the first mysql command, but changed 'delete from' to 'select * from' instead, as I believe that was your intent.

     

    For the fake routers generated by the NetFlow generation software tool, the PstID column was filled with '-1' values.

    For the real routers, there were 3 entries per router (I'm assuming 1 per enabled interface), with values of 0, -1 and -1.

     

    When I ran the second mysql command, the result was always a state of 'RebootRefresh'.

     

    Then, for the router connected directly to my lab network, I changed the NetFlow source interface to be the Ethernet port facing my network, whereas previously I had configured the source to a serial interface.

     

    Now, for this router, when I run the first mysql command using the IP address of the local router port, I receive PstID values of 0, 1 and 2.

    And the second mysql command gives a state of 'InitialPoll'.

     

    However, harvester-wrapper.log still shows 'Dropped Flows' values with an equal number of 'Map Failures'.

     

    I have CA PC configured with an SNMP profile that matches the router's SNMP settings.

     

    Here is a more complete log snippet:

     

    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ==================================
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -    Collection Stats   
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flow Collector Packet Count  : 788
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flow Archiver  Drop Count: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flow Balancer  Drop Count: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flow Processor Drop Count: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -   [0]: packetCount=788 dropCount=0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ==================================
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     FlowArchiver Stats  
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Archive Period : 1423692300
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Router Count   : 2
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Packet Count   : 6
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Failed Packet Count: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flushing 30 routers
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Flushing sequence numbers for 30 routers
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ==================================
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -   HarvesterFlowProcessor - 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Total Received Packets : 788
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Max Buffer Used Size   : 16096
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Processed Packets  : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Processed Flows    : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Processed Flows/Minute : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Dropped Packets    : 56
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     Parsing Failure: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     No Flow Found  : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     Pre-Process    : 56
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -    Header Errors  : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -    Invalid Version: 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -    Duplicate Packets  : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -    Router Reboot  : 56
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Dropped Flows      : 5875
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     Map Failure    : 5875
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] -     Bin Failure    : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Connection Pool Stats
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Max Connections : 200
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Active Connections  : 0
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - Idle Connections: 1
    INFO   | jvm 1| 2015/02/11 18:05:08 | 6:05:08 PM - [INFO] - ----------------------------------


  • 4.  Re: Harvester Dropping Flows

    Broadcom Employee
    Posted Feb 11, 2015 05:48 PM

    Sorry about the typo with the sql query.

     

    InitialPoll is good, so long as it doesn't change back to RebootRefresh, it should start showing data.

    You may see still some dropped flows messages, some may be normal.  These are cumulative and should reset after recycling the Harvester service.

     

    What I do in my lab for "Fake" routers which I cannot SNMP poll, is enable the "IgnoreReboots" setting in the parameter_descriptions table of the Harvester database.

     

    This will keep routers from going into the RebootRefresh state, but also has the side effect that it will not automatically poll devices when NFA detects a reboot on the device, which is detected by a change in sysuptime or a reset of flow sequence.  So the best option is to get SNMP working for these devices.

     

    To set the ignore reboots settings.

     

    mysql -P3308 harvester

     

    update parameter_descriptions set DefaultValue='true' where parameter='ignoreReboots';



    For existing devices you may need to reset the status with the queries below in the poller database.

     

    mysql poller

    update routers set State='InitialPoll', stateretry=0, reboottime=0 where state ='RebootRefresh';



  • 5.  Re: Harvester Dropping Flows

    Posted Feb 12, 2015 08:16 AM

    You should be sourcing your netflow from a loopback interface instead of a

    physical interface on the device.



  • 6.  Re: Harvester Dropping Flows

    Posted Feb 12, 2015 08:32 AM

    After a while my 'real' router's State change to Mapped in the routers table.

    Still could not see interfaces in the Console.

    I then enabled the ignore reboots setting in the database, and changed all routers that had an RebootRefresh state to InitialPoll, and restarted the Harvester service, but still nothing.

    However, now the interfaces table shows non-negative PstId values for the simulated routers (as well as the real routers).

    I also forced the Enabled value to 'Y' in the interfaces table for 1 real and 1 simulated router. This did not force the interfaces to appear in the Console as I'd hoped it would.

    I'm going to see if I can send flows to a freeware tool just to make sure that I am not doing something incorrectly.

     

    (Yup, I did initially have the source set to my be loopback interface as per best practices, but as it currently has a non-routable IP address, I experimented by changing it to be a physical interface - serial and then Ethernet.)



  • 7.  Re: Harvester Dropping Flows

    Posted Feb 13, 2015 08:26 AM

    The freeware NetFlow collection tool I set was able to capture and display flow data.

    At some point I may try installing a slightly older version of NFA (probably 9.1.3) to see whether that makes a difference.



  • 8.  Re: Harvester Dropping Flows

    Broadcom Employee
    Posted Feb 13, 2015 11:33 AM

    I'd recommend using the later versions of NFA, there were lots of fixes including in 9.2.1.



  • 9.  Re: Harvester Dropping Flows

    Posted Feb 18, 2015 01:16 PM

    Chris, I will mark your original reply as the correct answer.

    You had asked if all "CA NFA" services were running, and they were.

    However, a couple of "NetQoS Reporter/Analyzer" services were NOT running, namely the "NetQoS Reporter/Analyzer Query Service" and the "NetQoS Reporter/Analyzer Report Service".

    I clued into this when I looked at the System Status page, under the Harvester, which alerted me that:

    "The Query Service Service's current state is Stopped."

    So after manually starting these services I can see my interfaces appearing in the Console.