How does the new Data Retention Service work in NFA 9.1.x?

Document created by Christopher_Walsh Employee on Aug 7, 2014Last modified by Christopher_Walsh Employee on Apr 15, 2015
Version 2Show Document
  • View in full screen mode

A new Technical Document has been published on Support.ca.com at the following link.
https://support.ca.com/irj/portal/anonymous/kbtech?searchID=TEC594705&docid=594705&bypass=yes&fromscreen=kbresults

Title: How does the new Data Retention Service work in NFA 9.1.x?


Description:

In Reporter Analyzer 9.0 the Data Retention (registry settings) was set to a fixed amount controlled by the registry.

Raw .NFA (HarvesterArchive) files were saved for 24 hours maximum and then pruned.
1 Minute (ReaperArchive) files were saved for 30 days and then pruned.
15 Minute (Stored in the NQRPTR) database on the DSA was not pruned at all.
Solution:

In Network Flow Analyzer we now have the new Data Retention Service which the settings are controlled

**Disclaimer** Please note, increasing data retention can cause problems with server performance so it is recommended to stay with the default levels if possible. The default settings are what were tested by QA on the server specs documented in the Installation Guide**

The new pruning service checks are run every minute to determine if there is a need to start cleaning out data (to prevent the disk from being fully consumed), but the pruner only runs once a day to dial back down to the "max" threshold (in the 'data_retention.datastores' table). So for some time periods of the day, you can see more than 24 hours' worth of NFA files.
The data retention service provides the same features on the DSA as it does on the harvester - it is responsible for pruning the data to maintain the configured quantity of data.
On the DSA, the data retention service monitors only .rpa15 files.




**Note** 86400=24 hours. The amount of time is stored in seconds.
type_id 1= Harvester / DSA: ReaperArchive15; 15 minute data files

type_id 2= Harvester: ReaperArchive; 1 minute data

type_id 3= Harvester: HarvesterArchive; raw NetFlow data files use in Flow Forensics reports.

An example for changing the Retention service to retain HarvesterArchive raw Netflow data for 48 hours as opposed to 24 hours is as follows:


1)Log into MySql from a Windows Command Prompt using the command below on either the Harvester or DSA depending on your environment:

mysql -P3308 data_retention


2)When you get the welcome to MySql message do a 'Select' query first to make verify the current configuration.

The 'Select' query is as follows:
select * from datastores;


3)Once you verify the current settings you can start the 'Update' query.

For example to set the data retention for NFA files to 48 hours use the update statement below:
update datastores set maxThreshold='172800' where name='NFA';


4)Confirm by using the 'Select' query again from step 2.


5)Recycle the "CA NFA Data Retention" service on the Harvester/DSA for the changes to take effect.
Confirm by using the 'Select' query again from step 2.

Reserved Disk Space Configuration and Options

The service is also responsible for more aggressive pruning if the available free space on the drive falls to less than 5%. (5% is the default settings for reserved space on the harvester and DSA)

**We now recommend changing this setting to 10, so that NFA will maintain 10% free disk space on Harvesters and DSA's.**
In the data_retention.settings table we can adjust the reserved hard drive space.

Right now (by default) the value for freeSpacePercentage = '5'. If changed to '10', the drive will reserve 10% as opposed to 5%
Be careful changing these settings as you could lose 15 or 1 minute data.
An example for changing the Retention service to reserve 10% of the disk as opposed to 5% is as follows:

1)Log into MySql from a Windows Command Prompt using the command below on either the Harvester or DSA depending on your environment:
mysql -P3308 data_retention


2)When you get the welcome to MySql message do a 'Select' query first to make verify the current configuration.
select * from settings;


3)Once you verify the current settings you can start the 'Update' query.

For example to update the database to change the reserved disk space to 10% you can use the update statement below:

update settings set value='10' where name='freeSpacePercentage';


4)Confirm by using the 'Select' query again from step 2.

5)Recycle the "CA NFA Data Retention" service on the Harvester/DSA

Data Retention Priority

The Data Retention Service also has a much more broad way on controlling what data gets cleaned out first. This can be viewed here in the data_retention.types table.




The 'Priority' Column is what we will be looking at here. The higher the number, the lower the priority so essentially in the default instance above, 'NetflowArchive' will start cleaning out first.
This table is not recommended to be altered.

 

This document was generated from the following discussion: How does the new Data Retention Service work in NFA 9.1.x?

Attachments

    Outcomes