We have a very big smartstor DB on our production environment. We have done smartstor cleanups but that has n't helped much. We are working on fine-tuning agents to send only required metrics.
Due to huge smartstor, collectors are performing slowly which in turn causing entire cluster to slow down. Customers need the complete smartstor so there are some suggestions to work on this.
1) Create a query only cluster - Move the entire smartstor DB from live production cluster to query only cluster. The query only cluster can be used for historical query purposes. Drop smartstor DB in production and start from scratch. After a while, merge historical query only cluster and live production cluster. I know we can create query only cluster, move the data, drop the DB in production. But Is it possible to merge the older historical query cluster to live production environment if required in future?
2) As per CA's recommendations, only 10 collectors allowed per cluster. Can we add up more collectors if capacity requirements exceeds? I think this is not officially supported. Am I correct? What will be performance impact to APM if we add more than 10 collectors?
3) There is an option to set the metric clamp for agent, lets say for 1500. Is there way to specify what metrics can be allowed in this 1500? I mean is there a way to specify allow SQL, JSP in those 1500. I don't think we have an option to restrict metrics/allow metrics by individual metrics level. Is that right?