Hopefully, someone else has this problem.
We are running CA APM v10.0 and there are a number of agents that do not appear within the metric browser (webview or introscope workstation) but if we build a metric grouping with an agent expression for the missing agent, the metric grouping retrieves live/historic data with count = 1, but the agent itself does not appear in the metric browser.
Thought that it might be an overloaded Collector issue, but found the agent within the custom metric host - collector - agents and the collector the missing agent is reporting too has harvest times well under 1 sec, smartstor well under 1 sec, less than 12 % cpu usage, plenty of storage space and is responding rather well to queries for historic (7 day) data for agents reporting to it.
The agent name/metric labels are within the metadata storage and the metadata file is about 78 MB. Each of the collectors have free memory (RAM) already assigned to it. Typically running 80 to 85% of 6 GB. The GC report is showing less than 30% after GC and tops out at around 55%, so there should be plenty of room for the Java process, which is the enterprise manager.
The missing agent is listed within the workstation - investigator - manager - mount agent listing and the last time the agent reported varies from within the last hour to a few days ago. If we restart the agent, epagent, then the agent reappears within the metric browser.
If we mount the agent, the agent is grayed out, as if the agent isn't reporting but the metrics are all displaying current and live data.
Monitoring the mount agents, by opening the agent chooser, sort by last timestamp, screen capture it, then close it. Reopen the agent chooser, sort by last timestamp, the listing changes. Focusing on a specific agent, from the first listing, and then trying to find the same agent some times it is listed, some times it isn't present within the agent chooser.
Has anyone ever seen such?
Any suggestions on what might be wrong?