CA ecoMeter Administrator is the administration tool for polling data center devices. It is deployed as a UIM probe "ecometer_admin" on the same robot as the UMP portal.
The ecoMeter Administrator is going to scan all domain hubs and robots to discover deployed ecoMeter probes and present them in the Admin GUI:
2018-01-23 15:48:54,374 [pool-91-thread-8] INFO Get robots for hub...
2018-01-23 15:49:11,456 [pool-92-thread-7] INFO Get probes for robot...
In a large enviroment, this scan can create a huge amount of open descriptors for the wasp process. This will impact the UMP if we hit the max open files configured per process (4096 by default on a RedHat 7x box) and will produce that UMP is not available (404 error).
Some of the relevant log messages in the wasp.log are:
Jan 23 15:50:09:353 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/relationshipviewer]] java.io.FileNotFoundException: /UIM/Nimsoft/probes/service/wasp/webapps/relationshipviewer/WEB-INF/web.xml (Too many open files)
To check the amount of open files use: lsof -p <pid> (where <pid> is the proccess id of the wasp process).
To check what is the max open files assigned to a given process run: cat /proc/<pid>/limits
To fix this situation, increase the max number of simultaneous open files per process in the file (65536 should be enough):