NestorFalcon

"Too many open files" in wasp when deploying ecometer_admin probe

Discussion created by NestorFalcon Employee on Jan 25, 2018

CA ecoMeter Administrator is the administration tool for polling data center devices. It is deployed as a UIM probe "ecometer_admin" on the same robot as the UMP portal.

The ecoMeter Administrator is going to scan all domain hubs and robots to discover deployed ecoMeter probes and present them in the Admin GUI:

2018-01-23 15:48:54,374 [pool-91-thread-8] INFO  Get robots for hub...

...

2018-01-23 15:49:11,456 [pool-92-thread-7] INFO  Get probes for robot...

 

In a large enviroment, this scan can create a huge amount of open descriptors for the wasp process. This will impact the UMP if we hit the max open files configured per process (4096 by default on a RedHat 7x box) and will produce that UMP is not available (404 error).

Some of the relevant log messages in the wasp.log are:

Jan 23 15:50:09:353 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/relationshipviewer]] java.io.FileNotFoundException: /UIM/Nimsoft/probes/service/wasp/webapps/relationshipviewer/WEB-INF/web.xml (Too many open files)

 

To check the amount of open files use: lsof -p <pid> (where <pid> is the proccess id of the wasp process).

To check what is the max open files assigned to a given process run: cat /proc/<pid>/limits

 

To fix this situation, increase the max number of simultaneous open files per process in the file (65536 should be enough):

/etc/security/limits.conf

 

HTH,

Nestor

Outcomes