Clarity

Hi All,

We had that issue in the past and we never fixed it.

From time to time, almost daily, we have to restart our service background because of "java.lang.outofmemory" error.

After lots of discussion with CA support I got the feedback that the issue is my environment.

Today we have Weblogic servers in a different network of the background services server. According to CA they have to be in the same network in order for multicast to work. It seems that messages that are sent from the background services through multicast are not reaching the weblogic servers (app servers) and that eventually causes outofmemory.

I just would like to hear from you if you have had that issue before and if that is something you've considered.

About the environment configuration, I have inherited that application so I don't know why that was designed like that and it will be hard to change but can be done. I will work on that in a non-prod environment first to confirm this assessment and let you know if this fixed the issue.

Thanks for any insights.

Regards,

Fabrício De Marchi
Dell

Hi Fabrício.
Yes, the services absolutely have to talk to each other via multicast. An OoM error would be the least of my worries, I'd fear failed processes, jobs walking over each other, etc. Congratulations on inheriting this, but surprise! - it's now your problem to fix. Re-architecting our environment was the first thing I did when I walked in the door at my current employer...

Go make it right. Don't be afraid to question if the existing architecture is right for your organization and simplify if necessary.

Thanks for your response Rob.

It is reassuring to know someone had the same issue and it did get fixed by re-architecting it.

Regards,

Fabrício De Marchi

Fabricio,

Just a tip to help elevate the daily OOM errors, we have our services re-booted every evening and also have it set up to have them re-boot after 2 mins of the OOM error. It has prevented me from having to manually do it and I no longer worry about jobs that are still hanging out there. We need to re-architect our environment as well but lack the finances to do so at this time. Good luck to you!

That sounds like an innovative approach, but feet down.
Would you please share some more of it.
How it is actually arranged that the service are restarted after the OOM error?
What is your web server and database?
In you job schedule how many jobs are fired simultaneously eg by default many of of the jobs are scheduled at 00 mins.

Martti K.

Hi Katrina, thanks for sharing your experience.

I did tried to do the same with Java but the 1.5 does not have the necessary commands to do it, or I didn't do a good search for it. I know that 1.6 can do it.

Would you be able to share how have you set that up to restart after the OOM?

Thanks again for coming forward and not let me feel alone in this "dark moment".:-)

Regards,

Fabricio De Marchi

Fabricio,
In talking to our technical person I was mistaken that we automatically reboot after the OOM. But the Niku Server Service is set to automatically restart if the Clarity appcliation crashes. It is a scheduled task that calls a batch job that runs the following:

net stop "Niku Background Server"
net start "Niku Background Server"

Thanks Katrina.

I was able to create a BAT script that is able to restart that service for me.

I will share it so anyone can use it.

This script do a find in the background log files for "java.lang.outofmemory" and creates a file that lists any file that has OOM error.
Then it checks the file created above to see if it has a size greater then 0(zero) if so it means OOM was found and the service needs to be restarted.
It creates a folder to store the log files, move all log files in that folder, and restart the services. That allows you to keep all log files where OOM was detected for future reference.

It is quite simple and I don't mind anyone using it or modifying it.

Regards,

Fabricio De Marchi

@echo off
setlocal
set ctime=%time%
set cfixtime=%ctime::=_%
set cfixtime=%cfixtime: =%
set datetimenow=%date:~-4%%date:~4,2%%date:~7,2%_%cfixtime%
set maxbytesize=0
set logsfolder=D:\Clarity\niku81\logs
set oomfolder=D:\Clarity\niku81\logs\outofmemory\%datetimenow%
set oomlogfile=%logsfolder%\oomlogfile.log
set fileswithoom=%logsfolder%\fileswithoom.txt

REM Find java.lang.outofmemory inside a file and restart backgrouns services
echo Start %date%, %time% > %oomlogfile%

REM Find any file with the outofmemory error, 
REM this is limited to 5 in the log configuration
findstr /M "java.lang.OutOfMemoryError" %logsfolder%\bg-niku.log %logsfolder%\bg-niku.log.1 %logsfolder%\bg-niku.log.2 %logsfolder%\bg-niku.log.3 %logsfolder%\bg-niku.log.4 %logsfolder%\bg-niku.log.5 > %fileswithoom%

REM check if there was any OOM in the log files by checking the size of the result file
FOR /F "usebackq" %%A IN ('%fileswithoom%') DO set size=%%~zA

echo Files with OOM found. (%size%) >> %oomlogfile%

if %size% GTR %maxbytesize% (
   echo create folder to store log files  >> %oomlogfile%
   mkdir "%oomfolder%" >> %oomlogfile%

   echo Stop Background Service >> %oomlogfile%
   net stop "Niku Background Server" >> %oomlogfile% 2>&1

   echo Move log files to OOM folder >> %oomlogfile%
   move /Y %logsfolder%\bg-niku.log %oomfolder% >> %oomlogfile% 2>&1
   move /Y %logsfolder%\bg-niku.log.1 %oomfolder% >> %oomlogfile% 2>&1
   move /Y %logsfolder%\bg-niku.log.2 %oomfolder% >> %oomlogfile% 2>&1
   move /Y %logsfolder%\bg-niku.log.3 %oomfolder% >> %oomlogfile% 2>&1
   move /Y %logsfolder%\bg-niku.log.4 %oomfolder% >> %oomlogfile% 2>&1
   move /Y %logsfolder%\bg-niku.log.5 %oomfolder% >> %oomlogfile% 2>&1

   echo Start Background service  >> %oomlogfile%
   net start "Niku Background Server" >> %oomlogfile% 2>&1

   echo Move log file and findstr file to backup folder  >> %oomlogfile%
   move /Y %fileswithoom% %oomfolder% >> %oomlogfile% 2>&1
   move /Y %oomlogfile% %oomfolder%
)

Thanks Fabricio!
We are facing the same OOM err in beacon due to which beacon service get stopped

I have a doubt in here, it might sound dumb but since am new to this scripting will ask anyways :)

Will using the net start command for restarting service overwrite the services to be start by localuser? say for eg: i have configured niku services to be start using a service accnt, will using net start cmd overwrite it?

Thanks in advance!
JJ