Clarity

I am having strange issue with bg services after upgrading the ca clarity version from 14.2 to 14.4.

The following is what i am experience,

a) Bg service starts without any issue and continues to run.

b) No schedule jobs are running

c) Even if i manually start any job including the system jobs like 'Delete Process Instance' or 'delete investment job' nothing happens.

d) no processess are running, my process engine is always red. even if i restart the bg service or even all the services no use.

e) Also when i open the NSA webpage using the server name, when i login, after inserting the password, nothing happens. While at the same time, if i use IP or localhost, everything works. DNS is working fine, not sure where the issue might be. 

Any Suggest will be helpful.

Many thanks,

Aleem

Can you delete all the cancelled / completed processes and then restart the bg?

Check if it helps

NJ

Hi CA AEC,

My comments inline

a) Bg service starts without any issue and continues to run.

b) No schedule jobs are running - Suman: I am not sure if you paused all the jobs before upgrade, are your jobs going in waiting/scheduled state. You may check this document Clarity: Scheduled jobs stuck in waiting or scheduled status 

c) Even if i manually start any job including the system jobs like 'Delete Process Instance' or 'delete investment job' nothing happens.: Suman: Check based on comment on point b

d) no processess are running, my process engine is always red. even if i restart the bg service or even all the services no use. Suman: Are there any errors on the bg logs 

e) Also when i open the NSA webpage using the server name, when i login, after inserting the password, nothing happens. While at the same time, if i use IP or localhost, everything works. DNS is working fine, not sure where the issue might be. Suman: This is strange, when you use the server name and nothing happens do you see anything written on the logs, if IP and local host is working then it points to DNS or the http entry URL in CSA

Thanks Suman, I assume i have some problem in BG service, the following is the result from bg log i get when i try to run any job

SYS 2018-01-29 11:13:44,206 [WrapperSimpleAppMain] bgp.JobLogger (clarity:none:none:none) Tenant assigned: clarity
ERROR 2018-01-29 11:13:45,315 [WrapperSimpleAppMain] bgp.JobLogger (clarity:unknown:11039018__951F0AA7-0AA2-4C54-8997-644164D750FA:none)
java.lang.NullPointerException: null address || null buffer
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:693)
at org.apache.log4j.helpers.SyslogWriter.write(SyslogWriter.java:77)
at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:47)
at org.apache.log4j.helpers.SyslogQuietWriter.write(SyslogQuietWriter.java:53)
at org.apache.log4j.net.SyslogAppender.append(SyslogAppender.java:254)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.info(Category.java:663)
at com.niku.union.log.NikuLogger.info(NikuLogger.java:1339)
at com.niku.union.log.AuthLogger.logSimpleEvent(AuthLogger.java:404)
at com.niku.union.log.AuthLogger.internalLogEvent(AuthLogger.java:283)
at com.niku.union.log.AuthLogger.logEvent(AuthLogger.java:126)
at com.niku.union.log.AuthLogger.logEvent(AuthLogger.java:70)
at com.niku.union.log.AuthLogger.loginLogout(AuthLogger.java:391)
at com.niku.union.log.AuthLogger.login(AuthLogger.java:294)
at com.niku.security.cache.UserSessionCache.insertToPersistence(UserSessionCache.java:340)
at com.niku.security.service.AuthenticationService.init(AuthenticationService.java:355)
at com.niku.union.bgp.NikuBGServer.startServer(NikuBGServer.java:150)
at com.niku.union.bgp.NikuBGServer.initServer(NikuBGServer.java:103)
at com.niku.union.bgp.NikuBGServer.main(NikuBGServer.java:304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
at java.lang.Thread.run(Thread.java:748)

I guess you are having multiple servers and from one server to another server there is communication issue as I don't see session 

SYS 2018-01-29 11:13:44,206 [WrapperSimpleAppMain] bgp.JobLogger (clarity:none:none:none) Tenant assigned: clarity
ERROR 2018-01-29 11:13:45,315 [WrapperSimpleAppMain] bgp.JobLogger (clarity:unknown:11039018__951F0AA7-0AA2-4C54-8997-644164D750FA:none) 

Hi Aleem,

You mention that your process engine is always red. I think beacon service may not be working correctly.

Is there any error messages in beacon-system.log?

Regards,

Shoichi

No errors on Beacon service

java.lang.NullPointerException: null address || null buffer

might be a multi-cast issue - in case you have multi-cast, please check

NJ

I actually Cloned my test instance and did the upgrade exercise on the newly cloned machine.  May be this is an issue and causing some interference with beacon and bg service

The following is the error received in bg-system log;

log4j:WARN No such property [header] in org.apache.log4j.net.SyslogAppender.
2018/01/29 14:54:52.926 | log4j:ERROR Could not find localhost:514. All logging will FAIL.
2018/01/29 14:54:52.941 | java.net.UnknownHostException: localhost:514
2018/01/29 14:54:52.941 | at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2018/01/29 14:54:52.957 | at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
2018/01/29 14:54:52.972 | at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
2018/01/29 14:54:52.988 | at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
2018/01/29 14:54:52.988 | at java.net.InetAddress.getAllByName(InetAddress.java:1192)
2018/01/29 14:54:53.004 | at java.net.InetAddress.getAllByName(InetAddress.java:1126)
2018/01/29 14:54:53.019 | at java.net.InetAddress.getByName(InetAddress.java:1076)
2018/01/29 14:54:53.019 | at org.apache.log4j.helpers.SyslogWriter.<init>(SyslogWriter.java:47)
2018/01/29 14:54:53.035 | at org.apache.log4j.net.SyslogAppender.setSyslogHost(SyslogAppender.java:297)
2018/01/29 14:54:53.035 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2018/01/29 14:54:53.051 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2018/01/29 14:54:53.066 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2018/01/29 14:54:53.082 | at java.lang.reflect.Method.invoke(Method.java:498)
2018/01/29 14:54:53.082 | at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:205)
2018/01/29 14:54:53.097 | at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:164)
2018/01/29 14:54:53.113 | at org.apache.log4j.xml.DOMConfigurator.setParameter(DOMConfigurator.java:540)
2018/01/29 14:54:53.113 | at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:192)
2018/01/29 14:54:53.129 | at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:150)
2018/01/29 14:54:53.144 | at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:163)
2018/01/29 14:54:53.160 | at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:425)
2018/01/29 14:54:53.176 | at org.apache.log4j.xml.DOMConfigurator.parseCategory(DOMConfigurator.java:345)
2018/01/29 14:54:53.176 | at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:827)
2018/01/29 14:54:53.191 | at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:724)
2018/01/29 14:54:53.191 | at org.apache.log4j.xml.DOMConfigurator.configure(DOMConfigurator.java:553)
2018/01/29 14:54:53.207 | at com.niku.union.log.NikuLogger.init(NikuLogger.java:966)
2018/01/29 14:54:53.222 | at com.niku.union.log.NikuLogger.reinit(NikuLogger.java:922)
2018/01/29 14:54:53.222 | at com.niku.union.config.ConfigurationManager.reinit(ConfigurationManager.java:705)
2018/01/29 14:54:53.238 | at com.niku.union.config.ConfigurationManager.fireChangeEvent(ConfigurationManager.java:802)
2018/01/29 14:54:53.269 | at com.niku.union.config.ConfigurationManager.initChange(ConfigurationManager.java:915)
2018/01/29 14:54:53.285 | at com.niku.union.config.ConfigurationManager.fireConfigInitChangeEvent(ConfigurationManager.java:790)
2018/01/29 14:54:53.301 | at com.niku.union.config.ConfigurationManager.fetchAndUpdateConfigObjectIfChanged(ConfigurationManager.java:4856)
2018/01/29 14:54:53.301 | at com.niku.union.config.ConfigurationManager.access$600(ConfigurationManager.java:171)
2018/01/29 14:54:53.316 | at com.niku.union.config.ConfigurationManager$ConfigMonitor$1.run(ConfigurationManager.java:7175)
2018/01/29 14:54:53.316 | at java.util.TimerThread.mainLoop(Timer.java:555)
2018/01/29 14:54:53.316 | at java.util.TimerThread.run(Timer.java:505)

This is the error 2018/01/29 14:54:52.926 | log4j:ERROR Could not find localhost:514. All logging will FAIL.

If you have cloned the VM's did you go ahead and change the IP address on the properties.xml to the current VM for the Bind address 

I guess you are using linux vm, so check the ip of local server and set up the bind address correctly 

I have already set the bind address accordingly. We are using windows with sql server 2012 SP4.

Thanks I guess there is some issue with cloning of VM and there might be IP conflict. I don't see any other problem apart from the things I already mentioned 

I am now able to solve the issue of jobs. I edited the logger.xml file.  My SyslogHost parameter was pointing to localhost:514, i changed this to the server name and the jobs are now working. 

Great news, can you remove the BG and redploy and see

Done. Actually i had unchecked the run process engine flag in NSA.  The process engine issue also got resolved.  Thanks everyone for your suggestion and timely support.

Perfect. Please mark this thread as resolved so that it can help anyone else looking at this thread