Clarity

Hi all,

our application was running fine until this afternnon. Some users then began reporting system errors when they clicked on anything within the application.

Users are unable to login - as am I, even with the admin credentials.

In the logs I see this error recurring:

ERROR 2015-09-14 11:15:42,132 [http-bio-8080-exec-24] pagebuilder.UIThemeCache (clarity:X163479:200009667__A972AACC-DE1F-414E-B086-494542A76CB2:npt.overview) No default theme found

ERROR 2015-09-14 11:15:42,326 [http-bio-8080-exec-20] out.service (clarity:X163479:200009667__A972AACC-DE1F-414E-B086-494542A76CB2:GetNavigatorMenuBean) A RuntimeException encountered in Service. Converting to OSFApplicationException.

Also seeing errors like this:

Error message: [CA Clarity][SQLServer JDBC Driver][SQLServer]The cursor was not declared.

Executed:

select  d.id id, cc.name COLLATE Latin1_General_CI_AS_KS name, cc.description COLLATE Latin1_General_CI_AS_KS description, null avs_name, null avs_id, ua.TABLE_NAME, ua.PK_ID,  1 can_create_job, d.is_active

        from

                cmn_captions_nls cc,

                cmn_sch_user_assoc ua,

                cmn_sch_job_definitions d,

                ( select  id

                  from    cmn_lookups

                  where   lookup_type = 'SCH_JOB_TYPE'

                  and     lookup_code = 'REPORT') rt

        where   cc.language_code = ?

        and     d.id = cc.pk_id

        and     cc.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     d.job_type = rt.id

        and     ua.TABLE_NAME = 'CMN_SCH_JOB_DEFINITIONS'

        and     ua.USER_ID = ?

        and     ua.PK_ID = d.id

        AND     d.IS_SYSTEM_VISIBLE=?

        AND     d.IS_ACTIVE=?  and     ( (1 = niku.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT_DEFINITION', 'REPORT', 'CREATE', d.id ))

                or

                  (1 = niku.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT', 'REPORT', 'CREATE', d.id )) )  union

        select  d.id id, cc.name COLLATE Latin1_General_CI_AS_KS name, cc.description COLLATE Latin1_General_CI_AS_KS description, avs.name avs_name, avs.id avs_id, ua.TABLE_NAME, ua.PK_ID,

                1 can_create_job, d.is_active

        from    cmn_sch_job_definitions d, cmn_captions_nls cc, cmn_business_objects bo,

                cmn_attribute_value_sets avs, cmn_sch_user_assoc ua,

                ( select  id

                  from    cmn_lookups

                  where   lookup_type = 'SCH_JOB_TYPE'

                  and     lookup_code = 'REPORT') rt

        where   cc.language_code = ?

        and     d.id = cc.pk_id

        and     cc.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     d.job_type = rt.id

        and     bo.pk_id = d.id

        and     bo.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     bo.id = avs.business_object_id

        and     ua.TABLE_NAME = 'CMN_ATTRIBUTE_VALUE_SETS'

        and     ua.USER_ID = ?

        and     ua.PK_ID = avs.id

        AND     d.IS_SYSTEM_VISIBLE=?

        AND     d.IS_ACTIVE=?  and     ( (1 = niku.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT_DEFINITION', 'REPORT', 'CREATE', d.id ))

                or

                  (1 = niku.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT', 'REPORT', 'CREATE', d.id )) )

            order by name

Derived from statement:

<?xml version="1.0" encoding="UTF-8"?>

<statement location="schedulerPMD.xml" id="nmc.myReports_stmt" subjectToSlicing="true" sortColumnPath="/data/header/sortInfo/@sortColumn" sortDirectionPath="/data/header/sortInfo/@sortDirection" slicePath="/data/header/pagination/sliceInfo/@slice" sliceSizePath="/data/header/pagination/sliceInfo/@sliceSize" defaultSliceSize="10" inputSource="none" xmlns="http://schemas.niku.com/2002/pmd">

  <sql dbVendor="oracle" dbVersion="8">

    <text>

        select  /*+ RULE */ d.id id, cc.name name, cc.description description, null avs_name, to_number(null) avs_id, ua.TABLE_NAME, ua.PK_ID,

      </text>

  </sql>

  <sql dbVendor="oracle" dbVersion="!8">

    <text>

        select  d.id id, cc.name name, cc.description description, null avs_name, to_number(null) avs_id, ua.TABLE_NAME, ua.PK_ID,

      </text>

  </sql>

  <sql dbVendor="mssql">

    <text>

        select  d.id id, cc.name @COLLATE@ name, cc.description @COLLATE@ description, null avs_name, null avs_id, ua.TABLE_NAME, ua.PK_ID,

      </text>

  </sql>

  <sql dbVendor="all">

    <text>

                1 can_create_job, d.is_active

        from

                cmn_captions_nls cc,

                cmn_sch_user_assoc ua,

                cmn_sch_job_definitions d,

                ( select  id

                  from    cmn_lookups

                  where   lookup_type = 'SCH_JOB_TYPE'

                  and     lookup_code = 'REPORT') rt

        where   cc.language_code = ?

        and     d.id = cc.pk_id

        and     cc.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     d.job_type = rt.id

        and     ua.TABLE_NAME = 'CMN_SCH_JOB_DEFINITIONS'

        and     ua.USER_ID = ?

        and     ua.PK_ID = d.id

        AND     d.IS_SYSTEM_VISIBLE=?

        AND     d.IS_ACTIVE=?

      </text>

    <param replace="language" direction="IN" expressionListDelimiter=","/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param type="int" direction="IN" constantValue="1" expressionListDelimiter=","/>

    <param type="int" direction="IN" constantValue="1" expressionListDelimiter=","/>

  </sql>

  <sql dbVendor="mssql">

    <text>

        and     ( (1 = @DBUSER@.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT_DEFINITION', 'REPORT', 'CREATE', d.id ))

                or

                  (1 = @DBUSER@.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT', 'REPORT', 'CREATE', d.id )) )

      </text>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

  </sql>

  <sql dbVendor="oracle">

    <text>

        and     ( exists ( select object_instance_id

                           from   cmn_sec_chk_user_v0

                           where  object_instance_id = d.id

                           and    object_id = ?

                           and    permission_code = 'CREATE'

                           and    user_id = ?)

                  or

                  exists ( select 1 user_id

                           from   dual

                           where  1 = ?

                           or     1 = ? )

                )

      </text>

    <param direction="IN" expressionListDelimiter="," objectCode="REPORT_DEFINITION" objectType="REPORT" componentCode="CMN"/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param type="int" direction="IN" expressionListDelimiter=",">

      <globalRight permissionCode="CREATE" objectCode="REPORT" objectType="REPORT" componentCode="CMN"/>

    </param>

    <param type="int" direction="IN" expressionListDelimiter=",">

      <globalRight permissionCode="CREATE" objectCode="REPORT_DEFINITION" objectType="REPORT" componentCode="CMN"/>

    </param>

  </sql>

  <sql dbVendor="all">

    <text>

        union

        select  d.id id, cc.name @COLLATE@ name, cc.description @COLLATE@ description, avs.name avs_name, avs.id avs_id, ua.TABLE_NAME, ua.PK_ID,

                1 can_create_job, d.is_active

        from    cmn_sch_job_definitions d, cmn_captions_nls cc, cmn_business_objects bo,

                cmn_attribute_value_sets avs, cmn_sch_user_assoc ua,

                ( select  id

                  from    cmn_lookups

                  where   lookup_type = 'SCH_JOB_TYPE'

                  and     lookup_code = 'REPORT') rt

        where   cc.language_code = ?

        and     d.id = cc.pk_id

        and     cc.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     d.job_type = rt.id

        and     bo.pk_id = d.id

        and     bo.table_name = 'CMN_SCH_JOB_DEFINITIONS'

        and     bo.id = avs.business_object_id

        and     ua.TABLE_NAME = 'CMN_ATTRIBUTE_VALUE_SETS'

        and     ua.USER_ID = ?

        and     ua.PK_ID = avs.id

        AND     d.IS_SYSTEM_VISIBLE=?

        AND     d.IS_ACTIVE=?

      </text>

    <param replace="language" direction="IN" expressionListDelimiter=","/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param type="int" direction="IN" constantValue="1" expressionListDelimiter=","/>

    <param type="int" direction="IN" constantValue="1" expressionListDelimiter=","/>

  </sql>

  <sql dbVendor="mssql">

    <text>

        and     ( (1 = @DBUSER@.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT_DEFINITION', 'REPORT', 'CREATE', d.id ))

                or

                  (1 = @DBUSER@.cmn_sec_check_right_fct( ?, 'CMN', 'REPORT', 'REPORT', 'CREATE', d.id )) )

            order by name

      </text>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

  </sql>

  <sql dbVendor="oracle">

    <text>

        and     ( exists ( select object_instance_id

                           from   cmn_sec_chk_user_v0

                           where  object_instance_id = d.id

                           and    object_id = ?

                           and    permission_code = 'CREATE'

                           and    user_id = ?)

                  or

                  exists ( select 1 user_id

                           from   dual

                           where  1 = ?

                           or     1 = ?)

                )

                      order by name

      </text>

    <param direction="IN" expressionListDelimiter="," objectCode="REPORT_DEFINITION" objectType="REPORT" componentCode="CMN"/>

    <param replace="userId" direction="IN" expressionListDelimiter=","/>

    <param type="int" direction="IN" expressionListDelimiter=",">

      <globalRight permissionCode="CREATE" objectCode="REPORT" objectType="REPORT" componentCode="CMN"/>

    </param>

    <param type="int" direction="IN" expressionListDelimiter=",">

      <globalRight permissionCode="CREATE" objectCode="REPORT_DEFINITION" objectType="REPORT" componentCode="CMN"/>

    </param>

  </sql>

  <outputMapping collectionElementName="myReports" rowElementName="myReport"/>

</statement>

Referenced by:

<?xml version="1.0" encoding="UTF-8"?>

<statementRef id="nmc.myReports_stmt" inputSource="none" sortColumnPath="/data/header/sortInfo/@sortColumn" sortDirectionPath="/data/header/sortInfo/@sortDirection" slicePath="/data/header/pagination/sliceInfo/@slice" sliceSizePath="/data/header/pagination/sliceInfo/@sliceSize" xmlns="http://schemas.niku.com/2002/pmd"/>

Using input:

None

  at com.niku.union.persistence.PersistenceController.createException(PersistenceController.java:2019)

  at com.niku.union.persistence.PersistenceController.handleSQLException(PersistenceController.java:2123)

  at com.niku.union.persistence.PersistenceController.processSql(PersistenceController.java:2662)

  at com.niku.union.persistence.PersistenceController.processStatement(PersistenceController.java:852)

  at com.niku.union.persistence.PersistenceController.processStatements(PersistenceController.java:752)

  at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:560)

  at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:290)

  at com.niku.union.service.binding.ServiceControlBindingPMD.executePersistenceRequest(ServiceControlBindingPMD.java:204)

  at com.niku.union.service.binding.ServiceControlBindingPMD.getXMLOutput(ServiceControlBindingPMD.java:88)

  at com.niku.union.service.binding.ServiceControlBindingPMD.processRequest(ServiceControlBindingPMD.java:61)

  ... 39 more

Caused by: java.sql.SQLException: [CA Clarity][SQLServer JDBC Driver][SQLServer]The cursor was not declared.

  at com.ca.clarity.jdbc.sqlserverbase.ddb7.b(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddb7.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddb6.b(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddb6.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddr.v(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddr.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddq.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddm.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddr.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddm.t(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddm.c(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddn.f(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.tds.ddn.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserver.ddj.h(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.v(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.r(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.r(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.a(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.ddei.r(Unknown Source)

  at com.ca.clarity.jdbc.sqlserverbase.dddb.execute(Unknown Source)

  at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:68)

  at org.logicalcobwebs.cglib.proxy.Proxy$ProxyImpl$$EnhancerByCGLIB$$3fa11a93.execute(<generated>)

  at com.niku.union.persistence.PersistenceController.processSql(PersistenceController.java:2624)

  ... 46 more

ava:37)

  at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:95)

  at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)

  at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:84)

  at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:86)

  at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:348)

  at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:147)

  at com.ca.platform.osf.service.transport.servlet.OSFServletController.invoke(OSFServletController.java:82)

  at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)

  at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)

  at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)

  at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

  at com.niku.union.web.filter.CharsetFilter.doFilter(CharsetFilter.java:56)

  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

  at org.apache.catalina.filters.AddDefaultCharsetFilter.doFilter(AddDefaultCharsetFilter.java:88)

  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)

  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)

  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

  at com.niku.union.web.valves.ErrorPageRedirectValve.invoke(ErrorPageRedirectValve.java:69)

  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)

  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)

  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)

  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)

  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  at java.lang.Thread.run(Thread.java:745)

Has anyone else encountered similar?

Another error which seems to be cropping up quite a lot is:

(clarity:unknown:none:security.staticLogin) No default theme found

Hi Colin,

This sometimes happens when your custom theme gets corrupted. Please run the following queries to solve the issue, this will reset the default Clarity theme to default:

update cmn_ui_themes set is_default = 0 where is_default = 1;
update cmn_ui_themes set is_default = 1 where code = 'default';
commit;

Please let me know how it goes.

Kind Regards

Nika Hadzhikidi
CA Technologies
Principal Support Engineer

Thanks Nika.

I will try this and let you know how it goes.

Colin tried Nika's suggestion and reverted to the default UI theme but we are still seeing the issue. After a time, no mater what users click within the app system errors are thrown. Then, if if we log out we are unable to log in again.

I have tried rebooting the app server as well as restarting all services. This seems to fix the issue for a few minutes. Users can log in and navigate around ok, but before long the same system errors get thrown again.

We were also seeing errors in the app-system log concerning Wily.

SEVERE: Failed to initialize connector [Connector[AJP/1.3-8009]]

2015/09/14 12:15:11.132 | org.apache.catalina.LifecycleException: Failed to initialize component [Connector[AJP/1.3-8009]]

Caused by: java.net.BindException: Address already in use: JVM_Bind <null>:8009

It turned out that the Webview service was down and once this was restarted performance seems to have stabilized.

I don;t know if problems with Wily would impact the application in the way we saw though?

8009 is a really common initial/default AJP port used by Tomcat services.  Some services/applications will attempt to hit a range of ports (e.g. 8009 to 8019) looking for one that's available, others will just try the value they're configured to use and if already in use, report that message.

So the order in which you start/restart services can affect when and where you may see this error, but the best solution longer term is to find and define unique AJP connector port values for the services so that clashes are mitigated.

For Clarity you can alter this easily in the CSA by changing the highlighted value (Tomcat Connector Port of the Application properties for: app, app2, appN, etc. services).

Avoid 8000 and 8080 as other common ports, but many other random numbers in the 8*** range will be available to use.  A service stop / deploy / start is required after making these changes, after which you should not encounter the JVM bind address in use messages any more.

Hi Colin,

You actually had two issues : one was with the Theme, which we fixed, then you had the 8009 port conflict with Wily agent. I would say that the issue with the connection was most probably due to the Wily issue with the port since it was thrown many hundreds of times in the logs. We have seen issues in which third party apps would impact connection to Clarity previously.

Now when this is fixed you should not have any problem with it anymore.

Kind Regards
Nika Hadzhikidi
CA Technologies