DX NetOps

  • 1.  PM Error in PC service

    Posted Aug 14, 2018 12:03 PM

    Hi guys I have a problem with PM in the dasbhord I get an error id looking in the logs I found the following. I do not know if it's reconfiguration of the NFA

     

    I found this error in the PC folder

     

    INFO | jvm 1 | 2018/08/13 12:06:47 | SELECT .RouterID, .Router.AddressName, .Router.EOVBytes, .Router.Address, .Router.Name FROM CA.ReporterAnalyzer.RAWSParameters WHERE .GroupID = '99' AND .StartTime = 1534176360 AND .EndTime = 1534179960 AND .LimitCount = 10 ORDERBY .Router.EOVBytes DESC HAVING (.RAWebCall = 'EnterpriseTopRouters') LIMIT 10
    INFO | jvm 1 | 2018/08/13 12:06:47 | }}}
    INFO | jvm 1 | 2018/08/13 12:08:50 | ERROR | Model-Invoker-34 | 2018-08-13 12:08:50,714 | com.ca.im.portal.plugins.rib.models.RIBXYChartModel
    INFO | jvm 1 | 2018/08/13 12:08:50 | | RIB query failed
    INFO | jvm 1 | 2018/08/13 12:08:50 | Model ID/Type: {102029/RIBXYChartModel}
    INFO | jvm 1 | 2018/08/13 12:08:50 | Result status: {FAILED}
    INFO | jvm 1 | 2018/08/13 12:08:50 | Query ID: {edd8cc79-6ef9-43d7-8221-105f5c9248ae}
    INFO | jvm 1 | 2018/08/13 12:08:50 | RIB source URL: {http://1XX.XX.2XX.1XX:8681/NFARS/ribsource/rib/soap?wsdl}
    INFO | jvm 1 | 2018/08/13 12:08:50 | Query: {SELECT .RouterID, .Router.AddressName, .Router.EOVBytes, .Router.Address, .Router.Name FROM CA.ReporterAnalyzer.RAWSParameters WHERE .GroupID = 15831 AND .StartTime = 1534176480 AND .EndTime = 1534180080 AND .LimitCount = 10 ORDERBY .Router.EOVBytes DESC HAVING (.RAWebCall = 'EnterpriseTopRouters') LIMIT 10}
    INFO | jvm 1 | 2018/08/13 12:08:50 | Reason: {{
    INFO | jvm 1 | 2018/08/13 12:08:50 | Error occurred while running a RIB query on SQL RIB source.
    INFO | jvm 1 | 2018/08/13 12:08:50 | Possible reason: null
    INFO | jvm 1 | 2018/08/13 12:08:50 | Query:
    INFO | jvm 1 | 2018/08/13 12:08:50 | {
    INFO | jvm 1 | 2018/08/13 12:08:50 | SELECT .RouterID, .Router.AddressName, .Router.EOVBytes, .Router.Address, .Router.Name FROM CA.ReporterAnalyzer.RAWSParameters WHERE .GroupID = '119' AND .StartTime = 1534176480 AND .EndTime = 1534180080 AND .LimitCount = 10 ORDERBY .Router.EOVBytes DESC HAVING (.RAWebCall = 'EnterpriseTopRouters') LIMIT 10
    INFO | jvm 1 | 2018/08/13 12:08:50 | }}
    INFO | jvm 1 | 2018/08/13 12:08:50 | Caused by:
    INFO | jvm 1 | 2018/08/13 12:08:50 | {
    INFO | jvm 1 | 2018/08/13 12:08:50 | Error occurred while running a RIB query on SQL RIB source.
    INFO | jvm 1 | 2018/08/13 12:08:50 | Possible reason: null
    INFO | jvm 1 | 2018/08/13 12:08:50 | Query:
    INFO | jvm 1 | 2018/08/13 12:08:50 | {
    INFO | jvm 1 | 2018/08/13 12:08:50 | SELECT .RouterID, .Router.AddressName, .Router.EOVBytes, .Router.Address, .Router.Name FROM CA.ReporterAnalyzer.RAWSParameters WHERE .GroupID = '119' AND .StartTime = 1534176480 AND .EndTime = 1534180080 AND .LimitCount = 10 ORDERBY .Router.EOVBytes DESC HAVING (.RAWebCall = 'EnterpriseTopRouters') LIMIT 10
    INFO | jvm 1 | 2018/08/13 12:08:50 | }}}
    INFO | jvm 1 | 2018/08/13 12:12:17 | INFO | qtp1902924144-28195 | 2018-08-13 12:12:17,305 | com.ca.im.portal.api.services.datasource.DataSourcePoll
    INFO | jvm 1 | 2018/08/13 12:12:17 | | Test DataSource: Network Flow Analysis@harvester
    INFO | jvm 1 | 2018/08/13 12:12:17 | INFO | qtp1902924144-28195 | 2018-08-13 12:12:17,314 | org.apache.cxf.service.factory.ReflectionServiceFactoryBean
    INFO | jvm 1 | 2018/08/13 12:12:17 | | Creating Service {http://netqos.com/DataSourceWS}IDataSourceWSService from class com.ca.im.portal.api.datasources.interfaces.datasourcews.IDataSourceWS
    INFO | jvm 1 | 2018/08/13 12:12:17 | ERROR | qtp1902924144-28195 | 2018-08-13 12:12:17,399 | com.ca.im.portal.api.services.datasource.DataSourcePoll
    INFO | jvm 1 | 2018/08/13 12:12:17 | | Received WebServiceException from version check for data source Network Flow Analysis@harvester. CAUSE=java.net.UnknownHostException: UnknownHostException invoking http://harvester:80/ReporterDataSource/DataSourceWS.asmx: harvester. MESSAGE=Could not send Message.. Returning DS_COMM_FAILURE result.
    INFO | jvm 1 | 2018/08/13 12:12:17 | ERROR | qtp1902924144-28195 | 2018-08-13 12:12:17,399 | com.ca.im.portal.api.services.datasource.DataSourcePoll
    INFO | jvm 1 | 2018/08/13 12:12:17 | | javax.xml.ws.WebServiceException: Could not send Message.



  • 2.  Re: PM Error in PC service

    Posted Aug 14, 2018 02:25 PM

    looking for what could be the cause of the error I found this in the data agregator

     

    ERROR | entPush-thread-1 | 2018-08-08 03:00:09,083 | RootExceptionLog | .ca.im.core.util.ExceptionLogger 100 | m.ca.im.common.core.util | | A NEW application exception occurred (Key=51a7a204d601f823c34d67251f085effff5f3f03) : Caught exception during event push: javax.xml.ws.WebServiceException: org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05 : org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05
    javax.xml.ws.WebServiceException: org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:155)[154:org.apache.cxf.cxf-rt-frontend-jaxws:2.7.11]
    at com.sun.proxy.$Proxy400.push(Unknown Source)[212:portal-api.datasources.interfaces:3.5.0.RELEASE-103]
    at com.ca.im.connector.eventproducer.EventProducerWS.pushEvents(EventProducerWS.java:771)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at com.ca.im.connector.eventproducer.EventProducerWS.access$400(EventProducerWS.java:60)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at com.ca.im.connector.eventproducer.EventProducerWS$1$1.run(EventProducerWS.java:323)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
    at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
    Caused by: org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05
    at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:265)[149:org.apache.cxf.cxf-rt-databinding-jaxb:2.7.11]
    at org.apache.cxf.jaxb.io.DataWriterImpl.write(DataWriterImpl.java:221)[149:org.apache.cxf.cxf-rt-databinding-jaxb:2.7.11]
    at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:114)[145:org.apache.cxf.cxf-api:2.7.11]



  • 3.  Re: PM Error in PC service

    Broadcom Employee
    Posted Aug 15, 2018 11:05 AM

    Vladimir,

     

    Just in scanning over the errors, it sounds like there may be some inconsistencies between CAPC and NFA.  Are these errors constantly happening or is it a one time deal?  Are NFA / CAPC having issues syncing?  When was the last successful sync?  Have there been modifications or changes in NFA?

     

    I do not know the NFA database structure but you may want to open a case with the NFA team to get some assistance in manually running the queries that CAPC is saying is coming back as NULL.

     

    Troy



  • 4.  Re: PM Error in PC service

    Posted Aug 15, 2018 11:58 AM

    Hi Troy

    The error is constant, we have two harvesters in NFA and the problem only lies in one which is reconfigured and is where the problem is presented in the CAPM



  • 5.  Re: PM Error in PC service

    Broadcom Employee
    Posted Aug 15, 2018 12:02 PM

    Vladimir,

     

    What do you mean by reconfigured and do you have two separate NFA environments set up as data sources in CAPC?

     

    Troy



  • 6.  Re: PM Error in PC service

    Posted Aug 15, 2018 12:10 PM

    Troy 

    I have an NFA console and two computers that collect the flow "Harvester" the NFA console is connected to CAPM and it works correctly but only one harvester is not responding to those statistics within CAPM I found another log within the CAPM aggregator that indicates the following.

     

    ERROR | entPush-thread-1 | 2018-08-14 05:30:10,005 | ExceptionLog | .ca.im.core.util.ExceptionLogger 104 | m.ca.im.common.core.util | | An existing application exception RECURRED (Key=51a7a204d601f823c34d67251f085effff5f3f03), Recurrence count=191 : Caught exception during event push: javax.xml.ws.WebServiceException: org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05 : org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05
    WARN | -Minute-thread-3 | 2018-08-14 05:35:03,720 | FilteredEventProfileDAOImpl | impl.FilteredEventProfileDAOImpl 927 | .ca.im.aggregator.loader | | Dropped the rule because baseline is not calculated for '{http://im.ca.com/normalizer}NormalizedPortInfo.Availability' in '{http://im.ca.com/normalizer}NormalizedPortInfo' metric family
    WARN | entPush-thread-1 | 2018-08-14 05:35:17,985 | PhaseInterceptorChain | ache.cxf.common.logging.LogUtils 452 | org.apache.cxf.cxf-api | | Interceptor for {http://netqos.com/nqevents/EventManager}IEventManagerWSService#{http://netqos.com/nqevents/EventManager}Push has thrown exception, unwinding now
    org.apache.cxf.interceptor.Fault: Marshalling Error: stfadmpmo05
    at org.apache.cxf.jaxb.JAXBEncoderDecoder.marshall(JAXBEncoderDecoder.java:265)[149:org.apache.cxf.cxf-rt-databinding-jaxb:2.7.11]
    at org.apache.cxf.jaxb.io.DataWriterImpl.write(DataWriterImpl.java:221)[149:org.apache.cxf.cxf-rt-databinding-jaxb:2.7.11]
    at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:114)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.interceptor.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:570)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:479)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)[145:org.apache.cxf.cxf-api:2.7.11]
    at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)[153:org.apache.cxf.cxf-rt-frontend-simple:2.7.11]
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)[154:org.apache.cxf.cxf-rt-frontend-jaxws:2.7.11]
    at com.sun.proxy.$Proxy400.push(Unknown Source)[212:portal-api.datasources.interfaces:3.5.0.RELEASE-103]
    at com.ca.im.connector.eventproducer.EventProducerWS.pushEvents(EventProducerWS.java:771)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at com.ca.im.connector.eventproducer.EventProducerWS.access$400(EventProducerWS.java:60)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at com.ca.im.connector.eventproducer.EventProducerWS$1$1.run(EventProducerWS.java:323)[240:com.ca.im.NPCConnector.bundle:3.5.0.RELEASE-191]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
    at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
    Caused by: javax.xml.bind.MarshalException
    - with linked exception:



  • 7.  Re: PM Error in PC service

    Broadcom Employee
    Posted Aug 15, 2018 12:22 PM

    Vladimir,

     

    So the integration between NFA and CAPC has two high level sides.  The data sent over from NFA like the flow data as well as the devices that are in NFA being passed to the DA (if the option is selected) so it can be discovered by the DA and polled via SNMP.  Your first messages indicate that the query sent to NFA to gather data returned nothing or NULL. The second one mentions a marshalling error but no real error message or fault.  This last message mentions the marshalling error again but also mentions that there was not baseline calculated meaning either there was no historical data back x days for it to create the baseline or there is a fault in doing so.

     

    I would start with a sync between CAPC and NFA.  Next, go after your first error and see if you can get the queries to work in NFA directly.  As I mentioned, you may need to get a NFA Assisted Support Case opened to get assistance in doing so.  Also check into whatever the reconfiguration you mentioned entails and what it could do to CAPC's view into NFA.

     

    In the end, I think that an Assisted Support Case, likely with the NFA team would be a good next step, especially when only half of NFA (1 of 2 harvesters) seems to be working.

     

    Troy



  • 8.  Re: PM Error in PC service

    Posted Aug 15, 2018 12:45 PM

    Thanks Troy 

     

    I'll open the case and perform tests on the NFA console to see statistics