Clarity

Hi Friends,

I noticed this behavior in one of the lower development single-server environments of CA PPM. After a user logs in, it takes much time for the "Administration" menu to appear at the top between HOME and Favorites. Same is the case with any dropping menu sections under HOME or Favorites.

This happens mostly in the first login of the day and the delay is not so much in the subsequent logins. This literally means I have to wait for one or two minutes in my first login to go to ANY menu item either at HOME side or Administration side.

This issue happens in whichever web browser I try. We are on CA PPM 14.2 with Oracle 11G in this particular environment. What can be the possible cause for this huge delay? How to troubleshoot this one? Any ideas? 

Regards,

Georgy

Hi Georgy,

Yes you are correct but this happens if you reboot your services every day as when it logs in 1st time it caches lot of data and hence the delay. 

Regards

Suman Pramanik 

admin warmup?

I do see a little of that, but never more than a minute.

For me it happens on a system which has a high number of custom attributes  and also on Oracle.

When it is worst I have related it to network response.

The other thing is my home page: I do have user tabs and portlets on it which I believe slows down the download of the page even if the default tab is simple.

One way I can reduce the problem is not logging out: SSO keeps me logged even if I close my browser and shut down my PC. I understand that that is not recommended.

Hi Georgy,

Just make sure to delete everything from your browsing history before you leave for the day. This can be helpful.

Suhail.

Hi Friends,

Thanks a lot for all the suggestions and for mentioning some possible causes for this slow loading of CA PPM menu. Today we had to wait for a whopping 4 minutes for the Clarity menus to load in this environment 

suhail-sayed I tried your suggestion today before accessing this particular PPM environment. But no luck, the delay is the same.

urmas Yes, you are right. A little delay is understandable but a delay of more than 2min is just too much! No SSO in this lower environment and not more than the regular default 3-4 portlets in this Overview page with no extra user tabs. This delay was not there before and started all of a sudden three weeks back, so not sure if the network is to blame as all other actions work just fine even now.

SumanPramanik No, we don't reboot the PPM services everyday either manually nor have any scheduled daily reboot of them in this environment.

Dave_3.0  Interesting suggestion!!!  I've never used the admin warmup command before. Upon some research, I got the below discussions from this community on the same warmup command.

nikuadmin warmup  

NSA stops  

When i try this warmup command on the only application server, I get the below error:

java.net.ConnectException: Connection refused: connect

As SumanPramanik had suggested in the 2nd link above, I have deleted the tomcat folder and unzipped it again and the JAVA_HOME path is already correct. But this error still remains! However I will now wait for the next day's login to see if this tomcat fix has improved the situation.

Will keep you all posted. Thanks again.

Regards,

Georgy

Just wondering if clearing the browser cache can be expected to fix the problem if that occurs on one system only.

You can test if is the launch page, by creating a link to a page that loads fast. Then if you open the link when you are not logged in  it should first request logging in and after that take you to the specific page and not to the normal landing page.

If there is no difference it is not the landing page.

Could it be some mix up in net work set up or configuration?

Is it for one user, one workstation only?

Do the rights have any effect?

Hi Friends,

Checked today morning to see if the issue is resolved after the previous Tomcat folder fix; no luck still!

Yesterday I was even able to successfully run the admin warmup command from the CA PPM application server when I tried it in the below format, but still the PPM Menu issue remains when I login to the appln today. So the admin warmup command doesn't help me here.

admin warmup -url <applnURL> -user <UserID> -password <Password> -verbose

And this issue is not just for me , but for multiple users on different browsers like Chrome and Firefox as well. Clearing the security caches doesn't help either. It's not the issue of the landing page as the result is the same if I try to logon to some other url of the appln directly instead of the Overview page.

Dave_3.0  I would really love to know where the actual bottleneck is. But how can I go finding that? All I can see from the access logs is a loooooong delay time when this action is being performed.

[27/Mar/2017:07:03:44 +0200]|POST /niku/nu?uitk.vxml.form=1&action=security.loginAction&uitk.navigation.location=Workspace&uitk.navigation.parent.location=Workspace&uitk.navigation.last.workspace.action=homeActionId HTTP/1.1|200|25339|57861|-

Regards,

Georgy

Put trace on with the option that displays the response times.

Georgy N Joseph wrote:

 

 

I would really love to know where the actual bottleneck is. But how can I go finding that? All I can see from the access logs is a loooooong delay time when this action is being performed.

Its hard - even harder for someone not on your system  - you need to investigate what exactly is going on when the application "hangs". So far all you have done is see the symptom (the stalling app), you need to know what else is going on on your infrastructure at that point in time.

All GUESSes obviously, but things like...
1) Whats happening on the app server ; is CPU maxed out, is memory maxed out (could be badly configured and the app is swapping memory in and out of disk for example).

2) Is the app rebuilding its cache because the system has restarted (what everyone assumed was the issue)

3) is your database "waking up"

- e.g. is the app running its "rebuild the menu structure" query against the database and the DB is just taking a while to execute it - run the same SQL in a DB session only, does that take a while?

4) is your network "waking up"

5) is something else happening at browser-level even (e.g. your browsers are doing something unexpected, perhaps your network is doing something unexpected first thing in the day (authorizing internet traffic or something like that))

6) is your database badly optimised - take the "build menu SQL" and run it first thing in the morning before you start to connect from the app - does it take a long time? why? ; this would be a sign of badly optimised DB (refresh its statistics) and could explain symptoms (subsequent executions of that SQL are cached but it "falls out" of the cache overnight)

With all due respect to my more skilled community comembers what I hear is

 it is one page only for any user any time of the day any browser and first log on of the day only

Is that not correct?

If not please elaborate.

If yes then it does not sound like general slowness of the system, but just a single heavy or corrupted page.

Hi urmas,

As I said already, it is not page dependent and it's not the issue of the landing page (as I observed the same slowness in menu-building if I directly login to a different page of the application or even if I remove all the portlets in the Login page). 

Hi Dave_3.0,

Out of all your suggestions above, I guess, the point number 3 and 6 related to the PPM Database seem to be the real villain. Today, before connecting to the application, I ran the Menu Build SQL for my login (saved previously from trace) and it took a long 520 seconds from Oracle SQL Developer tool  And yea it worked because when I tried to login to this application after the above step, the PPM Menu was coming up in less than a second, in my first login of the day. So it all points to a bad DB now I guess 

Thanks a lot,

Georgy

My apologies, I missed the point that it is not page dependent.

DB is then the most likely course as Dave suggests and as you are pursuing.

App server file set up corruption is another but  less likely cause.

if you are not deliberately "rebooting" the system (app) overnight then it shouldn't be a cache problem - can you tell where the delay really is (obviously the "symptom" is the slow building menus) but where is the actual bottleneck?

I think everyone was assuming you were restarting the app and therefore assuming the delay was due to the application just rebuilding its cache the first time that it is used, but there could be other reasons I guess - app "falls over" overnight and recovers, slow network first thing, slow database that sort of thing?

Hi Georgy,

Are your browsers having the latest version? Did you check the logs? There is some issue of using IE for CA Clarity PPM. It works well with Firefox and Chrome. Are all the users facing this issue? Can you check with the IT Services Team if there is any network issue? Or it may be an issue related to your computer/laptop? What version of OS are you using? Also, try clearing the security caches on the CA Clarity PPM tool and see if this improves the speed.

Suhail.

Hi Georgy,

Can you check once if the initial elements of login (approximately 12-13 things that gets downloaded when the page gets loaded completely and see their time using chrome > F12 > Network section to see which component is taking time to load)

I had been in your project earlier and this was one of the observation i had made earlier where business was looking to reduce the logon time

Also check with HP Teams if this virtual servers are located in some place where there is a network issue only if you found the elements download time is higher)

You can also request BSM Team to capture this as a ad-hoc request in RUM to see what is happening

You would probably get more visibility towards root cause based on data you get

Also to identify the similar issue it had earlier used Tomcat Manager to see how the threads are behaving or maybe u can use jconsole to see this

Regards\\Avik

Hi Georgy ,

We are also facing the similar issue. Still we are not able to figure out what is the root cause of this problem. If this issue resolved please provide the reason for this issue.

Thanks in Advance.

Regards,

Rekha.