Came across this astounding issue and took a while to figure out. But here the jist of the problem and solution.
Problem Statement :
Post upgrading RHEL7 with Apache 2.4 and to R12.52 SP1 CR04/CR05 WebAgent; at random times the LLAWP CPU utilization spike to 100%. We would also see that Apache Error_log would be filled up every second with Semaphore errors. When WebAgent was started we did see that Semaphores and Shared Memory Segments were present. However when the CPU spike occurs we see that Semaphores were cleaned off, though Shared Memory Segments were present and LLAWP process was still showing running (but was really in a zombie state without any Semaphores).
Investigation :
We investigated the system logs on the Server and found these lines at the same timestamp about which the error_logs in Apache started spewing the missing Semaphores.
Aug 2 03:28:03 RHEL7-Server systemd: Removed slice user-986.slice.
Aug 2 03:28:03 RHEL7-Server systemd: Stopping user-986.slice.
Aug 2 03:30:01 RHEL7-Server systemd: Started Session 612 of user root.
Aug 2 03:30:01 RHEL7-Server systemd: Starting Session 612 of user root.
Aug 2 03:30:53 RHEL7-Server systemd-logind: Removed session 572.
Aug 2 03:30:53 RHEL7-Server systemd: Removed slice user-1000.slice.
Aug 2 03:30:53 RHEL7-Server systemd: Stopping user-1000.slice.
Aug 2 03:40:01 RHEL7-Server systemd: Started Session 613 of user root.
Aug 2 03:40:01 RHEL7-Server systemd: Starting Session 613 of user root.
Aug 2 03:50:01 RHEL7-Server systemd: Started Session 614 of user root.
Aug 2 03:50:01 RHEL7-Server systemd: Starting Session 614 of user root.
Aug 2 03:55:01 RHEL7-Server systemd: Created slice user-986.slice.
Investigating further we seem to have found a likely cause. It seems like a User Session Logout from the server causes Semaphores to be cleant out. After researching it was identified that RHEL7 has a setting which in prior versions of RHEL (upto 6) was disabled by default; however in RHEL7 is enabled by default.
https://docs.oracle.com/cd/E52668_01/E67200/html/section-t51_kcn_f5.html
https://www.linkedin.com/pulse/how-kill-your-database-seconds-os-update-rhel-oel-71-maciej-tokar
https://www.certdepot.net/rhel7-changes-between-versions/
https://www.freedesktop.org/software/systemd/man/logind.conf.html
[root@RHEL7-Server logs]# cat /etc/systemd/logind.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See logind.conf(5) for details.
[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
#HandleLidSwitch=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
The issue does reproduce itself. But there is no specific timing of when it would occur. It is sporadic in nature and is probably determined by a combination of when the last user logs out plus when the OS decides to kick a cleanup of User Sessions.
<SNIP> https://www.freedesktop.org/software/systemd/man/logind.conf.html
RemoveIPC=
Controls whether System V and POSIX IPC objects belonging to the user shall be removed when the user fully logs out. Takes a boolean argument. If enabled, the user may not consume IPC resources after the last of the user's sessions terminated. This covers System V semaphores, shared memory and message queues, as well as POSIX shared memory and message queues. Note that IPC objects of the root user and other system users are excluded from the effect of this setting. Defaults to "yes"
<SNIP>
Solution :
In RHEL7 when the OS is built for SiteMinder WebAgent, make sure to have these 2 changes done, inorder to disable removal of Semaphore when a User Session logs out.
There were 2 changes made.
A. Edit /etc/systemd/logind.conf and add an uncommented value RemoveIPC=no.
B. Do a restart of systemd-logind.
[root@RHEL7-Server ~]#
[root@RHEL7-Server ~]# cat /etc/systemd/logind.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See logind.conf(5) for details.
[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
#HandleLidSwitch=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
RemoveIPC=no
[root@RHEL7-Server ~]#
[root@RHEL7-Server ~]# systemctl restart systemd-logind
[root@RHEL7-Server ~]#
[root@RHEL7-Server ~]#
Regards
Hubert