Hi Chamara,
There is currently no way to filter ports on the PAM side. I understand that the delay in opening connections to the devices can be an annoyance. The KB doc below explains why this port is used with SSH and how to stop the connection from taking so long. Basically the only option here would be to set a firewall (outside of PAM) that will REJECT that port. Since you don't want to do this on a per-device basis, one other option would be to instead put the PAM appliances behind a firewall and set the firewall for all traffic over 8550 to be REJECTED.
Tip: Why is port 8550 being queried when starting - CA Knowledge
If you would like to see a feature added in a future release to control the PAM side firewall then please submit an Idea (enhancement request) here on the PAM communities page. Product Management regularly reviews these Ideas to decide which ones may be included in the product in future releases.
Regards,
Christian Lutz
Sr Support Engineer
Broadcom (CA) - ESD Support