Vlad,
I had a similar request recently with the federated apps on the SPS. Please review the solution in the following post.
CA SPS Federation Gateway Question
As you can see we handled the blocking of the inappropriate virtual host access with a auth scheme redirect to a static access denied page. We used agent group object to protect the realm associated with the federated app and applied the custom auth scheme to that realm. All VH on sps are using a upique Webagent.conf so we can use unique agent objects on SSO side. The appropriate virtual host is not in the agent group and not protected with this custom auth scheme and can access the app.
Thanks Hubert Dennis for the suggested method, it works great.
If you would like to see better delegation of these types of requests in the server.conf please upvote for my enhancement request here.
CA SPS - Block access to Federated Web Apps on Virtual Host Basis
thanks,
Adam