Hi,
There is no calculated hard limit for the maximum TPS that a VSE can handle. It all depends on the hardware configuration, load, complexity of the Virtual services, the size of the requests, system IO constraints and processor resource.
The complexity of your virtual service is viewable in the Workstation - make sure that you are not using any fields in the virtual service image (vsi) that you don't need, are not processing magic strings that you don't want, and so on.
The memory utilization is logged in the vse.log - as are errors should allocated memory be exhausted.
As for physical system constraints such as IO, processor etc, you should consult your system administration team.
Be aware that optimizing a system for performance is non-trivial, and you may wish to seek expert help to assist you on-premises.
Regards,
Javed