/restman is the same as other published APIs on gateway, you can disable/enable it (them) on port properties,
If you disable it, other normal APIs on gateway cannot be accessed via this port, too.
Another option is hardcoded in restman policy (or global policy) to validate the port using context variable ${request.tcp.localPort}