Hello!
These are heavy duty questions .... I will try.
Response to 1.:
I would suggest to use oauth instead of API Keys. There is more control with oauth clients than there is with API Keys. For example, when registering oauth clients in OAuth Manager, the value for 'Environment' could be used to mark this client as an 'internal' one. The client custom fields can be leveraged to express details of an oauth client in a JSON structure. That JSON message is then available at runtime and services can be configured to respect the message and details of it
Response to 2.:
In OTK you can register one single client but with multiple client_id's! Let's say 'MyFancyClient', one client_id for the web application, one for the mobile app and one for the server-2-server client (grant_type client_credentials). That allows you to have one client but with different access rules. Each client_id can be configured with different, valid SCOPE's, for example
Response to 3.:
I think this is more a question of available resources, maintenance, upgrades. On the other hand, you have to decide if it is acceptable to manage internal and external client through the same OTK instance (or gateway in general)
Response to 4.:
Trust can be created via user credentials (as in 'technical user that represents another machine), mutual SSL or via JWT. Those should be the most common solutions.
Maybe the list can be used as a starting point.
Regards,
Sascha