This is most likely due to the server within Domain Y, where you want to install an Agent, not having access to the Orchestrator server in Domain X.
To install an agent you first have to open a browser on the future agent server, access Process Automation GUI, and from the Installation palette on the Configuration tab, select Install Agent and allow the agent to be downloaded from the orchestrator server.
Does this work? I am guessing that you have attempted to install the agent directly on the server as the Domain URL value should have been copied from the Orchestrator and pre-populated:
Can you please attempt to install from the browser Install?