Hi Marko,
Since APM version 9.6 license files are no longer required so there is no need to get a new license for 10.5
You didn't mention if you use CEM or not, if not I would suggest doing a new clean install of 10.5 on RHEL 7 and then BEFORE starting the Enterprise Manager on 10.5 copy the existing smartstore data from the 9.5 environment the smartstore will then be automatically upgraded on startup. You can then also copy any custom management modules etc
If you do use CEM then there is also the issue of the APM database that needs to be upgraded and migrated to the new Postgres release if you use Postgres -
Upgrade the APM Database on PostgreSQL - CA Application Performance Management - 10.5 - CA Technologies Documentation
if you don't use CEM then the data in the APM database is not of so much importance (just App Map) and will be repopulated from the agents as they are used.
Hope this helps
Mike