Dear Navdeep,
This may not be feasible from OOTB perspective. Further, when you click on the Baseline link above it transitions onto the baseline revision list page.
A display condition cannot be applied onto it, neither could it be made secure. Probably you have already explored these options
If you only give an user read only access on project, i.e. not project edit management, then that user will not be able to see those links, however then again, the other aspects of the project will also be not available to the user.
Regards,
Samik