"The issue is that we can't remove the "Project - View Tasks - All" from *everyone* for all projects" - yes you can!
What I've done in the past is create a new OBS, lets call it "Security OBS" with a single node of "Open".
Tag every single project in the system (apart from your special one) with a Security OBS level of "Open" - then grant the "Project View Tasks" access right at OBS level to the "All Users" group.
( in my solution we also put the secret projects into a "Restricted" level of the OBS and then just allowed special-groups access to that, but you would not need that I think )
Note that this method can prevent system admins from accessing the special-projects too because no-one has a global project view/edit right (although system admins could in theory grant themselves the right so its not water-tight).