Looking at the documentation (of 14.4) at Studio Objects and Attributes under Prerequisites for Custom Object Development. It says
Verify that the attributes you create meet the needs of your users. After you create an attribute, you cannot remove it from the object. You can only deactivate it. The number of attributes affects the amount of time it takes to display pages. Do not create more attributes than you need.
Is that really so?
I just selected deleted a custom attribute for the project object in 13.3 and hit delete. After that there was no sign of it anywhere.
The other thing is I deactivated another and was kinda surprised to see that it remained in the list of active attributes and also in the edit property view and project list filter.