From a database perspective - you could add a unique index on 'name' - this would force the database to throw a unique constraint violation.
From a rules world -
1) you would create a parent table with 2 columns - name and count.
2) Then create a virtual relationship (schema page in LAC) between your table and this new parent table. The join is on name
3) Add a rule on this new parent table to count children
4) Add a rule - managed parent on the child table (select the parent table relationship) - aka Insert Parent if Not Found
5) Add a Validation on this new parent table - if (row.count > 1) {return false;} else {return true;}
message: This is a duplicate name: {name}
Now when you insert a name - the rules will insert a name into the parent (count = 1) - if a second name is inserted - the count > 1 and the validation rules fires with a message (instead of a database constraint validation error).