I'm inclined to agree with you, on the general grounds that applications generally contain if anything far too much code, but as always it depends. It depends for example on whether you want your collections to be read-only or read-write (although I guess if you want them read-only you could always return an unmodifiable wrapper), and on whether there are any other semantics associated with reading or writing the collection, and on whether it is sensible to grant access to multiple member collections separately considering they may leak into other threads for example.
There culd be partlly unmodifieable collections which allows for example only to remove elements like java.util.Map.keySet();.The collections could validate added objects. As you immentioned different users should have different view to providen collections, etc
I noticed, that in the praxis no one provide custom collection-implementation theire public APIs.