Segment Value Set Security Control for Ledger Set

We have enabled Segment Security control by ledger-wise and have also a ledger set. When we assign the ledger set to a user and they select a ledger from the LOV while making an entry, the segment security does not work for the ledger set. However, when we change the data access to a particular ledger, the segment security works correctly. Is this standard behavior, or am I missing some setup?