Strange issue with ISR and Cases with VBS/Redwood

We are trying to explore functionality to implement Internal Service Requests and Cases.

We are trying to use the Category (for ISR) and Case Type (for Cases) to hide fields using Visual Builder display logic rules. For now it's just simple rules to hide unwanted fields. The bigger picture is to control field visibility based on category / case type.

For Cases, when we setup the rules, they only apply to the first layout that we modify, but not the others. If we apply the rule to the Case Summary (Edit Mode), then it works but if we apply the exact same rule to the Case Summary (View Mode) or the Case Spotlight, it doesn't work. It only works on the first layout we apply the rule to. If we switch the order around, whatever had it first works and the others don't.