Indeed, to my knowledge, this is indeed not possible.
The authenticators are defined in the realm on domain level. You can have only one active realm in a domain and you cannot differentiate over managed servers.
Although you can have multiple authenticators, these are domain level.
And in essence, the user and the application do not bother about managed servers.
However, the only thing the authenticator does is to check if the user is valid within one of the authenticators, and to what groups they are assigned. So you could have two sql authenticators, one with users and groups for application A and one for Application B.
You'll need to define the access-rights through application specific roles that you map to the specific groups. So if you map the roles for Application A to the ApplicationA-groups and Application B accordingly, then I think you achieve the same as intended by Weblogic.
Weblogic always have had this restriction - security being applied per domain and not server instance - but this changed with the introduction of Domain Partitions, what you are looking for should be possible without Martien's clever workaround from above.
Check this link:
I haven't used it personally, though you can find plenty of documentation online.