I have tried to create a custom work manager for my web app for weblogic 10.3.5 to hvae throttling control and its showing correctly in the console too. In fact it starts rejecting the concurrent requests which are >12 with 503 http status code. but apparently it start throwing the OutofMemory Exception after some time. After looking into the issue and comparing the two heap dump snapshots, we found that weblogic is not crearing the rejected requests from its memory, hence more the rejected requests, more we are moving towards seeing the OutofMemory exception.
See the below heapdump snippet, in this case rejected requested were 3177,since server started up, but those objects were never cleared up. Not sure that why weblogic needs to keep the rejected requests/responses in the memory.