Suspect your only option, once the max message size is set at the WLS managed server level so you're protected from DOS attacks, is to get hold of the HTTP Content-Length header as the first action in your proxy service and choose to process the message or return immediately based on that.
Thanks for reply. I am looking at options outside of the proxy code , if that doesnt seems possible I would have to go with WLS level.
You cant change it outside the Proxy code and only at Proxy level. Either change it globally or if you want to do it at Proxy Level then validate it within Proxy Code.
If you still want to do it outside the Proxy Code and only at proxy level then you can expand HTTP transport using OSB transport SDK to a custom transport which has a limitation on message size as desired, not a valid use case for designing a custom transport but it will fulfill the requirement if that's all what you care about.