I think you are working with Oracle EBS.
I don't think so that we can able to call the custom form on clicking button.
We can call the form from the standard form using both custom.pll and form personalization
In custom.pll using zoom function we can do this and in form personalization we can call the form by using form function, In both we call the form by creating a special menu in standard from and we used to call.
So please check further, and let me know if any i can help you any.