Doing it that way the HTML fragment that your servlet generates would have to effectivley have to load a complete JET page - e.g. one that loads the JET CSS, RequireJS and an main that loads all the resources.
Instead why don't you either:
1. Just open a JET dialog and inside of that have an iframe with a src set to the HTML output of the servlet
2. Call the servlet, grab the HTML as a view and use that with ojmodule inside of the dialog
3) Call the servlet, gran the HTML and then set the innerHTML of a div in the dialog
I'd only do 2 or 3 if I was sure about the origin / cleanliness of the HTML as there is a potential security issue if you are just echo-ing HTML that might have been manipulated by a bad actor.