I'm not sure I understand your situation. DMStudio only runs on Windows.
If you are referring to running on Windows, then it could be that the user administrations rights have prevented the program from associating DXM files with DMStudio - something normally automatically done the first time you run DMStudio. So, you may have to manually assign DMStudio as the associated program via your Windows folder options.
Documaker is installed on linux server and it works fine and produces pdf when run through shell script. I want to create few new forms and am trying to launch Documaker for that. But am unable to open studio. Is there any other way to do development in linux environment.
The Documaker "engine" will run on Linux, but the development tool, DMStudio, only runs on Windows. It would be hard to develop resources manually outside of Windows by hand. You will need a Windows box (virtual or otherwise) in order to create your forms and sections with ease.