If we consider a set of threads running in the jvm, is there any way to detect if one of them is consuming more than a x percentage of cpu? In other words, I would like to take the control of the jvm's scheduler to avoid one thread could increase the cpu consumption of the jvm until 100%.
The RTSJ defines an optional "cost enforcement" mechanism which limits the amount of CPU execution time per release. If a thread exceeds this it is immediately suspended and a cost overrun handler (if installed) released. However immediate suspension makes this an extremely difficult mechanism to use effectively and to implement and consequently I'm not aware of any current RTSJ implementation that actually supports this notion of cost enforcement.
An alternative approach is to reserve the highest priority for a watchdog thread that can detect if the system does not appear to be making the right progress and take action - such as dropping the priority of the errant thread. However, how you go about detecting this and identifying the thread concerned is something you'd have to deduce in your application.
If running on a multi-cpu system where at least some cpu cycles are still available for other things, then you may be able to use the monitoring and management API's to monitor CPU usage of each thread (I believe that is an option for ThreadMXBean) and communicate that to a high priority management thread which could then take some action - such as dropping the priority.