Agent inconsistent with its responses

Summary:

While testing the agent (debugging), the instructions have been clearly defined on the prompt, yet the agent starts misbehaving sometimes. It fails one or two test cases once in a while.

So how can this be rectified..? Can Monitoring & Evaluation (new in 25D) help with this??

Is there a way that we can do something like supervised learning, wherein we tell the agents which of its responses are correct & which are incorrect, and that it should work more like the correct ones.
(I don't want to tamper much with the prompt as changing it even slightly worsens the condition more often than not)