A panel is not focusable by default. You can change that with
jpanel.setFocusable( true );
But you should think about your design here.
Why do you want to remove the focus from the textfield?
What is the purpose of making the panel focusable?
Does it make sense that tabbing through controls includes a panel in the components for the user?