We actually don't need to add or customize any class in osm-o2a-utility.jar. Instead, we can have our class file in any custom cartridge except solution cartridges(like TypicalSolution/SimpleSolution) because having the class file in solution cartridge won't get compiled/built.
Simple way could be, having the class file in any of the custom cartridge(as you know it should under the src folder, something like DSLCartridge/src/testPackage/abc.class) and the xquery being in same/different cartridge's resources folder.
From the XQuery inorder to invoke the class file, we have to declare the reference.
declare namespace callAbc = "java:testPackage.abc";