How do we do a controlled Skills assessment

Hello, Easier to make this an explanation of the problem. We are an offshore company that has to meet OPITIO and UKAS requirements globally for ROV Pilots/Techs among other things. This requires us to measure Competency/Skills (skill) for the technicians. They have up to 1,500 skills they have to master in their career.

Normally, I would do observation checklists, but each skill could be 15 minutes to assess or could be all day. On average, we have 2 hours per. In addition, we have a problem due to the observer constantly changing as we deal with who is available and a workforce constantly shifting between platforms and shifts based on work need.