Hi David, if your EM 18.104.22.168 install is successful them OMS, agent are up and running by default, you don't have to manually start them. Look at the command to manager EM components from here - Starting and Stopping Enterprise Manager Components
That error message means you are running emctl out of the database bin directory, not out of the OMS bin directory. Try running it out of the newly-installed OMS home and you will have better luck. The emctl binary in the database home controls EM Database Control, the single-instance monitoring tool, which you will not be using now that you have EM12c installed.