Make sure you are connecting to the right target database. Do as:
SQL> startup mount exclusive; ORACLE instance started. Total System Global Area 1610612736 bytes Fixed Size 2084336 bytes Variable Size 385876496 bytes Database Buffers 1174405120 bytes Redo Buffers 48246784 bytes Database mounted. SQL> alter system enable restricted session; sql> exit rman target / catalog rman@rmancatalog
export ORACLE_SID=<oracle SID> sqlplus / as sysdba startup force mount sql>alter system enable restricted session; sql>exit rman target / catalog sys/<pwd>@<catdb> rman>drop database including backups;
You set "restrict" mode two times, but never exclusive mode. So it's normal that you get "ORA-01586: database must be mounted EXCLUSIVE and not open for this operation"
SQL> startup restrict mount SQL> alter system enable restricted session;