Yes, it can be done however, the same is not recommended
If you are using DNS then all you need to do is make the changes at the DNS level. And then at the cluster level just remove and add the scan resources back.
srvctl remove scan_listener
srvctl remove scan
And then add the same back
srvctl add scan -n <scan_name> [-k <network_number>] [-S <subnet>/<netmask>[/if1[|if2|...]]]
srvctl add scan_listener [-l <lsnr_name_prefix>] [-s] [-p [TCP:]<port>[/IPC:<key>][/NMP:<pipe_name>][/TCPS:<s_port>] [/SDP:<port>]]
Also refer to the following
Ref: How to update the IP address of the SCAN VIP resources (ora.scan<n>.vip) (Doc ID 952903.1)
Ref: 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained (Doc ID 887522.1)
Vandana - Oracle
Well you definitely don't want to go down to 1 SCAN Listener. What happens if the one and only SCAN Listener crashes? Your end users won't be able to contact that single point of failure to be able to connect to an instance. One of the reasons for implementing RAC is for high availability. Intentionally introducing a single point of failure would be counter-intuitive here.
Why do you want do to this anyway? Even if you have a 2-node cluster, having 3 SCAN listeners is not a resource problem. What is the reason you desire to make this change?