Disable SFTP access without disabling ssh/scp

Strange requirement here, due to financial industry compliance....

Need to identify a set of users that would NOT be allowed to sftp but STILL be allowed ssh access. Due to the fact that we can have hundreds/thousands of OTHER users that should have no such restriction, we can't use the method that would use files with special group permissions that would ALLOW sftp to the users IN that file. We need the reverse, something along the lines of how normal ftp (ftpusers) file would work, where we identify the list of users that are NOT allowed access to sftp (but can still use ssh, scp, etc), and set THOSE users up