What could cause "Setting - Posted Job Apply Online Enabled KB33250" to not work?

Summary:

My client is in the process of migrating outside of Taleo Enterprise soon and in the meantime, want to continue posting new requisitions for visibility on their career site without allowing application on those new postings. I applied the instructions described in KB33250 and the "apply now"
button is still visible under requisitions whose checkbox in posting/sourcing is unchecked. What could cause this issue and how could I actually achieve what the document says should happen?