Standby is receiving logs even with primary listener down

Product -EE 19c db - hosted on linux servers

Query: Have 2 dbs configured in DG with Fast_start failover enabled. Now when the primary listener down, observer is not failovering the db to Standby.

Below warning is getting displayed in observer log and its trying to connect with primary only.

>Fast-Start Failover is not possible because primary last contacted the standby within FastStartFailoverThreshold seconds.

Even the standby db is getting logs from primary.

I understood that, since there is already connection established between primary and standby, there is no place for primary listener and they are running fine. But because of this, the failover is not happening to the standby db.