brcmfmac: make brcmf_p2p_detach() call conditional