I did that to at least two players today, we were 4 advancing in Stromgarde Keep and to be honest, between Elite Ogres and groups of Hordes I did not have time to chit chat, shift / click on names, found what we needed and moved on.
I also don't get replies to my whispers and I don't care, it's not like I'm sitting in SW waiting for a group, I'm questing, farming, PVP'ing...