Saying that there shouldn't be a mosque at ground zero because of extremist Muslim attacks is the same as saying there shouldn't be a Christian church in Oklahoma city because of extremist Christian attacks. Extremists are extremist, no matter what religion they follow, and do not represent the whole of that religion.
After his diatribe, Whoopi Goldberg and Joy Behar walked off the set.
O'Reilly's racist rant came after a particularly tense interview, during which he chastised Goldberg about her seemingly unfriendly or stand-offish body language and just generally acted like a creep. Whether or not Goldberg and Behar should have left the premises is debatable, but if it were me, I probably would have done the same thing.