Ways a Religious Character Could be Annoying

If it's a setting without overt supernatural elements, one way the religious person could irritate the atheist is by having prayer "work". Somehow, every time the religious person prays for guidance or support, some coincidence works out in their (and the atheist's) favor. Sure, it's probably just random chance, the cleric taking the time to think about the problem during prayer and maybe the placebo effect, but it's working way too often, and the cleric is getting kind of smug. (And then the cleric gets too smug, and things turn bad for them until they repent of the sin of pride.)

The religious character has been raised in an extremely religious setting for his entire life, so initially it's hard for him to even conceptualize someone who just doesn't believe in any gods. The atheist isn't exactly the most content with his life (for reasons unrelated to his lack of religion) and the cleric believes that religion will make him happier.

There are also people in his setting who would believe that his lack of respect towards the gods could anger them and cause them to hurt the people around him, but this isn't the main reason why the religious character does this.