At one point she asks Kazuma if he does in fact want power, and he (initially) turns it down. We don't actually see his flashbacks in full, at least in the anime, but we see two major images of her. Him swearing to protect her (which later is the question Ayano asked him on why he signed the contract, under Amor Piercing Question), and her getting fed to a youma. His eyes turn to red when he's possessed by evil, but if it were just anger fueling his power, it shouldn't be changing his power, or his eyes. Now, on the other hand, if the Spirit King that was the source of his power was tainted by evil, it would explain how Kazuma might be having problems like this when his heart is weak.