Charlie states several times throughout the film that his father was abusive and was very cold to him growing up. After it's revealed that Raymond was sent to the mental institution due to the hot bath water incident with Charlie, so soon after their mother died, his hatred makes perfect sense. He probably resented Charlie for causing him to lose another one of his loved ones.
