In the Stranger's confrontation with the town's hired guns in the bar, he grabbed a bottle in order to test their reaction. He knew who would probably draw first, who was hesitant, who was scared, and how fast they were. So when he tells them, "I'm faster than you'll ever live to be," it wasn't just bravado, it was the truth based on information that was given.
Callie's failure to injure the Stranger, despite shooting multiple times at him from less than 10 feet away, can either be chalked up to extremely bad aim or the possibility that the Stranger is not human.
If the Stranger is the ghost of Marshall Jim Duncan, and Mordecai finally giving the murdered Marshall a grave marker with the words "Rest in Peace" may be part of why the Stranger shimmers out of existence the way he shimmered inand it's possible that if the Townspeople had at the very least done that in the first place the Stranger might not have turned up at all.