In 1947, decorated war veteran Lucas "Luke" Jackson (Paul Newman), is arrested for cutting parking meters off their poles one drunken night. He is sentenced to two years in prison and sent to a Florida chain gang prison run by a stern warden, the Captain (Strother Martin), and a stoic rifleman, Walking Boss Godfrey (Morgan Woodward), whose eyes are always covered by a pair of mirrored sunglasses. Carr (Clifton James) the floorwalker, tells the rules to the new set of prisoners, with any violations resulting in spending the night in "the box", a small square room with limited air and very little room to move.
Luke refuses to observe the established pecking order among the prisoners and quickly runs afoul of the prisoners' leader, Dragline (George Kennedy). When the pair have a boxing match, the prisoners and guards watch with interest. Although Luke is severely outmatched by his larger opponent, he refuses to acquiesce. Eventually, Dragline refuses to continue the fight. Luke's tenacity earns the prisoners' respect and draws the attention of the guards. Later, Luke wins a poker game by bluffing with a hand worth nothing. Luke comments that "sometimes, nothing can be a real cool hand", prompting Dragline to nickname him "Cool Hand Luke".
After a visit from his sick mother, Arletta (Jo Van Fleet), Luke becomes more optimistic about his situation. He continually confronts the Captain and the guards, and his sense of humor and independence prove to be both contagious and inspiring to the other prisoners. Luke's struggle for supremacy peaks when he leads a work crew in a seemingly impossible but successful effort to complete a road-paving job in less than one day. The other prisoners start to idolize him after he makes and wins a spur-of-the moment bet that he can eat fifty hard-boiled eggs in one hour.
One day, Luke picks up a deadly rattlesnake from the grassy ditch and holds it up for Boss Godfrey to shoot with his rifle. Luke tosses the snake to the boss as a joke, before he hands him his walking cane. Dragline advises Luke to be more careful about his actions pertaining to the "man with no eyes." A rainstorm causes everyone to prematurely end their work. Before he joins the other prisoners in the truck, Luke shouts to God, testing him. On that same evening, Luke receives a letter stating that his mother has died.
The Captain, anticipating that Luke might attempt to escape in order to attend his mother's funeral, has him locked in the box. After being released from the box, receiving word that his mother's burial is completed and being told to forget about her, Luke is determined to escape. Under the cover of a Fourth of July celebration, he makes his initial escape attempt. He is later recaptured by local police and returned to the chain gang, but not before one of the three bloodhounds sent after him dies from heat and overexertion, including the strain of repeatedly going under the wire fences, in his attempt to trail Luke. After his capture and return, the Captain has Luke fitted with leg-irons and delivers a warning speech to the other inmates, explaining, "What we've got here is failure to communicate. Some men you just can't reach. So you get what we had here last week, which is the way he wants it. Well, he gets it. And I don't like it any more than you men."
A short time later, Luke escapes again by using string to shake a bush and deceive the guards about his location while taking a urination break behind said bush, afterwards visiting a nearby house where he uses an axe to remove his shackles. To keep the guard dogs from following his scent, he spreads curry powder and chili powder across the ground to send them into sneezing fits and overload their sensitive sense of smell. While free, Luke mails Dragline a magazine that includes a photograph of him with two beautiful women. He is soon recaptured, beaten, returned to the prison camp, and fitted with two sets of leg irons. Luke is warned by the Captain that if he ever attempts to escape again, he will be killed on the spot. Luke is now annoyed by the other prisoners fawning over the magazine photo and reveals it to be a fake. At first, the other prisoners are angry, but after a long stay in the box, when Luke is punished by being forced to eat a massive serving of rice, they come to help him finish it.
As further punishment for his escape, he is forced to repeatedly dig a grave-sized hole in the prison camp yard, fill it back in, and then be beaten. The prisoners observe his persecution, singing spirituals. Finally, as the other prisoners watch from the windows of the bunkhouse, an exhausted Luke collapses in the hole, begging God for mercy and pleads with the bosses not to hit him again. Believing Luke is finally broken, the Captain stops the punishment. Boss Paul warns Luke that he will be killed if ever he runs away again, which Luke promises in tears not to do. The prisoners begin to lose their idealized image of Luke, and one tears up the photograph of Luke with the women.
Seemingly broken, and again working on the chain gang, Luke stops working to give water to a prisoner. Watched by the disappointed prisoners, he runs to one of the trucks to take Boss Godfrey's rifle to him. After Boss Godfrey shoots a snapping turtle, Luke retrieves it from a slough for him, complimenting the boss for his shot. Luke takes one last stab at freedom when he is ordered to take the turtle to the truck. He steals the dump truck, as well as the keys to the other trucks. In the excitement of the moment, Dragline jumps in the dump truck and joins Luke in his escape. Later, after abandoning the truck, Luke tells Dragline that they should part ways. Dragline reluctantly agrees and leaves. Luke enters a church, where he talks to God and blames Him for sabotaging him so he cannot win in life. Moments later, police cars arrive. Dragline walks in and tells Luke that the police and bosses have promised not to hurt Luke if he surrenders peacefully. But instead of submitting, Luke opens a window facing the police and mocks the Captain by repeating the first part of his speech ("What we've got here is a failure to communicate."). He is immediately shot in the neck by Boss Godfrey. Dragline carries Luke outside and surrenders, but suddenly charges at Boss Godfrey and strangles him until he is beaten and subdued by the other guards. While Luke is loaded into the Captain's car, Dragline tearfully implores him to live. Against the protests of the local police, the Captain decides to take Luke to the distant prison infirmary instead of the local hospital, ensuring that he will not survive the trip. As the Captain's car drives away, a semi-conscious Luke weakly smiles while the tires crush Boss Godfrey's glasses. After Luke's implied death, Dragline and the other prisoners fondly reminisce about him. In the final scene, the prison crew is seen working near a rural intersection close to where Luke was shot. Dragline is now wearing leg irons, and there is a new Walking Boss. As the camera zooms out, the torn photograph of Luke grinning with the two women has been taped back together and is superimposed on a bird's eye view of the cross-shaped road junction.
