The ukiyo-e painter from Episode 5 shares an English voice actor with Mao.
Despite the fact that the show says it is not an accurate historical representation, there are characters throughout the series who did exist or were based off other famous people. Much of the background information, such as clan heraldry, is drawn from history. The list includes:
Manzo the Saw is a lampoon of Hanzo Itami.
Ando, the fashion designer in "War of the Words", is an homage to Andy Warhol. His full name, Uohori Andou, is pronounced like a Japanese rendition of "Warhol, Andy".
When Mugen is looking at an ukiyo-e, he mentions "doing it with a squid". While we never see what he is looking at, it is most likely Katsushika Hokusai's infamous painting "The Dream of the Fisherman's Wife◊" (warning: Link is NSFW).
Okuru is being pursued by members of the Matsumae clan, which did exist and administered the northernmost island of Japan, Hokkaido. Okuru's appearance and clothing are clearly parallel to the real-world Ainu ethnic minority in Hokkaido. The Ainu were some of the earliest human settlers of the archipelago but were pushed further and further to the margins by the arrival of the Jomon peoples, the ancestors of the ethnic Japanese. By the Tokugawa period Ainu language and culture were in decline due to external pressure and today both are nearly extinct.
Jin's sword style, the Mujuushin ("Sword of No Abiding Mind") kenjutsu, was an actual esoteric kendo school in the Edo era. There are at least 3 recorded masters of the style, the last of whom really was named Mariya Enshirou, although there's no indication that he died in a sword fight. The philosophy of the school was "to go beyond conventional ideas of winning and losing". The "ultimate technique" shown in episode 26 certainly counts as the user must accept that they cannot end the fight without sacrificing their life.
The heraldic symbols (or mon) on Jin and Yukimaru's battle attire show they are members of the real-life Takeda and Hojo clans, respectively. Both clans were involved in the 3-way power struggle that preceded the emergence of the Tokugawa shogunate. The Takeda clan was wiped out in the conflict, making Jin even more of a relic from another era. The Mariyas were, incidentally, linked to the Takeda clan in the Warring States era.
In Episode 21, Johnnythe hermit claims (at first) to be Musashi Miyamoto. Miyamoto was a legendary swordsman and certified Kensei (a title given to a person considered perfect in swordsmanship). In addition, one of Johnny's lessons to Jin was fluidity of movement on the battlefield, a technique that Miyamoto triumphed throughout his life. This technique led to the prevalent use of one-handed swords for all samurai, thus popularizing the katana.
It should also be noted that Miyamoto is often quoted in Cowboy Bebop, a show many consider to be the ancestor of Champloo.
Kasumi Seizou, the Samurai who smells of sunflowers was part of the Shimabara Rebellion, which did happen and did involve the uprising of Catholic Christian Japanese. After the rebellion, Christianity was officially banned for decades. The government raids and the use of fumi-e to force secret Christians to reveal themselves are accurate to the time period.