These are what we call the 'YMMV items.' Things that some people find in this work. We call them 'your mileage might vary' because not everyone sees these things in the same way. This starts discussions in the trope lists, a thing we don't want. Please use the discussion page if you'd like to discuss any of these items.
Genius Bonus: Tolkien often has obscure linguistic in-jokes that are not noticeable at all unless you know; the "smaller works" especially are rife with references ranging from mythology to personal jokes.
Being on the evil side doesn't equate to being Always Chaotic Evil. In The Lord of the Rings, the dark-skinned Southrons serving Sauron (by extension, all "enemy" humans throughout the Ages) are strongly implied to be not really evil at heart, but rather deceived and manipulated. For sure, during wartime human enemies are to be fought (in self-defense), but after Sauron is no more, they are pardoned and peace is made. (Also, Aragorn doesn't claim their lands by right of conquest, but sticks to Gondor's old borders.)
The Silmarillion and Unfinished Tales of Númenor and Middle-earth provide historical context for the hostility of the men of the south and east towards Gondor. In an earlier Age the Númenóreans, ancestors of the men of Gondor, impose their rule and oppress the humans of the East and South. They also despoil their forests and take captives for human sacrifice after Sauron corrupts them as well.
Not all "non-white" humans serve evil. Many Easterlings are allied with the Elves in The Silmarillion. Although some betray them in battle, the rest remain loyal and are all killed or scattered for it. Unfinished Tales and Tolkien's collected ''Letters'' also state that in later Ages, Blue Wizards were sent to the South and East to support and spread rebellions against Sauron just as Gandalf the Grey was sent to the West to help defend it from him. The latter source also says Gandalf himself also (probably) went into the South spreading "Resistance" against Sauron.
It's also worth noting that the Easterlings were also described as being quite technologically advanced in terms of metal-working, matched only by Gondor, and the Easterling Variags are perhaps one of the most bold examples of Worthy opponents in fiction, given how effectively and long they fought Gondor after Sauron's fall and how the men of Gondor began to view them with respect.