1) Scary Caribs who cannabalized often, and
2) Lack of water. Nowadays, 91 percent of Antigua and Barbuda is African, descendants of slaves. Recently they have asked for reparations for their ancestors' enslavement.
Most of the population is in Antigua, by the way. The name of the country is Spanish for "Ancient and bearded". Besides the two main islands, there are a bunch of tiny islets around the place. The biggest one of them is Redonda ("Rounded one"), named so by Columbus because from his POV it was, well, round (if he had circled it a bit more, he would have realized it's not that round). The island is completely uninhabitated and yet it's very popular, not so much by the country but by the Kingdom of Redonda, a micronation whose foundation history lies in legends. The kingdom not only has three different claimants because of it, but it has also "ennobled" many famous writers, actors, directors and others. (In case you don't know, the official queen of Redonda, and the whole country, is Queen Elizabeth II). The Antiguan/Barbudan flag