Borrowers are displaced Lilliputians.
Borrowers are what has become of Hobbits.
From The Fellowship of the Ring prologue:"Hobbits are an unobtrusive but very ancient people, more numerous formerly than they are today;...now they avoid us with dismay and are becoming hard to find. They are quick of hearing and sharp-eyed, and though they are inclined to be fat and do not hurry unnecessarily, they are nonetheless nimble and deft in their movements. They possessed from the first the art of disappearing swiftly and silently, when large folk who they do not wish to meet come blundering by...They seldom now reach three feet; but they have dwindled, they say, and in ancient days they were taller.""Those days, the Third Age of Middle-earth, are now long past, and the shape of all lands has been changed; but the regions in which Hobbits then lived were doubtless the same as those in which they still linger: the North-West of the Old World, east of the Sea.From The Borrowers:"...my brother said that, underneath, he thought they were frightened. It was because they were frightened, he thought, that they had grown so small. Each generation had become smaller and smaller, and more and more hidden. In the olden days, it seems, and in some parts of England, our ancestors talked quite openly about the "little people.'"
Connecting the dots here...Hobbits gradually shrank to the size of borrowers.
