"Big Brother Complex" is a problematic term due to usage of "complex" in some circles to refer to someone who idolizes (not necessarily romantically) the figure in question (their brother). All of these tropes can also apply to a big sister as well.
The trope you're looking for could be any one of these:
- Aloof Big Brother: A literal big brother who has no interest or time for the younger sibling
- Big Brother Bully: Exactly What It Says on the Tin: the big brother bullies or torments the younger sib.
- Big Brother Instinct: A male character has an instinctive desire to be a big brother to smaller, younger, weaker or female characters.
- Big Brother Attraction: For some reason a character wants to marry their sibling.
- Big Brother Worship: "My big brother is the best and I think he's perfect."
- Brother–Sister Incest: A romantic or sexual attraction between siblings.
- Incest Subtext: The implication of same.
- Big Brother Mentor: Exactly What It Says on the Tin, a big brother figure who is also the mentor to the younger character.
- Father, I Want to Marry My Brother: A child too young to understand everything that marriage entails makes this announcement.
- Knight Templar Big Brother: A Big Brother Instinct carried to ridiculous extremes; the "big brother" will absolutely destroy anyone who has wronged or hurt their younger sib, no matter how minor the offense.
Other tropes that may be what you were thinking of:
- Big Brother Is Watching: An all-seeing governmental agency monitors everyone.
- Big Brother: The TV show.
If an internal link led you here, please correct it to point to the right page.