If the characters are canonically related, there is usually a father cast as "Daddy Finger", a mother as "Mommy Finger", the two characters' son as "Brother Finger", his sister as "Sister Finger", and their younger sibling as "Baby Finger". However, if the family only has two kids, a pet or a completely different character may be cast as "Baby Finger". Sometimes, the family canonically has three kids, but the youngest is a girl, so she already has the role of "Sister Finger", or even the youngest child is older than three, which can result in "Baby Finger" being played by someone who appears too old.
Another problem arises in Finger Family videos centred around families with a Missing Mom or Disappeared Dad, an only child, sons but no daughters, or a combination. In those cases, they will usually use different characters (generally from the same work) to fill in the missing roles (a Finger Family video of Finding Nemo, for instance, might use Dory as "Mommy Finger", Deb as "Sister Finger", and Squirt as "Baby Finger"), and/or end up with a character being mis-gendered (e.g. one of the boys gets cast as "Sister Finger" or an unrelated character of the wrong gender gets cast to fill in a missing role).
If none of the characters are canonically related, it's basically a free-for-all, although generally their ages and genders align with the roles they're given.