The film doesn't really focus on the ethics of time travel, especially the men keeping it secret from their wives.
Mary and Tim have completely different perspectives on many key moments in their lives which were supposedly "shared", since Tim has replayed those events several times. In particular, Tim's "firsts" with Mary are rarely her firsts with him.
Similarly, Mary has no awareness that those moments were at least partly "manufactured" by Tim rather than arising naturally - some might say "genuinely".
It's okay to change things before conception right up until the baby is born. Which means that from Tim's perspective, there may be many different versions of his offspring over that 9 months, each one erased when he makes a change. How horrifying this is depends on your attitudes about the personhood of fetuses. It's possible Tim himself is an alternate universe child.
Because of this caveat, this also means that if Tim himself alters something from anyone else's timelines from before 9 months, he might be changing their child and not know it.
Tim could have easily slept with Charlotte, then rewound time so it never happened. Tim was too much in love to do something like that, but what if he wasn't? And it doesn't stop there, what if another male relative developed a taste for committing all sorts of horrible crimes just for the experience, then changing the timeline so they never happened. To the outside world, those things never happened, but the time traveler would live in a world where he could get away with literally anything.
Tim's father minimizes the power of time travel by the fact that he can only travel within his own lifetime, so he can't "kill Hitler or anything". Sure, but what about all the disasters within Tim's lifetime? What about World Trade Center attacks, or the London bombings, or the Norway shootings? What about the Boxing Day Tsunami, the the Haiti earthquake, or the Fukushima disaster? Sure, these couldn't be easily prevented, but surely he could come up with some way to at least warn the authorities and have those warnings taken seriously, especially since he has an unlimited number of attempts to find something that works. And, since he can undo those as soon as he learns about them, it's not like it would undo all of history. We get no indication that Tim even thinks about any of this, choosing, instead to use his power to impress girls and avoid awkward encounters. Does this kind of power over time and space come with no responsibility?
Also, it seems odd that Tim and his father (at least when they were younger) wouldn't have used their ability to pass information about and play the stock market or win the lottery at least once or twice...
As long as there are two such time travelers alive at any given moment, they can be both be immune to most types of accidental death. If Tim died in a plane crash, his father just undoes the event. The only things that can get them would require irrevocable action begun far in the past (like James' cancer), or something that gets all of them at once. The film never really delves into how far these genes have spread. For that matter, the women can't do it, but can they pass it on to their own sons?
They actually can change events prior to their own birth. James can't kill Hitler, but he can certainly tell his father to go back and tell "his" father, repeat however many times it takes to get an able-bodied adult in the 1920's. This would certainly erase the subsequent descendents from existence, but if the stakes were high enough, every link in the chain might consider the sacrifice to be worth it.
In that same vein, the ability can serve as a sort of fail-safe for the entire human race. If there is any event so terrible that it threatens the entire species, they can daisy-chain as far back as they need to go, handing 'up' knowledge and skills from son to father. Massive asteroid on a collision course for Earth in 2030, with no possible way to alter its course or evacuate? Send instructions on how to kick-start the industrial and scientific revolution back to the 1300's and get humanity into the Space Age by the 17th century, that kind of thing.
