That being said, you got some major problems from a storytelling standpoint. I know you and many others probably like the idea of just 1 man being the leader of a harem... but its already starting to stretch credibility of the story by insisting that the Human race will survive more than a couple of generations with only 1 man to serve as the primary genetic sire for the entire race. There's gonna be a huge genetic bottleneck (ya know, like what happened with the cheetahs that makes it hard for them to successfully reproduce), even if there weren't just one male; combine that with the effects of inbreeding, and humanity will for certain go extinct as each successive generation becomes more and more sterile because of these factors.
You could make it work, but you would need a flow chart and a lot of genetic luck.
Say you had three specimens: A(m), B(f), and C(f).
A(m) would breed with B(f) to create AB(m) and AB(f)
A(m) would breed with C(f) to create AC(m) and AC(f)
AB(m) could breed with C(f) with no inbreeding issues. This would give aBC(f) and aBC(m)
AC(m) could breed with B(f) with no inbreeding issues. This would give aCB(f) and aCB(m)
Since there are two degrees of separation; A(m) could breed with aBC(f) with minimal inbreeding issues and create AbC(f) and AbC(m)
Since there are two degrees of separation; A(m) could breed with aCB(f) with minimal inbreeding issues and create ABc(f) and ABc(m)
ABc(m) and AbC(m) could then breed with AB(f) and AC(f).
This is how selectively breeding rabbits was
(yes I used to do that, don't judge). You find a male and two females that have traits that you desire and breed them with two degrees of separation. Now usually when you're breeding animals...rabbits or not you want to add new blood every few generations or so or you will start having problems. However, that can be mitigated in this scenario by extrapolating this system out to 3 or 4 other females. If aBC(m) reproduced with D and aCB(m) reproduced with E. You would have three degrees of separation and that any extre degree of separation you can squeeze out would make humanity far more likely to survive.
Now, keep in mind that I'm not saying that you're not going to have problems:
The age of A by the time aBC(f) comes around to being old enough to have kids. He's gonna be like 54 by the time he bangs aBC(f) assuming that he was 18 when AB was conceived and 36 when AB reproduced with C(f) to create aBC(f). We can technically fix that by making the ages younger which would probably happen in an extinction-level scenario where there's only one male, but that's not allowed on the site. It also assumes twins although that's fairly rare. The twin thing can again be fixed a bit with more than a couple of females who can all be pregnant at the same time.
You are basically going to have to plan out the reproductive future of your denizens for literal generations....hundreds of years. So that nobody is stuck banging their half-sister. That is no easy task and assumes that everyone is going to go along with it and everyone in the future is going to go along with it.
You would have a genetic bottleneck. Genetic disorders which are normally recessive are going to be problematic. This is where genetic luck comes into play. You would essentially have to bar the people with obvious genetic issues from reproducing...otherwise up to half of the next generation could have the genetic problem. Then much more in the following generation. The problem is that not all genetic problems are visible and while certain genetic conditions aren't necessarily fatal. they could have a profound effect on future humans. Such as your heart bursting when you hit 30, or the aforementioned problems with cheetahs. These genetic problems would also occur several generations in (assuming they followed the plan) and originators of this hypothetical single-male line are all dead of old age. They may not know nor or care to stick to their ancestor's plan.
The best-case scenario is likely one where humanity survives with persistent genetic issues that makes life harder for them. The worst-case scenario would be losing the genetic lottery and getting a genetic disease that just wiped out most of a generation and then the next generation would likely be the last.
As far as the direct effects of inbreeding are concerned though, those can be mitigated. Inbreeding Depression will still happen, but the idea would be to drag out the effects of it gradually until the population is large enough and natural mutations can occur to begin increasing genetic diversity again.
Edit: It's called Linebreeding.