Hi Han!
Me, personally? I think that our species, Homo Sapiens Sapiens, has a lot of phenotype variety, from dwarfs and pygmies to Giants. I think that there probably were communities in which giantism was more common. Especially if you think the average of humanity was probably 5'4" for most of history.
I've read Arab travelogues claiming that Northerners, Scandinavians and Slavs were giants with heights like palm-trees. Ibn Fadlan, whose exploits Michael Crition's novel Eaters of the Dead (and the Movie the 13th Warrior) is based on said that the Scandinavian Rus he met were as tall as Date Palms. It seems like an exaggeration until you put yourself in the shoes of someone probably 5'2" meeting communities of people 6'1" or taller. Someone that tall, if you are very short, could very well seem almost as tall as trees. He wouldn't have been lying if that was his actual perception. If everyone around you is 5' or a bit higher, and you travel to a land in which everyone is between 5'7" and 6' (which was average based on exhumed remains of Vikings) then they seem like giants
I'm a short guy, and when I'm around very tall people they seem like giants to me.
Now what if these people with medieval nutrition, during a mini-Ice age in the Middle Ages, were the descendants of people who also - given proper nutrition - were capable of growing to even great heights in warmer less ecologically demanding eras? When there was less travel in the world, and the populations were all heavily isolated and less likely to mate with outsiders. Before the construction of Imperial Roads (Roman in Europe and Asia Minor, or the earlier Celtic roads which preceded them. Persian, or Assyrian, etc in the Middle East) or advances in ship-making.
The thing about being large and being small is that if you are very small you can survive with less food, if you are larger you need more calories to sustain you. In times of catyclism this could be a problem. If you are too small then you're easy prey for people who are bigger than you, or animals.
So I believe giants probably really did exist, maybe races or sub-races of giants or people for whom giantism was a norm or frequent occurrences. Just like we have pygmies in Africa and the Nigrito people in the Philippines. IF either mated with non-pygmies or non-giants their traits would become recessive, accounting for occasional massive shortness and massive tallness. Which has happened in historic times, you have in South East Asia and the Philippines, or Africa, cases in which pygmies people married or mated with non-pygmies. For example. The same would have happened in ancient times for giants and dwarf or hobbit like people.
Even in today's developed world you see people with access to the same amount of nutrition sometimes growing no higher than 4'5" or sometimes 6'8" or taller. For non-pathological reasons. Sometimes extreme tallness runs in families as does extreme shortness.
Perhaps in the past there were entire communities of giants, who depended on specific environmental conditions, and died out in parts of the world where those conditions were less common. But their genetic lineages still remain in places. We have spread-out communities around the world of people for whom extreme tallness is a near norm. Like the Woloof in West Africa/Senegal and Mauritania, and Scandinavians and Dutch in Northern Europe.
If we go back to pre-historic or early-historic times, say 4000 or 5000 years ago, when humanity was much smaller, and lived closer together, I could easily imagine small nations and cities of giants. Or dwarfs. I think that's a really possible, even likely, scenario.