There definitely was a bit of a scandal. Now I believe I heard some more substantiated rumours, but I am certainly not sure. A quick search on google didn't yield any information other than the rumours that he was either illegitimate or a test tube babe. In any case I would wash his mom off of any accusation of adultery. She simply had to produce a child, even if it was out of thin air. Everything else was a disaster. Dreadful times when the woman was officially blamed for the childlessness of a couple.
Yes, the pressure had to have been maddening when there was no baby. That's all women were for back then, perhaps companionship, and definitely housekeeping, sex, and babies, and the latter even more so the higher up on the social-political ladder one was. Can you imagine the pressure her Aunt Louise (Argyll) was under with Victoria?
Otherwise, there was no outright pressure as there would be for the direct heirs at the time. She was the younger daughter of the British monarch with an older brother and a sister, both with children, he the younger son of the Danish "mob," and neither of them had prospects on inheriting unless there was some sort of massive disaster. If they didn't have kids at that point in time, I don't think anyone would have batted an eyelash other than to say 'the poor darlings.' The Norwegian throne wasn't even a speck on the horizon at the time.
But the pressure's still there for the royal families even today. Just look at the tabs after these folks are married... but now I'm really veering off topic. Whoops.
