Viscount Lascelles was upset that his sons George and Gerald were in the room.
It was rumoured, and still is, that they had issues, and some even said George V, and Queen Mary sort of forced her into that marriage, and there the rumours stopped, their children talked on this regards, and they said they never saw any of that, and hinted what the movie shows, they both loved being in the country, and horses, and they thought there was more there, for any reason, those rumours remained but none gave any trouble. Same as his brother Bertie, then George VI, and Henry, Duke of Gloucester, we know George had his issues, and let's not dwell into the Duke of Windsor...
I think they all finished the show very early, the 20s, the 1929 stock's crashing, the 2WW, then add George being the right age to go and fight, you could get 20 season should you want it. And let's be honest, Maggie's career does not need much but the rest of the cast haven't done that much, except for Michelle Dockery. I, of course, don't think about those who left already, Dan and Lily.