We went here right when it opened before heading to nearby Pena Palace - it was empty. The night before it rained slightly so there was a slight fog in the forest path you have to walk to get to the cliff wall. There isn't much of a castle left, but there's a small garden, and the wall you can walk (if you don't mind heights). It was pretty magical that day with the plants glistening, fog rolling out, and the sun beams just peaking through and no one else around.
Then the line at Pena was crazy. You slowly trudge along. But the details were worth it for me.