First, King Charles is accused of removing Prince Harry and Meghan Markle‘s kids from the narrative and now their parents are looking to relocate them to a remote island.

It’s not quite as cushy as it looks being a Royal kid, that’s what we’ve been learning lately. What with Prince William and Kate Middleton taking their three children out of London in a desperate attempt to avoid the Royal sibling curse that struck William and Harry. Now, the Duke and Duchess of Sussex are allegedly planning on something a lot more extreme by whisking Prince Archie (5) and Princess Lillibet (3) away to a secluded island somewhere.

But is Harry right to fear for their safety so?

Prince Harry avoiding the U.K. as he “doesn’t feel his family is safe”

We thought Harry and Meghan’s general avoidance of the United Kingdom was simply because of the sour split from his family, but there’s reportedly a more significant reason behind it: the prince fears for his wife and children’s safety when he’s back on his native soil.

“Harry’s case is he does not feel safe. He doesn’t feel his family is safe when they’re in the U.K. because they do not have police protection officers with the Metropolitan Police with him,” Royal correspondent Cameron Walker claimed to The Mirror. “So perhaps that is the main reason why Prince Harry isn’t bringing Meghan and Archie and Lilibet over to the UK because he can’t guarantee, he would say he can’t guarantee their safety.”

As a reminder, when Harry and Meghan relinquished their Royal duties and moved to the States, they lost the right to have automatic security detail so now they are required to inform the Metropolitan Police 28 days ahead of any visit in order to receive any security protection. His family’s wellbeing when in the U.K. was a major reason behind the Sussexes’ emigration to the U.S. in the first place, as Harry detailed his concerns for his “sanity and physical safety” in his memoir, Spare.

Harry’s fears seem to stem from the backlash in the media against himself and Meghan, and perhaps even from the death of his mother, Princess Diana, an incident which involved the press, in 1997. That said, there has been no attack on a Royal on British soil in modern history, unless you count Princess Anne’s recent incident with a belligerent horse.

Given Harry’s feelings, it’s probably no coincidence that Meghan’s plan for a summer getaway this year involves shipping the family off to a “remote island” and away from the public eye. According to a source, Ok! reports that now that Independence Day is out the way Meghan wants to take the kids away for the summer months to a secluded, private retreat where they can wallow in anonymous luxury.

“As usual, she and Harry celebrated with the kids, but she wants to use the next couple of months relaxing and having fun after a stressful year so far,” the source shares. “Meghan wants to get away with Harry and the kids to a remote island so they can just have fun as a family without any outside pressure and be in complete luxury.”

Clearly, there might be some big downsides to being a Royal kid, but there are definitely some big perks, too.