Untrim the edges of the hole.

So delete any surfaces attached to those edges, then select them and hit delete.