for the external faces, try to avoid to have "generic" curves on the Diff boolean result!
So rotate the sphere at the beginning... (rotation not shown here)