""""""""" It's usually not too much work because I can use the broken object to snap to,""""""""""""
Don't do that! What you want to do is alter your creation method to not create broken objects.
With Booleans, the "seam edges" of objects involved in the Boolean is most often the culprit. When you are making little cylinders to Boolean out a hole (Or using circles too), Make sure to rotate the seam edge OUT of the way, or farther away from other edges, to not create the little slivery result
Here is a video of repairing your object. I dont think you really needed a repair video (Others may find it useful) Mostly I speed through to the end where I wanted to make the "SEAM EDGE" point I was talking about
