""""""""" It's usually not too much work because I can use the broken object to snap to,""""""""""""

Don't do that! What you want to do is alter your creation method to not create broken objects.

With Booleans, the "seam edges" of objects involved in the Boolean is most often the culprit. When you are making little cylinders to Boolean out a hole (Or using circles too), Make sure to rotate the seam edge OUT of the way, or farther away from other edges, to not create the little slivery result