Here's a (mostly) repaired version that booleans with the rectangle ok.
The problem was that there were some really tiny edges and surfaces. Some of this junk was along the edge of the rectangle and so was confusing the boolean. I found the problems using this plugin:
I then separated out the bad surfaces, trimmed off the junk, entirely remade the large cylindrical surfaces, and then joined everything back together. I left in some junk outside the area of the boolean. One problem is marked with a red point.
""""""""" It's usually not too much work because I can use the broken object to snap to,""""""""""""
Don't do that! What you want to do is alter your creation method to not create broken objects.
With Booleans, the "seam edges" of objects involved in the Boolean is most often the culprit. When you are making little cylinders to Boolean out a hole (Or using circles too), Make sure to rotate the seam edge OUT of the way, or farther away from other edges, to not create the little slivery result
Here is a video of repairing your object. I dont think you really needed a repair video (Others may find it useful) Mostly I speed through to the end where I wanted to make the "SEAM EDGE" point I was talking about
Andre repaired.3dm