Try to imagine making it without the chamfers and fillets. Boolean together the basic forms and then chamfer/fillet as a final step(s). I think of this similarly to 2D vectors where you are trying to approximate a curve and its choppy and bumpy because you have too many points when the solution is to have just the "right" amount of points and carefully controlled Bezier handles. In 3D it can be more complicated because there are more factors than just points and handles you have things like chamfers/fillets and such.