Ok I fixed my own problem. Instead of a full side boolean I made a box around the weird line and just did a boolean diff on that part (see yellow box where I did it), then extruded it back out and the line was gone.