Hi Petr, I've been looking at this and it is a little bit difficult to figure out.
But to start with I can give you a different way to diagnose what is happening.
There are 2 different styles of shelling - one is when you do it to a solid, and one is when you do it to something that is not closed. For the solid one the shell is the equivalent to an offset plus a boolean operation. The non-closed case is an offset combined with lofted/ruled surfaces between the open edges.
So sometimes you can look at just using offset as a way to diagnose what is going wrong. In this case, select everything except for the bottom face (invert after selecting the bottom), and do offset distance 5, and check "flip" (The default for shell is to go inward, for offset outward).
You can see the problem corner there. Also it is pretty subtle but if you look closely at the edges that I marked with red dots next to them, they look a little heavier than a normal edge, this is because they are naked edges not joined between the surfaces there.
Anyway the thing is that I'm not quite sure yet what the problem is. At first I thought it might be a bad intersection calculation between the 2 top surfaces because they are relatively close to being tangent, sometimes tangent intersections are difficult to calculate. But after going in there and trimming one to the other and looking at the result, that doesn't seem to be wrong.
So what I'm thinking now is that the proper result should actually look like the surfaces that are existing there - but the problem is that there are 2 vertex junctures in that area instead of only one. It seems like the offset mechanism thinks that there should only be a single common vertex point. So this might be an area that causes failure, single vertex branching into multiple vertices in the offset result.
After fussing around with extracting trim edges, trimming some of the ends off and untrimming and retrimming surfaces, I got this (model attached):
Which appears to be the correct result. So yeah I think the problem is the offseter getting confused and assuming the original vertex offsets to another common vertex which is not the case here, one portion diverges to another vertex in this location. I think...
Anyway, hopefully doing the offset method might give you some way to diagnose and repair shelling failures like this in the future.
- Michael