<< So how do you explain the smooth result in Thea with the same file?>>

Thea and Octane are extremely different programs and use different smoothing algorithms. IMO, one of the weaker areas in Octane is that it requires a very high polygon count when using glossy or specular materials. Thea actually announced that they had rewritten their smoothing algorithms to get rid of the terminator artifact issue that many other renderers have. You can read about it here: http://www.thearender.com/forum/viewtopic.php?f=17&t=4331

<< The vase objects are certainly dense enough to show a proper result. It's a bug in Octane, a finer mesh does not fix the problem.>>

I seriously doubt that it is a bug in Octane. There are hundreds of amazing images that are coming out of Octane every month. Nobody on their forums is complaining about this issue. Why don't you post your issue on their forums. There are lots of friendly users that I am sure would be glad to help.