I know what you mean about the edges of the masks (alphas) not being as clean as I'm used to. I think the Anti-aliased edges are sacrificed a bit in the balance to get clean-ish renders fast.
If you're not using a super complicated scene, I sometimes render a native C4D standard or physical pass with no lights or GI afterwards. The masks are super-sharp and the render only takes a couple of seconds.