there is a big difference in a complete manifold block that is behind your back that you can't see and has all of the low pressure outlets, vs. one that is only outputting to the face mask. In sidemount using a switch block you still have your inflator hoses, can run a dedicated normal second stage out of each bottle and just put a third breathing hose on the first stage and have that go to the block.
Alternative to the second stage on each bottle for sidemount, is to put a QD second stage on a suicide strap and a long hose and have that come from a Y on the output of the switch block, that way your secondary is always coming from the same bottle that you are breathing on and if it is done with a QD, in a block failure you can plug the secondary on one side and mask on the other.
Either way you have to add failure points, the point of the masks are to allow communication or protection from exposure. Saying that you shouldn't do it because of added failure points is BS because it is a calculated risk for a larger reward. Do we have to use ball swivels in sidemount, no. Does it add an extra failure point, yes, I've had one. Do I still dive with it because the added comfort is worth the benefit of having the added failure point, absolutely. CCR's have many more failure points than OC, but the added benefit is worth the potential risk, same with the isolation manifold in traditional doubles over a nonisolation manifold or independent doubles. We add a few extra failure points with an SPG instead of a J-valve but it is worth the risk, same with an octo instead of buddy breathing, or a drysuit instead of a wetsuit. Balance it for yourself but there are many benefits to certain tools, FFM's aren't for every day diving, but go ask Matthew Johnston if it is worth the added risk, or any of the hazmat divers and survey/videography teams that use them.