See the safety and liveness arguments section here: Minimal VDF randomness beacon - #2 by Mikerah - Sharding - Ethereum Research. I believe that it is reasonable for the first two assumptions to apply ( Majority validator honesty and Minority validator liveness and thus for the 128 validator and 2^-33 safety failure probability to hold provided that these two assumptions apply, and thus independently of particular constructions. However, the RANDAO + VDF construction and plan seems reasonable and would recommend using this approach also for the beacon layer. With only 3 validators, then you have (1 - 2/3*1/4)^3 = 0.579, which is a not only non-negligible, but likely, probability of inconsistency/unsafety! A 2^-33 safety failure would correspond to one failure every 2^33 or 8589934592 slots; with 8 s slots this would be every 2^36 s = 68719476736 s = 2179 years.
Liveness is more important than consistency because if you have liveness you can get eventual consistency. (With Casper CBC and its safety oracles, users can ostensibly—it’s not on a main net yet—be able to choose probabilistically the level of consistency they wish to have, trading off on finality, with a higher probability of consistency resulting in a longer finality, and vice versa. So for high value transactions users could use a high consistency probability and longer finality, etc.)
While requiring more validators per session has the tradeoff of requiring more effort to start up and maintain the network, this has long term advantages of decentralization and security, so should not be lightly overlooked.