If you've already designed for a floating shaft, why not use a keyway for your drive? You can pick up castor wheels with keyways in them for fairly cheap, and belting online can cut keyways for you at a very reasonable cost.
Printable View
That might be an option, thanks Robert.
I have all the parts for the current design so I'll give it a go as-is, but if it doesn't look reliable I'll have a rethink.