I assume the castor wheel has a boss on both sides of the middle bit. If you could remove one side of that boss, you could then mount the main gear the other way round, which would give you the space to have a bearing supporting the wheel on its axis.