If you want to keep the pulleys as small as possible, that means using a timing belt. All belts have a minimum pulley radius and the thinner the belt, the smaller the radius. Just keep in mind that the smaller the motor pulley is, the more likely it is to slip and strip the teeth off the belt.

In theory, brushless motors should over-volt better than brushed motors; they are more efficient and generate less heat, plus they don't have brushes to burn up. The main problem is still overheating; most brushless motors don't like to get over 60 or 70 deg. C. If you used a motor that is power full enough for the job and has some built-in cooling, then over-volting shouldn't be an issue.

Keep in mind that the motor you linked to has a maximum power rating of 2,400W and the continuous rating will be half that or less.