Mixing 2 different mAh batteries is a very bad idea. Even mixing 2 different brands or aged batteries with the same mAh is a bad idea.

It all has to do with the internal resistance.
The battery with higher internal resistance will act as a resistor and will heat up. In no way any good.

On overvolting. Depends on the geartrain you want to use. as an overvolted motor wants to run higher in RPM, but will take more power when loaded- running at a lower rpm that the motor wants to run- If a 9.6V motor is 80% efficient on 9.6V and takes 10 amps, it will make 19.2Watts of heat. On its best.
When you add half of that voltage again -4.8V- the motor will put out 2.25 times more power, but the rpm will go up with 1.5 times (10000 rpm becomes 15000). Problem is obvious, can that motor take 43W of heat?

In the heavies it seems most motors can do that. but I dont want to pay for every Bosch, CIM or Litton that is burned out.

In total. Dont mix up batteries.
Overvolting can be done, but keep in mind, the higher RPM- with the same gearbox and wheels youll end up a lot faster, but will spend more time working the motor a lot harder, heating it up-