Actually, the problem has to do with discharging the pack. At one point one set of batteries will have charge left, while the others are depleted. Carrying current through depleted batteries can cause major damage to them, you could even reverse their polarities because of it.
You wouldnt be able to charge them either, as one set is fully charged way before the other.