To be extra safe I normally make most holes 0.5 bigger than needed on 4mm parts, but for some holes where strength is important I have gone to 0.3 oversize through 10mm thick hardox (higher taper) and they have mostly been ok after a small amount of work with a needle file, for your design id say go 0.4mm or 0.5mm the intended size and it should be fine.