If you only have to make a few, just bite the bullet and machine them. With a CNC it is not a big deal. However the 4-40 threaded holes are a pia. If anything the tapping will be a problem with potential breakage of taps. Use a bigger thread if possible, 10-32 would make it much easier...