Just joined so sorry about the delayed response.
Another possibility is you may be driving your drilling operation too hard and over heating the aluminum. You don't see it but there may be chemical changes happening to your aluminum at the grain level. When the part is put in a cleaner that...