If you are at "O" temper you are about as stressed relieved as possible. You don't need to do a stress relief at "O" temper prior to machining as the block should have heat treated after it was formed.
If you have done any secondary forming work on the block after it was brought to “O” temper, yes I would partially anneal it prior to machining.
You may want to do an intermediate "partial stress relief" after your rough in the part. This may or may not be necessary depending on the amount of hardening (strain) that is done by machining. With the proper tooling this should not be a problem unless you have thin sections.
The aircraft industry and others get by machining Al with very high speed ridged machines.