If you are trying to remove the ordinate set all together, then try showing the ordinate set name, right click the name in the main drawing view, and select delete.
In drafting mode, when the ordinate function is selected, the green arrows appear to indicate the reference origin to the ordinate system. But you can have multiple origins, how do you remove the unnecessary ones?