The code states that 'openings ... shall preferably be circular, elliptical or obround'. So even rectangular openings are allowed and can be checked according to code, starting at UG-36.
However the opening in the vessel wall does not need to be rectangular for a rectangular sight glass.
From the mechanical point of view rectangular openings are a source of stresses. The arc of these have a big effect on the stresses suffered by the vessel, that is why the codes go to circular (optimal) or at least elliptical ones. in any case despending on the material/pressure these can be used but always taking into account that it is not the best "mechanical" solution