Even in \(\RR^3\text{,}\) orthogonality already gets a bit trickier than just the rotations/reflection of \(\RR^2\text{.}\) Consider the following matrix.
\begin{equation*}
\begin{pmatrix}
-1 \amp 0 \amp 0 \\
0 \amp -1 \amp 0 \\
0 \amp 0 \amp -1
\end{pmatrix}
\end{equation*}
This matrix is orthogonal, but it isn’t a rotation or reflection in \(\RR^3\text{.}\) It is some kind of ‘reflection through the origin’ where every point is sent to the opposite point with respect to the origin. (The equivalent in \(\RR^2\) is a rotation by \(\pi\) radians). It isn’t even a physical transformation: it’s not something I can do with a physical object without destroying it. (If the object were, say, an inflatable beach-ball, this transformation would be equivalent to collasping the ball, turning it inside-out, and reinflating it that way). However, the transformations still satisfies the condition of orthogonality.