I need a description of the cone to use for the domain of the integral. In the \(x-y\) plane, the cone is a circle, so I can take \(x \in [-3,3]\) and \(y\) between the curves \(-\sqrt{9-x^2}\) and \(\sqrt{9-x^2}\text{.}\) For \(z\text{,}\) I need to start at \(z\) and stop at the cone, so I need the cone as the graph of a function of \(x\) and \(y\text{.}\) The function \(g(x,y) = 5 -
\frac{5}{3}\sqrt{x^2+y^2}\) describes the cone.
\begin{align*}
\amp \int_{-3}^3 \int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}}
\int_0^{5 - \frac{5}{3} \sqrt{x^2 +y^2}} xy(z-5) dz dy
dx \\
\amp = \int_{-3}^3 \int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}}
\frac{xy(z-5)^2}{2} \bigg|_0^{5 - \frac{5}{3} \sqrt{x^2
+y^2}} xy(z-5) dy dx\\
\amp = \int_{-3}^3 \int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}}
\frac{xy}{2} \left( \left(\frac{-5}{3} \sqrt{x^2 +y^2}
\right)^2 - (-5)2 \right) dy dx\\
\amp = \int_{-3}^3 \int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}}
\frac{xy}{2} \left( \frac{25(x^2+y^2)}{9} - 25 \right)
dy dx \\
\amp = \frac{25}{18} \int_{-3}^3
\int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} \left(x^3y + xy^3 -
xy \right) dy dx
\end{align*}
I can split this into three integral by linearity and deal with each integral in turn.
\begin{align*}
\amp \frac{25}{18} \int_{-3}^3
\int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} x^3y dy dx
= \frac{25}{18} \int_{-3}^3 \frac{x^3y^2}{2}
\bigg|_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} dx + \frac{25}{18} =
0 \\
\amp \frac{25}{18} \int_{-3}^3
\int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} xy^3 dy dx
= \int_{-3}^3 \frac{xy^4}{4}
\bigg|_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} dx - \frac{25}{2} =
0 \\
\amp \frac{-25}{2} \int_{-3}^3
\int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} xy dy dx
= \int_{-3}^3 \frac{xy^2}{2}
\bigg|_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} dx = 0
\end{align*}
All of these integrals evaluate to zero. In the evaluate of the bounds of the \(y\) integral, the \(y\) term is either \(y^2\) or \(y^4\) and the bounds are symmetric, positive and negative. When I put this into \(y^2\) or \(y^4\text{,}\) the negatives become positive, and the term becomes \((9-x^2) - (9-x^2)\) or \((9-x^2)^2 -
(9-x^2)^2\text{.}\) In either case, this evaluates to zero, making the remaining integral in \(x\) just an integral of zero.