To derive the continuity equation you will need use the conservation of mass principles as well as the Reynolds transport theorem. First, the conservation of mass principles states that a system is a collection of unchanging contents. In other words the time rate of change of the system mass will equal zero.
(Eq 1) $\frac{DM_{SYS}}{Dt}=0$
$M_{SYS}$ = system mass
The systems mass can be expressed in the following equation.
(Eq 2) $M_{SYS}=\int{_{SYS}}~ρdV$
As mentioned above, the continuity equation is also derived using the Reynolds transport theorem. The equation below represents the general form of the Reynolds transport theorem.
(Eq 3) $\frac{DB_{sys}}{Dt}= \frac{∂}{∂t}\int{_{CV}}~ρbV +\int{_{CS}}~ρbv·\hat{n}dA$
$ρ$ = fluid density
$v$ = fluid velocity
$V$ = volume
$A$ = Area
The purpose of the Reynolds transport theorem is to link system ideas with control volume ideas. In this equation there is a variable “B” which represents an extensive property of the system. In addition there is a second variable “b” which is the intensive property of the system. To derive the continuity equation B = mass while b = 1. As a result, when considering this, the Reynolds transform theorem will result in the following equation. This equation is for a non deforming fixed control volume.
(Eq 3) $\frac{D}{Dt}~\int{_{sys}}~ρdV= \frac{∂}{∂t}\int{_{CV}}~ρV +\int{_{CS}}~ρv·\hat{n}dA$
Incompressible Flow
The left side of equation 3 represents the change of mass in respect to time of the system. While on the right side of the equation the first part, $\frac{∂}{∂t}\int{_{CV}}~ρV$ represents the time rate of change of mass that is coincident to the control volume. Finally, the second part, $\int{_{CS}}~ρv·\hat{n}dA$, represents the mass flowing through the control surface. For steady flows $\frac{∂}{∂t}\int{_{CV}}~ρV=0$.
Next, the integrand, $V·\hat{n}dA$, is the mass flow integral which is used represent a component of velocity which is perpendicular to the control surface as well as the differential area. As a result $V·\hat{n}dA$ represents the volume flow rate that is moving through $dA$. This means the $ρV·\hat{n}dA$ represents the mass flowrate moving through $dA$. Using this information the net mass flowrate through the control volume can be determined. This in turn is the continuity equation for a steady state situation.
(Eq 4) $\int{_{CS}}~ρbv·\hat{n}dA = \sum{\dot{m}_{out}}-\sum{\dot{m}_{in}}$
Where $\dot{m} represents the mass flowrate.
(Eq 5) $\dot{m} = ρAv$
Finally, when the flow is not steady, than the continuity equation for non-deforming fixed control volume will become the following.
(Eq 6) $\frac{∂}{∂t}\int{_{CV}}~ρbV +\int{_{CS}}~ρbv·\hat{n}dA=0$
What equation 6 states is that in order to conserve mass than the time rate of change of mass for the contents of the control volume plus any mass passing through the control surface must equal zero.
Compressible Flow
The above is used when the fluid is incompressible. However, there can be cases when the fluid is incompressible. When the fluid is incompressible, $ρ$ is uniformly distributed. However, for a compressible fluid we will have consider a uniformly distributed fluid density to determine an average velocity.
(Eq 7) $\overline{v} = \frac{\int{_A}~ρv·\hat{n}dA}{ρA}$
Once the average velocity is determined the same continuity equations for incompressible flow can be used for compressible flow.