You can calculate the total noise using the Optical Channel Analyzer element, but to calculate the phase noise I believe you will have to use the spectrum analyzer element, either the optical or the electrical version.
I increased the sample rate and the square oscillations disappeared. Accurate results should not change with an increase in the sample rate, so I don’t think the square wave results you are seeing are reliable. If you would like to replicate the results in the paper, I think you will have to change the properties of the ring resonator, MZM and waveguide elements from their defaults to match their values in the paper. Some tuning of the gain elements will likely be required as well. It might also be helpful to try setting up this circuit using simpler elements, like the Fabry-Perot filter instead of the ring resonator, to make sure the simulation works as expected before using more complex elements
Let me know if you have any questions.