It sounds like the increased memory requirements could have been causing the error message that you were seeing since you are able to run the simulation using a coarser mesh which does not require as much memory.
One way you could reduce memory requirements by about half is to take advantage of the symmetry of the structure in the x-direction, and set the x min boundary condition can be anti-symmetric. The following page shows how to choose the symmetry boundary condition:
You could also slightly reduce the x and y span of the simulation region to reduce the simulation memory requirements since the mode is well confined in the waveguide.
Since your original simulation file was already using a mesh accuracy setting of 2, I wouldn’t recommend using an even coarser mesh than this because numerical error increases as the coarser is made coarser, and this numerical error can build up over a long propagation distance. A 300 um propagation distance for optical wavelengths is relatively long, and I assume that you will eventually move on to simulating a structure which is not a uniform waveguide in the propagation direction so that you would need to propagate light for the full 300 um distance.
If using symmetry and a smaller simulation region x and y span does not sufficiently reduce the memory requirements for running the simulation, you may want to consider using either the 2.5D varFDTD solver or the EME solver in MODE Solutions which are more suitable for simulating long propagation distances.