Run multiple simulations serially


#1

I’m trying to set up a workflow for multiple users on the same machine. The way we are currently using Lumerical requires a user to call another user telling him when his simulation is finished, etc, and we obviously want to automate this.

I started using the batch file here https://kb.lumerical.com/en/index.html?user_guide_run_win_solver_command_line_direct.html. I’m using the run_simulations.bat file to avoid the command prompt.

What I thought of: When a user finishes setting up a simulation, he “submits a job” is by saving the simulation file to a “Jobs” folder then double clicking the run_simulations.bat file (which will be saved in the “Jobs” folder). In case another user has another simulation, all he has to do is save his file in the Jobs folder and double click the run_simulations.bat file. But what that will do is run the second simulation in parallel (check out a second engine license) instead of queuing the job to run after the current run. Also, how can I add to the batch file code that copies the files that have been simulated to another folder to avoid re-running these simulations?

Thank you


#2

Hello Aya,
Thank you for your inquiry. This is an interesting, if somewhat challenging scenario. Here are a few thoughts and suggestions, which hopefully may help direct you.

If we are looking for true multi-user job scheduling system, then the proper solution is a 3rd party job scheduling product (eg. PBS). The FDTD Solution’s job manager is a very simple job scheduling system, with limited ability to manage jobs from multiple users. It is possible to run FDTD simulations with most job scheduling systems, although the exact functionality will depend on the specific scheduling tool you select. We can assist in providing some technical support with configurations, such as explaining what command syntax is required to run the simulation. However, it is important to recognize that we offer limited support with installation or configuration of 3rd party products, such as job schedulers. More information on this topic is covered in the article below:
http://docs.lumerical.com/en/index.html?install_linux_running_simulations.html

There may be some simpler alternatives, if you are willing to accept a less sophisticated solution.

For example, the simplest method that comes to mind would be to have all the simulation files and the run_simulations.bat put into the “Jobs” folder, which is run once, or periodically, throughout the day. Then the completed contents can be moved to a new folder for analysis, and to clear out the “Jobs” folder queue ready for the next job(s). A slight extension of this idea is to modify the run_simulations.bat file to automatically move completed files to another directory for analysis. This will effectively clear the contents of the “Jobs” folder, providing a visual cue for the next user that they may run their job. To do this, we can append the following to the run_simulations.bat file:
move *.fsp <…\Completed_Jobs_Folder>
move *.log <…\Completed_Jobs_Folder>

I hope this information may be helpful. Thank you.


#3

Hello Aya,

I agree with ppham in that the most convenient solution is to use a queue system like Torque, for example. These software is extensively used in cluster environments and provides a very efficient solution for managing the computational resources among many users. My experience is that it is difficult to configure for beginners (like myself) plus you would need a scheduler, like Maui, which requires an additional configuration.

To implement the folder solution suggested by ppham, I would go for python. There is a very powerful module, called multiprocessing, where you can easily set-up a queue and select how many jobs do you want to execute in parallel and so on. I can provide more info, if you were interested in exploring this option.