I am using a server with 256 cores, which FDTD is also installed on it. According to our licence I directly connect to software, set my simulation and run it. Based on my evaluation having only 8 cores is the ideal case to run my simulation in fastest way. Now I want to do parametric sweep on this simulation. So I want to divide my 256 cores into groups of 8 (256/8 = 32) and run multiple programs in parallel. How can I do this, is it possible to seperate the cores in this way? How can I be sure that the the parallel programs doesn’t intersect and run on the same cores? Do I need a new licence?
Welcome to Lumerical Knowledge Exchange.
To answer your inquiries;
Based on my evaluation having only 8 cores is the ideal case to run my simulation in fastest way. Now I want to do parametric sweep on this simulation. So I want to divide my 256 cores into groups of 8 (256/8 = 32) and run multiple programs in parallel. How can I do this, is it possible to seperate the cores in this way? How can I be sure that the the parallel programs doesn’t intersect and run on the same cores?
Yes you can try and utilize all your resources (256 cores) to run parameter sweep. You can add resources in the Resource Configuration Manager with
localhost as the Hostname for all the resources you are adding.
If you are running Windows, you can use the processor binding option to bind each process to a specific core. (not support on our default MPICH2 on Linux/macOS - these systems will autoselect which cores your simulation job will be running on).
Once you have configured the default launching preset, simply set the processes to 8 and “Duplicate” your resource.
Depending on the number of sweep you will be generating. 20-24 resource line might work? Try to run some tests on how many resource lines would be optimal for your sweep. i.e. 32 or lesser resources, same method to obtain the optimum FDTD resource configuration for a simulation. the individual sweep’s log file should show the FDTD speed (mnodes/S).
Do I need a new licence?
If you are only running the sweep on 1 machine, i.e. all
localhost on your resource configuration as shown above, you do need additional FDTD solver licenses.
Hope this helps.
Thank you for your response, I followed your response and I added 10 resources and allocated 8 processes of localhost to each one and I ran the sweep. But I saw that simulation times is proportionately slower. Is it because all of them using the same FDTD engine?
Which licence should I purchase ?
And also it is not clear for me that when I add multiple resources (as you described) how to allocate each resource to a desired sweep parameter, for example suppose I want to do the simulation on a sweep parameter on 20 values I know for 10 of them I need less computational power and they converge faster. So I create two groups of resources, group one with for example 4 processes and group two with 8 processes. Then I allocate the group one to the sweep values that converge faster and group two to the values that are slower.
thank you very much
What operating system are you using on this machine with 256 cores?
The slow down is typical since FDTD will try to use all available bandwidth between your CPU and RAM when running on the same machine.
When running sweeps/optimizations we highly recommend to do these on multiple computers/nodes on your network. We have these KX article for details.
The above scenario would require additional “HPC FDTD Acclerator” licenses, depending on how many machines/nodes you want to run simulations at the same time.
When using a single machine, you cannot run different simulations at the same time and use different resource settings. If these are all parameter sweeps you can run them directly in sequence using a batch/script file from the command line and indicate the processes you want to use for each simulation file or batch of simulations…
# example for Linux shown below cd /location_of_your_specific_sweep_files /opt/lumerical/2020a/mpich2/nemesis/bin/mpiexec -n 8 /opt/lumerical/2020a/bin/fdtd-engine-mpich2nem -t 1 simulationfile1.fsp /opt/lumerical/2020a/mpich2/nemesis/bin/mpiexec -n 16 /opt/lumerical/2020a/bin/fdtd-engine-mpich2nem -t 1 simulationfile02.fsp # or use the fdtd-run-local.sh script to run all simulation files on the current directory cd /location_of_your_specific_sweep_files /opt/lumerical/2020a/bin/fdtd-run-local.sh -n 16 *.fsp
Hope this helps.