The CHARGE solver always starts with an initial guess (the initialization stage) before it starts the iterations at each bias point. If this initial guess is a good one then the solver is able to reach a solution quickly. However, if the guess is a poor one then the solver will diverge or take a long time to reach the solution.
The solver always starts from the equilibrium case (zero bias everywhere). So when you are running a single simulation for a bias of 2.5 volt, the system is far from equilibrium and the initial guess becomes a poor guess. In the other case where you are sweeping from 0 to 5 V, the solver easily finds the solution for 0 V since it is in equilibrium. In the next step the solver uses the result from the previous bias point as the guess. Since 1 V is much closer to 0 V, the guess is a good one and the solver reaches the solution quickly. For 2V it uses the result of 1 V as the initial guess and so on. Since at all bias points the solver starts with a goo guess, the simulation in the second case is much faster even though you are sweeping through more points.
Hope this clarifies the situation.