What determines the speed of a Soul Challenge?

While in battle, under certain conditions, Soul Challenges will occur, where a shrinking orange circle appears on the screen around a target white circle. Correctly timing pressing the B button while the orange circle is overlapping or within the white circle will result in some positive effects for your party.

However, I have noticed that not all Soul Challenges' orange circles move at the same speed. Often, it will move considerably faster, making it much more difficult to time the button press correctly, as seen below:

At a crucial point in a battle, successfully completing a Soul Challenge could be the difference between defeating the enemy or getting wiped out. Therefore I would very much like to know when to expect faster or slower Soul Challenges.
What affects the speed of a Soul Challenge? Is it based on enemy difficulty, party moral, HP or something else, or is it completely random?

I have been unable to find any information on this. The wiki has little information, and the in-game manual says only this:

Soul Challenge
Under certain conditions, the Soul Challenge gauge will appear on the screen after you use an Art. Press B while the shrinking orange circle is overlapping or within the white circle to successfully complete the challenge and gain additional effects. You don't have long before the orange circle shrinks completely and the challenge fails, so act quickly!