Trjtdtk

I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs.

To decide what action to perform, the network must perform a forward pass.

It performs forward pass in time $t$, but in the meantime the game could have already skipped like 10 frames or more (a varying amount).

The situation is worsened if I run, say 100 games at once on the same computer.

I don't have the ability to stop the game at each frame to do forwardprop. Anyway it wouldn't be possible were I to train, say a real-life robot to walk.

Question:

Should I stick to a 'fixed timestep' approach, only asking to provide an action every 0.1 seconds? While it computes next action, I could pretend the network keeps outputting the most recent action for all the skipped frames. Good idea?

If that's the only option, then should I avoid at all costs situations where forward prop takes more than the 'fixed timestep'? (more than 0.1 sec in my case) So it's better to choose say, 0.2 seconds just to be safe.

Seems quite unreliable - is there a better way to do it?

Is there a paper that explores the alternatives? (I guess such a paper will be about real-life robot training)

Your "fixed timestep" idea is actually very similar to a common technique called frame skipping. Instead of waiting a fixed amount of time, agents wait a fixed number of frames $k$ before choosing a new action. In the meantime, they repeat their most recently chosen action.

Frame skipping was included as part of the Atari 2600 Arcade Learning Environment. It was also used in the foundational DQN paper. Common values of $k$ are 3, 4, and 5. The value chosen depended on the game being played, since different games had important events happen at different time resolutions. In these papers, frame skipping enabled training to happen roughly $k$ times faster. So this is definitely a valid technique to try.

I actually think this would generally be less of a concern in the robotics application. Forward propagation usually happens much more quickly than real-world event timescales. As an example, Stanford famously applies RL to fly small helicopters, which requires considerable precision.

Finally, if your forward propagation really is taking too long, you should consider a faster architecture. One approach would be just to make your neural net smaller. You might consider policy distillation for compressing a large, trained network into a smaller one. Also, make sure you're not using some ridiculously slow activation function like sigmoid or tanh. ReLU is the common choice if you don't need a bounded output for a given neuron. If you do, I recommend softsign.

If your time bottleneck is actually in action selection, due to a large action space and using a value network, you should seriously consider switching to a policy-based method (e.g. actor critic). This would help because sampling from a distribution over actions would potentially be much faster than the $max$ operation involved in value-based methods. You can read more about this in Section 13.7 of Sutton and Barto's RL book.