Trjtdtk

I'm simulating a very simple system, recommendation system, and I am running an actor-critic model to predict what item I should recommend next. The agent is learning and is doing just fine. However, I feel like I should be able to train faster than what I currently am.

I've looked into the A3C algorithm for training, and since I have an 8 core, 16 threaded Ryzen 7 CPU I assumed that I could simply do the hogwild updates, running several processes in parallel. However, when I run more than two processes in parallel the time it takes to execute an iteration per process becomes exponentially slower. From 0.04 seconds per iteration to over 3 seconds per iteration, when I go from two processes to three and beyond.

For now, I simulate one trajectory at a time and then I do an update to the model. What should I consider for speeding up the training? :)

I also have a powerful graphics card if that could be useful.

Parts of my code:

import torch.multiprocessing as mp

model = model_class(batch_size, position_dims, feedback_dims, args.actions_per_dim,
 n_keep_old_states, tensor_input_shape, accept_no_movement=args.accept_no_movement)

model.share_memory()

processes = []
for rank in range(args.n_processes):
 p = mp.Process(target=train_hogwild, args=(rank, args, model, d_model_arguments))
 # We first train the model across `num_processes` processes
 p.start()
 processes.append(p)
for p in processes:
 p.join()

and the training function run by each process:

def train_hogwild(rank, args, model, d_model_arguments, iterations= 10**7):

 env = StepEnv(user_type=args.user_type, accept_no_movement=args.accept_no_movement,
 actions_per_dim=args.actions_per_dim)

 optimizer = optim.Adam(model.parameters(), lr=args.lr)

 for i in range(iterations):

 start_position = env.reset()
 rewards, log_prob_list, actions, values = model.generate_trajectory(start_position, env, args.max_steps)

 value_losses = model.update(optimizer, rewards, log_prob_list, values)