Using torch.nn.DataParallel with a custom CUDA extension
This is kind of unusual, as commonly “Batch” is exactly defined as all operations of the network being invariant along that dimension. So you could, for example, just introduce another dimension. So you have the “former batch dimension” in which your operation is not invariant. For this keep your current implementation. Then, parallelize over the … Read more