Sometimes, some processes may occupy GPU memory even if no process is shown in nvidia-smi. (Especially if using a multiprocessing dataloader, pytorch 0.3) How to find the true processes behind?

The lsof command in Linux displays in its output information about files that are opened by processes.

1. lsof /dev/nvidia0
2. kill them