Получить все файловые дескрипторы открытых файлов

При исполнении кода на gpu возникает проблема:

48it [00:07,  8.79it/s]Traceback (most recent call last):
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 337, in reduce_storage
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files
50it [00:07,  9.96it/s]Traceback (most recent call last):
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/resource_sharer.py", line 149, in _serve
Traceback (most recent call last):
  File "my_siamese.py", line 182, in <module>
    send(conn, destination_pid)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/resource_sharer.py", line 50, in send
    for idx, data in tqdm.tqdm(enumerate(train_dataloader, 0)):
    reduction.send_handle(conn, new_fd, pid)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/reduction.py", line 179, in send_handle
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/tqdm/std.py", line 1107, in __iter__
    with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/socket.py", line 463, in fromfd
    for obj in iterable:
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
    nfd = dup(fd)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
OSError: [Errno 24] Too many open files
    idx, data = self._get_data()
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
    fd = df.detach()
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/reduction.py", line 185, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/eurvanov/anaconda3/envs/seamease/lib/python3.7/multiprocessing/reduction.py", line 155, in recvfds
    raise EOFError
EOFError
50it [00:07,  6.52it/s]

Исходя из лога, видно, что

OSError: [Errno 24] Too many open files

Хочу на каждом шаге мониторить, какие файлы открыты. Можно ли это как-нибудь сделать?


Ответы (1 шт):

Автор решения: Yury Nevinitsin

Поскольку это linux, можно использовать команду lsof -p pid. С помощью sysctl fs.file-max это ограничение можно менять, конечно, если это не ошибка в приложении.

→ Ссылка