Выдает ошибку ConnectionError requests

Question


    Exception in thread Thread-7:
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\connection.py", line 61, in create_connection
        for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\socket.py", line 748, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno 11001] getaddrinfo failed
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
        self._validate_conn(conn)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
        conn.connect()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 300, in connect
        conn = self._new_conn()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x000002B627AAC898>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 449, in send
        timeout=timeout
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py", line 436, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.e-register.am', port=443): Max retries exceeded with url: /en/companies/7 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x000002B627AAC898>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
        self.run()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
        self._target(*self._args, **self._kwargs)
      File "c:/Users/user/Desktop/project/newsite/parser.py", line 19, in main
        r = requests.get(url1)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 75, in get
        return request('get', url, params=params, **kwargs)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 60, in request
        return session.request(method=method, url=url, **kwargs)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 533, in request
        resp = self.send(prep, **send_kwargs)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 646, in send
        r = adapter.send(request, **kwargs)
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 516, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.e-register.am', port=443): Max retries exceeded with url: /en/companies/7 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x000002B627AAC898>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
    
    Exception in thread Thread-65:
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
        six.raise_from(e, None)
      File "<string>", line 3, in raise_from
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
        httplib_response = conn.getresponse()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1321, in getresponse
        response.begin()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 296, in begin
        version, status, reason = self._read_status()
      File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 265, in _read_status
        raise RemoteDisconnected("Remote end closed connection without"
    http.client.RemoteDisconnected: Remote end closed connection without response

Мой код:


    import requests
    from bs4 import BeautifulSoup
    import csv
    from threading import Thread
    from transliterate import translit, get_available_language_codes
    from googletrans import Translator
    
    url = "https://www.e-register.am/en/companies/"
    
    
    def main(hjk, ur):
        info_first = []
        fulll = list()
        translator = Translator()
        for i in range(1, 130):
            url1 = ur + str(i * hjk)
            info_only_one = list()
            for_ful = list()
            r = requests.get(url1)
            soup = BeautifulSoup(r.text, 'html.parser')
            items = soup.find_all('td', class_='fval')
            items1 = soup.find_all('td', class_='fnam')
            compname = soup.find('div', class_='compname')
            fullname = soup.findChildren('td', class_='datarow')
            if fullname:
                for_ful.append(url1)
                for k in range(0, len(fullname), 2):
                    for_ful.append(translator.translate(fullname.get_text(), src='hy', dest='en').text)
                    for_ful.append(fullname[k + 1].get_text())
                fulll.append(for_ful)
            if items:
                rus = translator.translate(compname.get_text(), src='hy', dest='ru').text
                eng = translator.translate(compname.get_text(), src='hy', dest='en').text
                info_only_one.append(url1)
                arm = str(compname.get_text())
                info_only_one.append(arm)
                rus1 = str(rus)
                info_only_one.append(rus1)
                eng1 = str(eng)
                info_only_one.append(eng1)
                for j in range(len(items1)):
                    info_only_one.append(items[j].get_text())
                info_first.append(info_only_one)
        with open('C:/Users/user/Desktop/project/newsite/files.csv', 'w', encoding='utf-8', newline='') as file:
            writer = csv.writer(file, delimiter=';')
            writer.writerow(['URL', 'Name.Arm', 'Name.Rus',
                             'Name.Eng', 'Status', 'Registration number', 'Tax ID', 'Z-Code', 'Obsoleted by'])
            for item in info_first:
                print(item)
                if len(item) == 9:
                    writer.writerow([item[0], item[1], item[2],
                                     item[3], item[4], item[5], item[6], item[7], item[8]])
                else:
                    writer.writerow([item[0], item[1], item[2],
                                     item[3], item[4], item[5], item[6], item[7], ''])
            writer.writerow(['URL', 'Full name: / Company name:',
                             'Country of Nationality:'])
            for awd in fulll:
                for sd in range(1, len(items1), 2):
                    writer.writerow([awd[0], awd[sd], awd[sd + 1]])
    
    
    def save_file(ur):
        for u in range(1, 101):
            t = Thread(target=main, args=[u, ur])
            t.start()
    
    
    save_file(url)

Answer 1

Скорее всего причина в том, что сервер разорвал подключение:

Remote end closed connection without response

А это может быть из-за защиты от dos'а, т.к. ваш скрипт делает запросы без ожидания и запросов очень много -- 12900:

print(len(range(1, 101)) * len(range(1, 130)))
# 12900

Вам нужно:

Сделать обработку ошибок через try/except, чтобы скрипт не падал и желательно, при получении ошибки, сделать запрос повторно через некоторое (тут удобно через цикл while True):
```
...
while True:
    try:
        r = requests.get(url1)
        break
    except:
        time.sleep(5)

soup = BeautifulSoup(r.text, 'html.parser')
...
```
Добавить задержку между запросами
Уменьшить количество запросов, если есть лишние

А если выше перечисленное не поможет, то дополнительно использовать прокси

БЛОГ НА HUSL

Выдает ошибку ConnectionError requests

Ответы (1 шт):