Decreasing performance

Hey folks,
Thanks for this awesome library. I had a long struggle with it though.
I’m coding a http scanner that is doing as many rps as possible to a lot of different domains. Unfortunately I experienced a constantly decreasing performance for long running scans. After weeks of debugging and refactoring my code I finally discovered a profiler and found the solution:
I had to use the dummy cookie jar. This simple thing is increasing the rps by a factor >10.
I guess a more experienced programmer would have found that solution a lot faster, but for noobs like me I’d propose to mention that in the docs or even better deactivate the cookie jar by default and instead opt-in. It makes much more sense to me to opt-in because if I need that functionality I would search the docs about it.

Cheers Merlin

PS: Another problem were streams or big files. They are causing memory overflows. I’m using the StreamReader now but maybe for better usability one could add a max bytes argument to the ClientResponse.read function.

Hi.
The cookie jar is important for many scenarios, switching to the dummy jar breaks existing code.
The documentation update to mention your case makes sense, please feel free to provide a pull request.

Regarding read() function – it reads the whole content and caches it. The next read() call returns the cached copy.
This behavior is fixed in stone. Again, if we change it we break an existing code. Using response.content is the canonical way to handle a huge payload. Again, if you want to improve the docs – you are welcome. Now we have a https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-response-content chapter. Maybe we need a link to this text somewhere also.

Hey Andrew,
I understand. I will add a section to the docs.
Topic read() function, I thought something like this:

async def read(self, max_bytes: int=-1) -> bytes:
    """Read response payload."""
    if self._body is None:
        try:
            self._body = await self.content.read(max_bytes)
            for trace in self._traces:
                await trace.send_response_chunk_received(self._body)
        except BaseException:
            self.close()
            raise
    elif self._released:
        raise ClientConnectionError('Connection closed')
    return self._body

It brings confusion I think. What is the result of

data1 = await response.read(1024)
data2 = await response.read(2048)
print(data2)

?