Support for Streaming a Response to a Request?

I am building a tool that leverages an aiohttp client to perform HTTP requests to an internal application with large file sizes, and then upload those files to S3.

In order to do this, we can using aiobotocore to generate a presigned request that allows us to use aiohttp to perform the upload, but this requires that we send the request using multipart-upload, as it should contain both the file data as well as various request parameters.

This is functional:

download_response = await session.get(...)
content = await download_response.content.read()
fileobj = io.BytesIo(content)
request = await generate_presigned_post(...)
upload_response = await session.post(
    url=request['url'],
    data={
        **request['fields'],
        'file': fileobj,
    },
)

Obviously, this is undesirable, as it requires us to load the entire data into memory before sending it off to S3 – which can be quite large in some cases for our application.

There appears to be a way to send post an asynchronous iterator, such as we could get fromresponse.content.iter_chunks():

upload_response = await session.post(
    url=request['url'],
    data=download_response.content.iter_chunks(),
)

But this would send the raw downloaded data, and not include any of the url-encoded POST parameters, which are required.

I went down the rabbit hole of trying to craft my own multipart request, and trying to wrap an asynchronous iterator into a file-like object for streaming, but I assume there must be some tool in the aiohttp toolchain that I am missing here.

For now, I’ve landed on downloading the file to a temporary disk, and then uploading it from that file, which is not ideal. An ideal API for me would look something like:

upload_response = await session.post(
    url=request['url'],
    data={
        **request['fields'],
        'file': download_response.content.iter_chunks(),
    },
)

Does anybody have a hunch?

I think you need to craft FormData and pass it to session.post().
The missing part is form_data._is_multipart = True which is set for IOBase value only.
I think we can support StreamReader as well.

Please feel free to create a pull request :slight_smile: