Is it possible to reuse an HTTP Request Body multiple times without reading it into memory?

My requirement is a large file upload, using curl -T <file>, the server side is an HTTP Server in Rust, the implementation can be actix-web (or hyper if other implementations are more suitable), using HTTP/1.1. In HTTP/1.1, the server can only read the file upload data stream once, and I want to calculate the file's MD5, SHA-1 and SHA-256 (or BLAKE3) without reading the file data stream into memory, and then upload it to S3. If there is a solution for parallel calculation, it will be better, but it may be more complex, the key is still to reuse the stream.

It is best not to use a channel to implement it, as the stream will flow through multiple channel to calculate the Checksum, which may have a lower performance than reusing the stream to calculate at the same time.

This needs to be based on the Rust 2021 Edition, actix-web 4.x version (the latest is 4.3.0), and if tokio is used, it must be at least 1.0.0+ (I am currently 1.25.0). This is because the compatibility of library versions in Rust is too poor, and there may be API incompatibilities between each minor version.

May I ask if my requirement can be implemented using Rust? In Go, it seems that there is no better solution besides io.Pipe, but the performance of io.Pipe does not look good.

Thank you very much for your advice and patience!

Update: You can use either Go or Java for this requirement.

There's no such thing as doing calculations without putting the file in memory, at least on standard hardware. The solution is reading it in chunks with an implementer of BufRead, such as BufReader.

This is fully possible in synchronous rust, but since you specified tokio, I made this async, using tokio's IO types and traits. You can make the same thing synchronously with the standard library's IO types and traits, which are named similarly.

First, a function that takes a reader, updates a slice of hash functions, and then writes those bytes to a writer. This is essentially tokio::io::copy_buf but with hash functions in the middle.

pub async fn calculate_hashes<R, W>(
    mut reader: R,
    hashers: &mut [&mut dyn Update],
    mut writer: W,
) -> Result<(), std::io::Error>
where
    R: AsyncBufRead + Unpin,
    W: AsyncWrite + Unpin,
{
    loop {
        let part = reader.fill_buf().await?;
        if part.is_empty() {
            break;
        }

        for hasher in hashers.iter_mut() {
            hasher.update(part);
        }

        writer.write_all(part).await?;

        let part_len = part.len();
        reader.consume(part_len);
    }

    Ok(())
}

This uses a trait from the digest crate: Update. Luckily, there's already implementations of your specific hash functions that use digest: md-5, sha1, and sha2 (make sure you use versions that depend on the same version of digest, which conveniently are all 0.10.5 at the moment).

Here's that function in use:

pub async fn md5_sha1_sha256<R, W>(
    reader: R,
    writer: W,
) -> Result<(Md5, Sha1, Sha256), std::io::Error>
where
    R: AsyncBufRead + Unpin,
    W: AsyncWrite + Unpin,
{
    let mut md5_hasher = Md5::new();
    let mut sha1_hasher = Sha1::new();
    let mut sha256_hasher = Sha256::new();

    calculate_hashes(
        reader,
        &mut [&mut md5_hasher, &mut sha1_hasher, &mut sha256_hasher],
        writer,
    )
    .await?;

    Ok((md5_hasher, sha1_hasher, sha256_hasher))
}

If you want to go further, you could make a wrapper type that implements AsyncWrite for all Update types, and then you wouldn't even need to separate hashers and writer. You could also go the other way and split hashers into your specific hash function types or a specific number of generic arguments, which avoids dyn. I used a dyn slice here so that it is helpful to others in less specific situations.

Here's a link to the whole thing on playground. It doesn't compile because of missing dependencies, though.

Here is one implementation idea in Go: Calculate the Checksum of the Request Body while writing it to the S3 storage.

// ...

md5Hasher := md5.New()
sha1Hasher := sha1.New()
sha256Hasher := sha256.New()

type Checksum struct {
	Md5    string `json:"md5"`
	Sha1   string `json:"sha1"`
	Sha256 string `json:"sha256"`
}

hasher := io.MultiWriter(md5Hasher, sha1Hasher, sha256Hasher)
tr := io.TeeReader(r.Body, hasher)

// Upload the file to S3.
_, err := uploader.Upload(&s3manager.UploadInput{
    Bucket: aws.String("bucket"),
    Key:    aws.String("key"),
    Body:   tr,
})
if err != nil {
    log.Printf("failed to upload file, %v", err)
    w.WriteHeader(http.StatusInternalServerError)
    return
}

checksum := Checksum{
	Md5:    fmt.Sprintf("%x", md5Hasher.Sum(nil)),
	Sha1:   fmt.Sprintf("%x", sha1Hasher.Sum(nil)),
	Sha256: fmt.Sprintf("%x", sha256Hasher.Sum(nil)),
}

respBody, _ := json.MarshalIndent(checksum, "", "  ")
fmt.Printf("%s\n", respBody)

w.Write(respBody)

But now I am thinking, can I calculate the Checksum first through io.MultiWriter and then decide whether to upload the Request Body. But at this point, it seems there is no good way to use a Request Body that has already been read as an upload logic without reading it into memory or writing it to a temporary file.