如何从ASP.NET Core控制器端点发送大文件

huangapple go评论87阅读模式
英文:

How to send large file from controller endpoint in ASP.NET Core

问题

我正在尝试从我的 asp.net api 控制器发送一个相当大的 .zip 存档文件。我的设置如下所示:

我有一个 Angular 前端,它发起了一个用于获取存档文件的 GET 请求:

public getArchive(archiveName: string): Observable<any> {
    return this.http.get(
        `${this.runtimeConfig.backendApiPath}api/tenants/${this.auth.profile.tid}/objects/archives/${archiveName}`,
        {
            headers: new HttpHeaders({ Authorization: `Bearer ${this.auth.accessToken}` }),
        });
}

然后,这个请求在后端的 BFF (Backend For Frontend) api 中处理:

[HttpGet("archives/{archiveName}")]
public async Task<IActionResult> GetArchive(
    [FromRoute] string archiveName)
{
    try
    {
        var result = await this._objectsServiceClient.GetArchive(archiveName);

        return Ok(result);
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Error while retrieving archive.");
        return BadRequest(ex.Message);
    }
}

该服务返回一个来自后端 Api 的 byte[],这是一个 GET 请求的结果:

public Task<byte[]> GetArchive(string archiveName)
{
    var url = $"{this.options.BackendUrl}"
                .AppendPathSegments("api", "tenants", $"{this.tenantInfo.TenantId}", "objects", "archives", archiveName)
                .SetQueryParam("api-version", "1.0")
                .ToString();

    var res = this.Get<byte[]>(url);

    return res;
}

我有一个名为 this.Get<byte[]> 的方法:

protected async Task<T> Get<T>(string url)
{
    var response = await _httpClient.GetAsync(new Uri(url));
    await response.EnsureSuccessStatusCodeExtendedAsync();

    var retVal = await response.Content.ReadAsAsync<T>();
    return retVal;
}

后端控制器的端点代码如下:

[HttpGet("objects/archives/{archiveName}")]
public async Task<IActionResult> GetArchive(Guid tenantId, [FromRoute] string archiveName)
{
    using (Infrastructure.Extensions.LoggerExtensions.BeginScope(
               this.logger,
               LoggingScopeField.TenantId(tenantId)))
    {
        try
        {
            var path = new BlobStorageRepoPathBuilder()
                .WithId(tenantId)
                .WithPath("archives")
                .WithId(this.userInfo.UserId)
                .WithPath($"{archiveName}.zip")
                .Build();

            var blob = await this.repo.DownloadAsync(path);

            using (MemoryStream ms = new MemoryStream())
            {
                await blob.Stream.CopyToAsync(ms);
                ms.Position = 0;

                return Ok(ms.ToArray());
            }
        }
        catch (Exception ex)
        {
            this.logger.LogError(ex, "Error occurred during report generation.");
            return this.BadRequest("Error occurred during report generation. Please check the logs.");
        }
    }
}

基本上,我从文件存储中下载文件,将其加载到 MemoryStream 中,然后将其作为 byte[] 发送。对于小文件来说这是可以的(我知道使用 MemoryStream 不是最佳决策,这就是我想要更改它的原因),但是当一个大文件出现(例如大于 2GB)时,我会收到 OutOfMemory 异常。我找到了一篇文章,描述了如何接收大文件,而不是将其作为响应发送。因此,我的问题是,如何处理大于 2GB 的文件,并从中删除 MemoryStream。感谢您提前的回答!

英文:

I'm trying to send a quite large .zip archive from my asp.net api controller.
My setup is something like this:
I have an Angular frontend that makes a GET request for an archive file.

public getArchive(archiveName: string): Observable<any> {
    return this.http.get(
        `${this.runtimeConfig.backendApiPath}api/tenants/${this.auth.profile.tid}/objects/archives/${archiveName}`,
        {
            headers: new HttpHeaders({ Authorization: `Bearer ${this.auth.accessToken}` }),
        });
}

Then this request is handled in a Backend For Frontend (BFF) api:

[HttpGet("archives/{archiveName}")]
public async Task<IActionResult> GetArchive(
    [FromRoute] string archiveName)
{
    try
    {
        var result = await this._objectsServiceClient.GetArchive(archiveName);

        return Ok(result);
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, $"Error while retrieving archive.");
        return BadRequest(ex.Message);
    }
}

the service returns a byte[] that is coming from a GET request from the backend Api:

public Task<byte[]> GetArchive(string archiveName)
{
    var url = $@"{this.options.BackendUrl}"
                .AppendPathSegments("api", "tenants", $"{this.tenantInfo.TenantId}", "objects", "archives", archiveName)
                .SetQueryParam("api-version", "1.0")
                .ToString();

    var res = this.Get<byte[]>(url);

    return res;
}

and the this.Get<byte[]> method I have:

protected async Task<T> Get<T>(string url)
{
    var response = await _httpClient.GetAsync(new Uri(url));
    await response.EnsureSuccessStatusCodeExtendedAsync();

    var retVal = await response.Content.ReadAsAsync<T>();
    return retVal;
}

The backend controller's endpoint has the following code:

[HttpGet("objects/archives/{archiveName}")]
public async Task<IActionResult> GetArchive(Guid tenantId, [FromRoute] string archiveName)
{
    using (Infrastructure.Extensions.LoggerExtensions.BeginScope(
               this.logger,
               LoggingScopeField.TenantId(tenantId)))
    {
        try
        {
            var path = new BlobStorageRepoPathBuilder()
                .WithId(tenantId)
                .WithPath("archives")
                .WithId(this.userInfo.UserId)
                .WithPath($"{archiveName}.zip")
                .Build();

            var blob = await this.repo.DownloadAsync(path);

            using (MemoryStream ms = new MemoryStream())
            {
                await blob.Stream.CopyToAsync(ms);
                ms.Position = 0;

                return Ok(ms.ToArray());
            }
        }
        catch (Exception ex)
        {
            this.logger.LogError(ex, "Error occurred during report generation.");
            return this.BadRequest("Error occurred during report generation. Please check the logs.");
        }
    }
}

Basically I download the file from a file storage, load it in a MemoryStream and send it as a byte[].This works fine with small files (I know that using MemoryStream is not the best decision, that's why I want to change that), but when a large file comes (like more than 2GB) I got an OutOfMemory exceptions. I found out an article that describes how to receive a large file, not to send one as a response.

So my question is, how can I handle large (>2GB) files, and remove the MemoryStream from the equation

Thanks in advance!

答案1

得分: 1

我不完全确定this.repo.DownloadAsync的内部工作原理,因为它没有包含在您的问题中,但似乎它只是提供了文件源的直接IO响应流(看起来像Azure Blob Storage)。如果是这样的话,为什么不直接将文件流传输到客户端呢?:

var blob = await this.repo.DownloadAsync(path);
return File(blob.Stream, "application/zip", fileDownloadName: $"{archiveName}.zip");

根据您对这个答案的评论,似乎this.repo.DownloadAsync是基于HttpClient的,您可以使用以下代码直接从HttpClient获取流:

HttpClient client; // 您已经在使用的HttpClient实例
var stream = await client.GetStreamAsync(theUrl);

流将在接收到HTTP响应读取器后返回给您,然后您可以将其返回给您的控制器方法,然后使用return File直接通过您的服务器将结果流式传输给客户端,而不需要先下载它。

英文:

I'm not 100% sure how the internals of this.repo.DownloadAsync work since it's not included in your question, but it appears that it just provides the direct IO response stream from the file source (which looks like Azure Blob Storage). If that's the case, why not just stream the file to the client?:

var blob = await this.repo.DownloadAsync(path);
return File(blob.Stream, "application/zip", fileDownloadName: $"{archiveName}.zip");

From your comment on this answer, it seems like this.repo.DownloadAsync is using HttpClient based on your comment. To directly get the stream from HttpClient you can use the following code:

HttpClient client; // your HttpClient instance that you're already using
var stream = await client.GetStreamAsync(theUrl);

The stream will be returned to you after the HTTP response readers have been received, and then you can return this to your controller method, and then to the client using return File to directly stream the result through your server without first downloading it.

huangapple
  • 本文由 发表于 2023年6月22日 15:03:38
  • 转载请务必保留本文链接:https://go.coder-hub.com/76529331.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定