Httpclient with HttpCompletionOption.ResponseHeadersRead and ProgressMessageHandler doesn't work

huangapple go评论74阅读模式
英文:

Httpclient with HttpCompletionOption.ResponseHeadersRead and ProgressMessageHandler doesn't work

问题

以下是您的代码的翻译部分:

var handler = new HttpClientHandler() { AllowAutoRedirect = true };
var ph = new ProgressMessageHandler(handler);
ph.HttpReceiveProgress += (_, args) => { GetProgress(args.ProgressPercentage); };
var httpClient = new HttpClient(ph);
var response = await _httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead, cancellationToken.Token);
response.EnsureSuccessStatusCode();
using (var zipInputStream = new ZipInputStream(response.Content.ReadAsStreamAsync()))
{
    while (zipInputStream.GetNextEntry() is { } zipEntry)
    {
        var entryFileName = zipEntry.Name;
        var buffer = new byte[4096];
        var directoryName = Path.GetDirectoryName(fullZipToPath);
        if (directoryName?.Length > 0)
        {
            Directory.CreateDirectory(directoryName);
        }
        if (Path.GetFileName(fullZipToPath).Length == 0)
        {
            continue;
        }
        using (var streamWriter = File.Create(fullZipToPath))
        {
            StreamUtils.Copy(zipInputStream, streamWriter, buffer);
        }
    }
}

在您的编辑中,您提到了问题出现在 StreamUtils.Copy 是同步的情况下,进度只在该行执行完毕后报告,但在完成时已经是100%。这似乎是因为 ZipInputStream 不提供将流复制到文件的异步选项。您找到了一个解决方案,将响应首先复制到文件流,然后读取该文件流并将其用作ZipInputStream的输入。这个解决方案看起来是有效的。

如果您需要进一步的建议或解决方案改进,请随时提出。

英文:

I am trying to implement download a zip file and unzip it with progressbar. Roughly below how my code looks like

var handler = new HttpClientHandler() { AllowAutoRedirect = true };
var ph = new ProgressMessageHandler(handler);
ph.HttpReceiveProgress += (_, args) => { GetProgress(args.ProgressPercentage); };
var httpClient = new HttpClient(ph);
var response = await _httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead, cancellationToken.Token);
response.EnsureSuccessStatusCode();
using (var zipInputStream = new ZipInputStream(response.Content.ReadAsStreamAsync()))
{
    while (zipInputStream.GetNextEntry() is { } zipEntry)
    {
        var entryFileName = zipEntry.Name;
        var buffer = new byte[4096];
        var directoryName = Path.GetDirectoryName(fullZipToPath);
        if (directoryName?.Length > 0)
        {
            Directory.CreateDirectory(directoryName);
        }
        if (Path.GetFileName(fullZipToPath).Length == 0)
        {
            continue;
        }
        using (var streamWriter = File.Create(fullZipToPath))
        {
            StreamUtils.Copy(zipInputStream, streamWriter, buffer);
        }
    }
}

My problem here is when I use ResponseHeadersRead instead of ResponseContentRead, ProgressMessageHandler is not reporting progress, using ResponseContentRead I can see the progress incrementing correctly.

It also works fine using ResponseHeadersRead and copy the stream directly to a file as below.

await using (var fs = new FileStream(pathToNewFile + "/test.zip", FileMode.Create))
{
    await response.Content.CopyToAsync(fs);
}

But I feel like this way is waste to download zip to a temp file and unzip again with another stream while i can directly pass the stream to ZipInputStream like I do above. I believe I do something wrong here as I possible misunderstand the usage of ZipInputStream or ResponseHeadersRead? Does ZipInputStream require entire stream loaded at once while ResponseHeadersRead can gradually download the stream, so at the end I cannot directly pass the stream like that?

Please give me a suggestion if that is bad usage or i miss something?

EDIT: Problem seems to be because StreamUtils.Copy is sync, and Progress is only reported when this line is executed completed but it is already 100% once it is done. It looks like that ZipInputStream doesn't provide any async option to copy stream into a file. I need to probably find an alternative.

EDIT 2: I have changed the code using the built in ZipArchive, but also implements as Sync

  using (var zipArchive = new ZipArchive(fileStream, ZipArchiveMode.Read))
   {                  
      zipArchive.ExtractToDirectory(directoryName, true)
   }

EDIT 3 Working solution:
like I said if I just copy the response first to filestream and write as zip file

await using (var fs = new FileStream(pathToNewFile + "/test.zip", FileMode.Create))
{
    await response.Content.CopyToAsync(fs);
}

then read this zip file into stream and use this stream as below. it works, I can see the progress.

 var fileToDecompress = new FileInfo(_pathToNewFile + $"/test.zip");
 var stream = fileToDecompress.OpenRead();

     using (var zipArchive = new ZipArchive(fileStream, ZipArchiveMode.Read))
       {                  
          zipArchive.ExtractToDirectory(directoryName, true)
       }

答案1

得分: 1

如您所发现,如果复制操作是同步执行的,UI 将不会更新。

不幸的是,ExtractToDirectory 目前还没有异步版本。关于此问题有一个开放的 GitHub 问题

与此同时,您可以使用以下代码。其中大部分代码来自原始源代码

public static async ValueTask ExtractToDirectoryAsync(
  this ZipArchive source,
  string destinationDirectoryName,
  bool overwriteFiles,
  CancellationToken cancellationToken = default
)
{
	var extractPath = Path.GetFullPath(destinationDirectoryName);

	// 确保提取路径的最后一个字符是目录分隔符。
	// 如果没有这个,恶意的压缩文件可能会尝试超出预期的提取路径。
	if (!extractPath.AsSpan().EndsWith(new ReadOnlySpan<char>(new[] { Path.DirectorySeparatorChar }), StringComparison.Ordinal))
		extractPath += Path.DirectorySeparatorChar;

	Directory.CreateDirectory(extractPath);

	foreach (var entry in source.Entries)
	{
		// 获取完整路径以确保移除相对段。
		var destinationPath = Path.GetFullPath(Path.Combine(extractPath, entry.FullName));

		if (!destinationPath.StartsWith(extractPath, StringComparison.Ordinal))
			throw new IOException($"Entry {extractPath} has path outside {destinationDirectoryName}");

		if (Path.GetFileName(destinationPath).Length == 0)
		{
			// 如果是目录:
			if (entry.Length != 0)
				throw new IOException("Entry is directory with data");

			Directory.CreateDirectory(destinationPath);
		}
		else
		{
			await entry.ExtractToFileAsync(destinationPath, overwriteFiles, cancellationToken);
		}
	}
}
public static async ValueTask ExtractToFileAsync(
  this ZipArchiveEntry source,
  string destinationFileName,
  bool overwrite,
  CancellationToken cancellationToken = default
)
{
    FileStreamOptions fileStreamOptions = new()
    {
        Access = FileAccess.Write,
        Mode = overwrite ? FileMode.Create : FileMode.CreateNew,
        Share = FileShare.None,
        BufferSize = 0x1000,
    };

    const UnixFileMode OwnershipPermissions =
        UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute |
        UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute |
        UnixFileMode.OtherRead | UnixFileMode.OtherWrite |  UnixFileMode.OtherExecute;

    // 恢复 Unix 权限。
    // 为了安全起见,限制为所有权权限,并遵守 umask(通过 UnixCreateMode)。
    // 我们不应用 UnixFileMode.None,因为在 Windows 上创建的 .zip 文件和以前版本的 .NET 创建的 .zip 文件不包括权限。
    var mode = (UnixFileMode)(source.ExternalAttributes >> 16) & OwnershipPermissions;
    if (mode != UnixFileMode.None && !OperatingSystem.IsWindows())
    {
        fileStreamOptions.UnixCreateMode = mode;
    }

    await using (var fs = new FileStream(destinationFileName, fileStreamOptions))
    await using (var es = source.Open())
    {
        await es.CopyToAsync(fs, cancellationToken);
    }
    File.SetLastWriteTime(destinationFileName, source.LastWriteTime.DateTime);
}

请注意,如果基础流不可寻址,ZipArchive 将同步缓冲到 MemoryStream 中。为了避免这种情况,您可以自行进行缓冲:

var mem = new MemoryStream();
await yourStream.CopyToAsync(mem, someCancellationToken);
await using var zip = new ZipArchive(mem);
await zip.ExtractToDirectoryAsync(...
英文:

As you have found, the UI will not update if the copying is done synchronously.

Unfortunately, there is no async version of ExtractToDirectory as yet. Ther is an open GitHub issue for this.

In the meantime, you can use the following code. Most of it is taken from the original source code:

public static async ValueTask ExtractToDirectoryAsync(
  this ZipArchive source,
  string destinationDirectoryName,
  bool overwriteFiles,
  CancellationToken cancellationToken = default
)
{
	var extractPath = Path.GetFullPath(destinationDirectoryName);

	// Ensures that the last character on the extraction path is the directory separator char.
	// Without this, a malicious zip file could try to traverse outside of the expected extraction path.
	if (!extractPath.AsSpan().EndsWith(new ReadOnlySpan&lt;char&gt;(in Path.DirectorySeparatorChar), StringComparison.Ordinal))
		extractPath += Path.DirectorySeparatorChar;

	Directory.CreateDirectory(extractPath);

	foreach (var entry in source.Entries)
	{
		// Gets the full path to ensure that relative segments are removed.
		var destinationPath = Path.GetFullPath(Path.Combine(extractPath, entry.FullName));

		if (!destinationPath.StartsWith(extractPath, StringComparison.Ordinal))
			throw new IOException($&quot;Entry {extractPath} has path outside {destinationDirectoryName}&quot;);

		if (Path.GetFileName(destinationPath).Length == 0)
		{
			// If it is a directory:
			if (entry.Length != 0)
				throw new IOException(&quot;Entry is directory with data&quot;);

			Directory.CreateDirectory(destinationPath);
		}
		else
		{
			await entry.ExtractToFileAsync(destinationPath, overwriteFiles, cancellationToken);
		}
	}
}
public static async ValueTask ExtractToFileAsync(
  this ZipArchiveEntry source,
  string destinationFileName,
  bool overwrite,
  CancellationToken cancellationToken = default
)
{
    FileStreamOptions fileStreamOptions = new()
    {
        Access = FileAccess.Write,
        Mode = overwrite ? FileMode.Create : FileMode.CreateNew,
        Share = FileShare.None,
        BufferSize = 0x1000,
    };

    const UnixFileMode OwnershipPermissions =
        UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute |
        UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute |
        UnixFileMode.OtherRead | UnixFileMode.OtherWrite |  UnixFileMode.OtherExecute;

    // Restore Unix permissions.
    // For security, limit to ownership permissions, and respect umask (through UnixCreateMode).
    // We don&#39;t apply UnixFileMode.None because .zip files created on Windows and .zip files created
    // with previous versions of .NET don&#39;t include permissions.
    var mode = (UnixFileMode)(source.ExternalAttributes &gt;&gt; 16) &amp; OwnershipPermissions;
    if (mode != UnixFileMode.None &amp;&amp; !OperatingSystem.IsWindows())
    {
        fileStreamOptions.UnixCreateMode = mode;
    }

    await using (var fs = new FileStream(destinationFileName, fileStreamOptions))
    await using (var es = source.Open())
    {
        await es.CopyToAsync(fs, cancellationToken);
    }
    File.SetLastWriteTime(destinationFileName, source.LastWriteTime.DateTime);
}

Note that if the base stream is not seekable then ZipArchive will synchronously buffer it into a MemoryStream. To avoid that, you can buffer it yourself

var mem = new MemoryStream();
await yourStream.CopyToAsync(mem, someCancellationToken);
await using var zip = new ZipArchive(mem);
await zip.ExtractToDirectoryAsync(......

huangapple
  • 本文由 发表于 2023年7月10日 23:23:58
  • 转载请务必保留本文链接:https://go.coder-hub.com/76655191.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定