如何在Java中从大型远程zip文件中下载单个文件?

huangapple go评论104阅读模式
英文:

How can I download a single file from a large remote zip file in Java?

问题

我正在尝试从大小约为3-5 GB的给定zip文件中下载一个小文件(0.3 KB)。

我目前一直在使用本地库libfragmentzip,使用JNA,这非常快,但使用本地库会有自己的问题(比如不跨平台)。

我尝试过这个解决方案,但速度要慢得多,需要几分钟,而不是使用libfragmentzip,后者似乎只需要几秒钟。

这个是一个测试zip文件的URL(扩展名是.ipsw,但实际上是zip格式)。我要下载的文件是BuildManifest.plist,位于zip文件的根目录中。

有没有一种快速的方法可以从远程zip文件中下载单个文件,而不使用本地库?

英文:

I'm trying to download a small file (0.3 KB) from a given zip file that's around 3-5 GB in size.

I have currently been using the native library libfragmentzip using JNA, which is very fast, but has issues of its own that come with using native libraries (like not being cross-platform).

I have tried this solution, but it is much slower and ends up taking minutes compared to using libfragmentzip, which only seems to take seconds.

This is a URL to a test zip file (the extension is .ipsw but it is really a zip). The file I am trying to download is BuildManifest.plist, in the root of the zip.

Is there a fast way to download a single file from a remote zip file without using a native library?

答案1

得分: 2

你可以将 BuildManifest.plist 插入到URL的末尾。

例如:

http://updates-http.cdn-apple.com/2021SpringFCS/fullrestores/071-34317/E63B034D-2116-42D0-9FBD-97A3D9060F68/BuildManifest.plist

英文:

You can insert BuildManifest.plist at the end of the URL.

For example:

http://updates-http.cdn-apple.com/2021SpringFCS/fullrestores/071-34317/E63B034D-2116-42D0-9FBD-97A3D9060F68/BuildManifest.plist

答案2

得分: 1

使用Apache Commons Compress和一个由HTTP支持的自定义ByteChannel实现:

  1. var url = new URL(...);
  2. var fileName = "file.txt";
  3. var dest = Path.of(fileName);
  4. try (var zip = new ZipFile(new HttpChannel(url), "zip", "UTF8", true, true);
  5. var stream = zip.getInputStream(zip.getEntry(fileName))) {
  6. Files.copy(stream, dest, StandardCopyOption.REPLACE_EXISTING);
  7. }

HttpChannel(修改自JCodec):

  1. public class HttpChannel implements SeekableByteChannel {
  2. private final URL url;
  3. private ReadableByteChannel ch;
  4. private long pos;
  5. private long length;
  6. public HttpChannel(URL url) {
  7. this.url = url;
  8. }
  9. @Override
  10. public long position() {
  11. return pos;
  12. }
  13. @Override
  14. public SeekableByteChannel position(long newPosition) throws IOException {
  15. if (newPosition == pos) {
  16. return this;
  17. } else if (ch != null) {
  18. ch.close();
  19. ch = null;
  20. }
  21. pos = newPosition;
  22. return this;
  23. }
  24. @Override
  25. public long size() throws IOException {
  26. ensureOpen();
  27. return length;
  28. }
  29. @Override
  30. public SeekableByteChannel truncate(long size) {
  31. throw new UnsupportedOperationException("在HTTP上不支持截断。");
  32. }
  33. @Override
  34. public int read(ByteBuffer buffer) throws IOException {
  35. ensureOpen();
  36. int read = ch.read(buffer);
  37. if (read != -1)
  38. pos += read;
  39. return read;
  40. }
  41. @Override
  42. public int write(ByteBuffer buffer) {
  43. throw new UnsupportedOperationException("在HTTP上不支持写入。");
  44. }
  45. @Override
  46. public boolean isOpen() {
  47. return ch != null && ch.isOpen();
  48. }
  49. @Override
  50. public void close() throws IOException {
  51. ch.close();
  52. }
  53. private void ensureOpen() throws IOException {
  54. if (ch == null) {
  55. URLConnection connection = url.openConnection();
  56. if (pos > 0)
  57. connection.addRequestProperty("Range", "bytes=" + pos + "-");
  58. ch = Channels.newChannel(connection.getInputStream());
  59. String resp = connection.getHeaderField("Content-Range");
  60. if (resp != null) {
  61. length = Long.parseLong(resp.split("/")[1]);
  62. } else {
  63. resp = connection.getHeaderField("Content-Length");
  64. length = Long.parseLong(resp);
  65. }
  66. }
  67. }
  68. }
英文:

Using Apache Commons Compress and a custom ByteChannel implementation backed by HTTP:

  1. var url = new URL(...);
  2. var fileName = "file.txt";
  3. var dest = Path.of(fileName);
  4. try (var zip = new ZipFile(new HttpChannel(url), "zip", "UTF8", true, true);
  5. var stream = zip.getInputStream(zip.getEntry(fileName))) {
  6. Files.copy(stream, dest, StandardCopyOption.REPLACE_EXISTING);
  7. }

HttpChannel (modified from JCodec):

  1. public class HttpChannel implements SeekableByteChannel {
  2. private final URL url;
  3. private ReadableByteChannel ch;
  4. private long pos;
  5. private long length;
  6. public HttpChannel(URL url) {
  7. this.url = url;
  8. }
  9. @Override
  10. public long position() {
  11. return pos;
  12. }
  13. @Override
  14. public SeekableByteChannel position(long newPosition) throws IOException {
  15. if (newPosition == pos) {
  16. return this;
  17. } else if (ch != null) {
  18. ch.close();
  19. ch = null;
  20. }
  21. pos = newPosition;
  22. return this;
  23. }
  24. @Override
  25. public long size() throws IOException {
  26. ensureOpen();
  27. return length;
  28. }
  29. @Override
  30. public SeekableByteChannel truncate(long size) {
  31. throw new UnsupportedOperationException("Truncate on HTTP is not supported.");
  32. }
  33. @Override
  34. public int read(ByteBuffer buffer) throws IOException {
  35. ensureOpen();
  36. int read = ch.read(buffer);
  37. if (read != -1)
  38. pos += read;
  39. return read;
  40. }
  41. @Override
  42. public int write(ByteBuffer buffer) {
  43. throw new UnsupportedOperationException("Write to HTTP is not supported.");
  44. }
  45. @Override
  46. public boolean isOpen() {
  47. return ch != null && ch.isOpen();
  48. }
  49. @Override
  50. public void close() throws IOException {
  51. ch.close();
  52. }
  53. private void ensureOpen() throws IOException {
  54. if (ch == null) {
  55. URLConnection connection = url.openConnection();
  56. if (pos > 0)
  57. connection.addRequestProperty("Range", "bytes=" + pos + "-");
  58. ch = Channels.newChannel(connection.getInputStream());
  59. String resp = connection.getHeaderField("Content-Range");
  60. if (resp != null) {
  61. length = Long.parseLong(resp.split("/")[1]);
  62. } else {
  63. resp = connection.getHeaderField("Content-Length");
  64. length = Long.parseLong(resp);
  65. }
  66. }
  67. }
  68. }

huangapple
  • 本文由 发表于 2020年9月24日 06:15:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/64037006.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定