How to write responses of multiple asynchronous get requests to a single file in asynchronous httpclient java 11?

huangapple go评论84阅读模式
英文:

How to write responses of multiple asynchronous get requests to a single file in asynchronous httpclient java 11?

问题

public class httptest {
    
   private static ExecutorService executorService = Executors.newFixedThreadPool(4); 
   private static HttpClient client = HttpClient.newBuilder() 
            .executor(executorService)
            .build();
    
   public static void main(String[] args) throws Exception {
        File fts = new File("P:/Spyder_directory/sample.ts");
        fts.createNewFile();
        List<URI> urls = Arrays.asList(
            new URI("Url of file 1"),
            new URI("Url of file 2"),
            new URI("Url of file 3"),
            new URI("Url of file 4"),
            new URI("Url of file 5"));

        Path file = Path.of("P:/Spyder_directory/sample.ts");
        List<CompletableFuture<Void>> results = urls.stream()
            .map(url -> client.sendAsync(HttpRequest.newBuilder(url).build(), BodyHandlers.ofFile(file)))
            .map(responseFuture -> responseFuture.thenAcceptAsync(response -> {
                if (response.statusCode() == 200) {
                    System.out.println("Downloaded: " + response.uri());
                } else {
                    System.out.println("Failed to download: " + response.uri());
                }
            }))
            .collect(Collectors.toList());
        
        CompletableFuture<Void> allOf = CompletableFuture.allOf(results.toArray(new CompletableFuture[0]));
        allOf.join();
        System.out.println("All downloads completed.");
   }
}

Please note that this is just the translated code you provided, but I've removed the translation of comments as per your request. Make sure to replace the placeholder URLs with actual URLs and adapt the file paths as needed.

英文:

I can download a single media file using httpclient in java 11 like this

public class Httptest {
private static HttpClient client = HttpClient.newBuilder().build();
public static void main(String[] args) throws Exception {
File fts = new File(&quot;P:/sample.ts&quot;);  //Destination of downloaded file
fts.createNewFile();
URI url = new URI(&quot;File url here&quot;); //File Url
HttpRequest request = HttpRequest.newBuilder()   //Creating HttpRequest using Builder class
.GET()
.uri(url)
.build();
Path file = Path.of(&quot;P:/samp.ts&quot;);
//BodyHandlers class has methods to handle the response body
// In this case, save it as a file (BodyHandlers.ofFile())
HttpResponse&lt;Path&gt; response = client.send(request,BodyHandlers.ofFile(file)); 
}
}

The above code snippet downloads the .ts file from the url. And it is downloaded properly.

Now, I have list of urls, List&lt;URI&gt; urls. I made asynchronous call to the list of urls, and also ensured concurrent calls by adding an Executor service.
The place where I'm stuck is, how to write the list of responses to a single file.

The code I wrote so far:

public class httptest{
// Concurrent requests are made in 4 threads
private static ExecutorService executorService = Executors.newFixedThreadPool(4); 
//HttpClient built along with executorservice
private static HttpClient client = HttpClient.newBuilder() 
.executor(executorService)
.build();
public static void main(String[] args) throws Exception{
File fts = new File(&quot;P:/Spyder_directory/sample.ts&quot;);
fts.createNewFile();
List&lt;URI&gt; urls = Arrays.asList(
new URI(&quot;Url of file 1&quot;),
new URI(&quot;Url of file 2&quot;),
new URI(&quot;Url of file 3&quot;),
new URI(&quot;Url of file 4&quot;),
new URI(&quot;Url of file 5&quot;));
List&lt;HttpRequest&gt; requests = urls.stream()
.map(HttpRequest::newBuilder)
.map(requestBuilder -&gt; requestBuilder.build())
.collect(toList());
Path file = Path.of(&quot;P:/Spyder_directory/sample.ts&quot;);
List&lt;CompletableFuture&lt;HttpResponse&lt;Path&gt;&gt;&gt; results = requests.stream()
.map(individual_req -&gt; client.sendAsync(individual_req,BodyHandlers.ofFile(file)))
.collect(Collectors.toList());
}
}

The file sample.ts created at the end of execution does not have the response of the requests made.
If you get the gist of my problem, can anyone suggest alternate solutions for this problem.

答案1

得分: 1

使用HttpResponse.BodyHandlers.ofByteArrayConsumer与一个Consumer<Optional<byte[]>>,将字节写入文件,是一种可能性。这将使您能够控制文件如何打开,从而允许您追加到现有文件而不是每次都创建新文件。

请注意,如果这样做,您不应该使用sendAsync,因为请求将并发发送,因此响应也将并发接收。如果您仍然希望并发发送请求,您需要缓冲响应并在将其写入文件时进行一些同步。

英文:

One possibility would be to use HttpResponse.BodyHandlers.ofByteArrayConsumer with a Consumer&lt;Optional&lt;byte[]&gt;&gt; that writes the bytes to a file. This would let you control how the file is opened, allowing you to append to an existing file rather than creating a new file each time.

Note that if you do that you should not use sendAsync because the requests will be sent concurrently, and the response will therefore be received concurrently too. If you still want to send the requests concurrently you will need to buffer the responses and impose some synchronization when writing them down to the file.

huangapple
  • 本文由 发表于 2020年9月30日 16:13:17
  • 转载请务必保留本文链接:https://go.coder-hub.com/64133490.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定