How to use scala sttp client to download file

I have task to check AWS pricing, it's a public json file. there is no auth needed.

I use Tapir Sttp library

    val request: Request[Either[String, String], Any] = basicRequest
        .header(HeaderNames.Accept, MediaType.ApplicationOctetStream.toString())
        .get(uri"https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json")


    val response = request.send(sttpBackend)
    val res = Await.result(response, 30.seconds)

above is my simple get call.
But I got error
Caused by: java.io.UnsupportedEncodingException: Unsupported encoding

I can download the file from browers, get the json from Postman.
Anybody know how to download file by sttp client
Thanks

try to download json file from amazon pricing endpoint

It seems the aws endpoint returns a response with an empty Content-Encoding header:

> GET /offers/v1.0/aws/index.json HTTP/1.1
> Host: pricing.us-east-1.amazonaws.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
(...)
< x-amz-server-side-encryption: AES256
< Content-Encoding:
< x-amz-version-id: RPn8P8KKFZDsbbPZSHSPEAxgOPJ4okmH
(...)

Our code tries to match one of the supported decoding algorithms to the value of the header, if it is present. This fails, as there is no algorithm for an empty string.

I've created an issue to fix that in sttp. The fix is to ignore empty values of the header.

There is also a work-around: you can provide a custom encoding handler to "handle" the empty value. For example, using the HttpURLConnectionBackend:

object Test extends App {
  val request: Request[Either[String, String], Any] = basicRequest
    .header(HeaderNames.Accept, MediaType.ApplicationOctetStream.toString())
    .get(uri"https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json")

  val backend = HttpURLConnectionBackend(customEncodingHandler = { case (is, "") => is })

  println(request.send(backend))
}

I actually tried curl on the above mentioned URL, the Content-Encoding: is coming as empty.

curl -X GET https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json --output ~/Desktop/pricing-index.json -v
Note: Unnecessary use of -X or --request, GET is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 54.192.150.79:443...
* Connected to pricing.us-east-1.amazonaws.com (54.192.150.79) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1

## <-- redacted -->

> GET /offers/v1.0/aws/index.json HTTP/1.1
> Host: pricing.us-east-1.amazonaws.com
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
< Content-Length: 69431
< Connection: keep-alive
< Date: Tue, 13 Jun 2023 06:29:27 GMT
< Last-Modified: Tue, 13 Jun 2023 01:42:43 GMT
< x-amz-server-side-encryption: AES256
< Content-Encoding:

## <-- redacted -->

<
{ [1455 bytes data]
100 69431  100 69431    0     0  41107      0  0:00:01  0:00:01 --:--:-- 41229
* Connection #0 to host pricing.us-east-1.amazonaws.com left intact

And the sttp.client3.HttpClientSyncBackend.standardEncoding is failing due to empty Content-Encoding : https://github.com/softwaremill/sttp/blob/v3.8.15/core/src/main/scalajvm/sttp/client3/HttpClientSyncBackend.scala#LL64C16-L64C16

So, you need to bypass the standardEncoding and create a backend with custom encoding to handle the empty Content-Encoding header by providing just a pass-through customEncodingHandler ( customEncodingHandler: [(InputStream, String) => InputStream] = (is, _) => is )

I

val backend = HttpClientSyncBackend(customEncodingHandler = (is, _) => is)

// create your request
val request = ???

val response = request.send(backend)
val res = Await.result(response, 30.seconds)