Any way to achieve what 'curl -s -w %{redirect_url} "some_url"' does with 'Invoke-WebRequest'?

I am trying to convert following from bash to PowerShell:

url="https://api.adoptium.net/v3/binary/latest/19/ga/linux/x64/jdk/hotspot/normal/eclipse"
fetch_url=$( curl -s -w %{redirect_url} "${url}" )
fname=$( curl -OLs -w %{filename_effective} "${fetch_url}" )

Executing the above with bash sets $fetch_url to https://github.com/adoptium/temurin19-binaries/releases/download/jdk-19.0.2+7/OpenJDK19U-jdk_x64_linux_hotspot_19.0.2_7.tar.gz or, if there is no URL available, to {"errorMessage":"No releases match the request"} and the command returns with a return code other than "0".

Executing the next line $fname will be OpenJDK19U-jdk_x64_linux_hotspot_19.0.2_7.tar.gz. The file is downloaded to this name, because curl was called with -OLs. (-O: output to name found within url; -L: follow redirects; -s: silent; -w %{filename_effective}: selects the resulting filename to be reported on stdout). $fname then holds this filename for later use.

Any idea how to make the same happen with Invoke-WebRequest? For the URL given this only returns

> Invoke-WebRequest -uri "https://api.adoptium.net/v3/binary/latest/19/ga/linux/x64/jdk/hotspot/normal/eclipse"
StatusCode        : 200
StatusDescription : OK
Content           : {31, 139, 8, 0…}
RawContent        : HTTP/1.1 200 OK
Connection: keep-alive
ETag: "0x8DAFB9194B2EB0B"
Server: Windows-Azure-Blob/1.0
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: a6f360ad-e01e-004e-7f8b-470ced000000
x-ms-version: …
Headers           : {[Connection, System.String[]], [ETag, System.String[]], [Server, System.String[]], [x-ms-request-id, System.String[]]…}
RawContentLength  : 200079543
RelationLink      : {}

There seems to be no way to find out the real download-url and filename as with curl. I've tried with ($curl set to C:\cygwin\bin\curl.exe):

$fetch_url = &$curl -s -w %{redirect_url} "$api_url"

Instead of using Invoke-WebRequest. But this leads the server asking for authentication -- not what was intended. Any idea how to make one or the other work as expected?

<!-- language-all: sh -->

It is worth noting that modern Windows versions now come with curl.exe, so you could simply translate your Bash script to PowerShell:

$url = 'https://api.adoptium.net/v3/binary/latest/19/ga/linux/x64/jdk/hotspot/normal/eclipse'
$fetch_url = curl.exe -s -w '%{redirect_url}' $url
$fname= curl.exe -OLs -w '%{filename_effective}' $fetch_url

# === Extra code for diagnostic output and error handling.

# Show the resulting variable values.
[pscustomobject] @{
  URL = $url
  FetchUrl = $fetch_url
  FileName = $fname
} | Format-List

# Throw an error, if the last curl call indicated failure.
if ($LASTEXITCODE) { throw "curl failed with exit code $LASTEXITCODE." }

Note:

• In Windows PowerShell, be sure to use curl.exe, i.e. include the filename extension, so as to bypass the built-in curl alias that refers to Invoke-WebRequest - this problem has been fixed in PowerShell (Core) 7+.

• The %{…} arguments are enclosed in '…', because { and } are metacharacters in PowerShell (they enclose script-block literals, {…})

It is possible to implement your task using Invoke-WebRequest, but it is more cumbersome:

• Windows PowerShell solution:

$url = 'https://api.adoptium.net/v3/binary/latest/19/ga/linux/x64/jdk/hotspot/normal/eclipse'

# Get the (immediate) redirection URL from the response header's 'Location' field.
# Note the need for -MaximumRedirection 0, to suppress the automatic
# redirection that Invoke-WebRequest performs *by default*.
$fetch_url = (Invoke-WebRequest -Method HEAD -MaximumRedirection 0  $url).Headers.Location

# Extract the last component from the redirection URL and use it as the file name.
$fname = Split-Path -Leaf $fetch_url

# Perform the download, silently.
& {
  $ProgressPreference = 'SilentlyContinue'
  Invoke-WebRequest -ErrorAction Stop -OutFile $fname $fetch_url
}

• PowerShell (Core) 7+ solution (verified in v7.3.2):

  • Unfortunately, the Invoke-WebRequest call requires extra work, due to changes in behavior:

    • A 302 HTTP status code, indicating redirection, is now considered a (statement-terminating) error by default, which must be suppressed with -SkipHttpErrorCheck in order to get the header data returned.

    • Even with -SkipHttpErrorCheck, a non-terminating error is also emitted, stating The maximum redirection count has been exceeded [...], which can be silenced with -ErrorAction Ignore.

    • Header field values are now [string[]] typed. Given that only one redirection URL is present, simply accessing element [0] is sufficient.

    • Official list of all changes

  • Therefore, replace the first Invoke-WebRequest call above with the following:

$fetch_url = (Invoke-WebRequest -Method HEAD -MaximumRedirection 0 -SkipHttpErrorCheck -ErrorAction Ignore $url).Headers.Location[0]

Taking a step back:

• Targeting the ultimately redirected-to URL returns a Content-Disposition header that contains a suggested download file name:

  attachment; filename=OpenJDK19U-jdk_x64_linux_hotspot_19.0.2_7.tar.gz
  

• You can let curl use this file name by combining the -O (--remote-name option with -J (--remote-header-name)

  • Note: The suggested file name happens to be the same as the last segment of the (immediate) redirection URL in the case at hand, but there is no guaranteed relationship between the two.

  • The curl man page contains a warning regarding -J:

> Exercise judicious use of this option, especially on
Windows. A rogue server could send you the name of a DLL or
other file that could be loaded automatically by Windows or some
third party software.

If you want to take advantage of curl's -J option, your code can be simplified to:

# Directly saves to file `OpenJDK19U-jdk_x64_linux_hotspot_19.0.2_7.tar.gz` 
# and prints its name.
$fname = curl.exe -sLJO -w '%{filename_effective}' $url

Note:

• Invoke-WebRequest has no support for honoring a Content-Disposition field.

  • You can extract the file name from it manually, in a separate step, as follows:

    (Invoke-WebRequest $url -Method Head).Headers['Content-Disposition'] -replace '^.*\bfilename\*?="?([^;"]+)?.*', '$1'
    

• In fact, up to at least PowerShell 7.3.2, it doesn't even have support for using the URL's last segment (no equivalent to curl's -O): a file name must explicitly be passed to the -OutFile parameter:

  • However, GitHub PR #19007 will bring such support to a future PowerShell version, namely the ability to specify only a directory path to -OutFile, with the file name being implied by the last segement of the URL, albeit of the redirected-to URL (unlike what curl's -O does when combined with -L)

A little bit of a hack, curious to see if anyone has a better approach

$url = "https://api.adoptium.net/v3/binary/latest/19/ga/linux/x64/jdk/hotspot/normal/eclipse"
$response = Invoke-WebRequest -Uri $url -UseBasicParsing -Method HEAD

$null = $response.RawContent -match 'filename=([^\r\n]*)'
$fileName = $matches[1]
Invoke-WebRequest -Uri $url -UseBasicParsing -OutFile "C:\temp$fileName"