Why do I get a timeout in PhpStorm HTTP Client and Laravel HTTP Client but not in Postman?

When I send a plain GET request with Postman this takes about 228ms consistently and yields a json response. As you can see I disabled cookies and headers in the request to have the exact same request. The generated curl seems to confirm this. Please have a better look at my first 2 screenshots of Postman for proof (I would doubt me as well). No headers at all, even in de 2nd image of the debug console you'll see no headers should be sent.

When I try the same thing with PhpStorm's HTTP client

I get a timeout (except for 1 time which worked after 30 seconds)

Same thing in code:

$result = Http::timeout(10)
    ->get('https://mobileapi.jumbo.com/v17/products');


dump($result->body());

Making the request in curl is even weirder:

So we have 3 different responses for seemingly the same request:

• Postman: fast response (< 400ms) with expected body
• both PhpStorm and php code: timeout even with more that 10 seconds of allowed time
• curl request generated by Postman: Access Denied

Update
Found a solution to my problem, but the reason why Postman works without it still baffles me: A User-Agent header containing exactly "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0" works. However as you can see in above screenshots I don't send this header in Postman and there it works without. This question as to why this happens still stands.

Why do you get different behaviors using seemingly the same request?
Surely, the web server has some request filtering rules based on HTTP headers.

Even if it looks that all HTTP clients send the same request, the reality is that each one attaches slightly different headers silently. Given so, I created a RequestBin to check the differences between the clients.

curl --location 'https://...'

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95ba2-4633081a75af397c63ece198
User-Agent: curl/7.71.1
Accept: */*

Postman with Host header only

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95bac-102cc1e826a026f328c6c583

PhpStorm HTTP Client

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95cc5-40d9098a346629e74f52f4d9
User-Agent: Apache-HttpClient/4.5.14 (Java/17.0.7)
Accept-Encoding: br,deflate,gzip,x-gzip

Checking against your domain, it seems that setting the User-Agent to curl/7.71.1 or Apache-HttpClient/4.5.14 (Java/17.0.7) causes a time-out. As mentioned in other responses, there is probably a blacklist for user agents.

It's often due to some subtle differences in the way the requests are being sent or received. Here are some possible reasons:

Headers: Postman might automatically include headers that PhpStorm, Laravel, or curl don't.

SSL/TLS configurations: If the server uses HTTPS, the SSL/TLS setup could cause differences.

Proxies: Differences in proxy settings across the clients could affect the results.

Cookies: Ensure cookies are truly disabled in all clients.

It's important to compare the actual requests being sent in each client to identify any discrepancies.
One other possible reason can be Firewall Settings

Postman has a default User-Agent that it sends with every request (temporary headers) if you don't specify a different one. This is likely why you're able to make the request in Postman without explicitly setting the User-Agent header. The api endpoint probably recognizes Postman's default User-Agent and allows the request.

You could try to enable the header in the header section but leave the value field blank. This will override the temp headers set by postman.

PhpStorm's built-in HTTP client does not automatically add a User-Agent header to the requests. If you want to include a User-Agent header, you need to add it manually to your request.

The api endpoint probably requires some headers to be set, otherwise it will block the request.

Weird indeed, but the answer is quite simple: The site uses a blacklist to filter User-Agent's and quite possible does so with a homemade solution.

First indicator is looking at their homepage's robots.txt, which may indicate that they've seen increased traffic due to ChatGPT bots or simply don't want to be scraped. It's quite "a bit" of an old-fashioned approach and the disallow-list in there looks like manually put together.

Second at least for the API they seem to parse the User-Agent and have some checks in place.
Setting the UA to \0 for instance produces a 400 Bad Request reply.

Third, using Curl and setting the User-Agent to Cool returns a proper result:

$ curl --header "User-Agent: Cool" https://mobileapi.jumbo.com/v17/products

Fourth, the letters php seem to be on that blacklist, which means any User-Agent that starts with it (PhpStorm , phpstorm , php2020) seems to lead to no reply. User Agents ThePhp or ThisIsA Php Bot work though ...

However and finally, they also seem to use a whitelist.
Setting headers to PhpStorm and the request times out. Setting it to Mozilla PhpStorm and a proper reply comes back. Not for PhpMozilla or Php Mozilla From that I can only assume they first check for known browser identifiers and reply if they find, then check for blacklisted strings (but without properly going through the substrings) and if they find neither, they still reply (which seems to be the case for Postman and my absurd examples).

For instance setting UA to StackOverflow also yields a good reply. Ah and guzzle seems to be on that blacklist as well (I guess that's your Laravel UA).

TL;DR Postman works, with the headers you've set, because EVERY User Agent works that is not on the (weirdly parsing) blacklist.

On a serious note, someone should tell them about about rate limiting or some other ways to protect against over the top traffic.