Is error handling in Rust all about when you can and can’t afford to return a Result<T,E> instance?

It feels like the decision of whether to use panic! or use Result<T,E> is kind of arbitrary. Any place where the panic! macro ends up being called, like trying to divide by zero or trying to access an index out of bounds, we could have returned a Result<T,E> instance instead and left the error handling responsibility to the user (the calling code). In the case of accessing an index out of bounds in a vector, why not have normal indexing code like &v[100] behave just like v.get(100) for a vector v of length 5. I reckon that returning a Result<T,E> in all situations would make the code a bit cumbersome with match expressions everywhere, but it feels like there's no clear line between the kind of scenarios that warrant panicking and those that warrant returning a Result<T,E> instance.

I've tried to find the answer to this on the internet but everywhere it states some version of - if a piece of code takes your system into an invalid state, or if it causes an unrecoverable error, you should panic. But what error is unrecoverable exactly? Accessing an index out of bounds or dividing by zero isn't unrecoverable by any means, we can just return Result<T,E> and leave the recovering responsibility to the user. The match expression is going to make sure that the user deals with all the possibilities, especially that of an error.

What concerns does this error handling decision boil down to? Does it boil down to pragmatic decisions like:
How rare is the error? Is it frequent (normal) enough to warrant changing the return type of an operation from a simple type T to Result<T,E> where you have to do extra steps to simply get out the data? How cluttered with match expressions do we want our code to be?

If the problem is inherent to the algorithm (i.e. index out of bounds), then we should rework the code, thus panic!() makes sense: there is no valid reason to use [] with an incorrect index.

On the other hand, when the error can be due to an external factor, where the algorithm cannot make anything better (file not found, permission denied...) then the error should be reported with Result in order to offer the possibility of an alternative (give up, retry...).

It is true that what constitutes an recoverable/unrecoveable error is hard to define and is often skipped when explaining different strategies of error handling in Rust. I would suggest reading an excellent blog post Using unwrap() in Rust is Okay. The TL;DR is to use panic when an error means that there is a bug in the program (caused by either caller or the callie) and returing Result otherwise.