Confusion around no of CUDA Cores and the number of parallel threads

I have NVIDIA Corporation TU117 [GeForce GTX 1650] as my GPU. The specs are as below.<br>

No of SM's = 14<br>
No of cuda cores per SM - 64<br>
Total no of cuda cores - 64*14 = 896<br>
<br>
I have read that blocks are executed sequentially and that one SM can execute one block at once<br>
Each cuda core can execute 32 threads at once simultaneously(warps). So, each SM can execute 64 times 32 = 2048 threads simultaneously. But the max no of threads per block is 1024. So, when i assign one block to one SM (as told earlier, cannot assign multiple blocks to one SM), that means that SM can max run 1024 threads at once(limited by max block thread size<br>

Doesn't that mean that SM only runs at half (2048/1024) of it's max capacity?<br>
So, the max no of threads i can run simultaneously is 1024(max block thread size) times 14(no of SM's). Is that correct?<br>
Also, is the data given here on NVIDIA Corporation TU117 [GeForce GTX 1650] correct?

>I have read that blocks are executed sequentially and that one SM can execute one block at once

I'm not sure where you read that. It's incorrect. An SM can have multiple blocks resident and the warp schedulers in an SM on a cycle by cycle basis can choose any assigned warps to issue instructions from.

>Each cuda core can execute 32 threads at once simultaneously(warps).

No, not correct. A cuda core as that term is used by NVIDIA marketing most closely corresponds to a single precision functional unit in an SM. On any given clock cycle, a cuda core can be issued a single FADD, FMUL, or FFMA instruction for a single thread in a warp. To handle such an instruction warp-wide in a single cycle would require 32 cuda cores, just for that one warp, for that one instruction.

>So, each SM can execute 64 times 32 = 2048 threads simultaneously.

The previous statements are incorrect, and so this calculation is also. That is not how you compute the number of simultaneous threads that can be handled by an SM. A Turing SM can handle 1024 threads max as indicated here ("maximum number of resident threads per SM"). It's a hardware limit.

>Doesn't that mean that SM only runs at half (2048/1024) of it's max capacity?

The max capacity of the Turing SM is 1024 threads. There is no inconsistency.

>So, the max no of threads i can run simultaneously is 1024(max block thread size) times 14(no of SM's). Is that correct?

Yes, the maximum number of threads that can be simultaneously "in-flight" on any CUDA GPU is the maximum number of threads per SM times the number of SMs. This doesn't mean that a kernel launch is numerically limited to that many threads. Additional threads will "wait in the wings" until threads currently running on a SM complete and retire, making space for additional threads to be deposited.

>Also, is the data given here on NVIDIA Corporation TU117 [GeForce GTX 1650] correct?

You can get the number of SMs and the max number of threads per multiprocessor on any CUDA GPU by running the deviceQuery sample code.

Unit 3 of this online training series may be of interest, as it covers GPU architecture and warp scheduling info. In addition, there are numerous question here on the cuda SO tag that cover these topics. Here is one example, there are many others.