Parallelize processing of collection

I have a Seq() with a significant number of entries(50K suppose) - each being an ID.

Seq[Int] = Seq(1,2,3,4,5,6 ..... 50000)

There is a blocking function, which takes a Seq as an input,loads the entities corresponding to each id, and return a separate Iterable[Entity].

def getEntityFromID(id: Seq[Int]): Ierable[Entity]

How do I parallelize this operation on the above Seq - in batches of x , and collect the entries into a single Seq().

The easiest way is to "slide" over the sequence using .sliding(size, step) which returns Iterator[Seq[Int]], you can then do the call to the getEntityFromID from each Seq and wrap it in a future. To then get the results you can await the futures.

As an example:

  val s = Seq(1,2,3,4,5,6,7,8)
  def getEntityFromID(w: Seq[Int]) = w.map(_+1)

  import scala.concurrent.duration._
  implicit val executionContext: ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(1))
  val s2: List[Future[Seq[Int]]] = s.sliding(3, 3).map(w => Future(getEntityFromID(w))).toList
  val s3: List[Int] = s2.flatMap(f => Await.result(f, 2 second))
  println(s3)

Of course it's still blocking and waiting for subsequent calls to finish, but it's one way to do it.
A good idea is always to wrap a call to an external service in a future as you don't know how long it will take (or if it will return a valid answer).

It's a basic answer here I can give at best without making assumptions about the use case.

you need to do grouped(i) on the initial seq and get subsequest sequences of i size

then iterate those and call getEntityFromID

  val list : Seq[Int] = Seq(1,2,3,4,5,6..... 50000)
  val groupedList = list.grouped(3)
  
  groupedList.flatMap{
    gp => getEntityFromID(gp)
  }

  def getEntityFromID(id: Seq[Int]): Iterable[Entity] = ???

to execute groupedList in parallel you can use

 groupedList.toList.par.flatMap{
    gp => getEntityFromID(gp)
  }

you can also control prallelism using

  val parallelCollection = groupedList.toList.par
  parallelCollection.tasksupport = new ForkJoinTaskSupport(
    new scala.concurrent.forkjoin.ForkJoinPool(20)
  )

  parallelCollection.flatMap{
    gp => getEntityFromID(gp)
  }

above code executes 20 items at a time