Which Azure service for batch processing?

Once a day I want to download multiple CSV files (could be several hundred of them), parse them, do some transforming and write them to another file or database. I want to have an overview which processing failed and which succeeded and prefereably trigger a retry on those that failed. So I would need some kind of batch/job processing in Azure. Which services could be used for that? Is that possible with these services?

• Storage queue with Azure functions
• Azure batch with spot VMs?

What other services could I look into?

There are quite a few Azure Services that could perform this task. Selection would really come down to your existing skills and preference. These are the one's I'd start evaluating (but not an exhaustive list 🤪);

1. Azure Data Factory - This service is designed for precisely your use case, ETL of files.

2. Databricks - This is a "big data" service preferred by data professionals and will likely be overkill for your scenario.

3. Logic Apps - A "low code" solution that can operate as you've described. I do worry that it'll end up being a bit clunky and costly for you, depending on the transformations you'll be doing. Often for more complex activities you'll end up needing to write code in Azure Functions and call those functions from Logic Apps.