How shall I efficiently submit a larger size viewmodel data to ASP.NET action method?

I have an ASP.NET web app. On the client side, I've an Index.cshtml page which currently gets populated with ~40 MB of IndexViewModel.cs data on the GET request.

On the server side, I have this action method to process the data:

public ActionResult Update(IndexViewModel modelData)
{
    // validation
    // submission of data to DB
    return RedirectToAction(nameof(Index), modelData)
}

Structure of Index.cshtml:

S#1: User inputs section
     It consists of HTML input elements
     and a Get Data button, which generates an Ajax call to get the data from db based on S#1
     and populate the table in S#3
----------------------------------------------------------------------------------

S#2: A table with filtered data (filtered using jQuery) from table data in S#3
     this table's data is editable
----------------------------------------------------------------------------------

S#3: A table that gets populated currently with somewhat closer to ~18000 records from DB on
     click of the Get Data button in S#1, and this is not editable
----------------------------------------------------------------------------------

IndexViewModel.cs:

public class IndexViewModel
{
    List<TableSelector> tableSelectors { get; set; }
    List<SourceTable> srcTbl { get; set; }
    List<DestinationTable> destinationTbl { get; set; }
}

While submitting the form data both srcTbl and destinationTbl consists of ~18000 records each.

The server is hosted as an Azure web app service. The timeout on the Azure app gateway is set to 3 minutes.

When debugging from the local machine, the process takes time but it executes as expected and everything works as it should. But in Azure, after 3 minutes, I get a response:

> 504 Gateway timed out

But when checking the db, after around 5 minutes, I can see that the submitted data is available and the db has been updated.

Therefore, I tried increasing the timeout limit to 5 minutes and then submitting the form data. But, after 4 minutes, I get a 500 Internal Server error. The data size will keep increasing so increasing the timeout on the Azure app gateway is not the solution. But I also need the whole model at once to work within the body of Update().

I measured different times from local machine, like:

• How much time is it taking for the request to reach the first line inside the body of the Update()?
  Around 5-7 minutes
• How much time is it taking to process the data inside the body of Update()?
  Less than 30 seconds

According to these results, I came to the conclusion that most of the time is getting consumed by ASP.NET to prepare that IndexViewModel modelData parameter from the request body. Therefore, my question is how should I submit the data from the client to the server so that I get the whole model and also I get it efficiently and respond with a valid response within the 3 minutes timeout limit?

Note: this whole submitted code is of a similar sample app to the original one since I do not have the rights to the original code to use/submit/publish it elsewhere. The original app runs on .NET Framework v4.7 and the sample app runs on .NET 7. However, these versions shouldn't affect my question, because my question is of logic, is for an efficient approach.

• You can consider implementing Data Pagination, and send the data in parts instead of sending it together this will help improve the performance of your App.

• You can update your client side code to send the data in chunks and update server side code to listen to that chunked data.

Sample code:-

Client side:-

function submitData() {
	  
  var formData = $('#yourForm').serializeObject();

  
  var chunkSize = 1024 * 1024; // Adjust the chunk size as needed
  var data = JSON.stringify(formData);
  var chunks = [];

  for (var i = 0, offset = 0; offset < data.length; i++, offset += chunkSize) {
    chunks.push(data.slice(offset, offset + chunkSize));
  }

  
  sendChunks(chunks, 0);
}

function sendChunks(chunks, index) {
  if (index >= chunks.length) {
    
    return;
  }

  var chunk = chunks[index];

  
  $.ajax({
    url: '/YourController/ProcessChunk',
    type: 'POST',
    data: chunk,
    contentType: 'application/json',
    success: function () {
      // Process the success response

      // Send the next chunk recursively
      sendChunks(chunks, index + 1);
    },
    error: function (xhr, status, error) {
      // Handle the error response
    }
  });
}

Server side code:-

private static StringBuilder _dataBuilder = new StringBuilder();

[HttpPost]
public ActionResult ProcessChunk([FromBody] string chunk)
{
    
    _dataBuilder.Append(chunk);

    return Ok();
}

public ActionResult CompleteData()
{
    var completeData = _dataBuilder.ToString();

    
    _dataBuilder.Clear();

    return RedirectToAction(nameof(Index));
}

• You can also consider running data processing task in a background as a asynchronous method. You can set up a Message queue, Service Bus Queue to process the data and then call it as a update action in IndexViewModel.

• You can also consider caching the client side and server side code to cache the data for processing and then send it, This will optimize the performance of your Asp.NET Web app.

I tried creating one Asp.Net Web app connected to Database and successfully retrieved the data, Refer below:-

My Courseservice.cs code:-

using MySql.Data.MySqlClient;
using sqlapp.Models;
using System.Data.SqlClient;

namespace sqlapp.Services
{

    // This service will interact with our Product data in the SQL database
    public class CourseService
    {
        private static string db_connectionstring = "server=siliconservermysql.mysql.database.azure.com;user=username;password=password;database=appdb;";

        private MySqlConnection GetConnection()
        {
            return new MySqlConnection(db_connectionstring);
        }
        public List<Course> GetCourses()
        {
            List<Course> _lst = new List<Course>();
            string _statement = "SELECT CourseID,CourseName,rating from Course;";
            MySqlConnection _connection = GetConnection();
            // Let's open the connection
            _connection.Open();

            MySqlCommand _sqlcommand = new MySqlCommand(_statement, _connection);

            using (MySqlDataReader _reader = _sqlcommand.ExecuteReader())
            {
                while (_reader.Read())
                {
                    Course _course = new Course()
                    {
                        CourseID = _reader.GetInt32(0),
                        CourseName = _reader.GetString(1),
                        Rating = _reader.GetDecimal(2)
                    };

                    _lst.Add(_course);
                }
            }
            _connection.Close();
            return _lst;
        }


    }
}

Program.cs:-

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddRazorPages();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

Index.cshtml:-

@page
@model IndexModel
@{
    ViewData["Title"] = "Home page";
}

<div class="text-center">
    <h1 class="display-4">This is a list of Courses</h1>    
    <table class="table table-bordered">
  <thead>
    <tr>
      <th scope="col">Course ID</th>
      <th scope="col">Course Name</th>
      <th scope="col">Rating</th>      
    </tr>
  </thead>
   <tbody>
    
        @foreach(var course in Model.Courses)
                {
                    <tr>
                    <th scope="row">@course.CourseID</th>
                    <td>@course.CourseName</td>
                    <td>@course.Rating</td>
                    </tr>
                }                
    
    </tbody>
  </table>
</div>

Output:-

You can also run Diagnose and Solve Problem and get insights to improve the performance of your Web app like below:-