"Error - Wkhtmltopdf failed (error code: -11). Memory limit too low or maximum file number of subprocess reached."

I am working on an Odoo project where I need to generate PDF reports for 10 000 records. When I try to print a report with a high number of records, I encounter the following error:

> Wkhtmltopdf failed (error code: -11). Memory limit too low or maximum file number of subprocess reached. Message: b''.

The error occurs during the execution of the _run_wkhtmltopdf method, which is responsible for converting HTML content to a PDF document using Wkhtmltopdf.

I have noticed that the error only occurs when the number of records exceeds a certain threshold (around 10,000). Below that point, the report generates successfully without any issues.

I suspect that the error is related to resource limitations, such as memory or subprocess limits. However, I am unsure about how to address this problem and optimize the report generation process.

Could someone please provide guidance on how to handle large-scale report generation in Odoo or in general using Wkhtmltopdf? Are there any specific settings or configurations that need to be adjusted to avoid this error? Additionally, are there any best practices for pagination or generating reports in smaller batches to avoid resource limitations?

def print_pdf(self):
    data = {
        'inner': self.sale_ids or self.env['sale.order.line'].search_read(
            [], ['salesman_id', 'product_id', 'name', 'price_unit',
                 'qty_to_deliver', 'price_subtotal', 'state'],
            limit=10000)}
    return self.env.ref('coc_bot.sale_order_report'). \
        report_action(self.sale_ids.ids or None, data=data)

XML:

    <tr>
       <td t-out="da['salesman_id'][1] or ''"/>
       <td t-out="da['product_id'][1] or ''"/>
       <td t-out="da['name'] or ''"/>
       <td t-out="da['price_unit'] or ''"/>
       <td t-out="da['qty_to_deliver'] or ''"/>
       <td t-out="da['price_subtotal'] or ''"/>
       <td t-out="da['state'] or ''"/>
      </tr>

This is the code from where the error is raised.

  try:

        wkhtmltopdf = [_get_wkhtmltopdf_bin()] + command_args + files_command_args + paths + [pdf_report_path]
        process = subprocess.Popen(wkhtmltopdf, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        print(process.returncode, 'process',wkhtmltopdf,'wkhtmltopdf',out,'-----------',  err[-1000:])
        if process.returncode not in [0, 1]:
            if process.returncode == -11:
                message = _(
                    'Wkhtmltopdf failed (error code: %s). Memory limit too low or maximum file number of subprocess reached. Message : %s')
            else:
                message = _('Wkhtmltopdf failed (error code: %s). Message: %s')
            _logger.warning(message, process.returncode, err[-1000:])
            raise UserError(message % (str(process.returncode), err[-1000:]))
        else:
            if err:
                _logger.warning('wkhtmltopdf: %s' % err)
    except:
        raise

Let me give you 2 possible solutions

1. Run Odoo using systemd service unit and add this line:
   LimitNOFILE=100000

Change the value to address your limitations and that will play around the issue of the limit of open files.

Normally it's a memory issue due to number of records and the template size. That's when the second solution come into play

2. Find a way to split your PDF into several pages allowing you to be able to pick one or more records and create a PDF with that chunk of records and merge all the generated small PDFs into a single and final one to return

With this solution 2 you will be able to handle any kind of performance issue generating a PDF no matter how big it is or how many records you have to handle to build your PDF