SQLite best practices for dealing with multiple process access

We have a single SQLite database accessed by many (~20) processes on windows from C++ and C#.
The database is opened in WAL mode.

Each process open the database once, then perform continuous read and read/write transactions a few times a second. Each transaction is very short: a single select query, a single insert, or a single overwrite (performed as a delete followed by an insert). We perform each operation in a separate transaction.

All relevant fields and combinations for these queries have relevant indexes.

We observe quite poor performance, we need a huge amount of retries to get over SQL_BUSY - we went up to 10000 retries each followed by a random sleep of 1-10 ms, we sometimes need thousand of retries before the query gets through. (note the servers this runs on can get very busy in general).

We also observed the WAL file getting huge - occasionally gigabytes in size. To go around that every hour we run PRAGMA wal_checkpoint(TRUNCATE) followed by VACUUM and remove old entries. The database file itself stays reasonably steady (~100MB).

Are we doing anything wrong here or is there a way to increase performance. Should we instead so a database open and close for each transaction or are there other things we can try ?

I see different problems and I would face them one at a time.

1. Handling SQL_BUSY

I assume you are handling the error from the code, waiting and retrying. You could try letting the SQLite library handle the retry by setting PRAGMA busy_timeout = milliseconds; at the start of the connection. Docs here.

2. Managing file size

WAL file doesn't get automatically shrinked because overwriting it is faster than appending to it. Instead of truncating you could set PRAGMA journal_size_limit = N; to keep it at a manageable size. Doc here.
Also beware that VACUUM does generate a large WAL size, so vacuuming the database just after truncating the wal file will partially negate the truncate. Maybe after setting PRAGMA journal_size_limit you could discover that the db can be VACUUMed just once a day.

3. Improve concurrency

Since in WAL mode many readers and one writer can run concurrently, the conflicts come from multiple concurrent writers.
You should then work to keep writes as low and fast as possible.
First thing you can try is setting PRAGMA synchronous = NORMAL at the start of the connection. This should allow for faster writes without the risk of corrupting the database (see here).

After that you can look at the indexes in your db. Every write (insert, delete or update) will not only need to write the corresponding pages of the table, but also the modified pages of every index defined for that table.
Check if there are indexes which are not essential and can be removed.
Maybe you added some of them to speed up some SELECT query, but there is no point of speeding up a query from 10 to 5 ms, if it has to wait 100ms for a write transaction to complete. Better leave it at 10 ms and have the write transaction complete in 80 ms.

All of these suggestions should be easy to test with slight modifications to your code. If it doesn't work, then you probably need to make greater modifications to your implementation. In this case you could find out that switching to a different RDBMS, as suggested by mason, could be easier.