Trouble while saving large volume of data in coredata

I was trying to save a large volume of data in core data using async/await. The below function works fine for fewer records (5-10K), but when I push the limits and stress test the setup for more than 10K records I get errors on the random lines of code. Below are a few errors while saving the data.

1. Thread 1: EXC_BREAKPOINT (code=1, subcode=0x18a927e88)
2. value optimized out. Value is correctly passed in the function but for some random records gives this error
3. Gets an error while closing the app that triggers from applicationShouldTerminate function in appdelegate

I run the below code in a loop to add records in core data --

    @NSManaged public var categoryName: String
    @NSManaged public var categoryIcon: Data?
    @NSManaged public var categorySearchKey: String
    @NSManaged public var person: NSSet?
    @NSManaged public var country: NSSet
    

    class func addCategory(categoryName : String, categoryIcon: NSImage?, country : [Country], persons : [Person]?)async{
        let appDelegate = await NSApplication.shared.delegate as! AppDelegate
        let context : NSManagedObjectContext = appDelegate.persistentContainer.viewContext
        let entity = NSEntityDescription.entity(forEntityName: "Category", in: context)
        let CD = Category(entity: entity!, insertInto: context)
        CD.categoryName = categoryName
        CD.country = NSSet(array: country)
        if let persons = persons{
            CD.person = NSSet(array: persons)
        }
        if let categoryIcon = categoryIcon{
            CD.categoryIcon = categoryIcon.tiffRepresentation!
        }
        var s = [String]()
        s.append(categoryName)
        s.append(country.map({$0.countryName}).joined(separator: ", "))
        if let persons = persons {
            s.append(persons.map({$0.personName}).joined(separator: ", "))
        }
        CD.categorySearchKey = s.joined(separator: ", ")
    }

After adding the data, below code is used to save core data --

        func saveCoreData()async{
            let appDelegate = await NSApplication.shared.delegate as! AppDelegate
            let context : NSManagedObjectContext = appDelegate.persistentContainer.viewContext
            do {
                try await MainActor.run(body: {
                    try context.save()
                })
            }
            catch {
                print("Error in saving core data")
            }
        }
    }

I have tried saving without the main actor as well, same result, random errors. Also I have tried .newBackgroundContext() instead of .viewContext while saving the data, same random errors.

Because the errors are random, I cannot pinpoint the root cause to fix. Any guidance is appreciated. Thank you.

If you're adding a lot of data and not using batch inserts, you need to save changes from time to time, instead of waiting for the end of the process. All of your newly inserted objects are in memory, which keeps on growing, until you save. You could probably see this if you monitor your app's memory use in Xcode while it runs.

You can control this by saving changes at regular intervals-- every 100, or every 500, or some other interval. The interval depends on how much data each instance contains, so it depends on your app. Experiment a little to see what works best.

You said there are many random errors in the console so why not try creating a Singleton class to manage Database? You can create something like this:

class DataManager {

    static let shared = DataManager()

    lazy var persistentContainer: NSPersistentContainer = {
        let container = NSPersistentContainer(name: "Category") // Your Database Name
        container.loadPersistentStores(completionHandler: { (storeDescription, error) in
            if let error = error as NSError? {
                fatalError("Unresolved error \(error), \(error.userInfo)")
            }
        })
        return container
    }()

    // fetch data from the database
    func fetchData() -> [Category] {
        let fetchRequest: NSFetchRequest<Category> = Category.fetchRequest()
        do {
            let result = try persistentContainer.viewContext.fetch(fetchRequest)
            return result
        } catch {
            print("Failed to fetch data: \(error)")
            return []
        }
    }

    // save data
    func saveData() {
        let context = persistentContainer.viewContext
        if context.hasChanges {
            do {
                try context.save()
            } catch {
                print("Failed to save data: \(error)")
            }
        }
    }

    // Add data item into the database
    func addData(categoryName: String, categoryIcon: Data?, categorySearchKey: String, person: NSSet?, country: NSSet) {
        let context = persistentContainer.viewContext
        let entity = Category(context: context)
        entity.categoryName = categoryName
        if let icon = categoryIcon {
           entity.categoryIcon = icon
        }
        entity.categorySearchKey = categorySearchKey
        if let categoryPerson = person {
           entity.person = categoryPerson
        }
        entity.country = country

        saveData()
    }

    // Delete Data
    func deleteAllData() {
        let fetchRequest: NSFetchRequest<NSFetchRequestResult> = NSFetchRequest(entityName: "Category")
        let deleteRequest = NSBatchDeleteRequest(fetchRequest: fetchRequest)

        do {
            try persistentContainer.viewContext.execute(deleteRequest)
            try persistentContainer.viewContext.save()
        } catch {
            print("Failed to delete data: \(error)")
        }
    }
    
}

Then just call DataManager.shared.addData(...)

This is just a simple singleton class to manage database. You may need to modify it according to your database attributes. I just added those methods based on looking at your attributes so feel free to modify addData() method according to your requirements and then save() is called automatically. The above code will add single entity containing categoryName, person etc. And you can keep adding as much of them as you wish and later fetch like this: let categoriesArray = DataManager.shared.fetchData() and you will get array of [Category].