Skip to content

SQLite failures and excessive disk write reports due to low cache_size and observations? #1790

@hannesoid

Description

@hannesoid

Our app uses a hand ful of GRDB observations with some larger queries to react to database updates.
We're currently on GRDB 6.29.3. On iOS we noticed some issues:

Issue 1: n Xcode organizer we see some Disk Write reports. This means that there were excessive disk writes.

Image It looks like they originate not from the writing itself but from fetches due to observations triggered by writes.

Issue 2: For some users we also had issues reported that sqlite write failures ocurred

They get an error like SQLite error 14: unable to open database file - while executing 'INSERT OR ROLLBACK INTO ….

Analysis

In one case we got both issues reported from the same user so I think issue 1 causes issue 2.
The Xcode organizer mentioning a spill file lead me (& chatgpt) to the assumption that perhaps memory-expensive queries cause sqlite to do excessive temporary file writing due to limited in-memory caching allowances during such observations (issue 1). The system stops allowing disk writes if some limit is exceeeded and then following sqlite writes fail (issue 2)

We are not aware of any recursions or observation->write->observation->write (…) loops in our code but of course that may exist too and we will keep looking for that, as it would help explain the excessive data amounts.

I used a helper function and found out that cache size in bytes (totalCacheBytes) is by default

  • 512KB on iOS <- this seems small
  • 8000KB on macOS
    /// Logs SQLite memory usage, cache size, temp store mode, journal mode, and soft heap limits.
    /// Call this from anywhere (e.g. debug screen, launch, after a write-heavy task).
    static func logSQLiteMemoryStats(of database: DatabaseQueue) {
        do {
            try database.read { db in
                let pageSize = try Int.fetchOne(db, sql: "PRAGMA page_size") ?? 4096
                let cacheSize = try Int.fetchOne(db, sql: "PRAGMA cache_size") ?? 0
                let tempStore = try Int.fetchOne(db, sql: "PRAGMA temp_store") ?? 0
                let autoCheckpoint = try Int.fetchOne(db, sql: "PRAGMA wal_autocheckpoint") ?? 0
                let journalMode = try String.fetchOne(db, sql: "PRAGMA journal_mode") ?? "unknown"

                let totalCacheBytes = cacheSize < 0
                    ? abs(cacheSize) * 1024        // cache size is in KB → convert to bytes
                    : cacheSize * pageSize         // cache size is in pages → multiply by page size

                let tempStoreDescription: String = {
                    switch tempStore {
                    case 0: return "DEFAULT (check SQLite config)"
                    case 1: return "FILE (temp tables on disk)"
                    case 2: return "MEMORY (temp tables in RAM)"
                    default: return "Unknown (\(tempStore))"
                    }
                }()

                // let memoryUsed = sqlite3_memory_used()
                // let memoryHighWater = sqlite3_memory_highwater(0)
                let heapLimit = sqlite3_soft_heap_limit64(0) // read-only

                print("🧠 SQLite Memory Stats:")
                print("  - Page size:         \(pageSize) bytes")
                print("  - Cache size:        \(cacheSize) (\(totalCacheBytes) bytes total)")
                print("  - Temp store mode:   \(tempStoreDescription)")
                print("  - Journal mode:      \(journalMode.uppercased())")
                print("  - Auto-checkpoint:   \(autoCheckpoint) pages")
                print("  - Soft heap limit:   \(heapLimit) bytes")
                // memoryUsed & memoryHighWater require setting sqlite3_config(SQLITE_CONFIG_MEMSTATUS, 1) very early in the program (AppDelegate init)
                // print("  - Current mem used:  \(memoryUsed) bytes")
                // print("  - Peak mem used:     \(memoryHighWater) bytes")
            }
        } catch {
            print("⚠️ Failed to fetch SQLite memory stats: \(error)")
        }
        print() // good to set a breakpoint here
    }

Also I read that WAL mode typically causes less file-io so it may help relax the issue.

Ideas

Idea 1: Increase cache size on iOS and try WAL journaling mode

We'll increase the sqlite cache size and enable WAL mode and see if this helps.
This is what we will try next.

    /// Configures cache/memory/journaling settings of the SQLite connection for our purposes
    static func configureDatabaseConnection(of database: DatabaseQueue) throws {
        logSQLiteMemoryStats(of: database)
        // The default cache_size on iOS is just ~512kB which (hypothesis) potentially lead to a lot of
        // spill files, leading potentially to excessive disk writes. Default on macOS was ~8MB.
        // We increase both to hope for fixing excessive disk writes and increase DB performance.
        // temp_store is by default set to default, we explicitly set it to memory to reduce disk writes.
        _ = try database.writeWithoutTransaction { db in
            try db.execute(sql: """
                PRAGMA journal_mode = WAL;   -- WAL mode is prevents excessive file IO (fsync) when writing
                """)
        }
        try database.write { db in
            try db.execute(sql: """
                PRAGMA cache_size = -32768;  -- 32 MB cache (negative numbers are used for specifying KB as unit)
                PRAGMA temp_store = MEMORY;  -- Use memory for temp tables/sorts
                """)
        }
        // activate only when inspecting:
        Self.logSQLiteMemoryStats(of: database)
    }

Idea 2: switch from observations to invalidations

We could use sqlite triggers to create records in an invalidation table to replace larger observations with manual book-keeping.
This is more fragile and cumbersome to set up but will be more performant.

Does anybody have experience with the topic of cache size and disk write reports when using GRDB / SQLite?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      pFad - Phonifier reborn

      Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

      Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


      Alternative Proxies:

      Alternative Proxy

      pFad Proxy

      pFad v3 Proxy

      pFad v4 Proxy