-
-
Notifications
You must be signed in to change notification settings - Fork 769
Description
Our app uses a hand ful of GRDB observations with some larger queries to react to database updates.
We're currently on GRDB 6.29.3
. On iOS we noticed some issues:
Issue 1: n Xcode organizer we see some Disk Write
reports. This means that there were excessive disk writes.

Issue 2: For some users we also had issues reported that sqlite write failures ocurred
They get an error like SQLite error 14: unable to open database file - while executing 'INSERT OR ROLLBACK INTO …
.
Analysis
In one case we got both issues reported from the same user so I think issue 1 causes issue 2.
The Xcode organizer mentioning a spill file lead me (& chatgpt) to the assumption that perhaps memory-expensive queries cause sqlite to do excessive temporary file writing due to limited in-memory caching allowances during such observations (issue 1). The system stops allowing disk writes if some limit is exceeeded and then following sqlite writes fail (issue 2)
We are not aware of any recursions or observation->write->observation->write (…) loops in our code but of course that may exist too and we will keep looking for that, as it would help explain the excessive data amounts.
I used a helper function and found out that cache size in bytes (totalCacheBytes) is by default
- 512KB on iOS <- this seems small
- 8000KB on macOS
/// Logs SQLite memory usage, cache size, temp store mode, journal mode, and soft heap limits.
/// Call this from anywhere (e.g. debug screen, launch, after a write-heavy task).
static func logSQLiteMemoryStats(of database: DatabaseQueue) {
do {
try database.read { db in
let pageSize = try Int.fetchOne(db, sql: "PRAGMA page_size") ?? 4096
let cacheSize = try Int.fetchOne(db, sql: "PRAGMA cache_size") ?? 0
let tempStore = try Int.fetchOne(db, sql: "PRAGMA temp_store") ?? 0
let autoCheckpoint = try Int.fetchOne(db, sql: "PRAGMA wal_autocheckpoint") ?? 0
let journalMode = try String.fetchOne(db, sql: "PRAGMA journal_mode") ?? "unknown"
let totalCacheBytes = cacheSize < 0
? abs(cacheSize) * 1024 // cache size is in KB → convert to bytes
: cacheSize * pageSize // cache size is in pages → multiply by page size
let tempStoreDescription: String = {
switch tempStore {
case 0: return "DEFAULT (check SQLite config)"
case 1: return "FILE (temp tables on disk)"
case 2: return "MEMORY (temp tables in RAM)"
default: return "Unknown (\(tempStore))"
}
}()
// let memoryUsed = sqlite3_memory_used()
// let memoryHighWater = sqlite3_memory_highwater(0)
let heapLimit = sqlite3_soft_heap_limit64(0) // read-only
print("🧠 SQLite Memory Stats:")
print(" - Page size: \(pageSize) bytes")
print(" - Cache size: \(cacheSize) (\(totalCacheBytes) bytes total)")
print(" - Temp store mode: \(tempStoreDescription)")
print(" - Journal mode: \(journalMode.uppercased())")
print(" - Auto-checkpoint: \(autoCheckpoint) pages")
print(" - Soft heap limit: \(heapLimit) bytes")
// memoryUsed & memoryHighWater require setting sqlite3_config(SQLITE_CONFIG_MEMSTATUS, 1) very early in the program (AppDelegate init)
// print(" - Current mem used: \(memoryUsed) bytes")
// print(" - Peak mem used: \(memoryHighWater) bytes")
}
} catch {
print("⚠️ Failed to fetch SQLite memory stats: \(error)")
}
print() // good to set a breakpoint here
}
Also I read that WAL mode typically causes less file-io so it may help relax the issue.
Ideas
Idea 1: Increase cache size on iOS and try WAL journaling mode
We'll increase the sqlite cache size and enable WAL mode and see if this helps.
This is what we will try next.
/// Configures cache/memory/journaling settings of the SQLite connection for our purposes
static func configureDatabaseConnection(of database: DatabaseQueue) throws {
logSQLiteMemoryStats(of: database)
// The default cache_size on iOS is just ~512kB which (hypothesis) potentially lead to a lot of
// spill files, leading potentially to excessive disk writes. Default on macOS was ~8MB.
// We increase both to hope for fixing excessive disk writes and increase DB performance.
// temp_store is by default set to default, we explicitly set it to memory to reduce disk writes.
_ = try database.writeWithoutTransaction { db in
try db.execute(sql: """
PRAGMA journal_mode = WAL; -- WAL mode is prevents excessive file IO (fsync) when writing
""")
}
try database.write { db in
try db.execute(sql: """
PRAGMA cache_size = -32768; -- 32 MB cache (negative numbers are used for specifying KB as unit)
PRAGMA temp_store = MEMORY; -- Use memory for temp tables/sorts
""")
}
// activate only when inspecting:
Self.logSQLiteMemoryStats(of: database)
}
Idea 2: switch from observations to invalidations
We could use sqlite triggers to create records in an invalidation table to replace larger observations with manual book-keeping.
This is more fragile and cumbersome to set up but will be more performant.
Does anybody have experience with the topic of cache size and disk write reports when using GRDB / SQLite?