As described in the previous post, the primary now uses a high-I/O Ultra disk. Reads and writes took fractions of a millisecond, yet PostgreSQL WAL synchronization averaged 25-30 ms. Removing synchronous-standby waits didn't help: synchronous_commit=local still requires local WAL durability. With device-read constraints reduced and remote acknowledgments excluded, I traced the primary's synchronization call across the guest-OS boundary.
To investigate those durable-commit timings, I returned to synchronous_commit=on with normal HA for a full repeat, then used a separate pilot prepared for controller tracing.
Measure the synchronization, not only the disk
The relevant cumulative PostgreSQL counters are:
SELECT wal_write, wal_sync, wal_write_time, wal_sync_time, stats_reset
FROM pg_stat_wal;
With track_wal_io_timing=on, the interval mean is:
mean WAL sync ms = delta(wal_sync_time) / delta(wal_sync)
The full runs showed:
| Run | NOPM | Mean WAL sync ms | WALWrite observation share |
|---|---|---|---|
| Ultra, normal synchronous commit | 55,784 | 25.443 | 83.18% |
| Ultra, local commit | 54,556 | 29.550 | 87.29% |
| Full normal-HA repeat | 54,163 | 24.995 | 83.17% |
For example, the repeat's in-window counter endpoints contained 45,266 additional syncs and 1,131,416.247 additional milliseconds of synchronization time. Their ratio is 24.995 ms.
The repeat was measured on November 8, 2026, 10:26:56-11:11:58 UTC with the same 512-worker, 15+45-minute protocol. These durable-regime traces and the later pilot were collected after the asynchronous-commit experiment described in part 4.
Why request completion and fdatasync disagree
fdatasync must synchronize a file's data. With buffered I/O, that can involve preparing dirty pages, submitting writeback, waiting for previously submitted work, and completing the necessary persistence operations. The entire call can therefore include:
PostgreSQL calls fdatasync
dirty-page writeback and submission
admission waits before block dispatch
issued requests and their completion
remaining file-synchronization work
fdatasync returns
The application call includes work before device dispatch. PostgreSQL can queue other backends behind its WAL write lock while the holder is delayed in that earlier work. I traced the caller to locate the delay.
The captured delay was before dispatch
A 30-second capture during the full normal-on repeat, 10:48:02-10:48:33 UTC, observed 536 completed fdatasync calls, averaging 24.504 ms, in PostgreSQL processes other than the separately classified WAL writer and checkpointer.
Captured sleep/rescheduling time inside those calls was classified as:
| Captured stack location | Share |
|---|---|
| Pre-dispatch WBT admission | 96.43% |
| File-writeback completion | 3.36% |
| Block-flush completion | 0.20% |
An abbreviated stack, from the sleeping function towards its callers, was:
io_schedule
rq_qos_wait
wbt_wait
__rq_qos_throttle
blk_mq_submit_bio
...
ext4_do_writepages
WBT is Linux block-layer writeback throttling. Its admission rules can make a submitting process sleep before work reaches the device. Fast completion of admitted requests does not measure that earlier wait.
These percentages describe captured sleep/rescheduling inside PostgreSQL fdatasync calls, including the delay until a process runs again. The syscall trace covers PostgreSQL synchronization calls broadly, while pg_stat_wal measures WAL synchronization specifically.
A healthy latency controller can still have admission waits
WBT controls write admission using read-latency feedback and write-depth limits. A separate normal-on pilot prepared the instrumentation before launch. Its five short measured captures showed:
| Observation | Value |
|---|---|
| PostgreSQL fdatasync calls | 867 |
| Mean syscall elapsed time | 26.123 ms |
| WBT share of captured sleep/rescheduling | 96.404% |
| Controller timer decisions | 482, all LAT_OK
|
| Observed read minima / target | 0.100-0.213 ms / 2 ms |
| Maximum / normal / background admission limits | 191 / 96 / 48 |
Read latency was healthy in the captured windows while callers still slept in the admission machinery. The captures recorded controller limits and aggregate wait paths. Slow synchronization was also observed before and after the probes ran.
The running kernel was 7.0.0-1017-azure. The upstream WBT implementation and request-QoS implementation explain the policy.
I’ll check the gain against the matching durable run, PostgreSQL’s current direct-I/O status, and what public sources support about HorizonDB’s buffering.
Ran terminal command: @'
import json
for label, durable in [('Ultra on', 55784), ('Ultra local', 54556), ('Ultra on repeat', 54163), ('Original Premium on', 50648)]:
print(json.dumps({'comparison': label, 'durable_nopm': durable, 'off_nopm': 66096, 'gain_percent': (66096 / durable - 1) * 100}))
'@ | python -
Searched the web for What is PostgreSQL's production status for direct I/O in PostgreSQL 18 and 19...
Completed with input: {
"query": "What is PostgreSQL's production status for direct I/O in PostgreSQL 18 and 19 as of November 2026? Distinguish asynchronous I/O from direct I/O and debug_io_direct. Use official PostgreSQL documentation or release notes."
}
Searched the web for Does Azure HorizonDB avoid operating-system page-cache double buffering? Find...
Completed with input: {
"query": "Does Azure HorizonDB avoid operating-system page-cache double buffering? Find official documentation on compute shared buffers, local NVMe cache, WAL-only writes to storage, direct I/O, and dirty-page persistence offload."
}
Fetched https://learn.microsoft.com/en-us/azure/horizondb/overview
Completed with input: {
"url": "https://learn.microsoft.com/en-us/azure/horizondb/overview",
"max_length": 18000
}
Fetched https://www.postgresql.org/docs/18/runtime-config-developer.html
Completed with input: {
"url": "https://www.postgresql.org/docs/18/runtime-config-developer.html",
"max_length": 10500
}
Fetched https://techcommunity.microsoft.com/blog/ADforPostgreSQL/rethinking-memory-ma...
Completed with input: {
"url": "https://techcommunity.microsoft.com/blog/ADforPostgreSQL/rethinking-memory-management-for-azure-horizondb-on-disaggregated-storage/4562638",
"max_length": 20000
}
Fetched https://www.postgresql.org/docs/19/runtime-config-developer.html
Completed with input: {
"url": "https://www.postgresql.org/docs/19/runtime-config-developer.html",
"max_length": 10500
}
Fetched https://www.postgresql.org/docs/18/release-18.html
Completed with input: {
"url": "https://www.postgresql.org/docs/18/release-18.html",
"max_length": 12500
}
Fetched https://www.postgresql.org/docs/19/release-19.html
Completed with input: {
"url": "https://www.postgresql.org/docs/19/release-19.html",
"max_length": 12500
}
Fetched https://learn.microsoft.com/en-us/azure/horizondb/configure-maintain/concepts...
Completed with input: {
"url": "https://learn.microsoft.com/en-us/azure/horizondb/configure-maintain/concepts-compute-replicas",
"max_length": 15000
}
vote_memory
Completed with input: {
"scope": "user",
"fact": "For technical articles, use verifiable factual comparisons rather than marketing positioning.",
"direction": "upvote",
"reason": "The user is drafting an article addition and explicitly asked me to verify the HorizonDB double-buffering claim and supply the correct measured gain. I calculated the matched Ultra comparison, checked PostgreSQL 18/19 documentation, and found Microsoft's published memory-design explanation explicitly supporting the scoped compute-side double-buffering statement."
}
Yes—but the asynchronous result is not a hard upper bound on tuning gains, and direct I/O is a possible direction rather than a demonstrated fix. Your scoped HorizonDB double-buffering claim is supported by Microsoft’s published explanation.
Investigation and tuning answer different questions
My objective here was to identify where the time was spent. I could have continued with a WBT on/off comparison, different WAL synchronization methods, another I/O scheduler, ha separate WAL device. Finding a robust read/write trade-off would require repeated measurements, and that trade-off could work only with this specific workload. Allowing more writes can increase read latency, which WBT is designed to protect. Letting commits return before WAL is durable is, of course, not a solution, but it provides a quick estimation of the maximum performance gain expected by the tuning approach.
What this establishes
In the captured durable regime, much of the synchronization delay was in Linux's write-admission path, before device dispatch.
Provisioning more disk IOPS leaves the guest-OS admission rule unchanged. The relevant question has changed from the disk's advertised capacity to the caller's progress through writeback. All captures used the original WBT policy.
HorizonDB takes a different architectural approach. Its documented memory design removes the duplication of permanent database pages between shared buffers and the compute node’s kernel page cache, while retaining a local SSD cache. Its storage services handle WAL durability and data-page persistence, replacing the compute-local buffered durability path investigated here. This explains why the workload achieves higher throughput than this PostgreSQL configuration can support.
The lab also tested asynchronous commit, where background persistence continues while commits can return before WAL is durable. Part 4 returns to that result and follows BufferContent into page identification.
Top comments (0)