Performance

Speed up get_member_last_page

Base case:

Scenario:

  1. Index node + Chronik to block 220’000 (main chain, using -stopatheight)
  2. Re-using the same datadir, index just node to block 221’000 (-chronik=0), so Chronik will sync to the node height later
  3. Make a backup at this point so we have a reproducible state
  4. Re-using the same datadir, run Instruments (perf for Mac) using -stopatheight=221000, so it syncs blocks 220’000 to 221’000
  5. Delete test datadir and replace with backup for next run, starting at 4. Note: Runs are reproducible down to the second

Results:

Function Run time [s]
insert 19.61 19.69
get_member_last_page 18.33 18.38
SeekForPrev 14.95 15.05

Option 1: Gate get_member_last_page with key_may_exist

Idea: SeekForPrev is slow, and key_may_exist could provide a good heuristic for skipping it entirely

Result: 👎

function base case [s] optimized [s]
insert 19.61 22.21
get_member_last_page 18.33 20.79
SeekForPrev 14.95 14.18
key_may_exist 3.13
total 647830
may_exist 642752

Option 2: Cache tx pages

Idea: We simply cache entire pages

function base case [s] cache=10’000 cache=100’000
insert 19.61 13.68 10.93
get_member_last_page 18.33 12.10 9.14
SeekForPrev 14.95 9.98 7.37
lru::put 0.105
lru::get 0.042
cache=10’000 cache=100’000
total 647830 647830
page_cache_hit 239136 355314