I'm trying to figure out why my Monero node is syncing so slowly. I have experience syncing other PoW nodes before (like BTC, for example), but it was never this slow (I've spent over 2-3 weeks at this point). The thing which confuses me is that I can't find any clear bottlenecks.
Granted, my setup may be somewhat unconventional; I'm running monerod on my desktop, while the blockchain/DB directory (~/.bitmonero) is mounted from my NAS over NFS (NAS is on the same local network).
First, I imported the blockchain.raw bootstrap file - this was also very slow, until I found a post recommending I add --db-sync-mode safe:sync. Then the import sped up significantly.
Next, I launched monero-wallet-gui and also made sure to start the daemon with --db-sync-mode safe:sync. Now the wallet says "Daemon is synchronized", but is still processing some stuff ("Wallet blocks remaining: XXXXXX" - this has been making slow progress over the past 12h and appears around 40% done). When I run sync_info, it says the blockchain is only ~81% synced and Downloading at 547 kB/s.
Meanwhile, looking at both monerod and monero-wallet-gui process', neither is using more than maybe 3% CPU at any time, 5-7% RAM, and 0 to a few kB/s of disk. It doesn't seem like it's doing much at all. What could be the reason for this?
For example, when importing the blockchain bootstrap file, I was reaching disk (NFS) r/w speeds of up to 50 MB/s (~500 Mbit/s). Note that the bootstrap file was also being read from the same NFS share during the import.
sync_info output:
>>> sync_info
[1/23/24 2:30 PM] 2024-01-23 13:30:30.616 I Monero 'Fluorine Fermi' (v0.18.3.1-release)
Height: 2509060, target: 3068254 (81.7748%)
Downloading at 751 kB/s
Next needed pruning seed: 6
13 peers
Remote Host Peer_ID State Prune_Seed Height DL kB/s, Queued Blocks / MB
162.218.65.134:18080 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
5.189.173.238:18080 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
220.132.135.54:18080 cb10bfd84ce77457 synchronizing 0 3068254 44 kB/s, 200 blocks / 17.1269 MB queued
136.56.35.26:18080 a6d2954e8eab7c47 synchronizing 0 3068254 45 kB/s, 200 blocks / 16.4237 MB queued
54.39.50.52:18080 194a379a027f962d synchronizing 0 3068254 44 kB/s, 160 blocks / 13.0151 MB queued
31.220.2.81:18080 8477dea589dd92cb synchronizing 0 3067880 482 kB/s, 0 blocks / 0 MB queued
136.53.98.191:18080 12e087d4e177788f synchronizing 0 3067948 43 kB/s, 240 blocks / 21.6031 MB queued
185.132.45.117:18080 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
163.172.60.109:18080 5d7a29a5bda149b9 normal 0 1 0 kB/s, 0 blocks / 0 MB queued
212.41.9.32:18080 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
23.112.140.90:18080 f1bddf1f63cc1f8d synchronizing 0 3068254 47 kB/s, 220 blocks / 20.5201 MB queued
84.75.29.59:18089 f0aa45688630c794 synchronizing 0 3068254 46 kB/s, 200 blocks / 17.0472 MB queued
178.220.118.87:18080 0000000000000000 before_handshake 0 0 0 kB/s, 0 blocks / 0 MB queued
64 spans, 111.467 MB
[mooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo]
136.53.98.191:18080 20/184 (2509060 - 2509079, 2129 kB) 393 kB/s (0.21)
23.112.140.90:18080 20/184 (2509080 - 2509099, 2046 kB) 1455 kB/s (0.64)
82.153.138.6:18080 20/184 (2509100 - 2509119, 1652 kB) 1384 kB/s (1)
136.53.98.191:18080 20/184 (2509120 - 2509139, 2592 kB) 274 kB/s (0.21)
136.53.98.191:18080 20/184 (2509140 - 2509159, 1800 kB) 281 kB/s (0.21)
136.56.35.26:18080 20/184 (2509160 - 2509179, 1950 kB) 997 kB/s (0.44)
220.132.135.54:18080 20/184 (2509180 - 2509199, 2493 kB) 1007 kB/s (0.37)
23.112.140.90:18080 20/184 (2509200 - 2509219, 2648 kB) 1801 kB/s (0.64)
82.153.138.6:18080 20/184 (2509220 - 2509239, 2570 kB) 3956 kB/s (1)
84.75.29.59:18089 20/184 (2509240 - 2509259, 1539 kB) 426 kB/s (0.22)
84.75.29.59:18089 20/184 (2509260 - 2509279, 1703 kB) 596 kB/s (0.22)
220.132.135.54:18080 20/184 (2509280 - 2509299, 1642 kB) 1041 kB/s (0.37)
23.112.140.90:18080 20/184 (2509300 - 2509319, 1743 kB) 1331 kB/s (0.64)
84.75.29.59:18089 20/184 (2509320 - 2509339, 1300 kB) 1102 kB/s (0.22)
82.153.138.6:18080 20/184 (2509340 - 2509359, 1507 kB) 1533 kB/s (1)
136.53.98.191:18080 20/184 (2509360 - 2509379, 1341 kB) 180 kB/s (0.21)
136.56.35.26:18080 20/184 (2509380 - 2509399, 1630 kB) 744 kB/s (0.44)
136.56.35.26:18080 20/184 (2509400 - 2509419, 1486 kB) 845 kB/s (0.44)
23.112.140.90:18080 20/184 (2509420 - 2509439, 1071 kB) 935 kB/s (0.64)
136.53.98.191:18080 20/184 (2509440 - 2509459, 1455 kB) 333 kB/s (0.21)
54.39.50.52:18080 20/184 (2509460 - 2509479, 1846 kB) 1798 kB/s (0.88)
220.132.135.54:18080 20/184 (2509480 - 2509499, 2113 kB) 940 kB/s (0.37)
84.75.29.59:18089 20/184 (2509500 - 2509519, 2243 kB) 56 kB/s (0.22)
136.53.98.191:18080 20/184 (2509520 - 2509539, 2446 kB) 939 kB/s (0.21)
220.132.135.54:18080 20/184 (2509540 - 2509559, 1705 kB) 944 kB/s (0.37)
23.112.140.90:18080 20/184 (2509560 - 2509579, 1701 kB) 395 kB/s (0.64)
54.39.50.52:18080 20/184 (2509580 - 2509599, 1979 kB) 1920 kB/s (0.88)
136.53.98.191:18080 20/184 (2509600 - 2509619, 1385 kB) 287 kB/s (0.21)
136.56.35.26:18080 20/184 (2509620 - 2509639, 2146 kB) 1055 kB/s (0.44)
84.75.29.59:18089 20/184 (2509640 - 2509659, 2073 kB) 681 kB/s (0.22)
220.132.135.54:18080 20/184 (2509660 - 2509679, 2203 kB) 1273 kB/s (0.37)
54.39.50.52:18080 20/184 (2509680 - 2509699, 1657 kB) 1851 kB/s (0.88)
84.75.29.59:18089 20/184 (2509700 - 2509719, 2146 kB) 454 kB/s (0.22)
136.56.35.26:18080 20/184 (2509720 - 2509739, 2419 kB) 1061 kB/s (0.44)
23.112.140.90:18080 20/184 (2509740 - 2509759, 1480 kB) 1203 kB/s (0.64)
136.53.98.191:18080 20/184 (2509760 - 2509779, 1538 kB) 557 kB/s (0.21)
23.112.140.90:18080 20/184 (2509780 - 2509799, 2501 kB) 580 kB/s (0.64)
84.75.29.59:18089 20/184 (2509800 - 2509819, 2329 kB) 389 kB/s (0.22)
220.132.135.54:18080 20/184 (2509820 - 2509839, 1498 kB) 988 kB/s (0.37)
23.112.140.90:18080 20/184 (2509840 - 2509859, 2041 kB) 497 kB/s (0.64)
54.39.50.52:18080 20/184 (2509860 - 2509879, 1257 kB) 766 kB/s (0.88)
136.56.35.26:18080 20/184 (2509880 - 2509899, 1036 kB) 691 kB/s (0.44)
136.53.98.191:18080 20/184 (2509900 - 2509919, 1678 kB) 239 kB/s (0.21)
220.132.135.54:18080 20/184 (2509920 - 2509939, 1415 kB) 988 kB/s (0.37)
54.39.50.52:18080 20/184 (2509940 - 2509959, 1345 kB) 1514 kB/s (0.88)
136.56.35.26:18080 20/184 (2509960 - 2509979, 1600 kB) 1076 kB/s (0.44)
220.132.135.54:18080 20/184 (2509980 - 2509999, 1837 kB) 709 kB/s (0.37)
23.112.140.90:18080 20/184 (2510000 - 2510019, 1309 kB) 974 kB/s (0.64)
54.39.50.52:18080 20/184 (2510020 - 2510039, 1442 kB) 1629 kB/s (0.88)
84.75.29.59:18089 20/184 (2510040 - 2510059, 945 kB) 290 kB/s (0.22)
136.53.98.191:18080 20/184 (2510060 - 2510079, 946 kB) 140 kB/s (0.21)
136.56.35.26:18080 20/184 (2510080 - 2510099, 988 kB) 771 kB/s (0.44)
54.39.50.52:18080 20/184 (2510100 - 2510119, 1771 kB) 1935 kB/s (0.88)
136.56.35.26:18080 20/184 (2510120 - 2510139, 1118 kB) 822 kB/s (0.44)
220.132.135.54:18080 20/184 (2510140 - 2510159, 996 kB) 863 kB/s (0.37)
84.75.29.59:18089 20/184 (2510160 - 2510179, 1274 kB) 612 kB/s (0.22)
23.112.140.90:18080 20/184 (2510180 - 2510199, 1365 kB) 1060 kB/s (0.64)
136.53.98.191:18080 20/184 (2510200 - 2510219, 1740 kB) 463 kB/s (0.21)
54.39.50.52:18080 20/184 (2510220 - 2510239, 1712 kB) 1952 kB/s (0.88)
220.132.135.54:18080 20/184 (2510240 - 2510259, 1220 kB) 709 kB/s (0.37)
84.75.29.59:18089 20/184 (2510260 - 2510279, 1491 kB) 422 kB/s (0.22)
136.53.98.191:18080 20/184 (2510280 - 2510299, 2545 kB) 526 kB/s (0.21)
23.112.140.90:18080 20/184 (2510300 - 2510319, 2609 kB) 1762 kB/s (0.64)
136.56.35.26:18080 20/184 (2510320 - 2510339, 2047 kB) 1020 kB/s (0.44)
EDIT:
htop screenshots showing average resource usage
UPDATE:
~14h since launching Monero, monero-wallet-gui has apparently recognized that the blockchain isn't fully synced. However, nothing else changed.
