Attic updates fail when using 2026-06-06 database clone

[b1tw153]

When using the 2026-06-06 database clone with Overpass osm-3s_v0.7.62.11, attic updates fail with a core dump from update_from_dir while processing the batch of updates from 7142153 to 7142904.

Steps to reproduce:

1. Download https://dev.overpass-api.de/releases/osm-3s_v0.7.62.11.tar.gz, unpack, build, and install Overpass
2. Download a database clone using download_clone.sh --db-dir="/opt/overpass/db" --source="https://dev.overpass-api.de/api_drolbr/" --meta=attic; confirm that the download was successful
3. Start Overpass from the command line and confirm that the processes are running correctly:
   • dispatcher --osm-base --attic --db-dir="/opt/overpass/db" >>/opt/overpass/db/base_dispatcher.out 2>&1
   • dispatcher --areas --db-dir="/opt/overpass/db" >>/opt/overpass/db/areas_dispatcher.out 2>&1
   • apply_osc_to_db.sh "/opt/overpass/diff" auto --meta=attic >>/opt/overpass/db/apply_osc_to_db.out 2>&1
   • fetch_osc.sh auto "https://planet.openstreetmap.org/replication/minute/" "/opt/overpass/diff" >>/opt/overpass/diff/fetch_osc.out 2>&1
4. Allow Overpass to process updates until a core dump message is written to /opt/overpass/db/apply_osc_to_db.out

Complete apply_osc_to_db.log file:

2026-06-16 03:46:28: updating from 7141498
2026-06-16 03:46:33: updating from 7141498
2026-06-16 03:46:38: updating from 7141498
2026-06-16 03:46:43: updating from 7141498
2026-06-16 03:46:48: updating from 7141498
2026-06-16 03:46:53: updating from 7141498
2026-06-16 03:46:58: updating from 7141498
2026-06-16 03:47:03: updating from 7141498
2026-06-16 03:47:08: updating from 7141498
2026-06-16 03:47:13: updating from 7141498
2026-06-16 03:47:18: updating from 7141498
2026-06-16 03:47:23: updating from 7141498
2026-06-16 03:47:28: updating from 7141498
2026-06-16 03:47:33: updating from 7141498
2026-06-16 03:47:38: updating from 7141498
2026-06-16 03:47:43: updating from 7141498
2026-06-16 03:47:48: updating from 7141498
2026-06-16 03:47:53: updating from 7141498
2026-06-16 03:47:58: updating from 7141498
2026-06-16 03:48:03: updating from 7141498
2026-06-16 03:48:08: updating from 7141498
2026-06-16 03:48:13: updating from 7141498
2026-06-16 03:48:18: updating from 7141498
2026-06-16 03:48:23: updating from 7141498
2026-06-16 03:48:28: updating from 7141498
2026-06-16 03:48:33: updating from 7141498
2026-06-16 03:48:38: updating from 7141498
2026-06-16 03:48:43: updating from 7141498
2026-06-16 03:48:48: updating from 7141498
2026-06-16 03:48:53: updating from 7141498
2026-06-16 03:48:58: updating from 7141498
2026-06-16 03:49:03: updating from 7141498
2026-06-16 03:49:08: updating from 7141498
2026-06-16 03:49:08: updating to 7141500
2026-06-16 03:49:34: update complete 7141500
2026-06-16 03:49:34: updating from 7141500
2026-06-16 03:49:34: updating to 7141510
2026-06-16 03:51:19: update complete 7141510
2026-06-16 03:51:20: updating from 7141510
2026-06-16 03:51:20: updating to 7141551
2026-06-16 03:58:12: update complete 7141551
2026-06-16 03:58:12: updating from 7141551
2026-06-16 03:58:13: updating to 7141711
2026-06-16 04:20:07: update complete 7141711
2026-06-16 04:20:07: updating from 7141711
2026-06-16 04:20:13: updating to 7142153
2026-06-16 05:56:08: update complete 7142153
2026-06-16 05:56:08: updating from 7142153
2026-06-16 05:56:17: updating to 7142904

Tail from apply_osc_to_db.out file:

...
Version 1 has a later or equal timestamp (2026-06-04T15:44:49Z) than version 2 (2026-06-04T15:44:49Z) of Way 1526160858
Version 1 has a later or equal timestamp (2026-06-04T15:59:01Z) than version 2 (2026-06-04T15:59:01Z) of Way 1526164446
Version 1 has a later or equal timestamp (2026-06-04T15:59:02Z) than version 2 (2026-06-04T15:59:02Z) of Way 1526164447
Version 1 has a later or equal timestamp (2026-06-04T15:59:03Z) than version 2 (2026-06-04T15:59:03Z) of Way 1526164448
Version 2 has a later or equal timestamp (2026-06-04T16:41:29Z) than version 3 (2026-06-04T16:41:29Z) of Way 1526173955
Version 2 has a later or equal timestamp (2026-06-04T16:41:28Z) than version 3 (2026-06-04T16:41:28Z) of Way 1526173956
Version 1 has a later or equal timestamp (2026-06-04T17:02:10Z) than version 2 (2026-06-04T17:02:10Z) of Way 1526178664
Version 2 has a later or equal timestamp (2026-06-04T17:02:10Z) than version 3 (2026-06-04T17:02:10Z) of Way 1526178664
Version 1 has a later or equal timestamp (2026-06-04T17:02:12Z) than version 2 (2026-06-04T17:02:12Z) of Way 1526178666
Version 3 has a later or equal timestamp (2026-06-04T17:02:13Z) than version 4 (2026-06-04T17:02:13Z) of Way 1526178666
Version 2 has a later or equal timestamp (2026-06-04T17:02:35Z) than version 3 (2026-06-04T17:02:35Z) of Way 1526178765
Version 4 has a later or equal timestamp (2026-06-04T17:02:36Z) than version 5 (2026-06-04T17:02:36Z) of Way 1526178765
Version 1 has a later or equal timestamp (2026-06-04T17:02:41Z) than version 2 (2026-06-04T17:02:41Z) of Way 1526178772
Version 3 has a later or equal timestamp (2026-06-04T17:02:42Z) than version 4 (2026-06-04T17:02:42Z) of Way 1526178772
Version 1 has a later or equal timestamp (2026-06-04T17:02:54Z) than version 2 (2026-06-04T17:02:54Z) of Way 1526178782
Version 1 has a later or equal timestamp (2026-06-04T17:40:48Z) than version 2 (2026-06-04T17:40:48Z) of Way 1526188992
Version 1 has a later or equal timestamp (2026-06-04T18:14:11Z) than version 2 (2026-06-04T18:14:11Z) of Way 1526196829
Version 1 has a later or equal timestamp (2026-06-04T18:14:11Z) than version 2 (2026-06-04T18:14:11Z) of Way 1526196830
Version 1 has a later or equal timestamp (2026-06-04T18:14:11Z) than version 2 (2026-06-04T18:14:11Z) of Way 1526196831
Version 1 has a later or equal timestamp (2026-06-04T18:14:22Z) than version 2 (2026-06-04T18:14:22Z) of Way 1526196864
Version 1 has a later or equal timestamp (2026-06-04T18:53:31Z) than version 2 (2026-06-04T18:53:31Z) of Way 1526204671
Version 1 has a later or equal timestamp (2026-06-04T19:29:16Z) than version 2 (2026-06-04T19:29:16Z) of Way 1526211879
Version 1 has a later or equal timestamp (2026-06-04T19:59:53Z) than version 2 (2026-06-04T19:59:53Z) of Way 1526218542
Version 1 has a later or equal timestamp (2026-06-04T19:59:55Z) than version 2 (2026-06-04T19:59:55Z) of Way 1526218543
Version 1 has a later or equal timestamp (2026-06-04T19:59:55Z) than version 2 (2026-06-04T19:59:55Z) of Way 1526218544
Version 1 has a later or equal timestamp (2026-06-04T20:00:14Z) than version 2 (2026-06-04T20:00:14Z) of Way 1526218564
Version 1 has a later or equal timestamp (2026-06-04T20:00:16Z) than version 2 (2026-06-04T20:00:16Z) of Way 1526218566
Version 2 has a later or equal timestamp (2026-06-04T20:00:22Z) than version 3 (2026-06-04T20:00:22Z) of Way 1526218569
Version 3 has a later or equal timestamp (2026-06-04T20:00:22Z) than version 4 (2026-06-04T20:00:22Z) of Way 1526218569
Version 3 has a later or equal timestamp (2026-06-04T21:22:17Z) than version 4 (2026-06-04T21:22:17Z) of Way 1526238204
Compute current ... ready. Flushing to database ....... done. Compute attic ...Way 920867804 has changed at timestamp 2026-06-03T12:06:43Z in two different diffs.
terminate called after throwing an instance of 'std::length_error'
  what():  vector::reserve
bin/apply_osc_to_db.sh: line 92: 1581440 Aborted                 (core dumped) ./update_from_dir --osc-dir=$1 --version=$DATA_VERSION $META --flush-size=0

[b1tw153]

FWIW.

cat /var/crash/_opt_overpass_bin_update_from_dir.1000.crash
ProblemType: Crash
Architecture: amd64
Date: Tue Jun 16 08:12:01 2026
DistroRelease: Ubuntu 24.04
ExecutablePath: /opt/overpass/bin/update_from_dir
ExecutableTimestamp: 1781529189
ProcCmdline: ./update_from_dir --osc-dir=/tmp/osm-3s_update_yaYNMc --version=2026-06-05T01\\:45\\:34Z --keep-attic --flush-size=0
ProcCwd: /opt/overpass/bin
ProcEnviron:
 LANG=C.UTF-8
 PATH=(custom, user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
ProcMaps:
 57f583c11000-57f583c1a000 r--p 00000000 08:20 856188687                  /opt/overpass/bin/update_from_dir
 57f583c1a000-57f583e52000 r-xp 00009000 08:20 856188687                  /opt/overpass/bin/update_from_dir
 57f583e52000-57f583ea0000 r--p 00241000 08:20 856188687                  /opt/overpass/bin/update_from_dir
 57f583ea0000-57f583ea3000 r--p 0028f000 08:20 856188687                  /opt/overpass/bin/update_from_dir
 57f583ea3000-57f583ea4000 rw-p 00292000 08:20 856188687                  /opt/overpass/bin/update_from_dir
 57f583ea4000-57f583ea6000 rw-p 00000000 00:00 0
 57f5a299b000-57f63c605000 rw-p 00000000 00:00 0                          [heap]
 783cc8ee8000-783cd5d39000 rw-p 00000000 00:00 0
 783cd63fd000-783cdd3ff000 rw-p 00000000 00:00 0
 783cdfc00000-783cdfc28000 r--p 00000000 08:01 6506                       /usr/lib/x86_64-linux-gnu/libc.so.6
 783cdfc28000-783cdfdb0000 r-xp 00028000 08:01 6506                       /usr/lib/x86_64-linux-gnu/libc.so.6
 783cdfdb0000-783cdfdff000 r--p 001b0000 08:01 6506                       /usr/lib/x86_64-linux-gnu/libc.so.6
 783cdfdff000-783cdfe03000 r--p 001fe000 08:01 6506                       /usr/lib/x86_64-linux-gnu/libc.so.6
 783cdfe03000-783cdfe05000 rw-p 00202000 08:01 6506                       /usr/lib/x86_64-linux-gnu/libc.so.6
 783cdfe05000-783cdfe12000 rw-p 00000000 00:00 0
 783ce0000000-783ce009d000 r--p 00000000 08:01 6499                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 783ce009d000-783ce01e5000 r-xp 0009d000 08:01 6499                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 783ce01e5000-783ce026c000 r--p 001e5000 08:01 6499                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 783ce026c000-783ce0277000 r--p 0026b000 08:01 6499                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 783ce0277000-783ce027a000 rw-p 00276000 08:01 6499                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33
 783ce027a000-783ce027e000 rw-p 00000000 00:00 0
 783ce0281000-783ce0285000 rw-p 00000000 00:00 0
 783ce0285000-783ce0289000 r--p 00000000 08:01 6501                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 783ce0289000-783ce02ad000 r-xp 00004000 08:01 6501                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 783ce02ad000-783ce02b1000 r--p 00028000 08:01 6501                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 783ce02b1000-783ce02b2000 r--p 0002b000 08:01 6501                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 783ce02b2000-783ce02b3000 rw-p 0002c000 08:01 6501                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
 783ce02b3000-783ce02b5000 rw-p 00000000 00:00 0
 783ce02b5000-783ce02c5000 r--p 00000000 08:01 6509                       /usr/lib/x86_64-linux-gnu/libm.so.6
 783ce02c5000-783ce0344000 r-xp 00010000 08:01 6509                       /usr/lib/x86_64-linux-gnu/libm.so.6
 783ce0344000-783ce039c000 r--p 0008f000 08:01 6509                       /usr/lib/x86_64-linux-gnu/libm.so.6
 783ce039c000-783ce039d000 r--p 000e7000 08:01 6509                       /usr/lib/x86_64-linux-gnu/libm.so.6
 783ce039d000-783ce039e000 rw-p 000e8000 08:01 6509                       /usr/lib/x86_64-linux-gnu/libm.so.6
 783ce039e000-783ce03a2000 r--p 00000000 08:01 6260                       /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 783ce03a2000-783ce03bf000 r-xp 00004000 08:01 6260                       /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 783ce03bf000-783ce03c7000 r--p 00021000 08:01 6260                       /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 783ce03c7000-783ce03c9000 r--p 00028000 08:01 6260                       /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 783ce03c9000-783ce03ca000 rw-p 0002a000 08:01 6260                       /usr/lib/x86_64-linux-gnu/libexpat.so.1.9.1
 783ce03ca000-783ce03cd000 r--p 00000000 08:01 5145                       /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
 783ce03cd000-783ce03e8000 r-xp 00003000 08:01 5145                       /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
 783ce03e8000-783ce03ea000 r--p 0001e000 08:01 5145                       /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
 783ce03ea000-783ce03eb000 r--p 00020000 08:01 5145                       /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
 783ce03eb000-783ce03ec000 rw-p 00021000 08:01 5145                       /usr/lib/x86_64-linux-gnu/liblz4.so.1.9.4
 783ce03ec000-783ce03ee000 r--p 00000000 08:01 5155                       /usr/lib/x86_64-linux-gnu/libz.so.1.3
 783ce03ee000-783ce0400000 r-xp 00002000 08:01 5155                       /usr/lib/x86_64-linux-gnu/libz.so.1.3
 783ce0400000-783ce0406000 r--p 00014000 08:01 5155                       /usr/lib/x86_64-linux-gnu/libz.so.1.3
 783ce0406000-783ce0407000 r--p 0001a000 08:01 5155                       /usr/lib/x86_64-linux-gnu/libz.so.1.3
 783ce0407000-783ce0408000 rw-p 0001b000 08:01 5155                       /usr/lib/x86_64-linux-gnu/libz.so.1.3
 783ce040e000-783ce0410000 rw-p 00000000 00:00 0
 783ce0410000-783ce0414000 r--p 00000000 00:00 0                          [vvar]
 783ce0414000-783ce0416000 r--p 00000000 00:00 0                          [vvar_vclock]
 783ce0416000-783ce0418000 r-xp 00000000 00:00 0                          [vdso]
 783ce0418000-783ce0419000 r--p 00000000 08:01 6503                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 783ce0419000-783ce0444000 r-xp 00001000 08:01 6503                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 783ce0444000-783ce044e000 r--p 0002c000 08:01 6503                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 783ce044e000-783ce0450000 r--p 00036000 08:01 6503                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 783ce0450000-783ce0452000 rw-p 00038000 08:01 6503                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
 7ffdb8150000-7ffdb8171000 rw-p 00000000 00:00 0                          [stack]
 ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
ProcStatus:
 Name:  update_from_dir
 Umask: 0002
 State: S (sleeping)
 Tgid:  1581440
 Ngid:  0
 Pid:   1581440
 PPid:  1556728
 TracerPid:     0
 Uid:   1000    1000    1000    1000
 Gid:   1000    1000    1000    1000
 FDSize:        256
 Groups:        4 24 27 30 105 988 1000 10001
 NStgid:        1581440
 NSpid: 1581440
 NSpgid:        1556728
 NSsid: 1556677
 Kthread:       0
 VmPeak:         2854712 kB
 VmSize:         2854712 kB
 VmLck:        0 kB
 VmPin:        0 kB
 VmHWM:  2777488 kB
 VmRSS:  2777488 kB
 RssAnon:        2770920 kB
 RssFile:           6568 kB
 RssShmem:             0 kB
 VmData:         2845588 kB
 VmStk:      132 kB
 VmExe:     2272 kB
 VmLib:     4008 kB
 VmPTE:     5504 kB
 VmSwap:               0 kB
 HugetlbPages:         0 kB
 CoreDumping:   1
 THP_enabled:   1
 untag_mask:    0xffffffffffffffff
 Threads:       1
 SigQ:  0/15341
 SigPnd:        0000000000000000
 ShdPnd:        0000000000000000
 SigBlk:        0000000000000000
 SigIgn:        0000000000001007
 SigCgt:        0000000000004000
 CapInh:        0000000000000000
 CapPrm:        0000000000000000
 CapEff:        0000000000000000
 CapBnd:        000001ffffffffff
 CapAmb:        0000000000000000
 NoNewPrivs:    0
 Seccomp:       0
 Seccomp_filters:       0
 Speculation_Store_Bypass:      vulnerable
 SpeculationIndirectBranch:     always enabled
 Cpus_allowed:  3
 Cpus_allowed_list:     0-1
 Mems_allowed:  00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
 Mems_allowed_list:     0
 voluntary_ctxt_switches:       877037
 nonvoluntary_ctxt_switches:    5685
 x86_Thread_features:
 x86_Thread_features_locked:
Signal: 6
SignalName: SIGABRT
Uname: Linux 6.17.0-1018-azure x86_64
UserGroups: adm cdrom dip docker lxd sudo
_HooksRun: no

[b1tw153]

Also FWIW.

database.log:2026-06-16 05:56:17 [1581440] write_start() start version='2026-06-05T01\:45\:34Z'
database.log:2026-06-16 05:56:17 [1556703] write_start of process 1581440. Considered as reading:.
database.log:2026-06-16 05:56:17 [1581440] write_start() end
database.log:2026-06-16 05:56:26 [1581440] write() nodes_finished
database.log:2026-06-16 06:09:26 [1581440] write() compute_started
database.log:2026-06-16 06:09:27 [1581440] write() compute_finished
database.log:2026-06-16 06:17:04 [1581440] write() update_started
database.log:2026-06-16 06:17:04 [1581440] write() prepare_delete_tags_finished
database.log:2026-06-16 06:32:20 [1581440] write() update_ids_finished
database.log:2026-06-16 06:34:05 [1581440] write() update_coords_finished
database.log:2026-06-16 06:35:07 [1581440] write() meta_finished
database.log:2026-06-16 06:36:25 [1581440] write() tags_local_finished
database.log:2026-06-16 06:37:40 [1581440] write() tags_global_finished
database.log:2026-06-16 06:37:40 [1581440] write() current_update_finished
database.log:2026-06-16 06:47:34 [1581440] write() compute_attic_started
database.log:2026-06-16 06:47:36 [1581440] write() compute_attic_finished
database.log:2026-06-16 06:47:36 [1581440] write() attic_update_started
database.log:2026-06-16 07:01:37 [1581440] write() update_ids_finished
database.log:2026-06-16 07:02:28 [1581440] write() update_coords_finished
database.log:2026-06-16 07:02:40 [1581440] write() undeleted_finished
database.log:2026-06-16 07:03:23 [1581440] write() meta_finished
database.log:2026-06-16 07:04:23 [1581440] write() tags_local_finished
database.log:2026-06-16 07:07:09 [1581440] write() tags_global_finished
database.log:2026-06-16 07:07:10 [1581440] write() changelog_finished
database.log:2026-06-16 07:07:42 [1581440] write() update_finished
database.log:2026-06-16 07:07:42 [1581440] write() parser_started
database.log:2026-06-16 07:07:51 [1581440] write() ways_finished
database.log:2026-06-16 07:40:55 [1581440] write() compute_started
database.log:2026-06-16 07:40:57 [1581440] write() compute_finished
database.log:2026-06-16 07:40:57 [1581440] write() update_started
database.log:2026-06-16 07:40:57 [1581440] write() prepare_delete_tags_finished
database.log:2026-06-16 07:48:52 [1581440] write() update_ids_finished
database.log:2026-06-16 07:50:31 [1581440] write() update_coords_finished
database.log:2026-06-16 07:51:19 [1581440] write() meta_finished
database.log:2026-06-16 07:54:30 [1581440] write() tags_local_finished
database.log:2026-06-16 08:06:16 [1581440] write() tags_global_finished
database.log:2026-06-16 08:06:17 [1581440] write() current_update_finished
database.log:2026-06-16 08:11:59 [1581440] write() compute_attic_started
database.log:2026-06-16 08:12:21 [1556703] hangup of process 1581440.

[mmd-osm]

Might be related to #661. Running minutely updates and clone generation on the dev server at the same time could trigger some rare bug related to long running transactions. Maybe pause diff processing for some time while clone generation is ongoing. This needs to happen on the dev server, OP has no way to influence it as a consumer of clone files.

[b1tw153]

I reproduced the failure using the 2026-06-06 clone and did a RCA by instrumenting the code. Here are the results.

The Crash

update_from_dir aborts with std::length_error: vector::reserve during the way attic computation pass when processing update batches against a 2026-06-06 OSM database clone.

Crash Location

File: src/overpass_api/core/basic_types.h, line 354

target.reserve(reference.size() - removed.size() + added.size());

When removed.size() > reference.size() + added.size(), unsigned integer underflow produces a value exceeding vector::max_size(), throwing std::length_error.

Call Chain

update_from_dir
  → Way_Updater::update()
    → compute_new_attic_skeletons()           way_updater.cc
      → adapt_newest_existing_attic()         way_updater.cc
        → existing_delta.expand(existing_reference)
          → Way_Delta::expand()               type_way.h
            → expand_diff(...)                basic_types.h:354  ← CRASH

The Specific Failure

Way: 1050606078
Attic delta timestamp: 2026-06-03T12:05:09Z
Triggered by: diff 7142581 (explicit modification of this way)

The crash is geometry-only — the nds delta is valid:

existing_delta:  geom_removed=9  nds_removed=6  nds_added=2  geom_added=5
existing_reference:  geometry=0 (24 nodes, all in same tile)

geom_removed=9 > reference.geometry=0 + geom_added=5 → underflow → crash.

The attic delta asserts that 9 geometry entries can be removed from the reference skeleton, but the reference skeleton has no geometry entries at all.

Root Cause

Corrupt data in the 2026-06-06 clone.

Way 1050606078 does not appear in any diff from 7141498 through 7142468 (all batches between the clone and the crash). The inconsistency was present in the clone before any updates were applied.

The most likely cause: an uncontrolled process termination (OOM kill, SIGKILL, power loss, etc.) during a prior update on the clone generation. The update process commits current-data before computing attic data. If the process is killed during or after the current-data flush but before the attic write, the database is left with:

• Current skeleton: updated (in this case, nodes moved into the same tile → geometry=0)
• Attic delta: stale (still encodes geom_removed=9 relative to the old skeleton)

The 2026-06-06 clone captured this inconsistent state. Output to stderr from update_from_dir has many missing element references earlier in processing (68057 member ways missing from relations). This is further evidence of pre-existing data corruption in the clone. If there was an uncontrolled termination during a way current-data flush, that would leave an arbitrary number of ways absent or inconsistent.

The Code Bug

expand_diff has no precondition check for this underflow. The crash message (std::length_error: vector::reserve) gives no indication of which way is affected or why. The correct behavior is to detect the inconsistency and fail fast with a meaningful error message identifying the corrupt element, rather than crashing with an opaque exception.

Recommended Fix

In adapt_newest_existing_attic() (way_updater.cc), before calling existing_delta.expand(existing_reference), check:

if (!existing_delta.full)
{
  if (existing_delta.nds_removed.size() >
      existing_reference.nds.size() + existing_delta.nds_added.size()
      || existing_delta.geometry_removed.size() >
         existing_reference.geometry.size() + existing_delta.geometry_added.size())
  {
    std::cerr << "FATAL: corrupt attic delta for way " << existing_delta.id.val()
              << " (ts=" << Timestamp(existing_delta.timestamp).str() << "):"
              << " delta is inconsistent with current skeleton."
              << " Database may have been corrupted by an uncontrolled process"
              << " termination. Restore from a clean source and retry.\n";
    exit(1);
  }
}

This is not a recoverable situation. The attic delta cannot be correctly expanded without the original reference skeleton it was computed against. The process should abort with a clear error message so the operator knows to restore from a clean source rather dumping core with an opaque error message.

[b1tw153]

Might be related to #661. Running minutely updates and clone generation on the dev server at the same time could trigger some rare bug related to long running transactions. Maybe pause diff processing for some time while clone generation is ongoing.

That's an interesting hypothesis if it could result in attic data that is inconsistent with the current skeleton data.

In this particular case, it seems that there was much more missing from the 2026-06-06 clone. In the processing leading up to the crash, the update_from_dir output had a lot of error messages about missing elements that were referenced by other elements (e.g., relations that referenced missing ways). That suggests that the 2026-06-06 clone was broken in a substantial way, not in a subtle way.

[mmd-osm]

I’m not suggesting that #661 is about a subtle change. Once you start processing data that has been overwritten by mistake, all sorts of inconsistencies could happen.

Good to know trust there were many errors before the crash was happening. Attic data is fairly picky about inconsistencies, and the smallest issue could break everything.

I see if I can take a look at the log files. Maybe there was some issue reported during clone generation.

[b1tw153]

I’m not suggesting that #661 is about a subtle change. Once you start processing data that has been overwritten by mistake, all sorts of inconsistencies could happen.

Agreed. The other error messages in the log implied that there were inconsistencies in the base data as well as the attic data. It's just that the base data inconsistencies did not cause crashes.

It would be interesting to know if the log files point to a cause of the corruption!