Running out of memory during reindexing (allow reindex of parts of the mailbox)

[petterreinholdtsen]

Hi.

I noticed the statement "Initial reindexing is memory-hungry and it could fail on some humongous mailbox (but I doubt anybody has such a mailbox)" in the README, and can confirm that this is a real problem. When trying, I ran out of memory with my 1.4 million email mailbox.

To work around it, I try a patch to only reindex parts of the mail box, using a patch like this to accept a tag argument to the reindex command:

diff --git a/notmuch-response b/notmuch-response
index 9d0f759..c7fbef7 100755
--- a/notmuch-response
+++ b/notmuch-response
@@ -69,25 +69,30 @@ def index():
         print('IDs to untag ({}): {}'.format(len(replied), ', '.join(replied)))
         untag_ids(replied)

-def reindex():
+def reindex(tags = None):
     # reindexes EVERYTHING, it may take some time
     #1. tag everything with unreplied
-    cmd_run([ 'notmuch', 'tag', '+noresponse', '-response', '*' ])
+    if tags is None:
+        tags = '*'
+    cmd_run([ 'notmuch', 'tag', '+noresponse', '-response', tags ])
     #3. get all reply ids
-    replied = get_replied_ids('*')
+    replied = get_replied_ids(tags)
     print('Replies to untag: {}'.format(len(replied)))
     #4. untag these ids
     untag_ids(replied)

 if __name__ == '__main__':
-    if len(sys.argv) != 2:
-        print('Usage: {} <index|reindex>'.format(sys.argv[0]))
+    if len(sys.argv) != 2 and len(sys.argv) != 3:
+        print('Usage: {} <index|reindex [tags]>'.format(sys.argv[0]))
         sys.exit(1)
     cmd = sys.argv[1]
     if cmd == 'index':
         index()
     elif cmd == 'reindex':
-        reindex()
+        tags = None
+        if 2 < len(sys.argv):
+            tags = sys.argv[2]
+        reindex(tags)
     else:
         print('Unknown command: {}'.format(cmd))
         sys.exit(1)

[thinred]

Hi,
I've pushed the devel branch, which contains a fix to this problem. Could you try it?
Unfortunately, it requires 'ijson' library (https://pypi.python.org/pypi/ijson/).

Reindexing takes ~ 1 minute for my 60000 messages, so I predict ~ 20 min for your mail.

[petterreinholdtsen]

[Tomasz Buchert]

Hi,
I've pushed the devel branch, which contains a fix to this
problem. Could you try it? Unfortunately, it requires 'ijson' library
(https://pypi.python.org/pypi/ijson/).

Thank you. Unfortunately, ijson is not available in the version of
Debian I am using, so it will take a while before I can test it. :)

Reindexing takes ~ 1 minute for my 60000 messages, so I predict ~ 20
min for your mail.

Do your index fit in memory? Mine is around 5x available RAM, causing a
lot of operations to take forever. :)

Happy hacking
Petter Reinholdtsen

[thinred]

Yeah, same here, I guess I'll have one more package to push to Debian...
In the meantime you can try it by just copy-pasting ijson directory in notmuch-response. Actually, I could just take relevant code directly, since it is under MIT.

My notmuch index has just 1GB, so it should fit just fine in RAM, but I doubt that notmuch does it. Still, when I think that each my operation takes maybe x20 with your index...

[thinred]

ijson is now in NEW queue :)
https://ftp-master.debian.org/new/python-ijson_2.2-1.html

[thinred]

And now in unstable => https://tracker.debian.org/pkg/python-ijson (and so in 5 days in testing!).