feat: add IMAP IDLE support for getmail - #4675

Documentation preview for this PR is ready! 🎉

Built with commit: b9477ef

Thank you for your contribution. Just wanted to let you know, that we are quiet busy and review will take same time.

I will also take a look, but all PRs are currently blocked for me because of #4536

Oh I thought this was scheduled for v16 😓

I've had a glance over the PR diff but I would like to try take a simpler approach which I've outlined here: #4713 (comment)

That's dependent upon upstream and DMS switching back to using the PyPi package instead of the Debian one to benefit from that.

Taking the approach I've outlined would roughly reduce down to this:

[program:getmail-parallel]
numprocs = 2
numprocs_start = 1
process_name = %(program_name)s-%(process_num)s
autorestart = true
stopasgroup = true
command = bash -c 'getmail --getmaildir /var/lib/getmail --rcfile getmailrc-account-%(process_num)s && sleep %(ENV_GETMAIL_POLL)sm'

NOTE: Until such a PR is feasible, any DMS user could add the above as a config mount into /etc/supervisor/conf.d/ and with a complimentary user-patches.sh file (docs) that places the --rcfile referenced entries into the --getmaildir location (using a naming convention that can utilize %(process_num)s), should be simple enough to iterate over and rename/symlink such file names.

For the IDLE support, add the --idle=INBOX option. If you need something more flexible, a script should work. Instead of an ENV to split up, you could use the service process_num expression (perhaps via the supervisord environment setting, similar to what you already did for SERVICE_NAME) and take a list of files and sort them ($(ls -1 /var/lib/getmail | sort)?) then use the index on that to select the appropriate file-name. You could then have a regex for the filename suffix .idle-INBOX to detect when to add --idle and the value to assign it? (seemed simpler than the ENV or a JSON/YAML/TOML mapping)

Or adapt the approach you've got here. It's just more of a hassle to merge a feature when it could be greatly simplified by getting support for the idle setting in config files upstream 😅

I spent more time on investigating this integration with upstream.

• As it stands the --idle option has only been relevant for what mailbox to monitor over IMAP for notification of when to restart Getmail's processing loop.
• mailboxes must be configured as this is what actually is used to retrieve over IMAP, the --idle is just a trigger event. As such the default mailboxes = ("INBOX",) would result in whatever mailbox --idle subscribes to as pulling from the primary INBOX instead. Something this PR with it's test implementation was not covering, thus not caught.

The latest unreleased Getmail commits have added the ability to use --idle= (no value assigned) to exclude a mailbox value and infer it from the first item of mailboxes. I'd still rather push for a breaking change to treat --idle as a boolean toggle like Fetchmail and have that supported in configs, but upstream has been a bit reluctant on this (the project being named getmail6 doesn't lend too well to semver and breaking changes..).

Arguably the feature itself as I have described it (and as it is vaguely documented at the Getmail6 project) is in itself broken and not working as implied/documented.

Furthermore, I'd like to know the benefit of enabling/supporting IDLE, given the single mailbox limitation?

• What advantages are being sought out with long-lived connections over multiple Getmail instances, instead of polling at a shorter interval?
• Or rather... Is there an actual problem that IMAP IDLE is solving vs using Polling? How many mailboxes are you configuring, is it more than the standard INBOX per account?

I assume it's very few remote accounts and quite possibly only the default INBOX being monitored. I'm just curious how important the feature actually is to you, or if it's more of just an ick with polling often instead of receiving a notification from the remote server?

Related history for Fetchmail idle support:

• Solve Fetchmail imap idle issue #1728 (Feature request)
• Solve Fetchmail imap idle issue #1730 (merged via this commit)

Personally I don't consider the single folder a limitation.
As you say, it will mostly only monitor the inbox, except in some odd use cases someone might want to change this (to only get mails filtered by sieve).
If we can get this feature implemented I plan to configure it for 7 accounts to download mails.

The issue I find with polling is that you need to get the interval down to a minute maximum.
Due to all these security featues on various sites where they send you a mail to confirm it's you but these links are only valid for a couple of minutes. If your interval is too long you have to make several attempts before you can successfully login (hoping you don't get locked out after the 5th attempt).

The issue I find with polling is that you need to get the interval down to a minute maximum.

Fair enough, thank you :)

As you say, it will mostly only monitor the inbox, except in some odd use cases someone might want to change this (to only get mails filtered by sieve).

I've yet to get around to filing a bug report properly upstream, but I have pointed it out to the maintainer already that the --idle flag is implemented poorly.

It is designed to specify a mailbox to receive a notification about updates from, and yet there's no documentation to note that configuring that alone is insufficient if your mailbox is not INBOX. Fetchmail implemented the feature far better.

As this PR lacks a test case for catching that, users are just as likely to make that mistake unaware of it (I'm kind of surprised that it hasn't been raised as a bug report upstream thus far, but perhaps nobody is bothering to use Getmail with mailboxes other than INBOX 🤷‍♂️)

I'll return to this PR after I've approved the Dovecot 2.4 / Debian 13 upgrade PR. I'd much prefer convincing upstream to improve so integration in DMS has reduced complexity.

Apologies for the delay that causes here, but thus far I think you're the only user that's requested support for the feature in DMS 😅 I don't mind supporting integrations that can justify the convenience for users well enough, but I also would very much like it done right so that it's minimal burden to maintainers to carry.

@brainkiller It took me a while, but we are now ready to look at this PR again. Please rebase onto the latest master to resolve the conflicts. I will then be reviewing this PR so we can go forward with it.

Note: In general, a discussion is not sufficient to get a new feature in like that because it's likely to be unnoticed by maintainers like me. But looking at the discussion at hand, I think it's fair. Please open an issue next time, though :)

Damn looks like I closed this PR accidently while trying to re-base. I'm re-opening it with my proposed changes

No worries, I'll be looking into it shortly.