Unable to add home.pl IMAP account - parsing error

Sync Issues
, , , ,
#1

There used to be a problem syncing home.pl accounts (issue #967), which was solved. However, I encountered another error last week. I tried to send an e-mail from my account using a slightly unstable network connection. The mail spent over an hour in the Draft folder and Mailspring kept showing the “Sending message” indicator, and couldn’t synchronize the account, and same thing happened if I restarted the app. I decided to remove the account and add it again, since it had sometimes helped with some issues in the past. However, now I’m unable to add the account at all - I get the Parsing error (IMAP) message. I’m using Mailspring 1.7.2-5422b259 on Fedora 31. Log attached below. Interesting fact: I can add the account without any hiccups to Thunderbird and Evolution, and I can add it to Geary, but there it fails to sync.

----------IMAP----------
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot ready.
1 LOGIN "leszek@dziubek.biz.pl" *********
1 OK [CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT SORT=DISPLAY UIDPLUS UNSELECT QUOTA MOVE WITHIN LIST-EXTENDED LIST-STATUS SPECIAL-USE XLIST STARTTLS] Completed
2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK Completed
3 XLIST "" "*"
* XLIST (\HasNoChildren \) "." "Argos"
* XLIST (\HasNoChildren \Draft) "." "DRAFTS"
* XLIST (\HasNoChildren \) "." "Euron"
* XLIST (\HasNoChildren \) "." "Gambit"
* XLIST (\HasNoChildren \Inbox) "." "INBOX"
* XLIST (\HasNoChildren \) "." "Inne"
* XLIST (\HasNoChildren \) "." "Ksi&ARk-gowo&AVsBBw-"
* XLIST (\HasNoChildren \) "." "Lionbridge"
* XLIST (\HasNoChildren \) "." "Locaria"
* XLIST (\HasNoChildren \) "." "Proz"
* XLIST (\HasNoChildren \) "." "SDL"
* XLIST (\HasNoChildren \Sent) "." "SENT"
* XLIST (\HasNoChildren \Spam) "." "SPAM"
* XLIST (\HasNoChildren \) "." "Sopoltrad"
* XLIST (\HasNoChildren \Trash) "." "TRASH"
* XLIST (\HasNoChildren \) "." "Test"
* XLIST (\HasNoChildren \) "." "confirmed-ham"
* XLIST (\HasNoChildren \) "." "confirmed-spam"
3 OK Completed
XLIST (\HasNoChildren \) "." "Argos"

I’m not sure if this will be of any help, but I’ll also attach logs from Geary:

Account identifier: account_01
Account provider: GEARY_SERVICE_PROVIDER_OTHER
Service type: GEARY_PROTOCOL_IMAP
Service host: serwer1913971.home.pl
Error type: GearyImapError 2
Message: a003 xlist: Connection channels closed

Back trace:
 * unknown
 * unknown
 * unknown
 * unknown
 * g_closure_invoke
 * g_signal_handler_disconnect
 * g_signal_emit_valist
 * g_signal_emit
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * g_subprocess_communicate_utf8
 * g_task_attach_source
 * unknown
 * unknown
 * g_list_sort_with_data
 * g_main_context_dispatch
 * g_main_context_dispatch
 * g_main_context_iteration
 * g_application_run
 * unknown
 * __libc_start_main
 * unknown

(Originally posted by ldziubek on GitHub.)

#2

Same here. Imap error with home.pl

`----------IMAP----------

* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot ready.
1 LOGIN "xxx@xxx" *********
1 OK [CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT SORT=DISPLAY UIDPLUS UNSELECT QUOTA MOVE WITHIN LIST-EXTENDED LIST-STATUS SPECIAL-USE XLIST STARTTLS] Completed
2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK Completed
3 XLIST "" "*"
* XLIST (\HasNoChildren \Draft) "." "DRAFTS"
* XLIST (\HasNoChildren \Inbox) "." "INBOX"
* XLIST (\HasChildren ) "." "Mailspring"
* XLIST (\HasNoChildren ) "." "Mailspring.Snoozed"
* XLIST (\HasNoChildren \Sent) "." "SENT"
* XLIST (\HasNoChildren \Spam) "." "SPAM"
* XLIST (\HasNoChildren \Trash) "." "TRASH"
* XLIST (\HasNoChildren ) "." "confirmed-ham"
* XLIST (\HasNoChildren ) "." "confirmed-spam"
3 OK Completed
XLIST (\HasChildren ) "." "Mailspring"`

Any solution for this?

(Originally posted by PanchoKce on GitHub.)

#3

Hey folks! Thanks for filing this—the log excerpts you’ve provided all have this sort of pattern, where one of the XLIST response lines is arriving AFTER the OK Completed message:

* XLIST (\HasNoChildren \) "." "confirmed-spam"
3 OK Completed
XLIST (\HasNoChildren \) "." "Argos"

I’m not sure why that’s the case - it seems like the home.pl server is sending back items after it’s said that it’s finished, which is an IMAP error. Thunderbird might just ignore those folders (Argos in this example), instead of complaining - hard to say. I’m afraid I don’t have any suggestions - we can try to make Mailspring less strict in parsing these responses, but we use an open-source library to read/write IMAP commands so it’s not part of this immediate project.

(Originally posted by bengotow on GitHub.)

#4

Everyone, if you’d still like to see this problem fixed, the best chance may be to write a message to home.pl tech support, providing description of the problem and link to this issue thread. The problem is actually caused by incorrect IMAP configuration on the home.pl end, it can be worked around, but they should still fix it. I have sent them a message, if more people complain about this, they may do something about it.

(Originally posted by ldziubek on GitHub.)

#5

Yes, it still doesn’t work, completely preventing the use of Mailspring … and home.pl is one of the largest hosting company in Poland - they won’t be interested either … as I know them.

but the weird thing is that mailspring worked perfectly for several months … up to a point … :frowning:

(Originally posted by PanchoKce on GitHub.)

#6

I know, it was the same for me - home.pl changed something in their configuration. I don’t hold much hope that they will respond to my message, if they get a number of complaints, they may eventually fix it. It’s definitely not just a Mailspring issue, I had the same problem with Geary and their dev actually managed to prepare a working branch for me today, they will probably include a fix in a future release - but as I mentioned, it is just a workaround allowing the client to ignore a faulty response from server.

(Originally posted by ldziubek on GitHub.)

#7

Reported this problem to home and finally they have admitted that their current platform Idea is “not supported” by springmail/spark.
They proposed to change my account to dovecot, which should fix this issue (no changes on my side), so i guess they will not be fixing it on all accounts by default anytime soon.

Anyone who still has this issue should contact home support directly to fix it.

(Originally posted by FilipCEZOS on GitHub.)

#8

I’m still not convinced it is a “not-us” issue. I mean: home.pl does have its problems, but in this case I did some research and I wouldn’t be so sure it’s not the Mailspring fault. Let me explain: I made a test account and on it I replicated some of my other problematic account’s folder structure (using home.pl provided webmail interface). Then I attempted to add this test account to Mailspring. Unfortunately, Mailspring reported a parsing problem right before the account configuration could be finalized. I clicked on the “View Log” link and this is what I saw:

----------IMAP----------
connect <mailcore::IMAPSession:00BFF7A0>
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
ssl connect <server>.home.pl 993 2
OpenSSL version: OpenSSL 1.1.0f  25 May 2017
connect ok
login
1 LOGIN "mailspring@<server>.pl" *********
1 OK [CAPABILITY IMAP4rev1 SASL-IR LITERAL+ CHILDREN I18NLEVEL=1 ID IDLE SORT SORT=DISPLAY UIDPLUS UNSELECT QUOTA MOVE WITHIN LIST-EXTENDED LIST-STATUS SPECIAL-USE XLIST STARTTLS] Completed
2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK Completed
login ok
3 XLIST "" "*"
* XLIST (\HasChildren \) "." "Archive"
* XLIST (\HasNoChildren \) "." "Archive.2020"
* XLIST (\HasNoChildren \) "." "Archive.2021"
* XLIST (\HasNoChildren \) "." "Courses"
* XLIST (\HasNoChildren \Draft) "." "DRAFTS"
* XLIST (\HasNoChildren \Inbox) "." "INBOX"
* XLIST (\HasChildren \) "." "Mailspring"
* XLIST (\HasNoChildren \) "." "Mailspring.Snoozed"
* XLIST (\HasChildren \) "." "PhD"
* XLIST (\HasNoChildren \) "." "PhD. Journals"
* XLIST (\HasNoChildren \) "." "PhD. Thesis"
* XLIST (\HasNoChildren \) "." "PhD.Advisor"
* XLIST (\HasNoChildren \) "." "PhD.Archive"
* XLIST (\HasNoChildren \) "." "PhD.Articles"
* XLIST (\HasNoChildren \) "." "PhD.Conferences"
* XLIST (\HasChildren \) "." "Projects"
* XLIST (\HasNoChildren \) "." "Projects.Book in LaTeX"
* XLIST (\HasNoChildren \) "." "Projects.E-maps"
* XLIST (\HasNoChildren \) "." "Projects.FTP"
* XLIST (\HasNoChildren \) "." "Projects.Sklep"
* XLIST (\HasNoChildren \Sent) "." "SENT"
* XLIST (\HasNoChildren \Spam) "." "SPAM"
* XLIST (\HaXLIST (\HasChildren \) "." "Archive"

Well… fine! Might be the server. As somebody mentioned home.pl indeed has/had their “own” server solution (infamous Idea*Server), but this time it’s Dovecot reporting itself right after the connection is made. So I turned to the bare terminal (openssl s_client -connect .home.pl:993) and got the following (I omitted the TLS handshake details and redacted password):

* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
1 LOGIN "mailspring@<server>.pl" "**********"
1 OK [CAPABILITY IMAP4rev1 SASL-IR LITERAL+ CHILDREN I18NLEVEL=1 ID IDLE SORT SORT=DISPLAY UIDPLUS UNSELECT QUOTA MOVE WITHIN LIST-EXTENDED LIST-STATUS SPECIAL-USE XLIST STARTTLS] Completed
2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK Completed
3 XLIST "" "*"
* XLIST (\HasChildren \) "." "Archive"
* XLIST (\HasNoChildren \) "." "Archive.2020"
* XLIST (\HasNoChildren \) "." "Archive.2021"
* XLIST (\HasNoChildren \) "." "Courses"
* XLIST (\HasNoChildren \Draft) "." "DRAFTS"
* XLIST (\HasNoChildren \Inbox) "." "INBOX"
* XLIST (\HasChildren \) "." "Mailspring"
* XLIST (\HasNoChildren \) "." "Mailspring.Snoozed"
* XLIST (\HasChildren \) "." "PhD"
* XLIST (\HasNoChildren \) "." "PhD. Journals"
* XLIST (\HasNoChildren \) "." "PhD. Thesis"
* XLIST (\HasNoChildren \) "." "PhD.Advisor"
* XLIST (\HasNoChildren \) "." "PhD.Archive"
* XLIST (\HasNoChildren \) "." "PhD.Articles"
* XLIST (\HasNoChildren \) "." "PhD.Conferences"
* XLIST (\HasChildren \) "." "Projects"
* XLIST (\HasNoChildren \) "." "Projects.Book in LaTeX"
* XLIST (\HasNoChildren \) "." "Projects.E-maps"
* XLIST (\HasNoChildren \) "." "Projects.FTP"
* XLIST (\HasNoChildren \) "." "Projects.Sklep"
* XLIST (\HasNoChildren \Sent) "." "SENT"
* XLIST (\HasNoChildren \Spam) "." "SPAM"
* XLIST (\HasNoChildren \) "." "Studio"
* XLIST (\HasNoChildren \Trash) "." "TRASH"
* XLIST (\HasNoChildren \) "." "confirmed-ham"
* XLIST (\HasNoChildren \) "." "confirmed-spam"
3 OK Completed
4 LOGOUT
* BYE IMAP4rev1 logging out
4 OK Goodbye
closed

Pay close attention the the end of the * XLIST block. Something is fishy about what’s being reported by Mailspring in the log. Either the log is not giving a proper look at what happens behind the scenes or something is off with the communication routines/buffers, since the terminal connection appears to be fine.

It should be noted that I also redacted my server name.

Could somebody, please take a look at that? I’m more than willing to provide you my test account (details on the priv) if you’d like.

And one more thing: I did use Mailspring with my home.pl accounts a couple of versions back.

1 Like
#9

Indeed your output looks strange (log gets weird right after 1024 characters of XLIST output if I count correctly). For me home.pl fails for much simpler folder structure without such artifacts, just like in the original case.

I don’t know if it helps, but I made a mock-up service and tried to play around with how Mailspring reacts on various XLIST output, and it turns out that the whole problem with home.pl output could be that it contains the “(\HasNoChildren \)” part, especially the second backslash there seems to be the culprit.

Have a look at the log output for an attempt where I sumulated such output and it failed on the such entry (I made it appear as second on the folder list). This is the log saved by Mailspring (I cut it so it starts after login):

2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK List completed (0.001 + 0.000 secs).
login ok
3 XLIST "" "*"
* XLIST (\HasNoChildren \Test) "." "Test"
* XLIST (\HasNoChildren \) "." "ARCHIVE"
* XLIST (\HasNoChildren \Other) "." "Other"
3 OK Completed
XLIST (\HasNoChildren \) "." "ARCHIVE"

If failed right after retrieving the output of the 3 XLIST command, and the extra line at the end seems to be the entry added to log by Mailspring to show where it had problems parsing, because that was not part of the actual transmission.
One more example with folders rearranged (problematic entry as the last one this time):

2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK List completed (0.001 + 0.000 secs).
login ok
3 XLIST "" "*"
* XLIST (\HasNoChildren \Test) "." "Test"
* XLIST (\HasNoChildren \Other) "." "Other"
* XLIST (\HasNoChildren \) "." "ARCHIVE"
3 OK Completed
XLIST (\HasNoChildren \) "." "ARCHIVE"

And that also failed on the same entry.

Now when I changed that entry to say just “(\HasNoChildren)” without the extra backslash, it went through just fine, Mailspring apparently parsed the XLIST just fine , because it went on to issuing the fourth command, and basically reached the end of my mock-up:

2 LIST "" ""
* LIST (\Noselect) "." ""
2 OK List completed (0.001 + 0.000 secs).
login ok
3 XLIST "" "*"
* XLIST (\HasNoChildren \Test) "." "Test"
* XLIST (\HasNoChildren) "." "ARCHIVE"
* XLIST (\HasNoChildren \Other) "." "Other"
3 OK Completed
4 LIST "" "INBOX"
Mock-up ends here

So IMHO whatever fails here is the parsing of the XLIST output for cases where the bracketed part contains a lone backslash as the second parameter (which happens a lot for home.pl). Unfortunately I couldn’t find any RFCs to see if such structure is OK or not for IMAP. I also tried to look at the libetpan parts in the Mailspring-Sync code hoping to pinpoint the problem and fix it myself, but it quickly turned out that the C code there was just too much for me…

1 Like
#10

Thank you for taking time to check that yourself! I’m very grateful.

I took a look at RFCs and some Google and Microsoft docs and as far as I can understand this lone backslashes are indeed fishy. Not that they appear after \HasNoChildren flag, but because they are “empty” so to speak. I failed to find any mention of such a bizarre flag in RFC 6154, RFC 3348 or RFC 3501. \Drafts, \All etc. yes, but not that. But, oh well… it is what it is and we have to deal with it anyway ;}.

I think your intuition that there’s something wrong with the parser is more than justifiable (in fact bordering “spot on”). That backslash in front of parenthesis looks awfully like an escape character and I’d bet that’s exactly how parser treats it. In that case we would have a sequence interpreted as a literal ) instead of closing parenthesis for the flags section… which never really terminates… unlike parser ;}.

One question though: did you use Dovecot for your experiments? Perhaps it’s a Dovecot problem? But that’s simply out of curiosity. Whether it’s “Dovecot or not” we have a problem to be dealt with. Even though Google says that XLIST is deprecated it is still used and even if we’d switched to the recommended special-use list extension, we’d most probably hit the same thing anyway.

I’d love to help more, but my time is quite limited (surprise, surprise ;}), but that being said, I could take a look at the C code if you’d be so kind as to precisely point with your finger suspicious place… or file at least :}. Perhaps I’d see something interesting.

#11

I wasn’t using dovecot but a simple tcp api stitched together in node-red to simulate simple request-response communication (I just quickly wanted to have something to talk with Mailspring). This could be a dovecot bug, so I may try setting one up just to check… when I find time :slight_smile:

Nevertheless, I was able to reach this point (line 212 in Mailspring-Sync/Vendor/libetpan/src/low-level/imap/xlist.c). I tried to go deeper but quickly got lost on the next step (which I guess is this).

1 Like
#12

I see. Well, whether it’s Dovecot bug or nor, it’s out there and we can’t count on it disappearing anytime soon, so… should we handle it like a browser handles some malformed HTML? I think that’s the way (beside optional complaining to Dovecot devs :wink:).

I took a look at libetpan parts earlier and I pretty much followed your path arriving at mailimap_mbx_list_flags_parse definition, but… before we dive deeper into that (yes, I’m not thrilled to set up and compile a stub parsing machine on my own if I can avoid it :wink:), I think we should make sure the backend parser really gets what it should get. I’m saying that because Claws Mail also uses libetpan and it doesn’t have any problems with home.pl. It could be that it does some pre-processing to the IMAP response, but I don’t think that is the case.

There’s also mailcore2 with MCIMAPSession.cpp and a fair bit of folder list processing, but nothing obvious.

One more thing: I noticed that home.pl in its capabilities lists, beside XLIST (which is deprecated anyway), also LIST-EXTENDED. I tried it in terminal and it seems to work fine, that is: it produces extended list without ominous backslash preceded closing parentheses. I think the XLIST problem cannot be ignored, but that’s a different matter. In the meantime, how about giving preference to utilizing LIST-EXTENDED when both this and XLIST capabilities are available? That would be sensible and more future proof in light of XLIST deprecation.

#13

Oh man… This is a can of worms…

So, it appears that it is a “not-us” issue after all… kinda… sorta…

I did some digging in the code, especially in mailcore2 and libetpan which, admittedly are “not us”, so you were right all along. My bad ;}. I compiled libetpan and slightly modified their sample program to do folder listing.

So, what did I find? Two problems, one in each of the mentioned libraries. First, there’s a code in mailcore2 which gives preference to XLIST over LIST even if the latter may be extended. Not the most elegant of behaviours, if you ask me. Second, there is a problem (?) in libetpan in mailimap_xlist function. I gave the word “problem” a question mark, because, as we’ve already seen, it’s not so much the library’s problem—more like a bizarre Dovecot behaviour. Anyway, I did my check with sister function mailimap_list and all appeared fine, but when I tried to do the same with mailimap_xlist… parser error (surprise, surprise ;}).

I think this calls for a fair bit of nagging on mailcore2 and libetpan issue tracker or home.pl support (I’ll do it).

I understand you won’t like to mess with 3rd party libraries (I would avoid this rabbit hole as well), but if I may recommend a quick (and dirty) fix, it would be changing this:

    if (mXListEnabled) {
        r = mailimap_xlist(mImap, MCUTF8(prefix), "*", &imap_folders);
    }
    else {
        r = mailimap_list(mImap, MCUTF8(prefix), "*", &imap_folders);
    }

to this:

r = mailimap_list(mImap, MCUTF8(prefix), "*", &imap_folders);

in Vendor/mailcore2/src/core/imap/MCIMAPSession.cpp, lines 1554-1559. It’s a piece of code in IMAPSession::fetchAllFolders method which gets called by the main MailSync process in test mode in runTestAuth function. And that’s why we can’t add the darned account :}.

The change basically cuts away the use of XLIST entirely. It may cause its own problems, but, let’s face it: XLIST is in the death row since at least 2013. Besides, Claws Mail does not use mailimap_xlist at all and I doubt there are any complaints about it.

I leave the rest to you :}

Oh! And thank you for taking interest in this matter! :+1: :vulcan_salute: :grinning:

#14

Hey man, great investigation job over there!
I guess now it’s time to learn how to compile mailsync myself to finally add my own home.pl account :slight_smile: (BTW - I’m not the owner/contributor - just a regular home.pl’s dovecot implementation victim as you, but with much less C++ exposure :wink:, so I don’t feel confident making pull requests)
I think that in my case, instead of taking the XLIST entierly, I’ll just disable it for Dovecot servers (basically extending this line: 4292)…

#15

My pleasure! :blush:

“home.pl’s Dovecot implementation victim” is spot on description of… well… all this :rofl:. Let’s just hope, that soon we will be able to replace “victim” with “survivor” ;}

Aaaand suddenly you made C/C++ programming look like something measured in sieverts :wink:. Let’s just say in my case it’s “not great, not terrible” :wink:.

Seriously, though: extending the checks you’ve mentioned for Dovecot looks like a sensible workaround. In the meantime, I intend to try to convince mailcore2 devs to try to use LIST-EXTENDED instead of XLIST wherever possible and have a talk with home.pl and perhaps suggest to libetpan to be more forgiving in flag parsing.

#16

I reported the relevant issues to libetpan, mailcore2 and home.pl.

For reference:

Let us see what will stick to the wall :smiley:

#17

Thanks so much, @thebodzio! We really appreciate your help diagnosing this weird issue.

I’ve manually bumped you up to “regular user” instead of “new user”. That should lift most of those restrictions. I think your effort above more than earns that. :smiley:

1 Like
#18

It’s my pleasure! :smiley:

That’s the least I can do to thank for your work put in Mailspring and publishing the source :smiley:.

And thank you for the promotion :smiley:. It’s much appreciated!

1 Like
#19

@thebodzio I see that none of the issues have any follow-up, have you been able to resolve the problem otherwise?

#20

Nope. Nothing. Unfortunately: el zoicho :neutral_face:. I had no further response from home.pl (when I posted about the problem on their forum, they said that “they will look into that”, so…). The same with the libetpan and mailcore2.

I wish I could have better news, but it is as it is :neutral_face: