Hi,

Nice script! Is there a way to get it to iterate through all folders automatically? I have quite a large number of folders and I would prefer no to have to list them all out. The other question is - if I do have to list them all out, how does one specify nested folders in your structure. Unforutunately I am no programmer so help in its simplest form would be great!

Thanks again

Hi, i received these errors: I have ruby-1.8.6 and cyrus-imapd-2.3.9

$ imapGmail [localhost] connecting... [localhost] logging in... [imap.gmail.com] connecting... [imap.gmail.com] logging in... [localhost] selecting folder 'INBOX.HUP'... [imap.gmail.com] selecting folder 'INBOX.HUP'... [imap.gmail.com] analyzing existing messages... /usr/local/lib/ruby/1.8/net/imap.rb:982:in `pick_up_tagged_response': Could not parse command (Net::IMAP::BadResponseError) from /usr/local/lib/ruby/1.8/net/imap.rb:973:in `get_tagged_response' from /usr/local/lib/ruby/1.8/net/imap.rb:1031:in `send_command' from /usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize' from /usr/local/lib/ruby/1.8/net/imap.rb:1016:in `send_command' from /usr/local/lib/ruby/1.8/net/imap.rb:1169:in `fetch_internal' from /usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize' from /usr/local/lib/ruby/1.8/net/imap.rb:1167:in `fetch_internal' from /usr/local/lib/ruby/1.8/net/imap.rb:716:in `uid_fetch' from /home/gyula/bin/imapGmail:83 from /home/gyula/bin/imapGmail:53:in `each' from /home/gyula/bin/imapGmail:53

Hello,

imapsync 1.223 was buggy with dates and --syncinternaldates, 1.219 wasn't, 1.233 isn't.

Your Ruby code is nice. Is it GPL? Can I make a reference ti it in imapsync distribution?

Aha! I was indeed using version 1.223. Thanks.

Please consider this code public domain (and unsupported). You're more than welcome to refer to it if you'd like.

Man, I hate to use you as a source of tech support, but I just can't figure this out for myself, being a PHP guy and not a Ruby guy. I managed to synch my inbox just fine, but I'm having no luck with any other folders. For instance, this mapping:

imap/Receipts => Receipts

spits out the following error:

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/imap.rb:982:in `pick_up_tagged_response': Could not parse command (Net::IMAP::BadResponseError)

on my Leopard install. (It goes on from there, but I've clipped it for brevity's sake.) I'm under the impression that Leopard has a brilliant default Ruby setup, so I imagine that everything's A-OK there. Might you -- or some other commenter -- be able to point me in the right direction to figure out how to fix this?

I too am getting the same thing. Seems to be happening when the Inbox is a hierarchical one. My inbox structure looks like this: INBOX INBOX.folder1 INBOX.folder2 INBOX.folder3

etc etc... But only the main INBOX is migrated.. Here is the full output below..

Anyone figured this out?? Thanks

[imap.gmail.com] selecting folder 'INBOX.DHA'... [imap.gmail.com] folder not found; creating... [imap.gmail.com] analyzing existing messages... /usr/local/lib/ruby/1.8/net/imap.rb:982:in `pick_up_tagged_response': Could not parse command (Net::IMAP::BadResponseError) from /usr/local/lib/ruby/1.8/net/imap.rb:973:in `get_tagged_response' from /usr/local/lib/ruby/1.8/net/imap.rb:1031:in `send_command' from /usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize' from /usr/local/lib/ruby/1.8/net/imap.rb:1016:in `send_command' from /usr/local/lib/ruby/1.8/net/imap.rb:1169:in `fetch_internal' from /usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize' from /usr/local/lib/ruby/1.8/net/imap.rb:1167:in `fetch_internal' from /usr/local/lib/ruby/1.8/net/imap.rb:716:in `uid_fetch' from ./rubyimapsync:79 from ./rubyimapsync:50:in `each' from ./rubyimapsync:50

Hey guys, I don't mean to be rude, but you're on your own when it comes to debugging these problems. I wrote the script to meet my own needs and shared it in the hope that other people might find it useful, but I don't have the time or the inclination to test it on every IMAP server and every folder configuration you might want to use it with.

If you can't get this script working and don't want to debug it yourself, I recommend you use the latest version of imapsync, which is far more robust and doesn't have the date problem I experienced in an earlier version.

A huge thank you to you for this code. It's the first imap copy tool I've found that does just what I need.

I was having the same problem as the people above with the BadResponseError exception. I finally tracked it down : the problem is the script assumes there is at least one message in all source and destination folders. If you hand an empty uid list to uid_fetch, the Ruby imap library happily issues a bogus EXAMINE request on your behalf, and then we get this exception. I have a patch to fix it; your blog is eating it as I'm trying to paste it in here though, so you can grab it here. Note I did diff -ub so the indention/whitespace changes aren't generating tons of diff output.

Finally I have my mail copying up to Gmail. Many thanks again!

Thanks for the patch! I've updated the post to include your changes.

That was precisely the solution that I was hoping somebody would provide, Steve. Thank you for providing that patch, and thank you for working it into your original Ruby script, wonko.

Thank you for writing the script, and to Steve for the helpful patch. Unfortunately I am still getting errors on several other folders. Does anyone have any thoughts on these:

[mail.stmintz.com] selecting folder 'INBOX.Personal.Parents'...
[imap.gmail.com] selecting folder 'old-parents'...
[imap.gmail.com] analyzing existing messages...
c:/ruby/lib/ruby/1.8/net/imap.rb:949:in `receive_responses': * BYE [ALERT] Fatal
 (Net::IMAP::ByeResponseError)l: No such file or directory
        from c:/ruby/lib/ruby/1.8/monitor.rb:238:in `synchronize'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:933:in `receive_responses'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:918:in `initialize'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:917:in `start'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:917:in `initialize'
        from C:/ruby/samples/gmail_imap_import.rb:44:in `new'
        from C:/ruby/samples/gmail_imap_import.rb:44

[mail.stmintz.com] selecting folder 'INBOX.Personal.Mailing-List-Crafty'...
[imap.gmail.com] selecting folder 'old-crafty-list'...
[imap.gmail.com] analyzing existing messages...
c:/ruby/lib/ruby/1.8/openssl/buffering.rb:178:in `syswrite': An existing connect
ion was forcibly closed by the remote host. (Errno::ECONNRESET)
        from c:/ruby/lib/ruby/1.8/openssl/buffering.rb:178:in `do_write'
        from c:/ruby/lib/ruby/1.8/openssl/buffering.rb:219:in `print'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1042:in `put_string'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1016:in `send_command'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1015:in `each'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1015:in `send_command'
        from c:/ruby/lib/ruby/1.8/monitor.rb:238:in `synchronize'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1012:in `send_command'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1165:in `fetch_internal'
        from c:/ruby/lib/ruby/1.8/monitor.rb:238:in `synchronize'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:1163:in `fetch_internal'
        from c:/ruby/lib/ruby/1.8/net/imap.rb:712:in `uid_fetch'
        from C:/ruby/samples/gmail_imap_import.rb:95
        from C:/ruby/samples/gmail_imap_import.rb:56:in `each'
        from C:/ruby/samples/gmail_imap_import.rb:56

Thanks again.

Sean

Do the errors also occur on folders without hyphens in the name? I wonder if that has anything to do with it.

These work:

INBOX.old-inbox to old-inbox, INBOX.Personal.Livejournal to Livejournal, INBOX.Personal.NYU to old-nyu, INBOX.Personal.UH to old-uh

These don't work:

INBOX.Personal.Parents to old-parents, INBOX.Personal.Mailing-List-Crafty to old-crafty-list, INBOX.Sent to old-sent

Sean

On the folders that fail, I only get this error message on Linux:

[imap.gmail.com] analyzing existing messages...
 (Net::IMAP::ByeResponseError)949:in `receive_responses': * BYE [ALERT] Fatal error: max atom size too small: File exists
        from /usr/lib/ruby/1.8/monitor.rb:238:in `synchronize'
        from /usr/lib/ruby/1.8/net/imap.rb:933:in `receive_responses'
        from /usr/lib/ruby/1.8/net/imap.rb:918:in `initialize'
        from /usr/lib/ruby/1.8/net/imap.rb:917:in `start'
        from /usr/lib/ruby/1.8/net/imap.rb:917:in `initialize'
        from gmail_imap_import.rb:46:in `new'
        from gmail_imap_import.rb:46

Sean

wonko, thanks for merging in that patch. My copy finished up without errors and I'm cut over completely to gmail imap now.

Saw this thing from Sean and couldn't help myself from butting in... Sean, do those failing folders have a lot of messages in them? googling for "max atom size too small" suggests it's a request size limit with (older?) Courier servers. You could try modifying the ruby code here to break the lookups into chunks, or an easier, low tech solution might be to break those folders up into several, smaller folders using your favorite gui mail app. Then try the script again.

Yes, I figured that as well. These are large folders and I am splitting them up as we speak.

Thanks

Sean

For the benefit of others, splitting folders in to chunks of roughly 2500 messages works without any problems.

Hey,

I'm using this script to try and migrate a fairly large set of mailboxes from Dovecot to Gmail. I know the script is unsupported but there seems to be some interest in the comments here so I'm hoping somebody else might have run into this problem and have some suggestions.

I'm able to copy some of my smaller folders but when I try and copy larger folders I get this error:

/usr/lib/ruby/1.9/net/imap.rb:996:in `raise': Unable to append message to folder (Failure) (Net::IMAP::NoResponseError)
        from /usr/lib/ruby/1.9/net/imap.rb:996:in `get_tagged_response'
        from /usr/lib/ruby/1.9/net/imap.rb:1047:in `block in send_command'
        from /usr/lib/ruby/1.9/monitor.rb:190:in `mon_synchronize'
        from /usr/lib/ruby/1.9/net/imap.rb:1032:in `send_command'
        from /usr/lib/ruby/1.9/net/imap.rb:632:in `append'
        from /home/adam/bin/imap-to-gmail.rb:110:in `block (2 levels) in '
        from /home/adam/bin/imap-to-gmail.rb:96:in `each'
        from /home/adam/bin/imap-to-gmail.rb:96:in `block in '
        from /home/adam/bin/imap-to-gmail.rb:57:in `each'
        from /home/adam/bin/imap-to-gmail.rb:57:in `'

Normally I can simply run the command again and it will make it a bit farther so I guess I could put it in a loop until it made it all the way through but that seems a little ugly :-)

When I was playing with imapsync I saw my account loose IMAP access a couple of times. I'm half wondering if Google is seeing me pound their IMAP service with sync requests and blocking me (though it always seem to come back an hour or so later).

Adam.

FWIW ... if anyone is curious here's the imapsync command line I've found which "kind of" works for migrating from a Dovecot IMAP server to Gmail.

imapsync --syncinternaldates --host1 localhost --user1 adam --password1 password  --host2 imap.gmail.com --user2 adam@example.net --password2 password --authmech2 plain --port2 993 --ssl2 --authmech1 PLAIN --authmech2 LOGIN

I say "kind of" because while it appears to complete successfully it doesn't actually seem to do a full sync of your mail (from manual tests and also running imapsync with the "--justfolders" flag and comparing the results for each account).

However for smaller mailboxes it seems to work okay.

Adam.

I'd like to thank you for this script! I've been eyeing imapsync and quite frankly, that's a lot of time and software installs just to get it working (I still don't have it right) and it's installing software I normally wouldn't use on my Mac.

This script worked right off the screen clip and I didn't have to install anything (Rails works on Leopard out of the box!)

Thanks!!!!

Thanks for the script. It was taking me forever to do this manually in Mail.app.

One small change. I was getting occasional "Unable to append message to folder" errors from Google, so I wrapped the append command (line 103) to catch the exception and retry:

# Append the message to the destination folder, preserving flags and
# internal timestamp.
dd "storing message #{mid}..."
success = false
begin
  dest.append(dest_folder, msg.attr['RFC822'], msg.attr['FLAGS'], msg.attr['INTERNALDATE'])
  success = true
rescue Net::IMAP::NoResponseError => e
  puts "Got exception: #{e.message}. Retrying..."
  sleep 1
end until success

I started hitting the "Fatal error: max atom size too small" error when I tried syncing my larger folders. This happens when you try to do a uid_fetch on a large number of uids. I added a uid_fetch_block method that breaks it up into UID_BLOCK_SIZE uids (1024 worked fine for me). Here's the diff of all my changes:

--- untitled
+++ (clipboard)
@@ -15,6 +15,8 @@
 DEST_USER = 'username@gmail.com'
 DEST_PASS = 'password'
 
+UID_BLOCK_SIZE = 1024 # max number of messages to select at once
+
 # Mapping of source folders to destination folders. The key is the name of the
 # folder on the source server, the value is the name on the destination server.
 # Any folder not specified here will be ignored. If a destination folder does
@@ -33,6 +35,14 @@
    puts "[#{SOURCE_HOST}] #{message}"
 end
 
+def uid_fetch_block(server, uids, *args)
+  pos = 0
+  while pos < uids.size
+    server.uid_fetch(uids[pos, UID_BLOCK_SIZE], *args).each { |data| yield data }
+    pos += UID_BLOCK_SIZE
+  end
+end
+
 # Connect and log into both servers.
 ds 'connecting...'
 source = Net::IMAP.new(SOURCE_HOST, SOURCE_PORT, SOURCE_SSL)
@@ -77,16 +87,18 @@
   
   dd 'analyzing existing messages...'
   uids = dest.uid_search(['ALL'])
+  dd "found #{uids.length} messages"
   if uids.length > 0
-    dest.uid_fetch(uids, ['ENVELOPE']).each do |data|
+    uid_fetch_block(dest, uids, ['ENVELOPE']) do |data|
       dest_info[data.attr['ENVELOPE'].message_id] = true
     end
   end
   
   # Loop through all messages in the source folder.
   uids = source.uid_search(['ALL'])
+  ds "found #{uids.length} messages"
   if uids.length > 0
-    source.uid_fetch(uids, ['ENVELOPE']).each do |data|
+    uid_fetch_block(uids, ['ENVELOPE']) do |data|
       mid = data.attr['ENVELOPE'].message_id
 
       # If this message is already in the destination folder, skip it.

Thanks Brad, Steve, and Wonko - It is now working quite well yae :)

 0
    uid_fetch_block(source, uids, ['ENVELOPE']) do |data|
      mid = data.attr['ENVELOPE'].message_id

not sure why this was left off of your patch - possibly formatting??

Hello, thanks for publishing this IMAP transfer script, I've recently moved to google apps and have used this transfer ~5k emails. I have made a few modifications, mainly for handling exceptions and server timeouts. If your interested have a look here

Hugh

This script only allow transfer messages from one IMAP-folder? How do this recursive on existing subfolders?

I've added some lines to implement the following features in order to hava a much complete sync tool:

1) Make a duplicate copy of ALL the folders(labels) included in the source

2) Remove from destination files that are no more included in the source

3) Remove from destination folders(labels) that are no more included in the source

you can have a look here

Thanks a lot for writing the rest :-) Diego

I Add follow lines to get imap server’s all sub folder and mapped to gmail folder. example map (INBOX.example to INBOX/example ), it work in my situation.

Comment “FOLDERS” array
#FOLDERS = {
# ‘INBOX’ => ‘INBOX’
#}

and add the line under dd ‘logging in…’
dest.login(DEST_USER, DEST_PASS)

### Added line ##
FOLDERS = Hash.new
OLDLIST=source.list(””, ”*”)
OLDLIST.each{ |i|
$NEWGMAIL_FOLDER=i.name
$NEWGMAIL_FOLDER = $NEWGMAIL_FOLDER.gsub(/[.]/,’/’)
FOLDERS[i.name]=$NEWGMAIL_FOLDER
}

Hi guys,

A solution for the “max atom size too small” problem is to require the ‘enumerator’ stdlib and to change the fetch iteration to:

uids.each_slice(1000) do |slice| source.uid_fetch(slice, [‘ENVELOPE’]).each do |data|

Note: I used slices of 1000 emails, but i think it supports more than 3000.

Thanks for the post… really good.

I’m having this error: c:/ruby/lib/ruby/1.8/net/imap.rb:982:in `pick_up_tagged_response’: CLOSE not allowed now. (Net::IMAP::BadResponseError)

I think maybe is because gmail server, isn’t it?

I’ve been looking for an easy way to migrate a BUNCH of mail to GMail’s IMAP server. This solved it with some quick awk and friends to generate the mailbox lists.

Thanks a bunch!

For the first very thanks for the script.
I’m small modified it (also thanks for the patches to recursive transfer and any other, I’m read all and make self mix of all it with small add-ons) and it transfer many mails.

But, I have got a problem: What can be done if message_id in Envelope is nil (data.attr‘ENVELOPE’.message_id)??? So, it may be Nil by FRC 822 (http://www.faqs.org/rfcs/rfc3501.html)
In this case we can not check presents its mail in dest_hash! I’m suppose there is another way to compute unique message-hash. There are anyone who known it?

And I’m very-very newbie in ruby, so, it is my first experience with it. Please, explain much more simply as you can.

great script. Worked like a dream!

Thanks a lot. Saved me a bunch of work!

I am trying to sync gmail to my local server as a backup. it seems if you have subfolders that this script chokes up…

here is an example output….

[imap.gmail.com] selecting folder ‘example/sub folder’…
[localhost] selecting folder ‘example/sub folder’…
[localhost] folder not found; creating…
[localhost] error: could not create folder: Invalid mailbox name
[localhost] /usr/lib/ruby/1.8/net/imap.rb:972:in `get_tagged_response’
/usr/lib/ruby/1.8/net/imap.rb:1023:in `send_command’
/usr/lib/ruby/1.8/monitor.rb:238:in `synchronize’
/usr/lib/ruby/1.8/net/imap.rb:1008:in `send_command’
/usr/lib/ruby/1.8/net/imap.rb:414:in `create’

anyone have a fix?

fyi: my local server is CentOS 5.x using dovecot imap server with ~/Maildir configured, not mbox.

I figured it out. dovecot stores sub dir separators as a ., not a slash. I reversed the code above with the gsub()

a note, gmail is my SOURCE, dovecot is my DEST

FOLDERS = Hash.new
OLDLIST=source.list(“", "*”)
OLDLIST.each{ |i|
$NEWGMAIL_FOLDER=i.name
$NEWGMAIL_FOLDER = $NEWGMAIL_FOLDER.gsub(/\//,‘.’)
FOLDERS[i.name]=$NEWGMAIL_FOLDER
}

Does anyone know how to modify the script to mark the messages as unread in Gmail?