Review

Migrating to Google Apps (part 4)

Miguel Garrido — Wed, 13 Jan 2010 12:30:00 GMT

Google Apps with Exchange ActiveSync

I am an iPhone user, having adopted the device with the iPhone 3G (as soon as Exchange ActiveSync support was added). Needless to say, I depend on the excellent ActiveSync protocol for push email, calendar and contact synchronization. I think I started considering Google Apps a viable solution for my personal email when I learned that Google had licensed this technology from Microsoft.

I have setup my mail account on my iPhone with minimal issues, following Google’s instructions. Everything works the way you would expect, and while I have noticed that some folders sometimes take a few minutes to reflect certain changes (only when I moved messages with more than one label, I think), everything else is quick and solid.

My messages aren’t being deleted!

After setting up the iPhone with my new Google Apps account I continued testing (mainly to confirm that all the Outlook issues from part 3 in this series were resolved) by sending myself messages and acting on those messages. For one or two test messages I deleted them from my phone without marking them as read. A few minutes later while looking at the web interface I see two unread messages in my “All Mail” label, when I clicked on the label I became confused. Both messages that I had previously deleted from my iPhone had apparently been archived instead of moved to the Trash. I started searching online.

It turns out that Google sync implements this as a “feature” to lock mobile users into not deleting messages and archiving them instead. I thought, “OK, how do I turn this off?” and found much to my chagrin that you couldn’t! This is one of the things I dislike about Apple (more on that in another post), I do not like anybody making decisions for me, especially when they affect the way I organize my mailbox.

The workaround

After coming to terms with the fact that I couldn’t do anything about this “feature” Google built into their sync product except complain with a post in their message forums along with many others, all asking Google to fix this soon. I started looking for a way to work around this annoyance. I personally never archive anything and all the messages in my mailbox are labeled, so this gave me the idea:

Since I don’t archive any messages without a label, I just need to identify archived message without any labels and that should yield any messages that I deleted from my iPhone that Google decided NOT to delete.

With this thought I set off to search for a way to identify archived mail without labels. When I found out that there is no way to identify unlabeled archived messages I started searching for another way to accomplish this and finally hit with someone who had the right idea. The person’s suggestion was a dirty hack, but it was a solution: create a search that contains a “–label:” parameter for each of my labels, including the inbox, sent, etc. I have over 30 labels right now, so it took me a few minutes, but I did it, and sure enough there were the messages that I had deleted from my iPhone!

Of course, this method only works if you always archive your mail with a label (which I do). Additionally, the search needs to be updated if there is a change to any of the labels in your mailbox. However, I think this is acceptable for now because I don’t change my labels every day and I really hope Google fixes this annoying issue soon.

Now that I had a working search yielding desirable results, I just needed a quick way to access it from anywhere. There are Firefox plug-ins for this, such as Better Gmail 2 which includes a Persistent Searches script created by Mihai Parparita (requires Greasemonkey to run on Firefox). However, this method required 1) Firefox and 2) That I have the plug-in and Greasemonkey installed. I really didn’t want to be tied down to either so I kept looking. Finally, I found someone who had the right idea, create the saved search as a bookmark. This worked well but it still required me to have the bookmark on any machine I wanted to use the search on, which could be any machine if I felt like cleaning out messages I had deleted from my iPhone then and there from the web.

I finally decided on a compromise which is to create a directory on one of my domains containing an index.htm file whose sole purpose was to redirect the browser to my Google Apps Mail account with the search in the query string (using an meta refresh). As long as I was logged in (or logged in while navigating to the link) I would be taken to a search results page with all my “–label:” search parameters already applied, providing me with a list of the messages ready for deletion.

Summary

All things considered, Google Apps Premier is a great product and I don’t regret making the switch or giving them my money. I hope the product continues to mature and that Google fixes some of the annoyances I had to overcome in order to achieve a decent workspace. Hopefully this series will serve someone other than myself should I or they have to set this up for anyone else in a similar manner.

Migrating to Google Apps (part 3)

Miguel Garrido — Tue, 12 Jan 2010 12:30:00 GMT

Outlook issue: duplicates everywhere

Once the Google Sync tool completes the first synchronization of the mailbox you will see a replica of your Google Mail inbox in the mail folders pane in Outlook. Aside from the built-in Outlook folders (of which Inbox, Sent Items, and Deleted Items are mapped to Inbox, Sent Mail, and Trash respectively in the cloud, there is also a new [Archived] folder created in Outlook mapping to All Mail online) you will see each of your Google Mail labels show up as a folder in the Outlook folder list. The Contacts and Calendar will also be synchronized with their counterparts in the cloud, all is well so far, until you start noticing duplicate messages.

Knowing that each label is equivalent to a folder is not a problem if you don’t use search folders (like Unread Mail) because you will see each message marked as unread in its folder at a time (read: a message with three labels will show as unread in three folders) and once you read the message it be marked read in every folder it belongs to automatically.

The problem for me is that I take advantage of search folders, specifically Unread Mail and For Follow Up to help me keep track of messages without having to switch from folder to folder. This is a problem because if, for example I sent myself an email and then marked it for follow up, it would show up twice in my For Follow Up search folder! Once for the message in the Sent Items folder and once for the message in the Inbox.

Another similar issue is when I send myself an email, before I mark the message as read it will show up as unread in the Sent Items folder as well as the Inbox. This too, was unacceptable because I am constantly sending myself reminders when I’m not near OneNote.

The final thing that bothered me was the task list in the To-Do Bar as it mimicked the For Follow Up search folder’s behavior showing multiple flags depending on where the messages were in the mailbox (ex. two flags for a message in Sent Items and the Inbox that was flagged for follow up.)

I tried to live with it for 5 minutes, then decided I needed to make some changes to mitigate this annoying side effect of the way Google decided Outlook should behave when dealing with labels. What I ended up doing was creating custom search folders to mimic the built-in Unread Mail and For Follow Up search folders and customized the Filter property of the task list view in the To-Do Bar.

Below are the criteria and display options I chose for the two search folders and the task list view.

Create your own custom search folders

I deleted the built-in Unread Mail search folder, for some reason the For Follow Up folder wasn’t created in the list of “Search Folders” when the Google Apps account was created, so that was one less search folder to delete.

I then proceeded to create my new custom search folders, starting with the Unread Mail search folder:

Right click on the “Search Folders” item in the mail pane and select “New Search Folder…”.
When presented with a dialog asking for the type of search folder you want to create, go all the way down to “Create a custom Search Folder” and click on “Choose…” to choose the criteria.
In the “Custom Search Folder” dialog name the new search folder “Unread Mail” (make sure to delete the built-in search folder to avoid a name conflict) and click on “Criteria…”.
When presented with the “Search Folder Criteria” dialog, click on the “More Choices” tab, then click on the “Only items that are:” checkbox, and make sure that the drop down reads “unread”.
Click on the “Advanced” tab, then click on the “Field” drop down, navigate to the “All Mail Items” sub menu and click on “In Folder” from the menu that opens up.
Set the condition to “doesn’t contain” and the value to “Sent Items” and click on “Add to List”.
Click on OK twice to go back to the main Outlook window. The search folder just created should be in the “Search Folders” section of the mail pane.
Right click the Unread Mail search folder and select “Add to Favorite Folders” to show the search folder in the “Favorite Folders” section of the mail pane (all the way at the top).
NOTE: This method will not remove any duplicates resulting from multiple label assignments in Google Mail, you will still see duplicate messages in the Unread Mail search folder if a message that is unread has been tagged with multiple labels in Google Mail.

The procedure to create the custom For Follow Up search folder is similar, see below:

Follow steps 1, 2 and 3 above, but name the search folder “For Follow Up” instead of “Unread Mail” and click on “Criteria…”
When presented with the “Search Folder Criteria” dialog, click on the “Advanced” tab.
Click on the “Field” drop down, navigate to the “All Mail Items” sub menu and click on “In Folder” from the menu that opens up. Set the condition to “doesn’t contain” and the value to “Sent Items” and click on “Add to List”.
Click on the “Field” drop down, navigate to the “All Mail Items” sub menu and click on “Flag Status” from the menu that opens up. Set the condition to “exists” and click on “Add to List”.
Click on the “Field” drop down, navigate to the “All Mail Items” sub menu and click on “Flag Completed Date” from the menu that opens up. Set the condition to “does not exist” and click on “Add to List”.
Follow steps 7 and 8 above, right clicking the For Follow Up search folder instead of Unread Mail to add to favorite folders.
One additional step to completely mimic the built in For Follow Up search folder is to right click on it (from the “Favorite Folders” section or under the “Search Folders” section in the mail pane) and click on “Properties”, then in the properties dialog for the search folder about halfway down the dialog make sure the “Show total number of items” radio button is selected as opposed to the default “Show number of unread items” and click OK.
NOTE: This method will not remove any duplicates resulting from multiple label assignments in Google Mail, you will still see duplicate messages in the For Follow Up search folder if a message that is flagged has been tagged with multiple labels in Google Mail.

One thing you should know about the search method above is that if you have messages which had been flagged for recipients or show with a flag status represented with that icon previously and they did not show up in the default For Follow Up search folder, they will show up now. The corrective action I took on these messages is to search for all flagged messages in “All Mail Items” (from Outlook 2007) and for each message that was flagged for recipient or had that icon I right clicked on them and from the “Follow Up” menu, clicked on “Clear Flag/Delete Task” (or similar, I don’t quite remember). This only kept the messages I had flagged for follow up myself (identifiable by the sole Flag icon) in my For Follow Up search folder.

Modify the view of the task list in the To-Do Bar

The final thing that was bothering me was that if there were any items (like messages I had sent to myself) that I had flagged, they would show up as duplicate items in the To-Do Bar’s task list. I had fixed the issues above by creating my own search folders, but this view wasn’t created using a search folder. Instead, it uses a filter, which can easily be modified by following the steps below:

Right click on any empty space in the task list and click on “Customize Current View…”.
In the “Customize View: To-Do List” dialog, click on the “Filter…” button.
In the “Filter” dialog, click on the “Advanced” tab.
This should look familiar, click on the “Field” drop down, navigate to the “All Task Fields” sub menu and click on “In Folder” from the menu that opens up. Set the condition to “doesn’t contain” and the value to “Sent Items” and click on “Add to List”.
Click OK twice to go back to the main Outlook window. You should no longer have duplicate task items for any messages that you have sent to yourself.
NOTE: This method will not remove any duplicates resulting from multiple label assignments in Google Mail, you will still see duplicate messages in the To-Do Bar task list if a message that is flagged has been tagged with multiple labels in Google Mail.

Additional considerations

It is important to remember that Outlook works with folders as while Google Mail works with labels, in my opinion labels are vastly superior to a traditional folder hierarchy, but you have to use what is there. With this knowledge, you should be tolerant and try to avoid assigning multiple labels to a message if you normally work in Outlook and rely on search folder functionality as you will often see duplicate messages in your views if you don’t, possibly throwing off your counts.

This following statement is my assumption based on my observations, feel free to correct me if I’m wrong: Google Sync takes care of making sure that each labeled message is physically located in each folder on the PST file that locally holds your mailbox, so the PST file will be larger than your mailbox size as reported by Google Mail due to this.

As a side effect of my statement above, whenever a message is unread, or when a message is flagged, if it has more than one label assigned to it the message will show up more than once in your search folders, and Outlook folder in general. A big problem with this is sent items. When you send a message to yourself using Outlook, the message will arrive as “read”. I have only seen this behavior when a message is sent through Outlook, sending yourself a message from the web or a mobile device keeps the message unread as you would come to expect. The reason for Outlook’s behavior in my opinion is that the Google Apps Sync tool discards messages that would traditionally be copied to the Sent Items folder is deleted and instead is replaced with a message with the Sent label (or the message itself is applied with a sent label) thus moving it to the Sent Items folder. Google goes out of its way to ensure that the state of messages is kept intact for everything that has a label in Google Mail. A sent message is tagged with the Sent label, and if it arrives in your inbox it is also tagged with the Inbox label. So any message you send yourself always has more than one label, and by Google’s reasoning has to be kept in sync. This is why my custom search folders above worked for the messages I had sent myself (as reminders to myself which were subsequently flagged), there were no duplicates because I effectively filtered out the Sent label within Outlook by excluding the Sent Items folder from the searches.

So far, I feel content with the migration on the desktop side. My next and final post in this series will detail the steps I had to take to get around a ridiculous Google mobile sync “feature” that annoys many iPhone (if not mobile) users, including myself.

Migrating to Google Apps (part 2)

Miguel Garrido — Mon, 11 Jan 2010 12:30:00 GMT

Migration pains

The Google Apps Sync tool for Microsoft Outlook creates a new Outlook profile so that you can keep any other accounts separate from your Google Apps account (Exchange promotes this type of profile use as well). This tool is what eventually keeps your mail, contacts and calendar synchronized between Outlook and the cloud.

The sync tool also provides the ability to import your existing Microsoft Exchange mailbox to your Google Apps account, I naively accepted Google Apps Sync’s offer to do this and let it get to work. The tool successfully migrated my calendar and contacts to their counterparts in the cloud, it created labels replicating my folder structure in Exchange and started uploading my mail messages to the cloud (while showing them in Outlook as it worked).

I started going through the messages and noticed that my name was missing in the To and From fields of my messages, depending on who the sender and recipient was. This was unacceptable and I thought I might have been doing something wrong or that there was a bug with the tool. It turns out that there is a bug with the tool (Outlook/Exchange may be to blame, who knows) where the aforementioned fields were not preserved for any messages in your mailbox if the sender or recipient was a member of your Exchange organization. I can only guess that this is because Exchange/Outlook strip out this information and replaces it with the unique contact information from the GAL. Either way, this was unacceptable and I quickly aborted the synchronization process, cleaned up that newly created profile and all messages from the Google Mail account and started searching for an alternative way to import (namely, IMAP) since I read that this was one of the methods for importing mail that preserved this information (as well as time/date information for messages).

The Google Apps Advanced tools section of the dashboard provides a section where you can provide account details for one (or many) users, a root folder to start importing from in the mailbox and a list of folders to exclude (for example: /Contacts, /Calendar, /Sync Issues) from the migration. The migration tool will then log into your mailbox through IMAP and download all the messages in the folders found under the root folder specified. When specifying a root folder using / as the root folder is usually enough (however, if you use / as the root, you will want to exclude /Contacts and /Calendar because the tool will import these items as mail), but for example if you wanted to migrate only messages found in the Inbox and its subfolders you could specify /Inbox as the root folder and specify any folder exclusions in a relative path to the root folder that you don’t want synchronized, like /Do not bring along if this folder is a subfolder of the Inbox.

Now, as long as your server is configured for IMAP and you have provided the settings (root folder and exclusions, among other options that are pretty self explanatory) you can kick off the migration tool and it will log into the server and download the messages as per your specifications. This process can reportedly be slow but since my mailbox wasn’t huge and it was only working on my mailbox, I left it for a few hours and when I sat back down the process had completed.

The results of the migration were impressive, the mailbox was almost completely there: the sender/recipient names, message times and dates were all correct (save for a few probably malformed messages that may have been sent from the future). I found that the only messages that were not migrated were messages with attachments that Google Mail does not support (such as executable attachments and zipped executables, etc), or attachments which had a size (including attachments) >25MB. Here is an overview of the IMAP migration process. The messages that were not migrated messages were very simple to filter and once I confirmed that I didn’t need them (since I would have the backup PST of my mailbox as it was on the Exchange server to reference these messages if I ever needed them anyway) I deleted those messages as well as the messages with invalid date information since I didn’t want them to sit out of order.

Finally, the tool created a label for each folder in my folder structure and tagged each message it found with its corresponding label (thus maintaining my folder structure to be displayed by Outlook and other clients that treated labels as folders such as the iPhone’s Mail.app). Another nice touch is that every imported message was labeled with the “Migrated” label so that you could easily differentiate messages that arrived while the migration was taking place and messages coming in from the Exchange migration.

Additional considerations

While in the end I achieved the results I was looking for, I feel that I had to unnecessarily jump through hoops to properly get my messages into my Google Mail account. The fact that I had to use the IMAP migration tool was also inconvenient since one of the features on the Google Apps Sync tool is that it facilitated client-side migrations to Google Mail. While the IMAP tool is infinitely useful for large migrations, small shops might prefer to do client-side migrations, not to mention it is the easier approach. The Google Apps Sync tool should have been able to query the GAL (if available) and fill in the information missing from the messages before sending them upstream to Google’s servers.

It should also be noted that if you have more than three or four nested folders in your mailbox unless the names of the folders are very short you will likely exceed Gmail’s 40 character label limit. The migration tool I assume has a special pass to bypass this limit for the sake of maintaining the original folder structure intact, however don’t expect that you will be able to edit this label (or create labels to nest deeper) from the Gmail interface. I didn’t test if you could create nested folders using Outlook (and have the Sync Apps tool create the labels for you automatically online) instead I just opted to modify my folder structure to better embrace the label system, having labels for nested folders instead of the hierarchy except where it was absolutely necessary. This helped me stay below the limit and so far I can’t complain.

My next post will look into some of the challenges of using Outlook with the Google Mail way of doing things.

Migrating to Google Apps (part 1)

Miguel Garrido — Sun, 10 Jan 2010 18:04:46 GMT

Why I made the decision

Google Apps is Google’s response to Microsoft Exchange environments. Each have their pros and cons, and honestly, in the workplace Google Apps is still playing catch-up to Exchange but I was looking for something I could use for my personal email account. Having access to all your emails, calendars and contacts in the cloud is almost a necessity, especially since we are constantly on the move and are utilizing our mobile devices more than ever before. If you are daring enough you can keep your documents online as well. If you spring out the $50/yr per user for Google Apps Premier you are additionally provided with additional services like [SSO to] Google Chat, Google Video, Postini, Google Sites and mobile device access for your Blackberry in addition to the iPhone and a few other platforms that Google’s Sync software supports. There are also other perks like Single Sign-On (SSO), forced SSL (configurable), your mailbox size is bumped up to 25GB per user, and they claim “Blackberry Support” which I can only assume includes Push email support since Google Sync only provides Contact and Calendar sync at the time of writing.

Everything I’d heard before about Google Apps while impressive still wasn’t enough for me to make a change, I was dependent on Outlook. When I learned about the Google Apps Sync tool (only available to Google Apps Premier and Education editions) is when I started giving the possibility of using Google Apps some serious consideration. In the end, I did some final research into the features I would be giving up on Exchange in favor of Google Apps Premier and finding them acceptable I signed up. The Apps Sync tool provides the user with Exchange-like access to Mail/Contacts/Calendar – changes are pushed and faster than IMAP for mail, and the Contact/Calendar synchronization is not far behind that as well. Your data is stored in a PST file and it just works – there are a few kinks relating to the way Outlook handles Gmail labels [as folders] where it duplicates any message that has more than one label in its corresponding folder and the email/follow up flag duplication you are likely to see, but I cover how I overcame these hurdles in a later post.

Now that I knew I wouldn’t have to give up Outlook on my main PC (a requirement for me since I was perfectly fine with using the Gmail interface everywhere else) I felt ready to make the jump, I signed up.

The sign up process

The sign up process as I remember it was really straightforward, the main requirement is that you own a domain name. You will be asked for billing information, to create an administrative user, some other details and finally be asked to confirm that you own your domain (either by uploading a file there or creating a CNAME record that Google provides you with). Once you confirm that you own the domain you are brought to the dashboard for the domain you just signed up with:

Everything is currently activated, however, when you first sign up, you need to manually activate Email and Postini Services by modifying your MX records from your domain’s DNS control panel. The mobile sync service (bottom right) also needs to be enabled by opting into it.

Google makes the configuration process fairly painless by providing a setup guide the first time you sign into the control panel which you don’t have to follow but if you need guidance it is invaluable.

Activating Email and Postini can take up to an hour each so you need to be patient while this process completes, in the meantime feel free to go through all the options offered to you by every other component of the Google Apps suite and configure them to your liking.

Additional Considerations

If you opt not to go through all the pages in the Google Apps dashboard at least be aware of the following:

The Support tab provides you with a Customer PIN# and Support PIN# (not sure if it applies to Google Premier only) to contact support and bulk user creation.
The Advanced tools tab provides you with many useful options such as: authentication options, reporting options, email migration.
The Service settings dropdown (last link on the navigation menu) provides you with quick access to each of your service’s settings pages.
It is wise to customize the URLs of your services so that you can access them without having to type out the Google Apps URL. For example: http://webmail.example.com instead of http://mail.google.com/a/example.com or http://cal.example.com instead of http://calendar.google.com/a/example.com. All you need to accomplish this is create CNAME records on your domain’s DNS control panel and let the Google Apps application you are changing know its new URL (so that Google can update its servers). The services will then be reachable via your custom CNAME and the long URL format. The URL change can easily be accomplished through the settings page of each service.

NOTE: I have not covered all the options for a Google Apps migration from Exchange, only the options that I chose to go with because they met my needs, this article has additional information regarding integration with Outlook and Exchange.

All things considered, the creation process and dashboard was very painless, the pain began once I attempted to get all my data into my account. I will touch on that in the next post.