Tell them, my song.

Once upon a time, two email auto-responders had spent an entire weekend continuously sending each other out-of-office messages. Whenever one mailbox said, ‘”I’m sorry, but my user is currently not in the office and cannot respond to your message”, the other mailbox replied, “I am sorry to hear that, but I cannot inform my user about it, because he is also not in the office at the moment.” Naturally, the first mailbox was a polite mailbox and replied, “Oh my bad, I cannot tell my user either, because he is not in the office.” Being also a polite automat, the second mailbox then said, “My user is not in the office, but I will let him know about it when he is back.”

Now this rather useless conversation went on for a couple of days, filling the mail server’s database with thousands of such automated emails. When the system administrator came to the office on Monday morning, he found that the mail server was offline and that nobody in the office could use Microsoft Outlook anymore, because it was not able to connect to the Microsoft Exchange Server Standard Edition that was supposed to be running somewhere down in the dungeon where all the servers were.

Actually, this is a true story, and it happened back in the old days of Windows NT 4.0 Server and Exchange Server 5.5 Standard Edition. The Standard Editions of Microsoft Exchange Server, including the current versions, have an artificial size limit on their database storage. In Exchange Server 2003, for example, this limit is 18 GB. As soon as the database reaches that limit, Exchange Server will take the storage engine offline for ’security reasons’.

Beginning with Service Pack 2 for Exchange Server 2003, the size limit can be increased to 75 GB. More information about this is available on Microsoft Technet in this article.

But back in the day, the only way for the IT consultants to fix the exhausted database was to upgrade the Exchange Server to its bigger brother, the Enterprise Edition. Running the Enterprise Edition, they were able to delete all the useless emails from the server. Once they had done this, they could use Eseutil, Microsoft’s command line maintenance utility for Exchange databases, to compress the database to an acceptable size. And after this painful and long procedure, they could go back to the licensed version of Exchange 5.5 Standard Edition.

The Exchange Server storage database is based upon Microsoft Access/Jet engine technology, which deservedly so does not have the reputation of being very reliable and sophisticated. Exchange Server suffers from the same problems Access suffers from. At a company where I worked a couple of years ago, we used to ask our customers: “Do you want to use Microsoft Access or can we give you a real database?” It sounds like a joke, but it isn’t. Later on, I worked in another job where I had to fix the corrupted database of a constantly crashing Microsoft Access-based application at least twice a day. Since then, I hate Microsoft Access with a passion. It is the worst piece of software that I have ever seen. Ever.

Now what are some of the issues with the Exchange Server storage?

For starters, Exchange does not immediately purge deleted items. For example, you can define how long ‘disconnected’ mailboxes should be kept alive before the system eventually deletes them. You normally ‘disconnect’ a mailbox by deleting the user account that belongs to it. The purpose of this feature is that, within the defined grace period, you can ‘reconnect’ a mailbox to a user account. This comes in handy when you accidentally deleted a user account in Active Directory.

Another problem is that the Exchange Server database does not free no longer required disk space. Already allocated disk space is marked for reuse, but the database file won’t automatically compress and shrink. If you really need to free disk space, you have to use Eseutil to compress the database file and wipe out all the zombie data in it. But to do so, you need to have at least enough space on the hard disk for a second copy of the database file, and you have to take the data storage offline, meaning that no users will be able to access their mails or calendar items while the process is running. And that process is slow.

However, the good news is that the server won’t miss any incoming mails during that time, as long as you only stop the Exchange Server Storage Service. Exchange will store all incoming mails in a transaction file and process them later when the storage engine is online again.

Besides that, Exchange Server will automatically run a database maintenance on a scheduled interval defined in the storage’s properties in the Exchange Server System Manager, which is part of the default installation. The maintenance does not compress the database, but at least it marks no longer required database pages for reuse. This prevents the database file from growing as long as there is space that it can reuse.

Logfiles are another source of problems. Exchange Server literally stores thousands of .LOG files in the same directory where the database file is located, and sooner or later these files might consume all the space on the servers hard disk. When that happens, Exchange Server will also take the storage engine offline. In my experience, it is safe to delete all the files with the extension .LOG and then bring the storage engine back online by restarting its service.

Learning these things the hard way is an unpleasant experience. If for some strange or unknown reasons your users can no longer connect to your Exchange Server, the Storage Service is always a good starting point for your trouble shooting, and checking the size of the database file and the remaining disk space won’t hurt either.

Comments RSS