Weird Problem of the Day: Windows 2008 R2 and Windows 2003 DC’s Not Replicating
OK, one of our techs was onsite upgrading a basic single server DC/File Server/App Server. He joined the server to the domain, promoted it and ensured that the files in the sysvol share were replicating. Everything looked fine. He transferred the FSMO roles, again without error. When he ran the dcpromo on the old server to demote it, he got an errors saying that
"The directory service was unable to transfer ownership of one or more floating single-master operation roles to other servers"
Weird. So I had him do all the normal steps, moving FSMO back and forth, waiting 15 minutes, restarting netlogon and FRS services. Nothing helped. FRS was running, stuff was replicating, but apparently something was still missing.
Running "repadmin /showreps" yielded a few errors:
"Last attempt @ 2009-11-12 18:45:37 failed, result 1256 (0x4e8): The remote system is not available. For information about network troubleshooting, see Windows Help."
Bottom line, R2 and 2003 servers sometimes have security problems replicating between them. There is a hotfix, probably the worst-titled one in history, to fix it.
Crap huh? No way you would find that if you were searching for a problem with AD replication. There is a tiny note in there that one of the problems you might experience is replication-related. So we requested and ran the hotfix, rebooted and magically dcpromo worked as the maker intended it to. I hope that this makes it into a service pack or critical update at some point soon, because we have a lot of Windows 2003 servers that are eventually going to need to be replaced.
Hyper-V R2 Vs. VMware vSphere 4.0
Windows 2008 R2 is available now and one of the 4 key points of improvement is in the area of Microsoft's HyperVisor, Hyper-V. What has been lacking until recently is a side by side comparison of the features and limitations of Hyper-V R2 Vs. VMare vSphere 4.0. The killer feature that everyone has been waiting for on Hyper-V has been the live migration, designed to compete with vMotion. The devil is in the details though, and there are lots of details about this feature and others that factor into the decision as to which product is better for your environment. CTI has a great white paper that discusses the differences. You hit the side by side comparison on about page 17 of the paper (no reg required). Bottom line: here are the top 6 differentiators as far as I am concerned:
- Memory optimization: VMWare has over-commit protection. Hyper-V really doesn't. It can reserve RAM, but it is pretty hokey compared to what VMware can do. Who cares right? RAM is cheap! Not so fast. Some RAM on these higher end virtualization and blade servers can break the bank.
- Live migration: VMware wins here too. Hyper-V R2 has it, but it can only do one machine at a time, and the way it handles shared storage is weak. VMware has been at the game longer and I expect that this gap will be closed but for now it could be a dealbreaker for those who load their hosts down.
- Guest support: Another score for VMware. The only non-Windows guest supported on Hyper-V is Suse Linux. Not a huge deal if you are running Web servers I guess, but those guest tools start to get really nice if you are needing to do any of the more advanced functionality. VMware supports most *ux's including SCO OpenServer, SCO Unixware, Free BSD, Debian and CentOS. Again, not necessarily a deal breaker.
- Ability to hot-add disks: VMware can add them easily. Hyper-V can only add virtual SCSI devices, not IDE
- Number of guests: This one is a mix. Hyper-V can have 512 loaded, but only 192 running ("only," he says). VMware can run 256 at the same time, with up to 8 virtual CPUs and 255GB of virtual memory, compared to 4 CPU's and 64GB on Hyper-V.
- Monitoring: Hyper-V wins here. Since it is based on a Windows Core box that is joined to the domain, you can capitalize on the tools built into Windows, which are legion.
So who wins? Honestly, it depends on who is judging. For implementations of 30 Windows virtual servers and under, I don't see why you would pick VMware, honestly. It comes out cheaper by most people's math, you have fewer vendors to beg for support from, and you have fewer new interfaces to learn. For larger implementations, it depends a lot on how heavily you intend to stack the VM's on the hosts and what kind of downtime you can tolerate should one of the hosts fail.
As far as dollars are concerned, most calculators will show that the initial cost for Hyper-V is cheaper for similar implementations. Most, of course, except for the one that VMware provides (big shock).
Dollars for initial implementation are small potatoes though, compared to supporting a poorly planned implementation. It is always going to be good to bring in an experienced party that can help guide you through some of the pitfalls.
SHUT DOWN THE PORTS!!!…..erm, the airports, that is.
Remote Island Village May Have Swine Flu Outbreak...
Redirect traffic on Exchange 2007 CAS to OWA Subdirectory and to HTTPS
In the context of a dedicated Client Access server with a public IP, an admin might want to make the default landing point for the server be the /OWA site, so that users can get to the login prompt without appending the /owa to the end (e.g. https://<Public Name>/owa).
Some sites will tell you to simply log in to IIS and set a redirect on the default site to /owa. Problem is, that setting will be pushed down to all the virtual directories, which will then have to be un-set manually. If you don't do this, you can expect the sub-sites to fail, including activesync. If you have ever worked as an admin on an Exchange or IIS server, you know that, if you are running a Microsoft web-based application, you change as little as you can get away with and you do it in the simplest manner possible. The best way to do this is to set up a redirect on the iisstart.htm file.
To do this,
- Select the default website in IIS (this assumes Server 2008 and Exchange 2007, btw).
- Select the content view at the bottom to see the documents in the root of the directory.
- Highlight the iisstart.htm file and hit the "switch to features view" Now we are looking at the features for just this one file.
- Now select the HTTP Redirect button and check the box for "Redirect request to this destination" and put in "/owa" (no quotes).
- Hit apply and test. This should not require a bounce of IIS.
To ensure that users get directed to the correct site even if they forget to type in https, you can force the error they get to redirect them to the correct site.
- In the IIS manager, click on the default site and choose "Error Pages."
- Select the 403 error and choose to edit it.
- Choose to "Respond with a 302 redirect" and put in the full path (including https and /owa) of your owa login page. Hit OK. No restart of IIS should be necessary.
- Test.
When to panic…
I am working on a 225 mailbox migration this weekend. The environment is basically the following:
Old Server: Windows 2003 Std/Exchange 2003 Std, all patches (pretty basic)
New environment: 2 Windows 2008 Enterprise Mailbox servers running the Exchange 2007 Enterprise mailbox role in CCR with a Windows 2008 Standard machine running the CAS/HT roles and serving as the File Share Witness host. Each of the mailbox servers have three volumes (CCR likes both machines to be as nearly identical as possible): a 40GB C:, a 20GB D: for log files (on a RAID10) and a 300GB E: (on a RAID6). These volumes were set up by a co-worker a few weeks ago and he did a great job with it. The servers are fast and they have great I/O on disk writes. All three machines are hosted in a ESX/Blade server environment with a SAN backend connected via Fibre Channel. This is becoming a pretty popular arrangement. The RAID10 logfile volume is considered best practice for performance reasons. The mailbox store lives on the big RAID6 volume for fault tolerance.
Anyway, all machines were updated and I had tested failing over the CCR cluster nodes successfully, so at about midnight last night, I started moving mailboxes. At around 2am, the old mail server went offline. It responded to ping, but I could not RDP to it or get to and SMB shares. Couldn't get to the services either. It was, for my purposes, dead. The big issue here is that the mailbox move process was still trying to work, for all 225 mailboxes. The lack of old server caused all kinds of issues to take place that had the effect of hammering the log files. And since log shipping is pretty much how CCR works, both servers started choking. In two hours, we generated 19.8GB of log files, which then knocked the mailstore offline. I could not remount it, since there was no room for more logfiles.
Panic mode.
I temporarily stopped the replication, created new log file folders on both of the cluster nodes, moved the location of the log files in AD, moved the files themselves over to the big data volume, and restarted replication. These steps were originally from EXPTA.com, but it appears that that site is down, so I am linking to the google cache. These should all be done in the Exchange Management Shell (launched as administrator), and only performed after the new log directories have been created on both cluster nodes in the exact same location. Obviously, you will need to also change the paths to match your environment.
Step 1: Suspend-StorageGroupCopy -Identity "First Storage Group" -SuspendComment "Moving transaction logs" -Confirm:$False
Step 2: Move-StorageGroupPath -Identity 'First Storage Group' -LogFolderPath 'E:\ExchangeLogs' -SystemFolderPath 'E:\ExchangeLogs' -ConfigurationOnly
Step 3: move [oldpath]\*.* [newpath]
Step 4: Resume-StorageGroupCopy -Identity "exchange1\First Storage Group"
After this was completed (step 3 took a while, since I had 20GB of logfiles) I was able to remount the store and test via OWA. Then it was time to figure out why the Ex2003 box went down. After the moves are complete, I will run a backup to commit those log files to the DB and then move them back to the correct drive, as 20GB should be enough in any normal case.
Proof
Positive proof that, despite all the attempts from GeoCities to FaceBook to make the internet easy for the common man to control, the internet is still run by old nerds.
Searching for "gopher" on google and choosing "I'm feeling lucky" returns:
A protocol for handling information requests that has not been in common use since the *first* George Bush was in office, named thusly because the task it performs (to go 'fer things) sounds somewhat like the word gopher.
NOT.A.REAL.FREAKING.GOPHER....
I get no respect...
Microsoft Virtualization Calculator
MS has released a calculator to help you figure out exactly what you will need in terms of licenses and dollars in order to meet your virtualization requirements. They have two calculators, one of which requires Silverlight (guess which one I did). The calculators are VM-technology agnostic (meaning that they are the same whether you are using hyper-v, VMWare, or VirtualBox)
On calculator 2, you can either put in the number of servers you have with the avg. VMs per server, or you can list each server individually, which is more likely to be the best scenario for smaller shops. In the below screencap, I entered in one dual Proc (note that cores are immaterial) physical server running 3 virtual machines.
Virtualization Results Example
It reports that we will need either 3 Standards, 1 Enterprise, or 2 Data center licenses (since Data center is licensed per proc).
The pricing columns report that the cheapest way to do this is with the 3 std licenses, but it you wanted to add a 4th vm, the Enterprise license would be cheaper. If you wanted to expand way beyond that in terms of Vm per physical server and core pair, the data center starts to be more cost-effective.
Both calcs are available here
Is that BlueSteel? Or Magnum?
This model has really found his niche. One of the key elements of the Hedgehog principle is finding the things that ignites your passions, utilizes your strengths and drives your economic engine. This guy has truly embraced that, and serves as an inspiration to all of us.
Bonus points for noticing how he subtly mixes it up in some of the later pictures...
Second Bonus- great comment in the source:
<!--cf THE CODE APPEARS TO START HERE. NOT SURE HOW THIS WORKS -->
Exchange troubleshooting
While there is no substitute for a full working lab, there are several tools that can help to make troubleshooting various elements of your Exchange environment easier.
MX ToolBox - Great for all-in-one checking of reverse pointers, blacklists, open relays and general diagnostics. If I was stuck on a desert island, this would be the troubleshooting website I would take.
TestExchangeConnectivity.com - Runs a test connection to your Exchange server the same way your Wi-Mo phone or iPhone would. Great for testing an environment when you are not really sure the phone should work (unsupported OS or patch level)
Hexillion.com - Good for looking up public records for DNS and such. The have a lot of options for how much data you want to see.
Telnet client - The fact that this has to be manually installed on Vista is a crime
Steps to send email via telnet - If you need to interact on the most basic level, without fear of spamfiltering or email clients muddying the water, this is a good place to start.
MPack and how to stop it
MPack is a software kit written and marketed by Russian code writers. It is unusual because it is a) written in php and b) sold and updated as if it was a regular legitimate software product. People run it on their websites as a means to install keystroke loggers on vulnerable computers. It will work with Firefox and IE and will test the visiting browser to see which vulnerabilities are available for it to exploit. This thing gets updated monthly and there are even plugin modules you can buy and add on to it for a more effective attack. Most of the time, most people know enough to not go to the sites that end in .ru:8080 or other strange domain names. Unfortunately, the attack has adjusted its tactics to make itself more effective. If you have weak ftp passwords on your site or if they can get a keystroke logger on your computer to get your ftp password, they will write scripts that will automatically inject an invisible iframe onto every one of your html pages (php pages are less vulnerable, since they are frequently broken out into several php_includes.
This is scary stuff, and Google and other search sites will punish you in search rankings and with popups when people clickthrough to your site if they find out you have been compromised.
The software is constantly being updated, so this goes back to the lesson every server or site admin has to learn. Get updates out as quickly as possible on public facing sites. There is no such thing as a low-maintenance site.
- Update your software packages (wordpress, drupal, joomla, etc.) and any libraries that they may use (imagemagick, etc.)
- Update PHP and MySQL, as well as Apache or IIS. If you use a virtual or shared host, pressure them to keep their software updated. If they don't, switch to a new host or better yet, get a dedicated server or colo somewhere. It only looks expensive until you have to wait three days to have your ftp password changed because the support people at your $3.33 a month host need to escalate such a complicated task to their senior engineers. And all that is after you tried to change the password through their web-based form and it didn't work. And they told you three times that it was changed. %^&*%^&*$ I am looking at you, IVChosting.com ಠ_ಠ Ahem.
- Use secure passwords and change them regularly, especially if you publish via FTP regularly.
- If you get compromised (it happens to everyone), change your FTP passwords immediately, preferably from a computer you do not typically use for publishing.
- Scan your computer for viruses with a different AV than you normally use. I like Malwarebytes anti-malware and Trend Micro's Housecall.
- From a different computer, re-upload the files for your site from your last good backup (you have a backup, right?)
- If you don't have a backup, download all the htm and html files to a Windows based computer and do a find and replace for any iframes that reference an external site. If you have legitimate iframes to external sites you should know what they are, and they will probably be larger than 0 by 0 pixels. Notepad ++ has a great feature for doing find and replace across multiple files in multiple directories.
- If, like most people, you are concerned because Google tagged you as a badware site, you will need to log in to their webmaster tools and set up your site on your account. Then you can request a review, where they will check your site for lingering traces. This is a slow process, so the sooner you get started the better. They will remove the flag when they determine that your site is clean.