When Server 2012 DeDuplication goes bad

This is a really weird one.

Our old SAN was really struggling with space despite having EMC DeDuplication switched on, so I commissioned a Server 2012 guest on some new SAN storage, and started serving files through this guest off a dynamic VHDX file (sat on another chunk of the same storage). Because our old SAN used NDMP backups, I had to use RoboCopy to migrate the data which is slow but did the job. The reason for the intermediate VHDX is simply for portability- we could pick it up off the SAN and dump it on any Windows server (even stand-alone), whereas obviously anything stored directly on the SAN would need to be connected up to servers on the iSCSI network. The VHDX file is connected to the guest on a SCSI bus as this enables hot-removal unlike the IDE buses.

We’d also been waiting for Backup Exec 2010 R3 SP3, as this was the first release of 2010 that at least enabled the agent to work on Server 2012.

This was all good so far; RoboCopy had- as expected- kept all the NTFS permissions correct so it was just a case of sharing the folders out. As both shares now also sat on the same volume, I though Windows Server 2012 DeDuplication could really get to work by DeDuping across the shares, which previously hadn’t been possible due to the configuration. The DeDupe process started slowly but I didn’t think anything of this, because the Celerra took ages to fully DeDupe all its volumes. I reasoned that this should be fine as it was the Hyper-V guest doing the DeDuplication; I presumed that the guest should see the VHDX as just another block of storage rather than trying to DeDuplicate the VHDX file from a host machine.

This is where it all went a bit strange. Nobody had complained about access speeds or anything, yet Backup Exec was taking absolutely ages to back up this volume (I did some rough maths and figured out it would actually never complete a full backup inside a week, compared to “just” taking 48 hours or so previously). Then it turned out that actually, it was doing full backups in less than an hour because it wasn’t actually doing full backups.

This is when it became apparent that there is a clash between the presentation layer of Server 2012 (what I would normally refer to as the Explorer shell, but this is Core), DeDuplication and dynamic VHDX files. The storage system within Windows knows how much actual data is on the volume because it reports x TB used, correctly. DeDuplication seems to go mad and actually un-DeDuplicate everything, so you end up with a space saving of 0bytes (from 11GB… so it’s gone backwards). And the… “explorer” shell goes further and actually loses all reason, reporting insanely small “size on disk” numbers for vast amounts of data (real-world example: 5.1TB of actual data takes up just 37GB on disk. Yeah, right). So to be fair, this is why Backup Exec is being so erratic- it’s asking Windows for the amount of data, and Windows replies that although there’s supposedly 5TB of data, it’s only taking up 37GB on disk.

The fix is currently unknown, as apparently even Microsoft haven’t encountered this one. I’ve stopped the actual DeDupe jobs (even though it’s still enabled against the disk) and set a RoboCopy job off to rehydrate (fingers crossed) everything onto yet another volume (this time pass-through to the new SAN) so that at least we can start getting valid backups. We’re passing information on to Microsoft to see if they can come up with anything, and have discovered that this is perfectly repeatable: enable DeDupe against another (live but non-backed up) dynamic VHDX and the same thing happens, Windows reports the actual amount of data as X but the size on disk as X-a-lot-of-space.

Update 02-NOV-2013: still unresolved. Found a few discrepancies in NTFS permissions but this doesn’t explain much. The data seems to rehydrate onto another volume, but that’s maybe the wrong word as “rehydrate” implies it was deduplicated in the first place, which it wasn’t really. Still, at least we can get some form of backup for now.

Live VHDX extension struggle

Given we had a server with a CIFS share that was fast approaching its limit (on the file server, not just the share) I built a Hyper-v machine around two iSCSI shares: one held the Windows guest, the other held the CIFS data. Now I knew the data was going to be about 4TB so created the CIFS share at that size, but because we’ve decided to use VHDX as the “visible” storage medium (rather than pass-through, purely for portability) I thought I’d start off with a 1TB VHDX file and then expand it “on the fly” to see how easy it was. I started RoboCopying all the data on the above CIFS share to the new VHDX, knowing it would start failing to copy at the 1TB mark.

And the answer was incredibly easy, if only I’d connected the 1TB VHDX to the guest’s SCSI connector first time. Basically, it’s impossible to disconnect an IDE drive while the VM is live, but no such problem with SCSI (should have realised this).

Step 1 was to use DISKPART on the guest to set disk 1 offline, then on the host ran Remove-VMHardDiskDrive -VMName xxxx -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 (no path for Remove-… as it’s only removing a disk from the guest). Next, I used a Resize-VHD script (possibly pointlessly as it was a dynamic VHDX, but it worked…or at least didn’t fail), used Add-VMHardDiskDrive -VMName xxxx -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 -Path X:\xxxx.vhdx. I could then use DISKPART on the guest to switch the disk back online, extend it with server manager (on another machine- the storage server is Server Core) to- again deliberately- just 3.5TB to put limits on the copy process for experimentation. As soon as the extend had finished, the RoboCopy process kicked back in. All that’s left now is for the Windows DeDuplication to sort out the copied data and we should be ready to migrate over (I’m not sure how much it can compress the data given that it’s all unique images, which would already be considerably compressed through the jpg format etc. Currently running at 2% deduplication..)

WSMan client error on a Windows 8 client accessing Server 2012

I’ve been getting this for about an hour:

“The WSMan client cannot process the request. Proxy is not supported under HTTP transport.  Change the transport to HTTPS and specify valid proxy information and try again.”

And was getting pretty annoyed. I just couldn’t access my 4 remote Server 2012 servers. I read (but not carefully enough!) this link:

http://social.technet.microsoft.com/Forums/windowsserver/en-US/88d0cf24-c5c4-41aa-945c-ae7de7ed119d/remote-powershell-proxy-error?forum=winserverpowershell

And thought “that’s great, but I need my proxy server!”. Went to lunch, came back, and re-read it. Yup, it says local machine. So… I ran ‘netsh winhttp show proxy’ and I must have set up my machine to use our proxy server for some reason.