A little about my environment:
I have a Windows Server 2008 R2 virtual machine hosted on ESX 4.0 that is utilized as a File Server. On this server, we use DFS to replicate a copy of our network file share to a satellite office that also has a server running Windows Server 2008 R2 for the purposes of Disaster Recovery on top of daily backups taken by Backup Exec 2010 R2.
This particular set-up has been in place for a couple years and has ran without issue until recently.
The event:
On Friday, 8/9/2013, a couple of people reported that certain folders on our network file share seemed to revert to a status of early-to-mid afternoon 8/7/2013 as versions of files they'd worked on were date/time-stamped and reflected as such. We had backups from that night (Wednesday (8/7)) and Thursday (8/8) of up-to-date files. The files and folders were restored for anything that was noticed.
At the same time, I was able to confirm a couple of other folders / files that were unaffected and did have data that remained in-tact as they were date/time-stamped Thursday (8/8).
Late that Friday afternoon, I had set-up a specific subfolder with a versions of an affected file from Wednesday, Thursday, and the current version as of that moment in time on Friday for a user to compare versions to make sure that nothing was in fact lost.
I happened to go into the office today, Sunday (8/11) to take care of a couple things and noticed my subfolder and the files I had set-up in there were gone and the rest of the contents of the folder I was working in with that subfolder had reverted back to where they were on 8/7.
Trying to figure it out:
Now that this has effectively happened twice, I'm at a loss for what mechanically/functionally on the file server could have caused this to happen. I've dug through the Application and System Logs on the file server.
I noticed several informational items from the source 'volsnap' of Event ID 33 with the following verbiage - "The oldest shadow copy of volume E: was deleted to keep disk space usage for shadow copies of volume E: below the user defined limit."
This clued me to look to see if the Volume Shadow Copy service was running, which it was. I looked at the server in the satellite office (the other end of our DFS replication) to see if the Volume Shadow Copy service was running and it wasn't. I stopped it because I have no idea why it would be running since we don't use the functionality of 'previous versions'.
When looking through the Application log, I found a couple of isolated Errors related to VSS (Volume Shadow Copy Service) Event ID 12289 with the following verbiage:
Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. . hr = 0x80042409, Writer status is not available for one or more writers. A writer may have reached the limit to the number of available backup-restore session states.
.
Operation:
PreRestore Event
Context:
Old snapshot set: {31496c24-9516-4a42-91e3-650484dd72c4}
Old operation: 1019
Old state: 1
Old failure: 0
Old extended failure hr: 0x1
Old extended failure message: 0
Old snapshot set: {b51d2b8c-d10b-447e-aee4-2223dd661eb1}
Old operation: 1019
Old state: 1
Old failure: 0
Old extended failure hr: 0x1
Old extended failure message: 0
Execution Context: Writer
Writer Class Id: {2707761b-2324-473d-88eb-eb007a359533}
Writer Name: DFS Replication service writer
Writer Instance ID: {0250ab29-2b77-40d5-af2c-6cafe866c9c8}
The rest of the logs look fine/clean.
While I have the peace-of-mind of good backups, that doesn't help my current situation. The only thing I've found thus far in my own research thinking originally DFS may have been the culprit, is Microsoft KB article 2450944, which I cannot link to since my account hasn't been verified here yet.
To the best of my knowledge DFS doesn't use VSS in any capacity, nor do I believe that Backup Exec 2010 R2 is using VSS in any capacity while doing backups or restores.
Trying to trace back any changes made to the server; the previous weekend I had a scheduled maintenance window where I applied some patches, Windows Updates, etc., to my servers. While things were fine all week and this didn't appear until Friday, I'm also uncertain of anything in the context of changes made to the server that would've caused this.
Needless to say, I feel like I'm trying to find a needle in the haystack. Beyond what I've talked about, are there any other recommendations you can make for directions for me to look in?
I have a Windows Server 2008 R2 virtual machine hosted on ESX 4.0 that is utilized as a File Server. On this server, we use DFS to replicate a copy of our network file share to a satellite office that also has a server running Windows Server 2008 R2 for the purposes of Disaster Recovery on top of daily backups taken by Backup Exec 2010 R2.
This particular set-up has been in place for a couple years and has ran without issue until recently.
The event:
On Friday, 8/9/2013, a couple of people reported that certain folders on our network file share seemed to revert to a status of early-to-mid afternoon 8/7/2013 as versions of files they'd worked on were date/time-stamped and reflected as such. We had backups from that night (Wednesday (8/7)) and Thursday (8/8) of up-to-date files. The files and folders were restored for anything that was noticed.
At the same time, I was able to confirm a couple of other folders / files that were unaffected and did have data that remained in-tact as they were date/time-stamped Thursday (8/8).
Late that Friday afternoon, I had set-up a specific subfolder with a versions of an affected file from Wednesday, Thursday, and the current version as of that moment in time on Friday for a user to compare versions to make sure that nothing was in fact lost.
I happened to go into the office today, Sunday (8/11) to take care of a couple things and noticed my subfolder and the files I had set-up in there were gone and the rest of the contents of the folder I was working in with that subfolder had reverted back to where they were on 8/7.
Trying to figure it out:
Now that this has effectively happened twice, I'm at a loss for what mechanically/functionally on the file server could have caused this to happen. I've dug through the Application and System Logs on the file server.
I noticed several informational items from the source 'volsnap' of Event ID 33 with the following verbiage - "The oldest shadow copy of volume E: was deleted to keep disk space usage for shadow copies of volume E: below the user defined limit."
This clued me to look to see if the Volume Shadow Copy service was running, which it was. I looked at the server in the satellite office (the other end of our DFS replication) to see if the Volume Shadow Copy service was running and it wasn't. I stopped it because I have no idea why it would be running since we don't use the functionality of 'previous versions'.
When looking through the Application log, I found a couple of isolated Errors related to VSS (Volume Shadow Copy Service) Event ID 12289 with the following verbiage:
Volume Shadow Copy Service error: Unexpected error An older writer session state is being removed. . hr = 0x80042409, Writer status is not available for one or more writers. A writer may have reached the limit to the number of available backup-restore session states.
.
Operation:
PreRestore Event
Context:
Old snapshot set: {31496c24-9516-4a42-91e3-6
Old operation: 1019
Old state: 1
Old failure: 0
Old extended failure hr: 0x1
Old extended failure message: 0
Old snapshot set: {b51d2b8c-d10b-447e-aee4-2
Old operation: 1019
Old state: 1
Old failure: 0
Old extended failure hr: 0x1
Old extended failure message: 0
Execution Context: Writer
Writer Class Id: {2707761b-2324-473d-88eb-e
Writer Name: DFS Replication service writer
Writer Instance ID: {0250ab29-2b77-40d5-af2c-6
The rest of the logs look fine/clean.
While I have the peace-of-mind of good backups, that doesn't help my current situation. The only thing I've found thus far in my own research thinking originally DFS may have been the culprit, is Microsoft KB article 2450944, which I cannot link to since my account hasn't been verified here yet.
To the best of my knowledge DFS doesn't use VSS in any capacity, nor do I believe that Backup Exec 2010 R2 is using VSS in any capacity while doing backups or restores.
Trying to trace back any changes made to the server; the previous weekend I had a scheduled maintenance window where I applied some patches, Windows Updates, etc., to my servers. While things were fine all week and this didn't appear until Friday, I'm also uncertain of anything in the context of changes made to the server that would've caused this.
Needless to say, I feel like I'm trying to find a needle in the haystack. Beyond what I've talked about, are there any other recommendations you can make for directions for me to look in?