Windows 2012 R2 Deduplication Garbarge Collection eats up space

I have enabled Deduplication on a volume and it's running for a couple of weeks now. Backups are stored on this volume and dedup is doing a good job on this data (Weekly full backups + incrementals).

But when the weekly garbage collection runs, it claims all free space and the job get cancelled when the disk is full (or memory is low). It's going down from 600GB to 50GB. The following event is logged:

Event ID:      4143
Description:
Data Deduplication cancelled job type "Garbage Collection" on volume "\\?\Volume{55bdbbaf-af4f-431f-878e-c6ead208b86c}\". Memory resource is running low on the machine or in the job.

If i run a Full Optimization afterwards, the space is freed again.

Any idea what is happening? Is garbage collection trying to unoptimize some files or something?

Currently running Full Optimization, but here's the volume info:
Volume                   : E:
VolumeId                 : \\?\Volume{55bdbbaf-af4f-431f-878e-c6ead208b86c}\
Enabled                  : True
UsageType                : HyperV
DataAccessEnabled        : True
Capacity                 : 2.7 TB
FreeSpace                : 395.41 GB
UsedSpace                : 2.31 TB
UnoptimizedSize          : 11.85 TB
SavedSpace               : 9.54 TB
SavingsRate              : 80 %
MinimumFileAgeDays       : 0
MinimumFileSize          : 32768
NoCompress               : False
ExcludeFolder            :
ExcludeFileType          :
ExcludeFileTypeDefault   : {bin, vsv, slp, xml...}
NoCompressionFileType    : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify                   : False
OptimizeInUseFiles       : False
OptimizePartialFiles     : True