Make DRS take into account affinity rules in an intuitive way

[bhouse-nexthop]

problem

It appears DRS doesn't take into account affinity rules when rebalancing. I've got 5 VMs in a non-strict host anti-affinity group (I do non-strict since I have 5 total hosts, and if I take one down for maintenance its ok to co-locate during that duration).

I manually put one VM on each host, then run DRS and I can see it tries to move those VMs in the anti-affinity group so they're co-located on the same host even though there are other eligible VMs that can be moved to rebalance instead.

versions

4.22.0.0 on Ubuntu 24.04 with KVM

The steps to reproduce the bug

Steps in problem description

What to do about it?

DRS should be affinity-aware

[vishesh92]

@bhouse-nexthop DRS is affinity aware. But since you are using non-strict affinity groups, it allows the live migration.

For non-strict affinity groups you will still be able to live migrate the VMs to any host because the affinity group is non-strict. So, this is the intended behavior as of now.

[StepBee]

@vishesh92 i am with @bhouse-nexthop and am surprised about the DRS behavior you describe, because it is in contrast to the documentation of non-strict host anti-affinity groups.

From a technical perspective your description makes sense.

From a functional perspective, breaking a non-strict host anti-affinity should only break it in an urgency / emergency situation, for example activating maintenance for a host resulting in too less hosts to assure host anti-affinity.
According to the documentation for non-strict:
"In that case VMs are deployed to different hosts as long as there are enough hosts to satisfy the requirement, otherwise they might be deployed to the same host."

[bhouse-nexthop]

if non-strict allows vms to be colocated on the same node when the cluster can otherwise support it from a capacity standpoint, then that means non-strict essentially does nothing what-so-ever. Its not an affinity rule at all, its equivalent to not having one defined.

[DaanHoogland]

@bhouse-nexthop , non-strict tries and strict demands. this should be the difference. If you require DRS to free machines in combination with non-strict affinity groups, then there is room for discussion. I think it is a strong statement to say it does nothing whatsoever. It may in your case but even on the surface (i.e. without testing specific scenarios) it seems a harsh claim.

1. You want non-strict affinity, so VMs get deployed no matter what.
2. You want DRS so VMs get relocated to optimise host usage.

If you expect VMs to not be relocated to hosts where a VM in the same anti-affinity is running it should be a strict group. It seems that DRS and non-strict anti-affinity would always conflict on the long run.

[bhouse-nexthop]

what happens though if I don't have enough hosts online at that time and I use a strict affinity rule? I take a node down for maintenance, won't it fail to migrate a vm and therefore not enter maintenance mode with a strict affinity rule?

[StepBee]

that's exactly the case, the host won't enter maintenance - and for my understanding that's the main reason why non-strict was invented, to prevent this case.

If non-strict would mean kind-of-best effort but in the end cloudstack does not bind to it even if enough hosts are available, then i am with @bhouse-nexthop , the use-cases for non-strict are close to zero.

I think the documentation of non-strict, as pasted above, is quite clear about fulfilling anti-affinity as long as there are enough hosts available.

Looking from the other side - maybe my understanding is wrong and non-strict (anti-)affinity groups should only be considered for the initial deployment and not over the lifetime of a VM, for example in combination with DRS. Then it would be great to discuss about the extension of DRS to respect non-strict maybe? I see huge benefit from it, as i encourage using non-strict in my environment to make operations like rolling maintenance easier.

[weizhouapache]

@bhouse-nexthop
can you do a quick testing ?
migrate a vm with non-strict affinity groups, but please do not specify the destination, then check if the destination host is expected

[weizhouapache]

migrate a vm with non-strict affinity groups, but please do not specify the destination, then check if the destination host is expected

During host maintenance, there are multiple vm work job processed concurrently, which could cause issues. I am not sure.

I do not know if DRS works in the similar way.

[DaanHoogland]

all-in-all there are some discrepancies between how it is documented and how users experience this functionality. Would be nice if we can find it and resolve. @bhouse-nexthop , i am renaming your subject and moving it to the category “ideas”. I hope you’ll forgive me one day ;)