Need help! Time synchronization issue.

[SviridoffA]

Dear CloudStack Community,

I would be very grateful if someone could help me investigate a problem I’m currently facing.

It is difficult to pinpoint exactly when this started, but I’ve noticed that CloudStack has stopped updating the state of virtual machines promptly when they are shut down from within the guest OS. Usually, it takes about 3 minutes for CloudStack to reflect the power state change. However, in my current infrastructure, this update now takes roughly 3 hours.

To clarify: from the moment I execute halt -p inside the guest system until the state actually changes in the UI, 3 hours pass. I found the following entries in the management server logs:

DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:[]) (logid:) vm id: 1328 - time since last state update(-10739085 ms) has not passed graceful period yet

As seen in the message, the management server received a state update from the future, and the time difference is approximately 3 hours. All my servers—management, KVM nodes, and database servers—are synchronized to the same time in the Europe/Moscow zone, which is UTC +3 hours. I have verified this using timedatectl. DB time settings is:

SELECT @@global.time_zone, @@session.time_zone;
+--------------------+---------------------+
| @@global.time_zone | @@session.time_zone |
+--------------------+---------------------+
| SYSTEM             | SYSTEM              |
+--------------------+---------------------+

But if I understand correctly, the timestamps should still be in UTC.
In my db.properties, I have two parameters:

db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serN'&serverTimezone=UTC

db.usage.url.params=serverTimezone=UTC

It would be great if someone could point out the error in my configuration or explain where this time offset might be coming from, as well as the correct way to fix it without causing any issues.

I'm using:
Ubuntu 24.04.3 on all hosts
ACS 4.20.2
NTP servers outer and internal - chrony and chrony as clients
mysql 8.0.39

[TadiosAbebe]

@SviridoffA, this sounds very similar to an issue I encountered while testing VM HA. As documented here: #12183 using a custom timezone can impact certain CloudStack operations. In my case, the root cause was a timezone mismatch in the following code:

long milliSecondsSinceLastStateUpdate = currentTime.getTime() - vmStateUpdateTime.getTime();

One of these values was returned in UTC, while the other was returned in local time because it relied on NOW() in MySQL. As a result, CloudStack ended up comparing timestamps from two different timezones, which caused a consistent offset (3 hours in my case). Switching the system and database timezone back to the default UTC resolved the issue. I suspect doing the same would likely fix your problem as well.

[SviridoffA]

Thank you very much, @TadiosAbebe !. This information is very helpful. Could you please clarify: is it enough to change the timezone only on the MySQL servers, or do we also need to update it on all management and KVM nodes? Also, were there any issues after changing the timezone on the MySQL server?

[TadiosAbebe]

As far as I understand, changing the database server’s timezone is sufficient, since that’s where the local time was being retrieved. The cloudStack management server will still obtain the correct UTC time even if a custom timezone is configured on the server itself. After updating the database server’s timezone, a MySQL restart is required for the change to take effect. In my case, once I changed the timezone and restarted MySQL, everything started working as expected.

[SviridoffA]

@TadiosAbebe Thank you so much for your help. Changing the time zone on the MySQL servers resolved the issue, and everything is now working as expected. It’s strange, because I’m pretty sure it worked with the custom time zone a year ago on version 4.19 — I’m sure of it because we did extensive testing to minimize VM running status update times. Regardless, the issue surfaced in 4.20, but everything is functioning correctly now.