Clock Synchronization Issues
Imagine two separate bank branches trying to process a shared account without knowing the exact time. If one branch updates a balance at 10:00 AM and another at 10:01 AM, the system must know which event happened first to avoid errors. In a distributed system, computers lack a single, shared clock to keep their internal time perfectly aligned. This gap creates massive confusion when multiple machines try to coordinate their daily digital tasks simultaneously. Without a way to agree on the sequence of events, data integrity fails entirely.
The Challenge of Physical Time
Every computer uses a local hardware oscillator to track the passage of time during its operation. These oscillators drift over time, meaning two identical machines will eventually report slightly different times. Even if you sync them today, they will diverge within hours due to temperature changes or small manufacturing flaws. In a network, you cannot simply ask every machine to check a central clock because network lag introduces delays. Requesting the time from a server takes a few milliseconds, which is an eternity for high-speed processors. This inherent uncertainty makes physical timekeeping unreliable for ordering events across a vast, distributed network of machines.
Key term: Clock drift — the natural tendency of independent computer hardware clocks to deviate from each other over time.
When we try to sync these clocks, we face the same problem as a global company trying to schedule a meeting across time zones. If the New York office thinks it is noon while the Tokyo office thinks it is midnight, they cannot easily agree on a deadline. Even with digital protocols, the time it takes for a message to travel across the wires creates a permanent gap. We cannot assume that a timestamp from one machine matches the reality of another machine. This reality forces us to stop relying on physical time and start looking for a better way to track event order.
Using Logical Clocks for Order
To solve this, engineers use logical clocks to track the relative order of events instead of wall-clock time. A logical clock acts like a simple counter that increments every time a local process performs an action. When one machine sends a message to another, it includes its current counter value in the data packet. The receiving machine updates its own counter to be higher than the value it just received. This process ensures that every machine maintains a causal relationship between events without needing perfect global time. It creates a chain of causality that everyone can follow accurately.
We can visualize this system through the following list of requirements for maintaining event order:
- Each process maintains a local counter that increases whenever a significant internal event occurs, ensuring that local actions are always uniquely numbered.
- Every message sent between machines carries the sender's current counter value, which allows the receiver to understand the state of the sender at that specific moment.
- When a machine receives a message, it sets its local counter to the maximum of its current value and the incoming value plus one, effectively synchronizing the logical timeline.
This method guarantees that if event A causes event B, then the timestamp for A will always be lower than B. It does not matter how much the physical clocks drift because the logic relies on message flow. This approach allows distributed systems to function as a single, cohesive unit despite having many independent parts. It turns a chaotic collection of machines into a predictable, ordered system that handles tasks in the correct sequence.
Logical clocks allow distributed systems to maintain a consistent order of events by tracking causality rather than relying on unreliable physical hardware clocks.
But what does it look like when these systems scale up to handle millions of microservices?