4-1. General Structure of Scheduling

4-2. Specific to Collective Communication Scheduling

4-3. Packet Level Scheduling (Packet level)

TLDR; ASTRA-sim+ns-3 is entirely controlled by the network layer’s timing which is determined by qlen_mon_interval variable, and the durationMicros attribute in Chakra ETs. Also, the system layer schedules all communication nodes with a 10 ns delay.

4-1. General Structure of Scheduling

There are two types of events that are always processed by ns3: control events and chakra events.

• Control events are stored as events with a callback function to monitor_buffer. This updates the simulator’s network parameters and also schedules another control event. This is first created in the network setup phase, and is repeatedly scheduled every qlen_mon_interval time units.
• Chakra events are called and scheduled by the Workload and System layer in ASTRA-sim. These are created by the ET Feeder. The chakra event nodes are created during Workload::Fire() in the network front end.

Let’s go through the starting phase of the simulation. First, we start with setting up the network. After initializing the network parameters, we schedule a control event (with a callback function to monitor_buffer()). This is done at qlen_mon_start time, which is set to 0 by default. This can be visualized as shown:

Control event is added to m_events queue in Network Layer

Once the control event is added, we exit the setup function, and start firing each of the nodes by going into a for loop. This starts the scheduling of each of the nodes to both system and network layers' event queues. This can be seen in the stdout:

[WORKLOAD]: The operation node is a COMP node. Caling sys->register_event.
issue,sys->id=7,tick=0,node->id=7,node->name=aten::slice
[SYSTEM - Register Event] sys id 7 with event General should_schedule: 1
[AstraSimNetwork::sim_schedule] Calling NS-3 Schedule()!

The scheduling here is quite subtle. The ChakraETs have a field called duartionMicros which is used to mark the relative time when the job finishes running. This is used as a wake-up timer for the ns3 scheduler. This can be visualized as follows:

Chakra Node is parsed then added to the Sys::event_queue and ns3’s m_events queue

For example, if the workload layer issues a COMP operation to finish at t=10ms or 10000 ns, the event is placed in the Sys::event_queue with its end time (tAbsolute = delay + m_currentTs) as the key. If this event hasn’t been scheduled yet in ns3, we place it in m_events by calling ns::Schedule.

ns3 event queue after simulator starts running

Once the simulator starts running, it repeatedly checks its m_events queue, where it will see a control event along with other chakra events queued by the system layer. The reason why we saw differing orders of events in our logs is because the system layer’s scheduling is completely dependent on the duartionMicros attribute of Charka nodes.

4-2. Specific to Collective Node Scheduling