Communication Routines. More...

#include <core/core.h>

Include dependency graph for communication.h:

This graph shows which files directly or indirectly include this file:

Data Structures
struct	_outgoing_t
	Per-LP buffer of newly-generated events. More...

Macros
#define	SLAB_MSG_SIZE 512
	Slab allocator max message size. More...

#define	is_control_msg(type) (type >= MIN_VALUE_CONTROL && type != RENDEZVOUS_START)
	This macro tells whether a message is a control message, by its type.

#define	INIT_OUTGOING_MSG 8

Typedefs
typedef struct _outgoing_t	outgoing_t
	Per-LP buffer of newly-generated events. More...

Enumerations
enum	_control_msgs { RESERVED_MSG_CODE = 65532, TOPOLOGY_UPDATE, ABM_UPDATE, ABM_VISITING, ABM_LEAVING, MIN_VALUE_CONTROL = 65537, RENDEZVOUS_START, RENDEZVOUS_ACK, RENDEZVOUS_UNBLOCK, RENDEZVOUS_ROLLBACK, RENDEZVOUS_GET_PAGE, RENDEZVOUS_GET_PAGE_ACK, RENDEZVOUS_PAGE_WRITE_BACK, MAX_VALUE_CONTROL }
	Simulation Platform Control Messages. More...

enum	_mpi_tags { MSG_NEW_GVT = 100, MSG_FINI }
	Internal MPI tags. More...

Functions
void	ParallelScheduleNewEvent (unsigned int, simtime_t, unsigned int, void *, unsigned int)
	Schedule a new message to some LP. More...

void	communication_init (void)
	Initialize the communication subsystem. More...

void	communication_fini (void)
	Finalize the communication subsystem. More...

void	Send (msg_t *msg)
	Send a message. More...

void	insert_outgoing_msg (msg_t *msg)
	Place a message in the temporary LP outgoing buffer. More...

void	send_outgoing_msgs (struct lp_struct *)
	Send all pending outgoing messages. More...

void	send_antimessages (struct lp_struct *, simtime_t)
	Send all antimessages for a certain LP. More...

void	msg_hdr_release (msg_hdr_t *msg)
	Release a message header. More...

msg_t *	get_msg_from_slab (struct lp_struct *)
	Get a buffer to keep a message. More...

msg_hdr_t *	get_msg_hdr_from_slab (struct lp_struct *)
	Get a buffer to keep a message header. More...

void	pack_msg (msg_t *msg, GID_t sender, GID_t receiver, int type, simtime_t timestamp, simtime_t send_time, size_t size, void payload)
	Pack a message in a platform-level data structure. More...

void	msg_to_hdr (msg_hdr_t hdr, msg_t msg)
	Convert a message to a message header. More...

void	hdr_to_msg (msg_hdr_t hdr, msg_t msg)
	convert a message header into a message More...

void	msg_release (msg_t *msg)
	Release a message buffer. More...

void	dump_msg_content (msg_t *msg)
	Dump the content of a message. More...

void	validate_msg (msg_t *msg)
	Perform some sanity checks on a message buffer. More...

Detailed Description

Communication Routines.

This file contains all the communication routines, for exchanging messages among different logical processes and simulator instances.

Copyright: Copyright (C) 2008-2019 HPDCS Group https://hpdcs.github.io

This file is part of ROOT-Sim (ROme OpTimistic Simulator).

ROOT-Sim is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; only version 3 of the License applies.

ROOT-Sim is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with ROOT-Sim; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

Author: Francesco Quaglia; Roberto Vitali; Alessandro Pellegrini

Definition in file communication.h.

Macro Definition Documentation

#define INIT_OUTGOING_MSG 8

For performance reasons, while executing an event, newly-generated events are packed and pre-buffered in a per-LP data structure defined by the outgoing_t type. This implements a resizable array of pointers to msg_t types. Anytime that an event generates more new events than currently supported, the array is doubled in size.

This macro tells the number of pointers to msg_t which is allocated in the resizable array, for each LP, when the simulation starts.

Definition at line 101 of file communication.h.

#define SLAB_MSG_SIZE 512

Slab allocator max message size.

This is the size in bytes of a slab from the message slab allocator. If messages are smaller than this size, then message buffers are taken from the slab. Otherwise, the buddy system is queried directly.

Definition at line 43 of file communication.h.

Typedef Documentation

typedef struct _outgoing_t outgoing_t

Per-LP buffer of newly-generated events.

For performance reasons, while executing an event, newly-generated events are packed and pre-buffered in a per-LP data structure defined by the outgoing_t type. This implements a resizable array of pointers to msg_t types. Anytime that an event generates more new events than currently supported, the array is doubled in size.

This structure is used by the communication subsystem to handle outgoing messages. At simulation startup, there is space for at most INIT_OUTGOING_MSG messages.

The structure keeps pointers to messages which have been packed using the pack_msg() function. This allows to implement from the beginning a zero-copy message passing policy.

Enumeration Type Documentation

enum _control_msgs

Simulation Platform Control Messages.

In some circumstances, ROOT-Sim wants to send to a certain LP a message which should not be actually delivered to the simulation model. These are what are referred to as control messages, and are used to implement some sort of distributed (asynchronous) state changes at specific LPs.

To properly manage control messages, they must be declared in this enum. Note that the special MIN_VALUE_CONTROL is used to reserve space for application message codes. Anything below that value is considered to be a model-specific message code.

Enumerator
TOPOLOGY_UPDATE	Used by the topology API to convey remote updates on costs/probabilities
ABM_UPDATE	Used by ABM API, right now these are treated as normal positive messages
ABM_VISITING	Used by ABM API, right now these are treated as normal positive messages
ABM_LEAVING	Used by ABM API, right now these are treated as normal positive messages
MIN_VALUE_CONTROL	Separation value between model and platform messages.
RENDEZVOUS_START	ECS protocol: start synchronizing two LPs for a page fault.
RENDEZVOUS_ACK	ECS protocol: the sender LP has been synchronized and is now blocked.
RENDEZVOUS_UNBLOCK	ECS protocol: the destination LP can resume its normal execution.
RENDEZVOUS_ROLLBACK	ECS protocol: an ECS synchronization should be rolled back.
RENDEZVOUS_GET_PAGE	ECS protocol: a remote LP is asked for a certain set of pages.
RENDEZVOUS_GET_PAGE_ACK	ECS protocol: the sender LP is giving a lease on a set of pages.
RENDEZVOUS_PAGE_WRITE_BACK	ECS protocol: modified pages are sent back to the owner LP.
MAX_VALUE_CONTROL	Anything after this value is considered as an impossible message.

Definition at line 58 of file communication.h.

enum _mpi_tags

Internal MPI tags.

This enum defines the MPI tags which are internally used by the simulation runtime environment to exchange messages which are used to synchronize upon specific activities.

Enumerator
MSG_NEW_GVT	Master notifies the new GVT.
MSG_FINI	One rank informs the others that the simulation has to be stopped.

Definition at line 86 of file communication.h.

Function Documentation

void communication_fini ( void )

Finalize the communication subsystem.

This function finalizes the communication subsystem. It is called by at simulation shutdown, both if the simulation was successful or if it failed. This is the place where to cleanly shutdown the communication subsystem.

Definition at line 75 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void communication_init ( void )

Initialize the communication subsystem.

This function initializes the communication subsystem. It is called by the init module upon simulation startup. Any initialization of this subsystem should be placed here.

Definition at line 59 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void dump_msg_content ( msg_t * msg )

Dump the content of a message.

This function dumps the content of a message. This is mainly used when some runtime error is encountered, to provide on screen information which might be used for debugging what is going on.

Parameters

msg	A pointer to the message to dump on screen.

Definition at line 589 of file communication.c.

Here is the caller graph for this function:

msg_t* get_msg_from_slab ( struct lp_struct * lp )

Get a buffer to keep a message.

This function allocates a buffer from the slab of the LP identified by the specified lp_struct to keep a message.

Warning: The slab allocator is configured at simulation startup to keep buffers of size SLAB_MSG_SIZE. The type msg_t uses a flexible array (the event_content member) to keep also the model-specific payload. Therefore, if the size of the payload is such that sizeof(msg_t)+payload is larger that SLAB_MSG_SIZE, relying on this function to allocate a msg_t will generate a memory overflow. ALWAYS check the size of the payload before getting a message buffer from here!

Parameters

lp	A pointer to the lp_struct where to take the message buffer from. The slab allocator of the LP is used.

Returns: A pointer to the freshly allocated buffer. It is large enough to keep a msg_t datatype, but it might be too small to also keep the event payload.

Definition at line 192 of file communication.c.

Here is the caller graph for this function:

msg_hdr_t* get_msg_hdr_from_slab ( struct lp_struct * lp )

Get a buffer to keep a message header.

Message headers are the compact way used to represent antimessages. This function retrieves a buffer to keep a message header. Antimessages are associated with the sender LP, so the lp_struct used here must be the one of the sender LP.

Parameters

lp	A pointer to the lp_struct where to take the message header from. The slab allocator of the LP is used.

Returns: A pointer to the freshly allocated buffer. It is large enough to keep a msg_hdr_t datatype.

Definition at line 160 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void hdr_to_msg	(	msg_hdr_t *	hdr,
		msg_t *	msg
	)

convert a message header into a message

This is a commodity function which prepares a message data structure from a message header. Both the header and the message buffers must be already allocated.

The purpose of this function is to prepare the sending of an antimessage. Indeed, an antimessage is sent as a message of size zero, and the information is taken from compact versions of the originally sent messages, kept in the msg_hdr_t type. When an antimessage must be sent out, this is done by copying the message header into a msg_t type. This is required because all message sending logic assumes that a msg_t data structure is being passed (this avoid having to perform multiple checks or multiple casts in the code base).

Parameters

hdr	A pointer to the message header from which the message information is taken.
msg	A pointer to the message where the header information is copied.

Definition at line 568 of file communication.c.

Here is the caller graph for this function:

void insert_outgoing_msg ( msg_t * msg )

Place a message in the temporary LP outgoing buffer.

To quickly finish the execution of events, once a simulation model calls ScheduleNewEvent(), the event is not actually immediately sent. On the other hand, the message is packed and placed in a temporary output queue. Once the event's execution is completed, this queue is scanned to send out all the generated events.

This function places a newly-scheduled event into this temporary queue, which is implemented as a resizable array of pointers to message buffers.

Parameters

msg	The packed message to insert in the temporary outgoing queue.

Definition at line 396 of file communication.c.

Here is the caller graph for this function:

void msg_hdr_release ( msg_hdr_t * msg )

Release a message header.

Message headers are taken always from the sender LP, as they are the compact representation of an antimessage. Therefore, the release function does not check whether the LP is local or not, but it frees memory directly from the sender slab allocator.

Parameters

msg	A pointer to the message header to release

Definition at line 130 of file communication.c.

Here is the caller graph for this function:

void msg_release ( msg_t * msg )

Release a message buffer.

This function releases a message buffer which is no longer needed (i.e., it was keeping a message annihilated by a corresponding antimessage, or a message which is now beyond the commit horizon identified by the GVT).

To free the message, this function checks againts the total size of the message, considering both the size of the msg_t structure and that of the payload kept in the event_content member of msg_t. If the total size is smaller than SLAB_MSG_SIZE, then the message was taken from a slab, otherwise it has been taken by the buddy system. Therefore, we free the buffer from the corresponding data structure.

Messages are freed using this function both if they are stable and transient in this simulation instance, i.e. if they were destined for a local LP or if they were temporarily allocated here to be transmitted to a remote rank using MPI. Therefore, the function which_slab_to_use() is queried to find out the proper slab to use for deallocating the buffer.

Parameters

msg	A pointer to the message buffer to release.

Definition at line 224 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void msg_to_hdr	(	msg_hdr_t *	hdr,
		msg_t *	msg
	)

Convert a message to a message header.

This function takes an already packed message pointed by msg and populates the relevant fields of the message header pointed by hdr to create a compact representation of the message being sent out. This is necessary to later send antimessages, upon a rollback operation.

Parameters

hdr	A pointer to a msg_hdr_t where to store the relevant information to represent an antimessage.
msg	A pointer to an already-packed message from which to take the relevant information to populate the header

Definition at line 532 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void pack_msg	(	msg_t **	msg,
		GID_t	sender,
		GID_t	receiver,
		int	type,
		simtime_t	timestamp,
		simtime_t	send_time,
		size_t	size,
		void *	payload
	)

Pack a message in a platform-level data structure.

This function takes all the parameters which represent a model-level event and pack it in a simulation-level datastructure representing a message (namely, a msg_t type).

This function also allocates the buffer for that message. To this end, it determines whether the buffer can be taken from some slab allocator or not (depending on the size of the payload, which determines whether the final message fits into a buffer of size SLAB_MSG_SIZE).

This is a uniform internal API which can be used in any situation. Indeed, it relies on the which_slab_to_use() internal function to find out whether this message will be kept in the local instance of a distributed simulation or not.

Parameters

msg	A double pointer to a msg_t type. Since this function allocates the buffer, a pointer to a `msg_t` `*` datatype should be passed, in order for the caller to receive the pointer to the message.
sender	The GID of the sender
receiver	The GID of the receiver
type	A numerical code identifying the event type. This can be a model-specific type, or a platform-level code used to identify a control message.
timestamp	This is the simulation time at which the destination LP will have to execute this event.
send_time	This event has been sent by `sender` at this particular simulation time. This information is used to handle antimessages upon a rollback operation.
size	The size of the model-specific payload.
payload	A pointer to the model-specific payload. It can be a pointer to whatever, e.g. the stack of a ULT in which the LP is running, or a data structure in the simulation state of the LP. This is not a problem because we make a full copy of the event payload. Problems might arise if a pointer is present in the payload, and the ECS subsystem is not running, but at that point it is the simulation model's responsibility to make the simulation inconsistent or to crash the run.

Definition at line 494 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void ParallelScheduleNewEvent	(	unsigned int	gid_receiver,
		simtime_t	timestamp,
		unsigned int	event_type,
		void *	event_content,
		unsigned int	event_size
	)

Schedule a new message to some LP.

This is one of the entry points from the application model, used in parallel/distributed simulations. The simulation model calls ScheduleNewEvent() which is a function pointer, set to point to this implementation if the –sequential flag is not passed as an option.

This function performs all the required sanity checks:

Is the destination LP id valid?
Are we sending an event to the past?
Is the event type in a valid range?

If all the checks pass, then the event content is copied in a platform-level buffer and a pointer to it is placed in the temporary LP outgoing buffer, for later delivery (possibly via MPI).

If the LP is running in silent execution, this function simply returns as the event has already been sent during a previous event execution.

Parameters

gid_receiver	Global id of logical process at which the message must be delivered
timestamp	Logical Virtual Time associated with the event enveloped into the message
event_type	Type of the event
event_content	Payload of the event
event_size	Size of event's payload

Definition at line 263 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void Send ( msg_t * msg )

Send a message.

This function sends a message. This is the decision point where a message receiver is checked to understand whether it must be sent using MPI, or if it is heading towards a local LP and therefore it can be placed in the bottom half buffer.

This function is therefore a uniform internal API function to implement message passing in a parallel/distributed simulation environment.

Parameters

msg	A pointer to the message to send

Definition at line 367 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void send_antimessages	(	struct lp_struct *	lp,
		simtime_t	after_simtime
	)

Send all antimessages for a certain LP.

This function send all the antimessages for a certain LP, provided a simulation time (which is associated with the time at which we are rolling back.

After that the antimessage is sent, the header is immediately removed from the output queue, as MPI guarantees that the antimessage is eventually received at the destination.

Parameters

lp	A pointer to the LP lp_struct for which antimessages should be sent
after_simtime	The simulation time instant after which to send antimessages

Definition at line 327 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void send_outgoing_msgs ( struct lp_struct * lp )

Send all pending outgoing messages.

This function sends all messages registered in the outgoing message queue during the execution of an event (see insert_outgoing_msg()) to the destination LPs. Also, this function records in the output queue of the sender LP that at a certain simulation time a certain message was sent—this is done using the msg_hdr_t type. This information is used upon a rollback to send out antimessages.

After the execution of this function, the temporary outgoing queue is considered as empty.

Parameters

lp	A pointer to the LP's lp_struct for which we want to finalize the event send operation.

Definition at line 431 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void validate_msg ( msg_t * msg )

Perform some sanity checks on a message buffer.

This is a debugging function which performs some sanity checks on a message buffer, and aborts the simulation if these checks do not pass.

The checks performed are:

Is the sender LP GID in a valid range?
Is the destination LP GID in a valid range?
Is the message kind of a valid type?
Is the sender associated with the message mark in a valid range?
Is the sender associated with a rendezvous mark in a valid range?
Is the message type in a valid range?

If a message is corrupted due to any reason, the likelihood that this function spots the corruption is very high.

Warning: This function is computationally costly! never ever use it in a production environment. The NDEBUG guard ensures that it is never compiled in a final version of the runtime environment, so keep it only as a debugging function.

Parameters

msg	A pointer to the message to validate

Definition at line 664 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

Data Structures

Macros

Typedefs

Enumerations

Functions

Detailed Description

Macro Definition Documentation

Typedef Documentation

Enumeration Type Documentation

Function Documentation