Communication Routines. More...

#include <stdlib.h>
#include <float.h>
#include <core/core.h>
#include <gvt/gvt.h>
#include <queues/queues.h>
#include <communication/communication.h>
#include <statistics/statistics.h>
#include <scheduler/scheduler.h>
#include <scheduler/process.h>
#include <datatypes/list.h>
#include <mm/mm.h>
#include <arch/atomic.h>
#include <communication/mpi.h>

Include dependency graph for communication.c:

Go to the source code of this file.

Functions
void	communication_init (void)
	Initialize the communication subsystem. More...

void	communication_fini (void)
	Finalize the communication subsystem. More...

static struct lp_struct *	which_slab_to_use (GID_t sender, GID_t receiver)
	Find a slab to allocate a message buffer. More...

void	msg_hdr_release (msg_hdr_t *msg)
	Release a message header. More...

msg_hdr_t *	get_msg_hdr_from_slab (struct lp_struct *lp)
	Get a buffer to keep a message header. More...

msg_t *	get_msg_from_slab (struct lp_struct *lp)
	Get a buffer to keep a message. More...

void	msg_release (msg_t *msg)
	Release a message buffer. More...

void	ParallelScheduleNewEvent (unsigned int gid_receiver, simtime_t timestamp, unsigned int event_type, void *event_content, unsigned int event_size)
	Schedule a new message to some LP. More...

void	send_antimessages (struct lp_struct *lp, simtime_t after_simtime)
	Send all antimessages for a certain LP. More...

void	Send (msg_t *msg)
	Send a message. More...

void	insert_outgoing_msg (msg_t *msg)
	Place a message in the temporary LP outgoing buffer. More...

void	send_outgoing_msgs (struct lp_struct *lp)
	Send all pending outgoing messages. More...

void	pack_msg (msg_t *msg, GID_t sender, GID_t receiver, int type, simtime_t timestamp, simtime_t send_time, size_t size, void payload)
	Pack a message in a platform-level data structure. More...

void	msg_to_hdr (msg_hdr_t hdr, msg_t msg)
	Convert a message to a message header. More...

void	hdr_to_msg (msg_hdr_t hdr, msg_t msg)
	convert a message header into a message More...

void	dump_msg_content (msg_t *msg)
	Dump the content of a message. More...

unsigned int	mark_to_gid (unsigned long long mark)
	Tell the GID of the sender of a message, given its mark. More...

void	validate_msg (msg_t *msg)
	Perform some sanity checks on a message buffer. More...

Variables
void(*	ScheduleNewEvent )(unsigned int gid_receiver, simtime_t timestamp, unsigned int event_type, void *event_content, unsigned int event_size)
	This is the function pointer to correctly set ScheduleNewEvent API version, depending if we're running serially or in parallel.

Detailed Description

Communication Routines.

This file contains all the communication routines, for exchanging messages among different logical processes and simulator instances.

Copyright: Copyright (C) 2008-2019 HPDCS Group https://hpdcs.github.io

This file is part of ROOT-Sim (ROme OpTimistic Simulator).

ROOT-Sim is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; only version 3 of the License applies.

ROOT-Sim is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with ROOT-Sim; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

Author: Francesco Quaglia; Roberto Vitali; Alessandro Pellegrini

Definition in file communication.c.

Function Documentation

void communication_fini ( void )

Finalize the communication subsystem.

This function finalizes the communication subsystem. It is called by at simulation shutdown, both if the simulation was successful or if it failed. This is the place where to cleanly shutdown the communication subsystem.

Definition at line 75 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void communication_init ( void )

Initialize the communication subsystem.

This function initializes the communication subsystem. It is called by the init module upon simulation startup. Any initialization of this subsystem should be placed here.

Definition at line 59 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void dump_msg_content ( msg_t * msg )

Dump the content of a message.

This function dumps the content of a message. This is mainly used when some runtime error is encountered, to provide on screen information which might be used for debugging what is going on.

Parameters

msg	A pointer to the message to dump on screen.

Definition at line 589 of file communication.c.

Here is the caller graph for this function:

msg_t* get_msg_from_slab ( struct lp_struct * lp )

Get a buffer to keep a message.

This function allocates a buffer from the slab of the LP identified by the specified lp_struct to keep a message.

Warning: The slab allocator is configured at simulation startup to keep buffers of size SLAB_MSG_SIZE. The type msg_t uses a flexible array (the event_content member) to keep also the model-specific payload. Therefore, if the size of the payload is such that sizeof(msg_t)+payload is larger that SLAB_MSG_SIZE, relying on this function to allocate a msg_t will generate a memory overflow. ALWAYS check the size of the payload before getting a message buffer from here!

Parameters

lp	A pointer to the lp_struct where to take the message buffer from. The slab allocator of the LP is used.

Returns: A pointer to the freshly allocated buffer. It is large enough to keep a msg_t datatype, but it might be too small to also keep the event payload.

Definition at line 192 of file communication.c.

Here is the caller graph for this function:

msg_hdr_t* get_msg_hdr_from_slab ( struct lp_struct * lp )

Get a buffer to keep a message header.

Message headers are the compact way used to represent antimessages. This function retrieves a buffer to keep a message header. Antimessages are associated with the sender LP, so the lp_struct used here must be the one of the sender LP.

Parameters

lp	A pointer to the lp_struct where to take the message header from. The slab allocator of the LP is used.

Returns: A pointer to the freshly allocated buffer. It is large enough to keep a msg_hdr_t datatype.

Definition at line 160 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void hdr_to_msg	(	msg_hdr_t *	hdr,
		msg_t *	msg
	)

convert a message header into a message

This is a commodity function which prepares a message data structure from a message header. Both the header and the message buffers must be already allocated.

The purpose of this function is to prepare the sending of an antimessage. Indeed, an antimessage is sent as a message of size zero, and the information is taken from compact versions of the originally sent messages, kept in the msg_hdr_t type. When an antimessage must be sent out, this is done by copying the message header into a msg_t type. This is required because all message sending logic assumes that a msg_t data structure is being passed (this avoid having to perform multiple checks or multiple casts in the code base).

Parameters

hdr	A pointer to the message header from which the message information is taken.
msg	A pointer to the message where the header information is copied.

Definition at line 568 of file communication.c.

Here is the caller graph for this function:

void insert_outgoing_msg ( msg_t * msg )

Place a message in the temporary LP outgoing buffer.

To quickly finish the execution of events, once a simulation model calls ScheduleNewEvent(), the event is not actually immediately sent. On the other hand, the message is packed and placed in a temporary output queue. Once the event's execution is completed, this queue is scanned to send out all the generated events.

This function places a newly-scheduled event into this temporary queue, which is implemented as a resizable array of pointers to message buffers.

Parameters

msg	The packed message to insert in the temporary outgoing queue.

Definition at line 396 of file communication.c.

Here is the caller graph for this function:

unsigned int mark_to_gid ( unsigned long long mark )

Tell the GID of the sender of a message, given its mark.

This function inverts the Cantor pairing function used to generate unique message marks. It can be used to perform sanity checks on the marks, to see whether they are correct or corrupted. Also, it can assist in debugging errors in the management of messages.

Warning: This function is computationally costly! never ever use it in a production environment. The NDEBUG guard ensures that it is never compiled in a final version of the runtime environment, so keep it only as a debugging function.

Parameters

mark	The mark to invert.

Returns: The GID of the sender of the message stamped with the mark. The GID is not actually represented as a GID_t, rather as an int.

Definition at line 628 of file communication.c.

Here is the caller graph for this function:

void msg_hdr_release ( msg_hdr_t * msg )

Release a message header.

Message headers are taken always from the sender LP, as they are the compact representation of an antimessage. Therefore, the release function does not check whether the LP is local or not, but it frees memory directly from the sender slab allocator.

Parameters

msg	A pointer to the message header to release

Definition at line 130 of file communication.c.

Here is the caller graph for this function:

void msg_release ( msg_t * msg )

Release a message buffer.

This function releases a message buffer which is no longer needed (i.e., it was keeping a message annihilated by a corresponding antimessage, or a message which is now beyond the commit horizon identified by the GVT).

To free the message, this function checks againts the total size of the message, considering both the size of the msg_t structure and that of the payload kept in the event_content member of msg_t. If the total size is smaller than SLAB_MSG_SIZE, then the message was taken from a slab, otherwise it has been taken by the buddy system. Therefore, we free the buffer from the corresponding data structure.

Messages are freed using this function both if they are stable and transient in this simulation instance, i.e. if they were destined for a local LP or if they were temporarily allocated here to be transmitted to a remote rank using MPI. Therefore, the function which_slab_to_use() is queried to find out the proper slab to use for deallocating the buffer.

Parameters

msg	A pointer to the message buffer to release.

Definition at line 224 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void msg_to_hdr	(	msg_hdr_t *	hdr,
		msg_t *	msg
	)

Convert a message to a message header.

This function takes an already packed message pointed by msg and populates the relevant fields of the message header pointed by hdr to create a compact representation of the message being sent out. This is necessary to later send antimessages, upon a rollback operation.

Parameters

hdr	A pointer to a msg_hdr_t where to store the relevant information to represent an antimessage.
msg	A pointer to an already-packed message from which to take the relevant information to populate the header

Definition at line 532 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void pack_msg	(	msg_t **	msg,
		GID_t	sender,
		GID_t	receiver,
		int	type,
		simtime_t	timestamp,
		simtime_t	send_time,
		size_t	size,
		void *	payload
	)

Pack a message in a platform-level data structure.

This function takes all the parameters which represent a model-level event and pack it in a simulation-level datastructure representing a message (namely, a msg_t type).

This function also allocates the buffer for that message. To this end, it determines whether the buffer can be taken from some slab allocator or not (depending on the size of the payload, which determines whether the final message fits into a buffer of size SLAB_MSG_SIZE).

This is a uniform internal API which can be used in any situation. Indeed, it relies on the which_slab_to_use() internal function to find out whether this message will be kept in the local instance of a distributed simulation or not.

Parameters

msg	A double pointer to a msg_t type. Since this function allocates the buffer, a pointer to a `msg_t` `*` datatype should be passed, in order for the caller to receive the pointer to the message.
sender	The GID of the sender
receiver	The GID of the receiver
type	A numerical code identifying the event type. This can be a model-specific type, or a platform-level code used to identify a control message.
timestamp	This is the simulation time at which the destination LP will have to execute this event.
send_time	This event has been sent by `sender` at this particular simulation time. This information is used to handle antimessages upon a rollback operation.
size	The size of the model-specific payload.
payload	A pointer to the model-specific payload. It can be a pointer to whatever, e.g. the stack of a ULT in which the LP is running, or a data structure in the simulation state of the LP. This is not a problem because we make a full copy of the event payload. Problems might arise if a pointer is present in the payload, and the ECS subsystem is not running, but at that point it is the simulation model's responsibility to make the simulation inconsistent or to crash the run.

Definition at line 494 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void ParallelScheduleNewEvent	(	unsigned int	gid_receiver,
		simtime_t	timestamp,
		unsigned int	event_type,
		void *	event_content,
		unsigned int	event_size
	)

Schedule a new message to some LP.

This is one of the entry points from the application model, used in parallel/distributed simulations. The simulation model calls ScheduleNewEvent() which is a function pointer, set to point to this implementation if the –sequential flag is not passed as an option.

This function performs all the required sanity checks:

Is the destination LP id valid?
Are we sending an event to the past?
Is the event type in a valid range?

If all the checks pass, then the event content is copied in a platform-level buffer and a pointer to it is placed in the temporary LP outgoing buffer, for later delivery (possibly via MPI).

If the LP is running in silent execution, this function simply returns as the event has already been sent during a previous event execution.

Parameters

gid_receiver	Global id of logical process at which the message must be delivered
timestamp	Logical Virtual Time associated with the event enveloped into the message
event_type	Type of the event
event_content	Payload of the event
event_size	Size of event's payload

Definition at line 263 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void Send ( msg_t * msg )

Send a message.

This function sends a message. This is the decision point where a message receiver is checked to understand whether it must be sent using MPI, or if it is heading towards a local LP and therefore it can be placed in the bottom half buffer.

This function is therefore a uniform internal API function to implement message passing in a parallel/distributed simulation environment.

Parameters

msg	A pointer to the message to send

Definition at line 367 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void send_antimessages	(	struct lp_struct *	lp,
		simtime_t	after_simtime
	)

Send all antimessages for a certain LP.

This function send all the antimessages for a certain LP, provided a simulation time (which is associated with the time at which we are rolling back.

After that the antimessage is sent, the header is immediately removed from the output queue, as MPI guarantees that the antimessage is eventually received at the destination.

Parameters

lp	A pointer to the LP lp_struct for which antimessages should be sent
after_simtime	The simulation time instant after which to send antimessages

Definition at line 327 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void send_outgoing_msgs ( struct lp_struct * lp )

Send all pending outgoing messages.

This function sends all messages registered in the outgoing message queue during the execution of an event (see insert_outgoing_msg()) to the destination LPs. Also, this function records in the output queue of the sender LP that at a certain simulation time a certain message was sent—this is done using the msg_hdr_t type. This information is used upon a rollback to send out antimessages.

After the execution of this function, the temporary outgoing queue is considered as empty.

Parameters

lp	A pointer to the LP's lp_struct for which we want to finalize the event send operation.

Definition at line 431 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void validate_msg ( msg_t * msg )

Perform some sanity checks on a message buffer.

This is a debugging function which performs some sanity checks on a message buffer, and aborts the simulation if these checks do not pass.

The checks performed are:

Is the sender LP GID in a valid range?
Is the destination LP GID in a valid range?
Is the message kind of a valid type?
Is the sender associated with the message mark in a valid range?
Is the sender associated with a rendezvous mark in a valid range?
Is the message type in a valid range?

If a message is corrupted due to any reason, the likelihood that this function spots the corruption is very high.

Warning: This function is computationally costly! never ever use it in a production environment. The NDEBUG guard ensures that it is never compiled in a final version of the runtime environment, so keep it only as a debugging function.

Parameters

msg	A pointer to the message to validate

Definition at line 664 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

static struct lp_struct* which_slab_to_use	(	GID_t	sender,
		GID_t	receiver
	)

static

Find a slab to allocate a message buffer.

Messages are kept in per-LP memory. This function is used to find out from what LP slab a message buffer should be allocated. The reason for such a function to exist is because messages can be targeted to local or remote LPs, but in both cases we need some memory. Therefore, this function takes the GID of two LPs and finds out whether the destination LP is local or not. If it is local, then the message will be incorporated in some local LP queue, therefore we take memory from there. On the other hand, if the LP is remote, it means that we are packing a message which will be later passed to MPI for remote transmission—as soon as the transmission is completed, that buffer will be released. Therefore, in that case, we take the memory from the source LP.

Parameters

sender	The GID of the sender LP of a message
receiver	the GID of the destination LP of the message

Definition at line 112 of file communication.c.

Here is the call graph for this function:

Here is the caller graph for this function:

Functions

Variables

Detailed Description

Function Documentation