Parallel Virtual Machine

From Parawiki

(Redirected from PVM)
Jump to: navigation, search

Contents

History

PVM was created as a message passing system that enables a network of heterogenous (in the first versions only supported Unix) computers to be used as a single large distributed memory parallel virtual machine.

The first version PVM 1.0 was created by Vaidy Sunderam and Al Geist in 1989 at the Oak Ridge National Laboratory. It was developed for internal use only and hence not released to the public.

PVM was completely rewritten in 1991 at the University of Tennessee in Knoxville and released as PVM 2.0. A cleaner specification and improvements in robustness and portability were the achievements of the new version.

For the third major release PVM 3.0 a complete redesign was considered necessary in order to obtain better scalability and portability. Beyond the implementation for Unix systems, PVM has now been ported to additional platforms such as Windows and Linux. Version 3 was released in March 1993. As of today the newest version is 3.4.5, which was published in September 2004.

Design

PVM provides a framework (libraries, runtime-environment and helper programs) for the creation of PVM-programs in C, C++ and Fortran. In addition there are projects for Perl, Python and Java (jPVM). PVM enables a collection of heterogenous computer systems to be viewed as one single parallel virtual machine, independent from the underlying architectures. This implies, that PVM and not the programmer cares about the specific platform features such as memory (shared or local-memory), network (Ethernet, FDDI, etc.) or architecture (vector computer, graphics engine or simple home PCs). PVM originally was designed for distributed parallel systems, although SMP support was introduced later on (with shared memory and threads on a few architectures), It provides easy to use functions for process-to-process and group communication. The first contains send (non-blocking by default) and receive (blocking or non-blocking), the latter provides barriers, broadcasting, scattering, gathering and reduction. A major feature of PVM is dynamic process creation. The number of processes can either be determined before program execution (with the PVM-console) or dynamically, spawning new tasks during runtime with the pvm_spawn() function.

Virtual machine

The main issue of PVM is to pretend that all machines create one virtual machine although having different architectures, memory usage and networks. This is realized by PVM daemons (pvmd). On each machine one pvmd must be started. Pvmds are responsible not only for communication. They are the central instance of task-management and communication for all PVM tasks on a machine. All communication between tasks on several machines is done via pvmds and not between the tasks directly. The first pvmd started in a network becomes the master pvmd. The master pvmd is responsible for starting the slave pvmds on the other machines in the net. Starting the master and slave pvmds is normally done by the PVM console, a bash-like command-line interpreter to manage the PVM environment. The PVM console enables users to add new hosts (with new slave pvmds) or to start (spawn) PVM programs. Instead of the usage of the PVM console, library functions are available for these tasks.

Independent of the way in which slave-pvmds are started, the master-pvmd connects to the new host either by rsh, ssh or rexec and starts the new pvmd on this machine. While a PVM program is running there are two sorts of communication: task-to-pvmd and pvmd-to-pvmd communication (no direct task-to-task). task-to-pvmd communication is done by the TCP-protocol and pvmd-to-pvmd communication by a proprietary protocol based on UDP.

Spawning tasks

The instance of parallel execution is a task. Tasks can be compared to processes. The machine a task is started on, is determined either by PVM or by the programmer. Tasks are internally realized as processes. With the TPVM-add-on PVM tasks can either be realized by processes or threads depending on the message passing functions used. To spawn a thread, a call to pvm_spawn() must be performed. The function takes parameters specifying the PVM-program file of the new task, the machine where to start the task and the number of tasks to spawn.

spawn_example.c:
        #include <pvm3.h>
        ...
        /* In this example the root process creates two children. *
         * After spawing the children do something in parallel    */
        int main() {
                int nchildren;
                int tids[2]; /* Holds task-IDs of the two tasks to be spawned */

                /* This section if the current task has no parent-task. *
                 * In this example this applies only to the root task   */
                if (pvm_parent() == PvmNoParent) {
                        nchildren = 2;
                        /* Spawn nchildren instances of "spawn_example" on a target machine determined by PVM. */
                        /* The task-ids (tids) of the new tasks are stored in tids[].                         */
                        pvm_spawn("spawn_example", NULL, PvmTaskDefault, 0, nchildren, tids);
                }
                /* This section is executed by the spawned children only */
                else {
                    /* Do something in parallel */
                }
                /* Tell PVM runtime that we are leaving it now (mandatory for root-task).  */
                /* If not specified root would not wait until children have finished work. */
                pvm_exit();
                return 0;
        }
   

Remember that the program must be compiled as the executable "spawn_example". Additionally this file must be placed in a directory defined by the environment variable PVM_PATH. A reference to the program as "./spawn_example" is not possible. As you can conclude from the above example, tasks form a hierarchic tree. Each task spawned by another task is a child of the latter. The task initially created by PVM when the program was started is the root of this hierarchy. This applies only in case the program has been executed directly (e.g. calling "./spawn_example" from the shell). If the task was called by the PVM console program (executed with the command "pvm") the process initially created will be a child of the PVM console, hence this example will not work.

Communication: Send and Receive messages

The concept of sending and receiving data to/from other tasks is rather easy:

  • Sending data is done in three steps:
  1. initialize sending
  2. pack data
  3. send packed data
  • Receiving data is done in two steps:
  1. initialize receiving
  2. unpack data

Data of several datatypes can be packed into one package send in a single pass. For each supported datatype there is an extra pack-function (pvm_pkint(), pvm_pk_double()). There even is a function pvm_packf() which packs different data in printf() fashion (pvm_packf("%+ %d %lf", PvmDataDefault, intVal, dblVal) inits sending and packs one int and one double value). Besides the datatype, the number of items of the specified type and a stride (a stride of one means no stride) must be given. On receiving the opposite functions can be used to unpack data (pvm_upkint(), etc.). The send and receive functions do not have to be used symmetrically (pvm_pkint() -> pvm_upkint()) they just have to be compatible.

Each message sent or received has a tag supplied as an extra parameter. This tag is used to distinguish and filter messages. A receiver only considers messages sent by the given task and having the right tag. Setting the values of one of those parameters to -1 ignores either the tag or the sender information of the message received.

For the standard case, data is sent using the XDR data exchange format to guarantee architecture independent datatypes (PvmDataDefault as parameter for pvm_initsend. See example). If all participating machines are of the same architecture the sending of data can be sped up using no XDR-encoding (PvmDataRaw). On machines using PVM's shared memory support (e.g. IBM AIX4/5) it is possible to send a pointer of the memory location of the data and the data's size only (PvmInPlace).

send_example.c :
        #include <pvm3.h>
        ....
        int main(int *argc, char *argv[]) {
                int tag = 99; /* tag used to distinct messages */
                ....
                if (pvm_parent() == PvmNoParent) {
                       int nprocs = 2;
                       int tids[1];
                       int int_array[] = { 1, 2, 3 };
                       double double_val = 1.5;
                       ....
                       nprocs = pvm_spawn("send_example", NULL, PvmTaskDefault, 0, nprocs-1, tids)+1;
                       printf("Tasks: %d\n", nprocs);
                       /* step 1: init sending using XDR-encoding */
                       pvm_initsend(PvmDataDefault);
                       /* step 2: pack data */
                       /*  - pack two int-Values with a stride of two (int_array[0] and int_array[0+2]) */ 
                       pvm_pkint(int_array, 2, 2);
                       /*  - pack one double-Value with a stride of one (no stride) */
                       pvm_pkdouble(&double_val, 1, 1);
                       /* step 3: send data to task with task-id tids[0] (first task spawned) and tag */
                       pvm_send(tids[0], tag);
                }
                else {
                        int int_first, int_last;
                        double double_val = 1.5;
                        ....
                        /* step 1: init receiving */
                        pvm_recv(pvm_parent(), tag);
                        /* step 2: unpack data */
                        /*  - unpack one int-Value with no stride (int_first = int_array[0] = 1) */
                        pvm_upkint(&int_first, 1, 1);
                        /*  - unpack one int-Value with no stride (int_last = int_array[2] = 3) */
                        pvm_upkint(&int_last, 1, 1);
                        /*  - unpack one double-Value with no stride */
                        pvm_upkdouble(&double_val, 1, 1);
                        printf("Received: %d %d %lf\n", int_first, int_last, double_val);
                }
                pvm_exit();
                return 0;
        }

As seen above no buffer of appropriate size for packaging needs to be allocated by the programmer. This is all done by PVM.

Sending in PVM is always buffered and non-blocking (pvm_send). The standard in receiving data is buffered and blocking receiving (pvm_recv). A non-blocking alternative exists (pvm_nrecv).

In contrast to MPI "user defined data types" are not supported by PVM. Alternatively this can be achieved by packing the members of the user datatype seperately or by packing the whole userstruct using pvm_pkbyte() with sizeof(userstruct) elements. The latter method should be used only in case the architectures of the participating machines are the same.

The next diagram shows the dependency of package sizes send with pvm_send and communication overhead. We tested the following situation: One task packs packSize of int-Values of the whole matrix with pvm_pack() and sends it to another task. The other task receives data and sends it immediately back to the sender. He will not wait until all fragments have been received. We measured the time between the sending and receiving of the whole matrix. All times have been measured on a 4 CPU, 1000 MHz architecture with 16 GB RAM. We do not know the underlying network type although the results are extremely dependant on this information. On slow networks execution times certainly will be worse than on very fast ones.

Packspeed.png

Sending big packages or too little packages is not efficient. Big packages consume a lot of main memory (around 2x matrix size because of a single large extra buffer). If there is not enough free memory the system will use virtual memory and slow down. Additionally the receiving task is busy with receiving data and can not do any calculation in parallel. With smaller sizes the task receives in the background and is only blocked for a shorter time when calling pvm_recv() because in the best case data has already been received by the pvmd so it can return immediately. If you send too small packages this causes a lot of overhead due to the high amount of TCP- and PVM-defined UDP headers. In addition it allocates memory for very small buffers over and over again. Buffers might not be freed immediately after usage so a big amount of memory is used. We observed that on our testing machines very big and a large bunch of very little packages caused memory allocation errors. In contrast to the times given in the chart above the time needed just for sending was about the half of the receiving time although pvm_send() is non-blocking. The time needed for allocation of matrix memory and creation of random int-values for all matrix-elements with rand() followed by a multiplication and division only took a fraction of the time needed for sending (6000x5000: 1.26s, 3000x3000: 0.38s, 1000x1000: 0.043s).

Communication: Group Communication

In addition to process-to-process (task-to-task) communication, PVM supports communication within a group of specified tasks.

To make group communication work a so called "PVM dynamic group server" (pvmgs) must be started in addition to the pvm-daemon. This daemon handles all calls to group communication functions and the mapping of tasks to groups. The server is started automatically by pvmd on the first call to a group communication function. Note that the executable pvmgs must be in one of the paths given in the environment variable PVM_PATH.

A group must be specified explicitly. There is no standard group as in other message passing systems (like the MPI_COMM_WORLD group in MPI). In order to use group communication tasks have to be assigned to one or more groups. This is done by one or more calls to pvm_joingroup("group_name") by the task. Now the task belongs to the group with name "group_name" (multiple memberships possible). Tasks belonging to a group are numbered beginning with group-id 0. Be aware that PVM has no local groups. When defining a group tasks of every other PVM program running in the PVM environment can join the group, even if this behaviour is not desired.

The name of the group then has to be passed to the specific group-communication function. The following group-communication functions are available:

  • pvm_barrier
  • pvm_bcast (mcast)
  • pvm_reduce
  • pvm_gather
  • pvm_scatter

Usage of these functions is not as clean as in MPI:

On one hand pvm_barrier(), pvm_gather(), pvm_scatter() and pvm_reduce() have to be called by all processes of the participating group, each defining the root task via an extra parameter. On the other hand pvm_bcast() only the (sending) root process calls this function and the receiving tasks fetch the broadcasted data via the task-to-task pvm_initrecv()/pvm_unpack()-functions. In addition a call to pvm_bcast has to be preceded by task-to-task manner calls to pvm_initsend() and multiple pvm_pack operations.

group_example.c:
        #include <pvm3.h>

        #define NPROCS 3

        int main(int argc, char* argv[]) {
                int tids[NPROCS-1];                /* task-IDs of spawned tasks */
                int gid;                           /* group id */
                int tag = 99;                      /* tag for messages */
                char* group = "barrierGroup";      /* name of group for all tasks */
                int work[] = { 1, 3, 4, 7, 2, 3 }; /* array of numbers to sum */
                int part_work[2];                  /* partial array of numbers to sum by each task */
                int result;                        /* local sum of each task */

                gid = pvm_joingroup(group);
  
                if(gid == 0) {
                        if(pvm_spawn("group_example", NULL, PvmTaskDefault, NULL, NPROCS-1, tids)+1 != NPROCS) {
                                fprintf(stderr, "Error: Could not spawn processes!");
                                exit(1);
                        }

                        /* wait until spawned tasks are ready */
                        pvm_barrier(group, NPROCS);

                        /* scatter (send) work into parts of 2 items for each task, task is root (gid=0) */
                        pvm_scatter(part_work, work, 2, PVM_INT, tag, group, 0);
                        /* calc local sum */
                        result = part_work[0] + part_work[1];
                        /* reduce local sums to one result (1) using PvmSum-operation, task is root (gid=0) */
                        pvm_reduce(PvmSum, &result, 1, PVM_INT, tag, group, 0);

                        printf("Sum: %d\n", result);
                }
                else {
                        /* tell other tasks that we have joined the group */
                        pvm_barrier(group, NPROCS);

                        /* scatter (recv) work into parts of 2 items for each task, task with gid=0 is root */
                        pvm_scatter(part_work, NULL, 2, PVM_INT, tag, group, 0);
                        /* calc local sum */
                        result = part_work[0] + part_work[1];
                        /* reduce local sums to one result (1) using PvmSum-operation, task with gid=0 is root */
                        pvm_reduce(PvmSum, &result, 1, PVM_INT, tag, group, 0);
                }

                /* leaving group to early might have bad consequences, so wait until all tasks are done */
                pvm_barrier(group, NPROCS);
                pvm_lvgroup(group);
                pvm_exit();
                return 0;
        }

The above example needs quite a lot of barriers. If the barrier in front of the scatter command would be ommited, the program might not work. The root-task of the scatter-function sends part_work to those tasks only that have joined the group already.

Do not use the above example as is. See the "experience"-section for possible failures when using this code and how to resolve them.

Experiences

Installation

During installation of PVM we were confronted with several difficulties. In order to (compile and) install and configure PVM you need to set several environment variables to tell PVM where its runtime binaries are located or where PVM user programs are located. Furthermore you need to set the location of the group server, if you want to use group-communication.

On all systems it took us a long time to add new hosts to our virtual machine using the PVM console program from the master-pvmd machine. Various errors concerning the environment variables emerged. Especially the rsh/ssh connection to start the slave pvmds did not work. PVM pretended that PVM_ROOT was not set correctly at the slave-machine, but is was.

We installed PVM on PowerPC using AIX5, i386 and AMD Opteron using Linux. Although there is an explicit shared memory mode and a mode in which PVM uses MPI (MPP-mode) for underlying communication on AIX4/5 architectures, we were not able to make it work not even with massive effort. After compilation of PVM with one of this options enabled PVM always stuck within program execution.

Although mentioned in the official user guide it was not possible to enable thread support. In none of the PVM source code file appears a line like pthread_spawn or similar which might create a new thread. Perhaps the user guide references to the TPVM add-on which is not supported any more. The documentation of TPVM sounds promising but the latest release was an alpha version in 1994. It could not even be compiled on a Linux operating system.

Support

Today, there are not many web sites left containing valuable information. The sites given in the link-section of this web-site were the only ones found on the internet that could help us to solve our problems. The sites dedicated to PVM we found seemed to be neglected by their owners. Most articles are older than five years (last updates mostly were done in 1995). In return the two official sites contain some very good reference guides and a FAQ concerning the most important issues, but it was very hard to find any programming examples written for PVM.

New versions of PVM only contain a few bug fixes or support for new architectures (for example the last two releases 3.4.4 and 3.4.5 in 2001 and 2004). The latest programming features (message boxes) were introduced in the year 1997.

The ORNL PVM website presents many add-ons for PVM such as threading support or distributed file system I/O access (PIOUS). Development of most of them has been quit in the mid of the 1990s. Due to the missing support for and adaption to modern architectures some of them can not even be installed.

One import add-on is XPVM. XPVM is the graphical counterpart to the PVM-console and assists in host management, spawning of tasks and graphical debugging.

Usability

Programming in PVM is not very different to programming in MPI. The number of functions available in PVM is far less than in MPI-I, although PVM has a much longer history than MPI. Usage of the few communication functions available is very easy, especially if you are familiar to MPI. On the one hand automatic creation and management of send and receive buffers is a great feature and makes non-blocking sending and receiving easier than it can be done in MPI. It is an additional feature that receiving is always non-blocking. Problems due to usage of buffers that are currently used by another non-blocking operation can not occur in PVM. On the other hand the lack of control in regard to buffer usage can cause extensive waste of memory. There are no applicable functions for buffer control to prevent this.

The man pages available for every PVM function were very helpful. Error messages thrown at execution time are not always rendering the correct cause of the problems, as a result debugging is quite difficult.

Important features such as user defined datatypes (elements of data-structs must be sent separately) are lacking. Due to the init-pack-send principle the lack of user defined datatypes can be compensated. It is a different situation if you want to use group-communication. Without user defined datatypes the scatter and gather functions are not applicable in many cases.

PVM user programs must be located in a path defined by the PVM_PATH environment variable. If you want to place a copy of an old version of one of your programs, you are not able to execute it without adding the new path to PVM_PATH. If you do this PVM might spawn children with the wrong version of the executable from the other path. It might take a lot of time to find the reason for a program with a wrong behaviour if the failure can not be found in the source codes. Some machines ignore the PVM_PATH variable, so all executables have to be located in PVM's main binary dir. This is not a very good solution when you have many different executables.

Group Communication

Group communication was added in the early public releases of PVM between 1993 and 1995. Most of the functions are not applicable in practice. The steps to take to define a new group are too complex considering the little ease group communication brings. It seems that group communication has never been brought into a usable state. Confirmation to this point of view gives a quotation found in the man-page for pvm_scatter()

 "The current algorithm is very simple and robust. A future implementation may make more efficient use of the architecture to allow greater parallelism."

Note the "future implementation" part. The man page was written in April 1994.

Better avoid the usage of PVM's group communication functions. They only produce overhead without making any use of the network's topology. pvm_scatter() for example is implemented mainly as a for-loop with group-size iterations. Each loop uses the pvm_send() primitive to send a part of the buffer to a group member without taking net topology into account. pvm_bcast is a simple wrapper for pvm_mcast, a non group-communication function using an array of tids instead of a group. So use pvm_mcast instead of pvm_bcast if possible.

If you have to use group communication, consider that group names are used globally. Usage of group names like "group" in several independent programs might lead to unforeseen but inevitable interferences if those programs are executed at the same time. So always use group names with a prefix unique to your application. This is not sufficient if you want to run several (not spawned) instances of your program or if the execution of the program failed and the processes have not been killed yet. This may result in group numbers not starting with 0 for the root or interference of pvm_scatter() or other calls. To avoid this always append the tid of the root task to the group name and check (although it should not be needed) if the root-task has gotten gid 0. Otherwise something went horribly wrong. Now you can use local groups in the fashion of MPI's MPI_COMM_WORLD.

Another problem is the inability of pvm_scatter to scatter arrays with size != n*#groupmembers (n is the partial size of the scattered array for each group member). MPI contains a function called MPI_Scatterv(). Another problem with pvm_scatter() is that the root-task always has to receive its part of the whole matrix into a new partial buffer although it already has the data. In MPI this problem can be solved in setting the result buffer to MPI_IN_PLACE. The root process then makes the send-buffer the receive-buffer. In PVM these problems can be solved using the following little trick:

  1. Calculate the rest: rest = array_length % ntasks
  2. Calculate the new array offset: offset = array + rest
  3. Let root scatter the following: pvm_scatter(offset, offset, array_length/ntasks, ...)

This is possible because PVM does not require different pointers to a send and receive buffer. Now root gets the first n+rest items of "array", each other task gets n items (n = array_length/ntasks).

Efficiency

We were not able to measure big speedups neither with a single machine nor with a cluster of identical machines. According to PVM's handbook SSH or RSH are only used to start the slave pvmds and not for the UDP pvmd-to-pvmd communication. Configuring PVM to use RSH instead of SSH brought no noticeable speedups, so encryption should not be the problem.

Speed-ups on a single machines might have been better if shared memory could have been enabled but it was not possible to do this.

Cowichan Problems

The abilities of PVM were tested with a set of parallel programs called the Cowichan-Problems. The experiences we made are discussed in the experience-section. This section considers issues concerning speed-ups obtained with PVM.

If you have not the slightest clue what the Cowichan-Problems are take a look at the following page of the Research Group Programming Languages / Methodologies.

Speed-ups

Speed-ups have been measured on two systems:

  • Cluster consisting of 3 quad-processor machines (IBM RS6000 pSeries 630). Specification of each machine:
    • CPUs: 4 x IBM PowerPC POWER4, 1000 MHz
    • Operating system: AIX 5.2.0
    • Main memory: 16 GB (2 machines), 2 GB (1 machine)
  • Cluster of 38 dual-processor machines. Specification of each machine:
    • CPUs: AMD Opteron 248, 2.2 Ghz
    • Operating system: Redhat Enterprise Linux WS 64 Bit
    • Main memory: 4 GB

All programs have been compiled under the same conditions (with gcc without optimzation-flags -On). On both systems PVM 3.4.4 was used for compilation and execution.

Although CPU-usage of other users was less than 5% there _were_ other processes which could have influenced our results. The programs were executed at several days for several times to calculate average speed-ups that are not influenced by other users. The single results did not differ very much from the average results, so the speed-ups should be representative.

The sizes of input data were chosen for target execution times of about 30s where applicable. In most cases this was not applicable because either not enough main memory or disk-space (an input file of 10.000x10.000 double-elements needs 2,3 GB) for bigger inputs was available. So some of the execution times are only up to 1s. That is not enough to be representative. The main algorithms of these programs could have been executed in a loop to achieve higher execution times. This has not been done because PVM needs large inputs to compensate communication times with a large amount of unsynchronized calculation time. In most of the programs larger inputs invoke more communication and synchronisation hence both effects are nearly compensating each other. So even if we had enough memory it is not likely that there will be enormous speed-ups.

We could have achieved better speed-ups if we could increase the number of iterations in each of the Cowichan-Problems but little of them are designed to do this. Those that are already have good speed-ups. We adjusted the number of iterations to an amount that gives a good compensation of synchronisation time.

Spu131-cw.png

Spu131-ms.png

Several other constellations have been tested (e.g. PVM with 3 machines, usage of MPI instead of PVM). Speed-ups obtained in these tests are available here:

Interpretation

Although the given speed-ups are reproducible some of the execution times achieved with the maximum input-data sizes have been to low to be representative, although the results are plausible.

In contrast to the serial implementation only with mandel, elastic and thresh-nohist real speed-ups could be obtained. In all other problems the parallel implementation was slower even if there is a speedup compared to the execution times of a smaller number of CPUs. We presume that the reason for this problem is the large amount of communication in contrast to the time needed for the actual calculation. In addition some of the execution times are rather low (less than a few seconds) which is due to the small amount of available memory. So we could not test most of the programs with matrices that need bigger execution times.

Some of the programs use static partitioning for work scheduling. On one hand this causes less network traffic than sending little partitions. On the other hand the receiver of a big data-package is blocked until he received all data. When using smaller packages a receiver is able to work on a packages while his local pvmd receives the next package in the background. See the "Communication: Send/Receive" section for more information.

Probably a wrong package size is the problem with mandel. We expected mandel to be the program with the best speed-ups but it is not (although it has real speed-up). The only synchronization done in mandel is performed at the beginning when a few bytes of initialization-data are broadcast to the coworkers and at the end when the partial matrices calculated by the coworkers are gathered by the root-process. All calculation is done without any need of synchronization and should be massively parallel. In the worker-coworker static work scheduling implementation speed-ups might be destroyed because of the sending time needed for big packages and a bad load balancing. Another implementation of mandel is using the master-slave paradigm and sends only small slices of data with dynamic work scheduling. It performes much better than the other version of mandel.

Load balancing in general might be another problem with static work scheduling but when looking at mandel this could not be observed without any extra tools (all tasks had a CPU-usage of about 25% which corresponds to 100% of one of the four CPUs). We experimented with the sizes of data-packages used in life. We obtained better execution times with packages of 10 data-rows than with 500 rows.

We even tried to find the problem with the graphical XPVM interface which shows the amount of time for each PVM-primitive. The results have been very peculiar and will not be evaluated on this page. In addition we ported some of the programs to MPI. Altough MPI is faster in general it shows almost the same speed-up behaviour.

Most likely the reason for the bad speed-ups is that message passing systems should only be used for tasks that have a large amount of calculation to be done unsynchronized and only little communication (large data blocks are only kept locally, small results are passed to other tasks). See the times needed for creating and filling a matrix with a little bit of additional calculation and those for sending and receiving them. Most of the Cowichan-Problems consist only of one loop in which trivial calculation is done with each element. The minimal time for sending is much higher than calculation (if you do not have super-high speed networks and main memories). This demonstrates randmat. It is almost the same algorithm than mandel. The difference is the amount of calculation. Calculation in mandel is much more complex than in randmat and needs four times longer to execute as a sequential program than a sequential randmat. On that account mandel has (real) speed-ups and randmat only slows down.

There are no advantages in using the 3 machines with each having 4 CPUs compared to a single 4 CPU machine. The speed-ups achieved when using a cluster can even be be seen when using only one machine with the same amount of tasks. We measured speed-ups of those problems that apparently take advantage of the 12 CPUs with a single machine with up to 12 tasks. The speed-ups for 12 tasks on the single 4 CPU machine were almost the same. So the reason of the speed-ups we gained with 12 CPUs is not that we used 12 CPUs. A reason might be a better usage of the CPUs another that we get more time slices from the scheduler when we run more processes.

We found two sites (which are as well almost the only ones we found with PVM speed-ups) whose authors faced the same problems we did. They only got slow-downs while calculating small programs using PVM. This was also the reason for Ferrari and Sunderam to develop a threaded library for PVM. Their motivation for this is described in the TPVM handbook:

Conclusion

Advantages

  • Lightweight framework with "easy" to use functionality
  • User defined data types are not needed in most cases when using the easy to use pack-functions for different data types
  • Transparent send/recv buffer control
  • All of the Cowichan-Problems could be implemented for parallel execution without bigger problems
  • The size of transmitted data need not to be known before receiving the data. The package size can be transmitted by packing an additional int value in a.

Disadvantages

  • Installation
  • User-programs must be located in directories defined by PVM_PATH
  • No remarkable speed-ups or speed-ups < 1 (even with test-programs delievered with PVM)
  • The amount of unsynchronized calculation has to be very high in comparison with communcation
  • Group communication:
    • Stuck in alpha state, just a wrapper for non-group PVM-functions (no optimizations considering network topology)
    • Not applicable in most cases due to lacking features
    • Difficult creation of local groups
  • Very little support
  • No further development (new features, improvements) since 1997
  • Not suitable for a bunch of Cowichan-Problems. PVM needs problems with more calculation/iterations and less communication

Future

Even if it might hurt hardcore PVM programmers: it seems that usage of PVM has decreased rapidly in the late 1990s in favor of MPI. New versions of PVM can be considered as maintenance builds with a few bug-fixes but without new features. Conferences about the future of PVM are held together with those of MPI. Although these conferences take place even nowadays, PVM does not seem to profit from them in contrast to MPI. The official PVM homepage is missing any announcements of new features and seems to be dead or at least orphaned. It can be concluded that MPI is the better choice for new projects using message passing. In the early 1990s PVM was state-of-the-art in message-passing programing, hence there might be still quite a lot of important programs running with PVM. Though the importance of PVM nowadays should not be underestimated.

Related Links

Personal tools