<div dir="ltr">Yeah, re-init or only finalize atexit(), like we did before, meaning shmem_finalize is really just shmem_tool_sync :-)<div><br></div><div>Jeff<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 22, 2017 at 6:43 AM, Dinan, James <span dir="ltr"><<a href="mailto:james.dinan@intel.com" target="_blank">james.dinan@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Also, if you use reference counting, you also need re-initialization, because:</div>
<div><br>
</div>
<div>FOO_init()</div>
<div>FOO_finalize()</div>
<div>BAR_init()</div>
<div>BAR_finalize()</div>
<div><br>
</div>
<div>This is likely, if FOO is an I/O library and BAR is some compute library.</div>
<div><br>
</div>
<div> ~Jim.</div>
<div><br>
</div>
<span id="m_4398497004614520665OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Openshmem-list <<a href="mailto:openshmem-list-bounces@openshmem.org" target="_blank">openshmem-list-bounces@<wbr>openshmem.org</a>> on behalf of Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, March 21, 2017 at 7:25 PM<br>
<span style="font-weight:bold">To: </span>Bob Cernohous <<a href="mailto:bcernohous@cray.com" target="_blank">bcernohous@cray.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openshmem-list@openshmem.org" target="_blank">openshmem-list@openshmem.org</a>" <<a href="mailto:openshmem-list@openshmem.org" target="_blank">openshmem-list@openshmem.org</a>><div><div class="h5"><br>
<span style="font-weight:bold">Subject: </span>Re: [Openshmem-list] Effects of removing implicit finalize from the specification<br>
</div></div></div><div><div class="h5">
<div><br>
</div>
<blockquote id="m_4398497004614520665MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div>
<div>
<div dir="ltr">I agree wholeheartedly with Nick that first-call is wrong for the reason he gives. Any we expect one library to be able to init-final, then any number need to be allowed to do this, and this means we need last-call finalization.
<div><br>
</div>
<div>The other option is, of course, to copy MPI and force every library to check the state and potentially init and thus be on the hook to finalize as well, but this means that libraries must be popped in the order they are pushed, or else this happens:</div>
<div><br>
</div>
<div><font color="#000000"> FOO_init() --> includes call to shmem_init() --> actually initializes, knows it<br>
BAR_init() --> includes call to shmem_init() --> no-op, knows it did not initialize<br>
FOO_finalize() --> includes call to shmem_finalize() --> knows it initialized, so might assume it should finalize</font></div>
<div><span style="color:rgb(0,0,0)"> BAR_finalize() --> includes call to shmem_finalize() --> if FOO finalized, cannot call SHMEM; if FOO didn't finalize, should BAR finalize?</span><font color="#000000"><br>
</font>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Obviously, we can generalize...</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div><font color="#000000"> FOO_init() --> includes call to shmem_init() --> actually initializes, knows it<br>
BAR_init() --> includes call to shmem_init() --> no-op, knows it did not initialize</font></div>
<div><span style="color:rgb(0,0,0)"> GOO_init() --> includes call to shmem_init() --> </span><span style="color:rgb(0,0,0)">no-op, knows it did not initialize</span><br style="color:rgb(0,0,0)">
<span style="color:rgb(0,0,0)"> BAZ_init() --> includes call to shmem_init() --> no-op, knows it did not initialize</span></div>
<div><span style="color:rgb(0,0,0)"> BAR_finalize() --> includes call to shmem_finalize() --> should I finalize?</span><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,0)"> I did not initialize...</span><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><span style="color:rgb(0,0,0)"> BAZ_finalize() --> includes call to shmem_finalize() --> </span><span style="color:rgb(0,0,0)">should I finalize? I did not initialize...</span><font color="#000000"><br>
FOO_finalize() --> includes call to shmem_finalize() --> </font><span style="color:rgb(0,0,0)">should I finalize? I initialized...</span></div>
</div>
<div class="gmail_extra">
<div class="gmail_extra">
<div><span style="color:rgb(0,0,0)"> GOO_finalize() --> includes call to shmem_finalize() --> </span><span style="color:rgb(0,0,0)">should I finalize?</span><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,0)"> I did not initialize...</span><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><font color="#000000"> // uh oh, nobody finalized</font></div>
<div><font color="#000000"><br>
</font></div>
<div><font color="#000000">The problem situation already occurs with Elemental (<a href="http://libelemental.org/" target="_blank">http://libelemental.org/</a>). If Elemental is called before MPI is initialized, Elemental takes ownership of MPI, which means that it initializes
MPI and sets a flag that it should also finalize it. However, that means that you have ugly termination because MPI_Finalize is called inside of a C++ dtor when the program exits, and this does not always go well (perhaps because Elemental needs to do more
refcounting than it does already...).</font></div>
<div><font color="#000000"><br>
</font></div>
<div><font color="#000000">Jeff</font></div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Mar 21, 2017 at 2:41 PM, Bob Cernohous <span dir="ltr">
<<a href="mailto:bcernohous@cray.com" target="_blank">bcernohous@cray.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="m_4398497004614520665gmail-m_5097612450858162674WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">In general, I think libraries with side effects like initializing/finalizing SHMEM are problematic. [It’s especially bad for something like dmapp_init which takes config parameter
so neither FOO nor BAR can count on controlling the initialized configuration. Same thing, I guess, if FOO or BAR were trying to set env’s internally before init.]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">But everything you say is true and if those are all the goals then reference counting is fine.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">But there was, I believe, a customer request to support exit w/finalize. Ie. if a process took an early exit, not a shmem_global_exit, wait for other PEs to finalize/exit normally.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">Or, imagine FOO and BAR didn’t explicitly have finalize functions but relied on atexit to finalize.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">If you want atexit(shmem_finalize) (or the current implicit finalize) to wait and truly finalize, then it can’t just be decrementing a reference count and executing no-op.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Nicholas Park [mailto:<a href="mailto:nspark@computer.org" target="_blank">nspark@computer.org</a>]
<br>
<b>Sent:</b> Tuesday, March 21, 2017 4:20 PM<br>
<b>To:</b> Naveen Ravichandrasekaran <<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>><br>
<b>Cc:</b> Bob Cernohous <<a href="mailto:bcernohous@cray.com" target="_blank">bcernohous@cray.com</a>>; Kuehn, Jeff <<a href="mailto:jakuehn@lanl.gov" target="_blank">jakuehn@lanl.gov</a>>; Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>>;
<a href="mailto:openshmem-list@openshmem.org" target="_blank">openshmem-list@openshmem.org</a></span></p>
<div>
<div class="m_4398497004614520665gmail-h5"><br>
<b>Subject:</b> Re: [Openshmem-list] Effects of removing implicit finalize from the specification<u></u><u></u></div>
</div>
<p></p>
</div>
</div>
<div>
<div class="m_4398497004614520665gmail-h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">I'm very concerned about first-call semantics on shmem_finalize(). I think this effectively breaks general-purpose library support for OpenSHMEM.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">Consider two libraries FOO and BAR that both build on OpenSHMEM.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Claim: Library interoperability has two goals:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 1) Libraries FOO and BAR can both initialize and finalize the OpenSHMEM library<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 2) Users of FOO need not make OpenSHMEM calls can can interact solely with FOO<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Under the proposed first-call semantic, Goal #1 is violated. A call sequence of:<u></u><u></u></p>
</div>
<p class="MsoNormal"> FOO_init() --> includes call to shmem_init() --> actually initializes<br>
BAR_init() --> includes call to shmem_init() --> no-op<br>
BAR_finalize() --> includes call to shmem_finalize() --> actually finalizes<br>
FOO_finalize() --> includes call to shmem_finalize() --> no-op<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">means that FOO_finalize() will fail if it calls
<i>any</i> OpenSHMEM function other than shmem_finalize(). It could non-trivially require calls to shmem_free(), shmem_my_pe(), shmem_barrier_all().<u></u><u></u></p>
</div>
<p class="MsoNormal">In contrast, the reference-counted semantic leads to valid behavior:<br>
FOO_init() --> includes call to shmem_init() --> actually initializes<br>
BAR_init() --> includes call to shmem_init() --> no-op<br>
BAR_finalize() --> includes call to shmem_finalize() --> no-op<br>
FOO_finalize() --> includes call to shmem_finalize() --> actually finalizes<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">Currently, Goal #2 is violated by the fact that calling shmem_init() more than once is undefined. Each library must make the assumption that OpenSHMEM is either initialized/finalized solely by them or external
to them. Only the latter is truly sane, but then means that a user interacting solely with FOO (without loss of generality) has to make OpenSHMEM calls (and may not even be aware that FOO relies on OpenSHMEM). I recognize we're trying to solve that in this
discussion, but I wanted to make this goal explicit for library support.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Nick<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Mar 21, 2017 at 12:54 PM, Naveen Ravichandrasekaran <<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">+1 for the first-call semantics – init/finalize on the first-call and then no-op after that.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">-Naveen N Ravi.</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid windowtext;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid windowtext;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Bob Cernohous
<br>
<b>Sent:</b> Tuesday, March 21, 2017 11:09 AM<br>
<b>To:</b> Kuehn, Jeff <</span><a href="mailto:jakuehn@lanl.gov" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">jakuehn@lanl.gov</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">>; Naveen Ravichandrasekaran
<</span><a href="mailto:nravi@cray.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">nravi@cray.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">>; Jeff Hammond <</span><a href="mailto:jeff.science@gmail.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">jeff.science@gmail.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">><br>
<b>Cc:</b> </span><a href="mailto:openshmem-list@openshmem.org" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">openshmem-list@openshmem.org</span></a><span style="font-size:11pt;font-family:calibri,sans-serif"><br>
<b>Subject:</b> RE: [Openshmem-list] Effects of removing implicit finalize from the specification</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">I would vote for first-call semantics. The email example wasn’t particularly interesting for atexit. The only reason to want atexit(shmem_finalize) is that there might be a code
path that doesn’t explicitly shmem_finalize. I think you want atexit to finalize, not decrement a reference count. So init/finalize on first-call and no-op after that?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid windowtext;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid windowtext;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Openshmem-list [</span><a href="mailto:openshmem-list-bounces@openshmem.org" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">mailto:openshmem-list-bounces<wbr>@openshmem.org</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">]
<b>On Behalf Of </b>Kuehn, Jeff<br>
<b>Sent:</b> Tuesday, March 21, 2017 10:48 AM<br>
<b>To:</b> Naveen Ravichandrasekaran <</span><a href="mailto:nravi@cray.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">nravi@cray.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">>; Jeff Hammond <</span><a href="mailto:jeff.science@gmail.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">jeff.science@gmail.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">><br>
<b>Cc:</b> </span><a href="mailto:openshmem-list@openshmem.org" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">openshmem-list@openshmem.org</span></a><span style="font-size:11pt;font-family:calibri,sans-serif"><br>
<b>Subject:</b> Re: [Openshmem-list] Effects of removing implicit finalize from the specification</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">I have a very small concern with nested semantics vs first call semantics in the case of init and finalize. If one considers the cases of an extra init or finalize and a missing
init or finalize, it seems that the observable side effects of those bugs, are more local to the problem code, and thus easier to debug in the case of first-call semantics, and more non-local in the case of nested semantics. It’s a small concern, but what
do the rest of you think?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">Regards,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">Jeff</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On 3/20/17, 19:51, "Openshmem-list on behalf of Naveen Ravichandrasekaran" <<a href="mailto:openshmem-list-bounces@openshmem.org" target="_blank">openshmem-list-bounces@opensh<wbr>mem.org</a> on behalf of
<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">Hi Jeff,</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">> We either need to allow multiple calls to initialize/finalize or we
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">> need to have is_initialized/is_finalized queries. MPI does the latter
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">> but since it is utterly trivial to ref count in this case, we should just
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in">> do better than MPI and do the former.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">My requirement is not to support multiple shmem_finalize calls. I was just</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">merely stating the fact that the side-effect of implicit finalize in the current
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">specification allows the usage of shmem_finalize multiple times.
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">What we really need is a semantics which successfully performs finalize or
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">do no-op when the user uses atexit(shmem_finalize) in their application.
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_init(); /* ref_count++ -> actual initialization */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_init(); /* ref_count++ -> no-op */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_finalize(); /* ref_count-- -> no-op */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_finalize(); /* ref_count-- -> actual finalization */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">FWIU, the reference counter method would be useful for handling</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">multiple shmem_init and shmem_finalize calls, but it won’t probably
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">work for handling atexit(shmem_finalize).
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">In this example, it is unclear how the reference counter works.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">int main(void) {</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_init(); /* ref_count++ -> actual initialization */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> atexit(shmem_finalize);</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_finalize(); /* ref_count-- -> actual finalization */
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> return 0; /* unclear what should happen here */</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">}</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">Irrespective of the number of times shmem_init is used, we need to
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">successfully handle atexit(shmem_finalize). So, I would prefer adding
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">new APIs in SHMEM to query is_initialized/is_finalized and let the</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">users handle atexit(shmem_finalize) scenario correctly.
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif">-Naveen N Ravi.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif"> </span><u></u><u></u></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid windowtext;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid windowtext;padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-left:0.5in"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Jeff Hammond [</span><a href="mailto:jeff.science@gmail.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">mailto:jeff.science@gmail.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif"><wbr>]
<br>
<b>Sent:</b> Monday, March 20, 2017 4:01 PM<br>
<b>To:</b> Naveen Ravichandrasekaran <</span><a href="mailto:nravi@cray.com" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">nravi@cray.com</span></a><span style="font-size:11pt;font-family:calibri,sans-serif">><br>
<b>Cc:</b> </span><a href="mailto:openshmem-list@openshmem.org" target="_blank"><span style="font-size:11pt;font-family:calibri,sans-serif">openshmem-list@openshmem.org</span></a><span style="font-size:11pt;font-family:calibri,sans-serif"><br>
<b>Subject:</b> Re: [Openshmem-list] Effects of removing implicit finalize from the specification</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in">If we permit multiple calls to finalize then we should also permit multiple calls to initialize. This makes it easy on applications that use libraries and the libraries do not know whether they should call initialize
or not.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">We either need to allow multiple calls to initialize/finalize or we need to have is_initialized/is_finalized queries. MPI does the latter but since it is utterly trivial to ref count in this case, we should just
do better than MPI and do the former.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">We could add return codes on initialize/finalize to allow users to know whether the calls were no-ops or not. This does not break any existing code because users do not have to assign return codes to variables.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Jeff<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On Mon, Mar 20, 2017 at 1:37 PM, Naveen Ravichandrasekaran <<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid windowtext;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<p class="MsoNormal" style="margin-left:0.5in">At present, the specification allows the multiple usage of shmem_finalize.<br>
This seems to be a defined behavior and multiple subsequent calls to finalize<br>
are no-ops.<br>
<br>
The following example should pass as per the current specification;<br>
<br>
Example:1<br>
int main(void) {<br>
shmem_init();<br>
shmem_finalize();<br>
shmem_finalize();<br>
return 0;<br>
}<br>
<br>
In Example:1, we have three finalize (2 explicit + 1 implicit) operations.<br>
Even any common SHMEM usage, will have two finalize operations<br>
(1 explicit + 1 implicit). Even though the specification is not clear about the<br>
above usage, this is the side-effect of having implicit finalize at exit() or at<br>
return from main.<br>
<br>
Example:2<br>
int main(void) {<br>
shmem_init();<br>
atexit(shmem_finalize);<br>
return 0;<br>
}<br>
<br>
During F2F meeting, we decided to drop implicit finalize in the specification.<br>
The argument was that the users can explicitly call "atexit" in the application<br>
to achieve the above behavior as shown in Example:2.<br>
<br>
Example:3<br>
int main(void) {<br>
shmem_init();<br>
atexit(shmem_finalize);<br>
shmem_finalize();<br>
return 0;<br>
}<br>
<br>
FWIU, if we remove implicit finalize from the specification - Example:2 is<br>
guaranteed to work but the behavior on Example:3 is undefined.<br>
<br>
To allow the explicit atexit usage, we either<br>
1. Need to specify that multiple calls to finalize are no-ops or<br>
2. We need to have a new API - "shmem_finalized()" so that the users can<br>
have a wrapper from atexit which checks shmem_finalized() before calling<br>
shmem_finalize().<br>
<br>
Please let me know, if this analysis looks valid.<br>
<br>
-Naveen N Ravi<br>
Cray Inc.<br>
<br>
______________________________<wbr>_________________<br>
Openshmem-list mailing list<br>
<a href="mailto:Openshmem-list@openshmem.org" target="_blank">Openshmem-list@openshmem.org</a><br>
<a href="http://www.openshmem.org/mailman/listinfo/openshmem-list" target="_blank">http://www.openshmem.org/mailm<wbr>an/listinfo/openshmem-list</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:0.5in">-- <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
______________________________<wbr>_________________<br>
Openshmem-list mailing list<br>
<a href="mailto:Openshmem-list@openshmem.org" target="_blank">Openshmem-list@openshmem.org</a><br>
<a href="http://www.openshmem.org/mailman/listinfo/openshmem-list" target="_blank">http://www.openshmem.org/mailm<wbr>an/listinfo/openshmem-list</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_4398497004614520665gmail_signature">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div></div></span>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div></div>