<div dir="ltr">Hi Jeff, Naveen,<div><br></div><div>Tools need some way of querying the init/fini reference counter to determine when it's actually safe to merge and dump data. is_{initia,fina}lized() seems like a nice way to expose the counter. Return values also make sense to me. Is there harm in doing both?</div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 21, 2017 at 12:15 AM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Mar 20, 2017 at 6:51 PM, Naveen Ravichandrasekaran <span dir="ltr"><<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="m_1689638789201716986gmail-m_1246775112113157736WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">Hi Jeff,</span><u></u><u></u></p><span class="m_1689638789201716986gmail-">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">> We either need to allow multiple calls to initialize/finalize or we
<u></u><u></u></p>
<p class="MsoNormal">> need to have is_initialized/is_finalized queries. MPI does the latter
<u></u><u></u></p>
<p class="MsoNormal">> but since it is utterly trivial to ref count in this case, we should just
<u></u><u></u></p>
<p class="MsoNormal">> do better than MPI and do the former.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
</span><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">My requirement is not to support multiple shmem_finalize calls. I was just<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">merely stating the fact that the side-effect of implicit finalize in the current
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">specification allows the usage of shmem_finalize multiple times.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">What we really need is a semantics which successfully performs finalize or
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">do no-op when the user uses atexit(shmem_finalize) in their application.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> </span></p></div></div></blockquote><div><br></div></span><div>Yes, and I think ref_count is the easiest way to achieve that.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="m_1689638789201716986gmail-m_1246775112113157736WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_init(); /* ref_count++ -> actual initialization */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_init(); /* ref_count++ -> no-op */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_finalize(); /* ref_count-- -> no-op */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">shmem_finalize(); /* ref_count-- -> actual finalization */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">FWIU, the reference counter method would be useful for handling<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">multiple shmem_init and shmem_finalize calls, but it won’t probably
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">work for handling atexit(shmem_finalize).
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">In this example, it is unclear how the reference counter works.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">int main(void) {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_init(); /* ref_count++ -> actual initialization */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> atexit(shmem_finalize);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_finalize(); /* ref_count-- -> actual finalization */
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> return 0; /* unclear what should happen here */<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">}<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> </span></p></div></div></blockquote><div><br></div></span><div>It seems perfectly clear to me:</div><div><br></div><div><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">int main(void) {<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_init(); /* ref_count:=1 */<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> atexit(shmem_finalize);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> shmem_finalize(); /* ref_count:=0 */<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"> return 0; /* atexit(shmem_finalize) sees that ref_count=0 and returns immediately */<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">}<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> </span></p></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="m_1689638789201716986gmail-m_1246775112113157736WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">Irrespective of the number of times shmem_init is used, we need to
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">successfully handle atexit(shmem_finalize). So, I would prefer adding
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">new APIs in SHMEM to query is_initialized/is_finalized and let the<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">users handle atexit(shmem_finalize) scenario correctly.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> </span></p></div></div></blockquote><div><br></div></span><div>Why? Having is_{initia,fina}lized force the user to do checks that the implementation can do for them, and I don't see what benefit they provide.</div><div><br></div><div>If we add return codes for init and finalize, then users who don't know how many times init was called can just do this:</div><div><br></div><div>int count;</div><div>do {</div><div> count = shmem_finalize(); // returns ref_count</div><div> if (count==0) break; </div><div>} while (1);</div><div><br></div><div><br></div><div>Jeff</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="m_1689638789201716986gmail-m_1246775112113157736WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif">-Naveen N Ravi.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif"><u></u> <u></u></span></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Jeff Hammond [mailto:<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><wbr>]
<br>
<b>Sent:</b> Monday, March 20, 2017 4:01 PM<br>
<b>To:</b> Naveen Ravichandrasekaran <<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>><br>
<b>Cc:</b> <a href="mailto:openshmem-list@openshmem.org" target="_blank">openshmem-list@openshmem.org</a><br>
<b>Subject:</b> Re: [Openshmem-list] Effects of removing implicit finalize from the specification<u></u><u></u></span></p>
</div>
</div><div><div class="m_1689638789201716986gmail-h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">If we permit multiple calls to finalize then we should also permit multiple calls to initialize. This makes it easy on applications that use libraries and the libraries do not know whether they should call initialize or not.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We either need to allow multiple calls to initialize/finalize or we need to have is_initialized/is_finalized queries. MPI does the latter but since it is utterly trivial to ref count in this case, we should just do better than MPI and
do the former.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We could add return codes on initialize/finalize to allow users to know whether the calls were no-ops or not. This does not break any existing code because users do not have to assign return codes to variables.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Jeff<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Mar 20, 2017 at 1:37 PM, Naveen Ravichandrasekaran <<a href="mailto:nravi@cray.com" target="_blank">nravi@cray.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">At present, the specification allows the multiple usage of shmem_finalize.<br>
This seems to be a defined behavior and multiple subsequent calls to finalize<br>
are no-ops.<br>
<br>
The following example should pass as per the current specification;<br>
<br>
Example:1<br>
int main(void) {<br>
shmem_init();<br>
shmem_finalize();<br>
shmem_finalize();<br>
return 0;<br>
}<br>
<br>
In Example:1, we have three finalize (2 explicit + 1 implicit) operations.<br>
Even any common SHMEM usage, will have two finalize operations<br>
(1 explicit + 1 implicit). Even though the specification is not clear about the<br>
above usage, this is the side-effect of having implicit finalize at exit() or at<br>
return from main.<br>
<br>
Example:2<br>
int main(void) {<br>
shmem_init();<br>
atexit(shmem_finalize);<br>
return 0;<br>
}<br>
<br>
During F2F meeting, we decided to drop implicit finalize in the specification.<br>
The argument was that the users can explicitly call "atexit" in the application<br>
to achieve the above behavior as shown in Example:2.<br>
<br>
Example:3<br>
int main(void) {<br>
shmem_init();<br>
atexit(shmem_finalize);<br>
shmem_finalize();<br>
return 0;<br>
}<br>
<br>
FWIU, if we remove implicit finalize from the specification - Example:2 is<br>
guaranteed to work but the behavior on Example:3 is undefined.<br>
<br>
To allow the explicit atexit usage, we either<br>
1. Need to specify that multiple calls to finalize are no-ops or<br>
2. We need to have a new API - "shmem_finalized()" so that the users can<br>
have a wrapper from atexit which checks shmem_finalized() before calling<br>
shmem_finalize().<br>
<br>
Please let me know, if this analysis looks valid.<br>
<br>
-Naveen N Ravi<br>
Cray Inc.<br>
<br>
______________________________<wbr>_________________<br>
Openshmem-list mailing list<br>
<a href="mailto:Openshmem-list@openshmem.org" target="_blank">Openshmem-list@openshmem.org</a><br>
<a href="http://www.openshmem.org/mailman/listinfo/openshmem-list" target="_blank">http://www.openshmem.org/mailm<wbr>an/listinfo/openshmem-list</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div class="m_1689638789201716986gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div></div></div>
<br>______________________________<wbr>_________________<br>
Openshmem-list mailing list<br>
<a href="mailto:Openshmem-list@openshmem.org">Openshmem-list@openshmem.org</a><br>
<a href="http://www.openshmem.org/mailman/listinfo/openshmem-list" rel="noreferrer" target="_blank">http://www.openshmem.org/<wbr>mailman/listinfo/openshmem-<wbr>list</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>John C. Linford, Ph.D.<br>Senior Computer Scientist</div><div><a href="http://www.paratools.com" target="_blank">ParaTools, Inc.</a></div><div>5520 Research Park Drive, Suite 100</div><div>Baltimore, MD 21228<br></div><div>Phone: <a value="+15408089250">540-808-9250</a><br>
-------------------------------------------------------------------------</div></div></div></div></div>
</div>