<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>shmem_tool_sync() would be nice for the tool developers since we could count on SHMEM at the time of dump. When shmem_finalize is in a DSO it's suddenly hard to assert that SHMEM calls are safe to use if the tool is dumping from a static destructor. From the user perspective, the tools should already provide their own mechanisms for dumping data, e.g. TAU has the TAU_DB_DUMP call and setting the TAU_TRACK_SIGNALS=1 environment variable will cause TAU to dump profiles if the application crashes.</div><div><br></div></div></blockquote><div><br></div><div>Yeah, and my past use of TAU_DB_DUMP with NWChem is certainly part of the motivation here. shmem_tool_sync provides a standard wrapper for any tools that want to do this.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>At F2F I floated the idea of a SHMEM_T interface to expose and manipulate SHMEM internal variables. I could imagine a user setting flags to instruct the tool to dump on certain events, every Nth collective, etc.</div><div><br></div></div></blockquote><div><br></div><div>Piggybacking on collectives is a means of last resort that I never want to see anyone implement, because it can lead to unexpected performance variation.</div><div><br></div><div>Given the slow adoption of MPI_T support, I'm not sure how effective that will be. I'd rather see implementers try some things and see if a best practice emerges.</div><div><br></div><div>Jeff</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>~John C.</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Mar 20, 2017 at 5:07 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div class="gmail_extra">The discussion of shmem_finalize made me wonder if it would not be helpful to have an explicit call to tell any active tools to dump their logs.</div><div class="gmail_extra"><br></div><div class="gmail_extra">For example, if I have a code like NWChem that goes through a series of phases, and the odds of phase i+1 crashing is nontrivial, but I want to know how phase i did with some tool, then I can call shmem_tool_sync between the phases to ensure that my tool state is (1) persistent and (2) in an analyzable state.</div><div class="gmail_extra"><br></div><div class="gmail_extra">It really sucks if an app crashes in such a way that tool state is not useable, particularly when said tool state would help figure out why the app crashed.</div><div class="gmail_extra"><br></div><div class="gmail_extra">One advantage of this call vs some hidden implementation is that the hidden implementation is either asynchronous or probably runs more often than it needs to. For example, if I automatically dump logs every Nth collective, then I might slow down the app noticeably. On the other hand, if I have to do the dump asynchronously, I cannot take advantage of I/O aggregation or any other coordinated analysis.</div><div class="gmail_extra"><br></div><div class="gmail_extra">It would be ideal if somebody from the TAU team could comment on this.</div><span class="m_-3055254331043714485HOEnZb"><font color="#888888"><div class="gmail_extra"><br></div><div class="gmail_extra">Jeff<br clear="all"><div><br></div>-- <br><div class="m_-3055254331043714485m_4459416961530889410gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Openshmem-list mailing list<br>
<a href="mailto:Openshmem-list@openshmem.org" target="_blank">Openshmem-list@openshmem.org</a><br>
<a href="http://www.openshmem.org/mailman/listinfo/openshmem-list" rel="noreferrer" target="_blank">http://www.openshmem.org/mailm<wbr>an/listinfo/openshmem-list</a><br>
<br></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_-3055254331043714485gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>John C. Linford, Ph.D.<br>Senior Computer Scientist</div><div><a href="http://www.paratools.com" target="_blank">ParaTools, Inc.</a></div><div>5520 Research Park Drive, Suite 100</div><div>Baltimore, MD 21228<br></div><div>Phone: <a value="+15408089250">540-808-9250</a><br>
------------------------------<wbr>------------------------------<wbr>-------------</div></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div>