Also use l_tls_dtor_count to decide on object unload (BZ #18657)

When an TLS destructor is registered, we set the DF_1_NODELETE flag to
signal that the object should not be destroyed.  We then clear the
DF_1_NODELETE flag when all destructors are called, which is wrong -
the flag could have been set by other means too.

This patch replaces this use of the flag by using l_tls_dtor_count
directly to determine whether it is safe to unload the object.  This
change has the added advantage of eliminating the lock taking when
calling the destructors, which could result in a deadlock.  The patch
also fixes the test case tst-tls-atexit - it was making an invalid
dlclose call, which would just return an error silently.

I have also added a detailed note on concurrency which also aims to
justify why I chose the semantics I chose for accesses to
l_tls_dtor_count.  Thanks to Torvald for his help in getting me
started on this and (literally) teaching my how to approach the
problem.

Change verified on x86_64; the test suite does not show any
regressions due to the patch.

ChangeLog:

	[BZ #18657]
	* elf/dl-close.c (_dl_close_worker): Don't unload DSO if there
	are pending TLS destructor calls.
	* include/link.h (struct link_map): Add concurrency note for
	L_TLS_DTOR_COUNT.
	* stdlib/cxa_thread_atexit_impl.c (__cxa_thread_atexit_impl):
	Don't touch the link map flag.  Atomically increment
	l_tls_dtor_count.
	(__call_tls_dtors): Atomically decrement l_tls_dtor_count.
	Avoid taking the load lock and don't touch the link map flag.
	* stdlib/tst-tls-atexit-nodelete.c: New test case.
	* stdlib/Makefile (tests): Use it.
	* stdlib/tst-tls-atexit.c (do_test): dlopen
	tst-tls-atexit-lib.so again before dlclose.  Add conditionals
	to allow tst-tls-atexit-nodelete test case to use it.

stdlib/tst-tls-atexit.c

@@ -16,12 +16,20 @@
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
-/* This test dynamically loads a DSO and spawns a thread that subsequently
-   calls into the DSO to register a destructor for an object in the DSO and
-   then calls dlclose on the handle for the DSO.  When the thread exits, the
-   DSO should not be unloaded or else the destructor called during thread exit
-   will crash.  Further in the main thread, the DSO is opened and closed again,
-   at which point the DSO should be unloaded.  */
+/* For the default case, i.e. NO_DELETE not defined, the test dynamically loads
+   a DSO and spawns a thread that subsequently calls into the DSO to register a
+   destructor for an object in the DSO and then calls dlclose on the handle for
+   the DSO.  When the thread exits, the DSO should not be unloaded or else the
+   destructor called during thread exit will crash.  Further in the main
+   thread, the DSO is opened and closed again, at which point the DSO should be
+   unloaded.
+
+   When NO_DELETE is defined, the DSO is loaded twice, once with just RTLD_LAZY
+   flag and the second time with the RTLD_NODELETE flag set.  The thread is
+   spawned, destructor registered and then thread exits without closing the
+   DSO.  In the main thread, the first handle is then closed, followed by the
+   second handle.  In the end, the DSO should remain loaded due to the
+   RTLD_NODELETE flag being set in the second dlopen call.  */
 
 #include <dlfcn.h>
 #include <pthread.h>
@@ -31,6 +39,14 @@
 #include <errno.h>
 #include <link.h>
 
+#ifndef NO_DELETE
+# define LOADED_IS_GOOD false
+#endif
+
+#ifndef H2_RTLD_FLAGS
+# define H2_RTLD_FLAGS (RTLD_LAZY)
+#endif
+
 #define DSO_NAME "$ORIGIN/tst-tls-atexit-lib.so"
 
 /* Walk through the map in the _r_debug structure to see if our lib is still
@@ -43,7 +59,10 @@ is_loaded (void)
   for (; lm; lm = lm->l_next)
     if (lm->l_type == lt_loaded && lm->l_name
 	&& strcmp (basename (DSO_NAME), basename (lm->l_name)) == 0)
-	  return true;
+      {
+	printf ("%s is still loaded\n", lm->l_name);
+	return true;
+      }
   return false;
 }
 
@@ -63,7 +82,9 @@ reg_dtor_and_close (void *h)
 
   reg_dtor ();
 
+#ifndef NO_DELETE
   dlclose (h);
+#endif
 
   return NULL;
 }
@@ -104,19 +125,30 @@ do_test (void)
       return 1;
     }
 
+#ifndef NO_DELETE
+  if (spawn_thread (h1) != 0)
+    return 1;
+#endif
+
+  void *h2 = dlopen (DSO_NAME, H2_RTLD_FLAGS);
+  if (h2 == NULL)
+    {
+      printf ("h2: Unable to load DSO: %s\n", dlerror ());
+      return 1;
+    }
+
+#ifdef NO_DELETE
   if (spawn_thread (h1) != 0)
     return 1;
 
-  /* Now this should unload the DSO.  FIXME: This is a bug, calling dlclose
-     like this is actually wrong, but it works because cxa_thread_atexit_impl
-     has a bug which results in dlclose allowing this to work.  */
   dlclose (h1);
+#endif
+  dlclose (h2);
 
-  /* Check link maps to ensure that the DSO has unloaded.  */
-  if (is_loaded ())
-    return 1;
-
-  return 0;
+  /* Check link maps to ensure that the DSO has unloaded.  In the normal case,
+     the DSO should be unloaded if there are no uses.  However, if one of the
+     dlopen calls were with RTLD_NODELETE, the DSO should remain loaded.  */
+  return is_loaded () == LOADED_IS_GOOD ? 0 : 1;
 }
 
 #define TEST_FUNCTION do_test ()