Fix datatype confusion with the new lossy GiST distance functions.

We can only support a lossy distance function when the distance function's
datatype is comparable with the original ordering operator's datatype.
The distance function always returns a float8, so we are limited to float8,
and float4 (by a hard-coded cast of the float8 to float4).

In light of this limitation, it seems like a good idea to have a separate
'recheck' flag for the ORDER BY expressions, so that if you have a non-lossy
distance function, it still works with lossy quals. There are cases like
that with the build-in or contrib opclasses, but it's plausible.

There was a hidden assumption that the ORDER BY values returned by GiST
match the original ordering operator's return type, but there are plenty
of examples where that's not true, e.g. in btree_gist and pg_trgm. As long
as the distance function is not lossy, we can tolerate that and just not
return the distance to the executor (or rather, always return NULL). The
executor doesn't need the distances if there are no lossy results.

There was another little bug: the recheck variable was not initialized
before calling the distance function. That revealed the bigger issue,
as the executor tried to reorder tuples that didn't need reordering, and
that failed because of the datatype mismatch.

doc/src/sgml/gist.sgml

@@ -818,10 +818,15 @@ my_distance(PG_FUNCTION_ARGS)
       </para>
 
       <para>
-       The result value can be any finite <type>float8</> value.  (Infinity and
-       minus infinity are used internally to handle cases such as nulls, so it
-       is not recommended that <function>distance</> functions return these
-       values.)
+       If the distance function returns *recheck=true for a leaf node, the
+       original ordering operator's return type must be float8 or float4, and
+       the distance function's return value must be comparable with the actual
+       distance operator. Otherwise, the distance function's return type can
+       be any finit <type>float8</> value, as long as the relative order of
+       the returned values matches the order returned by the ordering operator.
+       (Infinity and minus infinity are used internally to handle cases such as
+       nulls, so it is not recommended that <function>distance</> functions
+       return these values.)
       </para>
 
      </listitem>