Json_writer had two separate code paths: add_unquoted_str() for
numbers/bool/null, and add_escaped_str() which added the surrounding
quotes itself for strings. Single_line_formatting_helper, which
buffers consecutive array/object elements to decide if they fit on
one line, assumed only strings could ever be buffered and always
wrapped the flushed values in quotes. As a result, arrays of numbers
(e.g. "depends_on_map_bits", "rec_per_key") were incorrectly rendered
with their elements quoted as strings.
Unify both paths into add_escaped_quoted_str(): the caller now hands
over bytes that are already in their final on-the-wire form. String
escaping (json_escape_to_string) writes its own surrounding quotes,
while numbers/bool/null are passed through unquoted, so the one-line
helper just concatenates the buffered payloads on flush instead of
adding quotes itself. A DBUG_ASSERT in add_escaped_quoted_str() now
checks that every payload is already a quoted string or a bare
number/bool/null token, so a caller that violates the contract trips
an assertion in debug builds instead of silently producing invalid
JSON.
Also:
- Fix Json_writer_array::add(ulonglong)/(size_t), which went through
add_ll() with a cast to longlong and corrupted large unsigned
values (e.g. ULLONG_MAX); route them through add_ull() instead.
- Fix mysql-test/include/opt_context_schema.inc: "subquery_runs" was
nested inside the preceding object instead of being a sibling
member, and "rec_per_key" items are now declared as "number" to
match the corrected output.
- Update recorded .result files for opt_trace, opt_context_*, and
subselect_mat_analyze_json to reflect numbers/booleans no longer
being quoted inside JSON arrays.
- Extend unittest/sql/my_json_writer-t.cc with coverage for arrays
of primitives: plain integers, mixed types, sizes, multi-line
arrays, values flushed before a nested object, strings that were
already escaped by the one-line buffer, and invalid utf8mb4 input
through add_str().
- Rewrite the "multi-line array of integers" test to actually
overflow the one-line buffer (9 seven-digit numbers instead of 7),
since the previous version fit on one line and never exercised the
element-per-line flush path it claimed to test.
- Drop the now-redundant single-argument add_escaped_quoted_str()
overload; all call sites already know their length.
- Update json_escape_to_string()'s doc comments (my_json_writer.h,
sql_json_lib.h) to state that it quotes its output, not just
escapes it.
- Make Json_writer::add_str() do the quoting with json_escape_to_string().
= All other string writing methods are handled as they call this one.
- Move json_escape_to_string() from opt_histogram_json.cc to
my_json_writer.cc. This is needed because JSON writer unit tests do not
link with opt_histogram_json.
- Histogram code does its own escaping (with handling for special cases).
Add Json_writer::add_escaped_str() to accept escaped strings.
This is not intended to be commonly used.
- opt_trace_print_expanded_query() did its own escaping for
"expanded_query" string. It was done in MDEV-30354. Remove escaping
there to avoid double escaping.
- dump_record_to_trace() also did own escaping when writing table DDLs
into the optimizer trace. Remove double escaping.
- Adjust my_json_writer unit test.
Add 104 TAP unit tests for the Json_writer class covering:
- Invalid JSON detection (unnamed values in objects, named values
in arrays, mismatched open/close)
- Output verification for all value types (strings, integers,
unsigned integers, booleans, null, doubles, sizes)
- Edge cases: LLONG_MAX/MIN, ULLONG_MAX, special doubles (0.0,
-0.0), Latin-1 and UTF-8 mb4 encoded strings, embedded NUL
bytes in keys and values, duplicate key names
- add_member(name, len) explicit key length
- Single-line formatting helper including long-element fallback
and embedded-NUL disable behavior
- Nested structures (objects in objects, arrays of objects, mixed)
- RAII wrappers (Json_writer_object, Json_writer_array) including
ulonglong add overload preserving unsigned range
- String_with_limit truncation behavior
- Json_writer size limit enforcement with exact truncation count
- add_size formatting (bytes, Kb, Mb) with boundary tests
- Positive-path test for add(name, value, num_bytes) overload
Also fix a missing my_writer NULL guard in
Json_writer_object::add(const char*, const char*, size_t)
which would crash when used with a NULL writer.
Harden Single_line_formatting_helper against embedded NUL bytes:
on_add_member() now rejects keys containing NUL, and on_add_str()
disables single-line mode and flushes when a value contains NUL.
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <deepsainivarun@gmail.com>
Similar to #2480.
567b681 introduced safe_strcpy() to minimize the use of C with
potentially unsafe memory overflow with strcpy() whose use is
discouraged.
Replace instances of strcpy() with safe_strcpy() where possible, limited
here to files in the `sql/` directory.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
Changes:
- To detect automatic strlen() I removed the methods in String that
uses 'const char *' without a length:
- String::append(const char*)
- Binary_string(const char *str)
- String(const char *str, CHARSET_INFO *cs)
- append_for_single_quote(const char *)
All usage of append(const char*) is changed to either use
String::append(char), String::append(const char*, size_t length) or
String::append(LEX_CSTRING)
- Added STRING_WITH_LEN() around constant string arguments to
String::append()
- Added overflow argument to escape_string_for_mysql() and
escape_quotes_for_mysql() instead of returning (size_t) -1 on overflow.
This was needed as most usage of the above functions never tested the
result for -1 and would have given wrong results or crashes in case
of overflows.
- Added Item_func_or_sum::func_name_cstring(), which returns LEX_CSTRING.
Changed all Item_func::func_name()'s to func_name_cstring()'s.
The old Item_func_or_sum::func_name() is now an inline function that
returns func_name_cstring().str.
- Changed Item::mode_name() and Item::func_name_ext() to return
LEX_CSTRING.
- Changed for some functions the name argument from const char * to
to const LEX_CSTRING &:
- Item::Item_func_fix_attributes()
- Item::check_type_...()
- Type_std_attributes::agg_item_collations()
- Type_std_attributes::agg_item_set_converter()
- Type_std_attributes::agg_arg_charsets...()
- Type_handler_hybrid_field_type::aggregate_for_result()
- Type_handler_geometry::check_type_geom_or_binary()
- Type_handler::Item_func_or_sum_illegal_param()
- Predicant_to_list_comparator::add_value_skip_null()
- Predicant_to_list_comparator::add_value()
- cmp_item_row::prepare_comparators()
- cmp_item_row::aggregate_row_elements_for_comparison()
- Cursor_ref::print_func()
- Removes String_space() as it was only used in one cases and that
could be simplified to not use String_space(), thanks to the fixed
my_vsnprintf().
- Added some const LEX_CSTRING's for common strings:
- NULL_clex_str, DATA_clex_str, INDEX_clex_str.
- Changed primary_key_name to a LEX_CSTRING
- Renamed String::set_quick() to String::set_buffer_if_not_allocated() to
clarify what the function really does.
- Rename of protocol function:
bool store(const char *from, CHARSET_INFO *cs) to
bool store_string_or_null(const char *from, CHARSET_INFO *cs).
This was done to both clarify the difference between this 'store' function
and also to make it easier to find unoptimal usage of store() calls.
- Added Protocol::store(const LEX_CSTRING*, CHARSET_INFO*)
- Changed some 'const char*' arrays to instead be of type LEX_CSTRING.
- class Item_func_units now used LEX_CSTRING for name.
Other things:
- Fixed a bug in mysql.cc:construct_prompt() where a wrong escape character
in the prompt would cause some part of the prompt to be duplicated.
- Fixed a lot of instances where the length of the argument to
append is known or easily obtain but was not used.
- Removed some not needed 'virtual' definition for functions that was
inherited from the parent. I added override to these.
- Fixed Ordered_key::print() to preallocate needed buffer. Old code could
case memory overruns.
- Simplified some loops when adding char * to a String with delimiters.
sprintf() format of double changed from '%lg' to '%-.11lg'
The change was to make it easier to read optimizer trace output
with tables that has millions of records.
MemorySanitizer (clang -fsanitize=memory) requires that all code
be compiled with instrumentation enabled. The C runtime library
is an exception. Failure to use instrumented libraries will cause
bogus messages about memory being uninitialized.
In WITH_MSAN builds, we must avoid calling getservbyname(),
because even though it is a standard library function, it is
not instrumented, not even in clang 10.
The following cmake options were tested:
-DCMAKE_C_FLAGS='-march=native -O2'
-DCMAKE_CXX_FLAGS='-stdlib=libc++ -march=native -O2'
-DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug
-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF
-DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO
-DWITH_SAFEMALLOC=OFF
-DWITH_{ZLIB,SSL,PCRE}=bundled
-DHAVE_LIBAIO_H=0
-DWITH_MSAN=ON
MEM_MAKE_DEFINED(): An alias for VALGRIND_MAKE_MEM_DEFINED()
and in the future, __msan_unpoison().
For now, neither MEM_MAKE_DEFINED() nor MEM_UNDEFINED()
perform any action under MSAN. Enabling them will catch more bugs, but
will also require some more fixes or work-arounds.
Json_writer::add_double(): Work around a frequently occurring
failure in optimizer tests, related to EXPLAIN FORMAT=JSON.
dtoa(): Disable MSAN altogether. For some reason, this function
is triggering a lot of trouble, especially when invoked for
DBUG functions. The MDL default timeout is dd=86400 seconds,
and for some reason it is claimed to be uninitialized.
InnoDB: Define UNIV_DEBUG_VALGRIND also WITH_MSAN.
ut_crc32_8_hw(), ut_crc32_64_low_hw(): Use the compiler built-in
functions instead of inline assembler when building WITH_MSAN.
This will require at least -msse4.2 when building for IA-32 or AMD64.
The inline assembler would not be instrumented, and would thus cause
bogus failures.
- Move testing of my_writer to inline functions to avoid calls
- Made more functions inline. Especially thd->thread_started()
is now very optimized!
- Moved Opt_trace_stmt classe to opt_trace_context.h to get critical
functions inline
- Added unlikely() to optimize for not having optimizer trace enabled
- Made THD::trace_started() inline
- Added 'if (trace_enabled())' around some potentially expensive code
(not many found)
- Added ASSERT's to ensure we don't call expensive optimizer trace calls
if optimizer trace is not enabled
- Added length to Json_writer functions to speed up buffer writes
when optimizer trace is enabled.
- Changed LEX_CSTRING argument handling to not send full struct to writer
function on_add_str() functions now trusts length arguments
This task involves the implementation for the optimizer trace.
This feature produces a trace for any SELECT/UPDATE/DELETE/,
which contains information about decisions taken by the optimizer during
the optimization phase (choice of table access method, various costs,
transformations, etc). This feature would help to tell why some decisions were
taken by the optimizer and why some were rejected.
Trace is session-local, controlled by the @@optimizer_trace variable.
To enable optimizer trace we need to write:
set @@optimizer_trace variable= 'enabled=on';
To display the trace one can run:
SELECT trace FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
This task also involves:
MDEV-18489: Limit the memory used by the optimizer trace
introduces a switch optimizer_trace_max_mem_size which limits
the memory used by the optimizer trace. This was implemented by
Sergei Petrunia.
Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.
This fix excludes rocksdb, spider,spider, sphinx and connect for now.
- Added sql/mariadb.h file that should be included first by files in sql
directory, if sql_plugin.h is not used (sql_plugin.h adds SHOW variables
that must be done before my_global.h is included)
- Removed a lot of include my_global.h from include files
- Removed include's of some files that my_global.h automatically includes
- Removed duplicated include's of my_sys.h
- Replaced include my_config.h with my_global.h
Fix an issue in Single_line_formatting_helper: flush_on_one_line()
didn't clean up the buffered items which could cause them to be
printed for the second time.
This can't be ever observed by a user (see MDEV text for details).
- Single_line_formatting_helper should not accidentally exit the
DISABLED state. No JSON construct should be able to move the
Single_line_formatting_helper from DISABLED state.
Writing JSON:
- Fix a bug in Single_line_formatting_helper
- Add Json_writer_nesting_guard - safety class
EXPLAIN JSON support
- Add basic subquery support
- Add tests for UNION/UNION ALL.
- First code, "EXPLAIN FORMAT=JSON stmt" and "ANALYZE FORMAT=JSON stmt"
work for basic queries. Complex constructs (e.g subqueries, etc) not
yet supported.
- No test infrastructure yet
