1
0
mirror of https://github.com/nlohmann/json.git synced 2025-08-07 18:02:57 +03:00

BJData optimized binary array type (#4513)

This commit is contained in:
Nebojša Cvetković
2025-01-07 17:09:19 +00:00
committed by GitHub
parent 60c48755e3
commit 2e50d5b2f3
7 changed files with 508 additions and 289 deletions

View File

@@ -4,13 +4,16 @@
// (1) // (1)
static std::vector<std::uint8_t> to_bjdata(const basic_json& j, static std::vector<std::uint8_t> to_bjdata(const basic_json& j,
const bool use_size = false, const bool use_size = false,
const bool use_type = false); const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2);
// (2) // (2)
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false); const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2);
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o, static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false); const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2);
``` ```
Serializes a given JSON value `j` to a byte vector using the BJData (Binary JData) serialization format. BJData aims to Serializes a given JSON value `j` to a byte vector using the BJData (Binary JData) serialization format. BJData aims to
@@ -34,6 +37,9 @@ The exact mapping and its limitations is described on a [dedicated page](../../f
`use_type` (in) `use_type` (in)
: whether to add type annotations to container types (must be combined with `#!cpp use_size = true`); optional, : whether to add type annotations to container types (must be combined with `#!cpp use_size = true`); optional,
`version` (in)
: which version of BJData to use (see [draft 3](../../features/binary_formats/bjdata.md#draft-3-binary-format)); optional,
`#!cpp false` by default. `#!cpp false` by default.
## Return value ## Return value
@@ -68,3 +74,4 @@ Linear in the size of the JSON value `j`.
## Version history ## Version history
- Added in version 3.11.0. - Added in version 3.11.0.
- BJData version parameter (for draft3 binary encoding) added in version 3.12.0.

View File

@@ -2,10 +2,10 @@
The [BJData format](https://neurojson.org) was derived from and improved upon The [BJData format](https://neurojson.org) was derived from and improved upon
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized [Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 4 new type markers - array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 5 new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64` and `[h] - float16` - to unambiguously map common binary numeric types; `[u] - uint16`, `[m] - uint32`, `[M] - uint64`, `[h] - float16` and `[B] - byte` - to unambiguously map common binary
furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to avoid numeric types; furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
unnecessary conversions on commonly available platforms. avoid unnecessary conversions on commonly available platforms.
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare
combination of being both binary and **quasi-human-readable**. This is because all semantic elements in BJData and combination of being both binary and **quasi-human-readable**. This is because all semantic elements in BJData and
@@ -49,6 +49,7 @@ The library uses the following mapping from JSON values types to BJData types ac
| string | *with shortest length indicator* | string | `S` | | string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format/ND-array* | array | `[` | | array | *see notes on optimized format/ND-array* | array | `[` |
| object | *see notes on optimized format* | map | `{` | | object | *see notes on optimized format* | map | `{` |
| binary | *see notes on binary values* | array | `[$B` |
!!! success "Complete mapping" !!! success "Complete mapping"
@@ -128,15 +129,24 @@ The library uses the following mapping from JSON values types to BJData types ac
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data
types following the `$` marker in an optimized array and object container are restricted to types following the `$` marker in an optimized array and object container are restricted to
**non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of `UiuImlMLhdDC`. **non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of
This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an optimized array or object `UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an
in BJData. optimized array or object in BJData.
!!! info "Binary values" !!! info "Binary values"
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the BJData BJData provides a dedicated `B` marker (defined in the [BJData specification (Draft 3)][BJDataBinArr]) that is used
documentation. In particular, this means that the serialization and the deserialization of JSON containing binary in optimized arrays to designate binary data. This means that, unlike UBJSON, binary data can be both serialized and
values into BJData and back will result in a different JSON object. deserialized.
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled using
the `version` parameter of [`to_bjdata`](../../api/basic_json/to_bjdata.md).
In Draft2 mode (default), if the JSON data contains the binary type, the value stored as a list of integers, as
suggested by the BJData documentation. In particular, this means that the serialization and the deserialization of
JSON containing binary values into BJData and back will result in a different JSON object.
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/master/Binary_JData_Specification.md#optimized-binary-array)
??? example ??? example
@@ -171,11 +181,13 @@ The library maps BJData types to JSON value types as follows:
| int32 | number_integer | `l` | | int32 | number_integer | `l` |
| uint64 | number_unsigned | `M` | | uint64 | number_unsigned | `M` |
| int64 | number_integer | `L` | | int64 | number_integer | `L` |
| byte | number_unsigned | `B` |
| string | string | `S` | | string | string | `S` |
| char | string | `C` | | char | string | `C` |
| array | array (optimized values are supported) | `[` | | array | array (optimized values are supported) | `[` |
| ND-array | object (in JData annotated array format)|`[$.#[.`| | ND-array | object (in JData annotated array format)|`[$.#[.`|
| object | object (optimized values are supported) | `{` | | object | object (optimized values are supported) | `{` |
| binary | binary (strongly-typed byte array) | `[$B` |
!!! success "Complete mapping" !!! success "Complete mapping"

View File

@@ -2313,6 +2313,16 @@ class binary_reader
case 'Z': // null case 'Z': // null
return sax->null(); return sax->null();
case 'B': // byte
{
if (input_format != input_format_t::bjdata)
{
break;
}
std::uint8_t number{};
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U': case 'U':
{ {
std::uint8_t number{}; std::uint8_t number{};
@@ -2513,7 +2523,7 @@ class binary_reader
return false; return false;
} }
if (size_and_type.second == 'C') if (size_and_type.second == 'C' || size_and_type.second == 'B')
{ {
size_and_type.second = 'U'; size_and_type.second = 'U';
} }
@@ -2535,6 +2545,13 @@ class binary_reader
return (sax->end_array() && sax->end_object()); return (sax->end_array() && sax->end_object());
} }
// If BJData type marker is 'B' decode as binary
if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B')
{
binary_t result;
return get_binary(input_format, size_and_type.first, result) && sax->binary(result);
}
if (size_and_type.first != npos) if (size_and_type.first != npos)
{ {
if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first)))
@@ -3008,6 +3025,7 @@ class binary_reader
#define JSON_BINARY_READER_MAKE_BJD_TYPES_MAP_ \ #define JSON_BINARY_READER_MAKE_BJD_TYPES_MAP_ \
make_array<bjd_type>( \ make_array<bjd_type>( \
bjd_type{'B', "byte"}, \
bjd_type{'C', "char"}, \ bjd_type{'C', "char"}, \
bjd_type{'D', "double"}, \ bjd_type{'D', "double"}, \
bjd_type{'I', "int16"}, \ bjd_type{'I', "int16"}, \

View File

@@ -28,6 +28,13 @@ NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
{ {
/// how to encode BJData
enum class bjdata_version_t
{
draft2,
draft3,
};
/////////////////// ///////////////////
// binary writer // // binary writer //
/////////////////// ///////////////////
@@ -735,11 +742,14 @@ class binary_writer
@param[in] use_type whether to use '$' prefixes (optimized format) @param[in] use_type whether to use '$' prefixes (optimized format)
@param[in] add_prefix whether prefixes need to be used for this value @param[in] add_prefix whether prefixes need to be used for this value
@param[in] use_bjdata whether write in BJData format, default is false @param[in] use_bjdata whether write in BJData format, default is false
@param[in] bjdata_version which BJData version to use, default is draft2
*/ */
void write_ubjson(const BasicJsonType& j, const bool use_count, void write_ubjson(const BasicJsonType& j, const bool use_count,
const bool use_type, const bool add_prefix = true, const bool use_type, const bool add_prefix = true,
const bool use_bjdata = false) const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2)
{ {
const bool bjdata_draft3 = bjdata_version == bjdata_version_t::draft3;
switch (j.type()) switch (j.type())
{ {
case value_t::null: case value_t::null:
@@ -829,7 +839,7 @@ class binary_writer
for (const auto& el : *j.m_data.m_value.array) for (const auto& el : *j.m_data.m_value.array)
{ {
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata); write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
} }
if (!use_count) if (!use_count)
@@ -847,11 +857,11 @@ class binary_writer
oa->write_character(to_char_type('[')); oa->write_character(to_char_type('['));
} }
if (use_type && !j.m_data.m_value.binary->empty()) if (use_type && ((use_bjdata && bjdata_draft3) || !j.m_data.m_value.binary->empty()))
{ {
JSON_ASSERT(use_count); JSON_ASSERT(use_count);
oa->write_character(to_char_type('$')); oa->write_character(to_char_type('$'));
oa->write_character('U'); oa->write_character(use_bjdata && bjdata_draft3 ? 'B' : 'U');
} }
if (use_count) if (use_count)
@@ -870,7 +880,7 @@ class binary_writer
{ {
for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i) for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i)
{ {
oa->write_character(to_char_type('U')); oa->write_character(to_char_type((use_bjdata && bjdata_draft3) ? 'B' : 'U'));
oa->write_character(j.m_data.m_value.binary->data()[i]); oa->write_character(j.m_data.m_value.binary->data()[i]);
} }
} }
@@ -887,7 +897,7 @@ class binary_writer
{ {
if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end()) if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end())
{ {
if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata) if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata)
{ {
break; break;
} }
@@ -931,7 +941,7 @@ class binary_writer
oa->write_characters( oa->write_characters(
reinterpret_cast<const CharType*>(el.first.c_str()), reinterpret_cast<const CharType*>(el.first.c_str()),
el.first.size()); el.first.size());
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata); write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
} }
if (!use_count) if (!use_count)
@@ -1615,10 +1625,11 @@ class binary_writer
/*! /*!
@return false if the object is successfully converted to a bjdata ndarray, true if the type or size is invalid @return false if the object is successfully converted to a bjdata ndarray, true if the type or size is invalid
*/ */
bool write_bjdata_ndarray(const typename BasicJsonType::object_t& value, const bool use_count, const bool use_type) bool write_bjdata_ndarray(const typename BasicJsonType::object_t& value, const bool use_count, const bool use_type, const bjdata_version_t bjdata_version)
{ {
std::map<string_t, CharType> bjdtype = {{"uint8", 'U'}, {"int8", 'i'}, {"uint16", 'u'}, {"int16", 'I'}, std::map<string_t, CharType> bjdtype = {{"uint8", 'U'}, {"int8", 'i'}, {"uint16", 'u'}, {"int16", 'I'},
{"uint32", 'm'}, {"int32", 'l'}, {"uint64", 'M'}, {"int64", 'L'}, {"single", 'd'}, {"double", 'D'}, {"char", 'C'} {"uint32", 'm'}, {"int32", 'l'}, {"uint64", 'M'}, {"int64", 'L'}, {"single", 'd'}, {"double", 'D'},
{"char", 'C'}, {"byte", 'B'}
}; };
string_t key = "_ArrayType_"; string_t key = "_ArrayType_";
@@ -1648,10 +1659,10 @@ class binary_writer
oa->write_character('#'); oa->write_character('#');
key = "_ArraySize_"; key = "_ArraySize_";
write_ubjson(value.at(key), use_count, use_type, true, true); write_ubjson(value.at(key), use_count, use_type, true, true, bjdata_version);
key = "_ArrayData_"; key = "_ArrayData_";
if (dtype == 'U' || dtype == 'C') if (dtype == 'U' || dtype == 'C' || dtype == 'B')
{ {
for (const auto& el : value.at(key)) for (const auto& el : value.at(key))
{ {

View File

@@ -171,6 +171,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
using error_handler_t = detail::error_handler_t; using error_handler_t = detail::error_handler_t;
/// how to treat CBOR tags /// how to treat CBOR tags
using cbor_tag_handler_t = detail::cbor_tag_handler_t; using cbor_tag_handler_t = detail::cbor_tag_handler_t;
/// how to encode BJData
using bjdata_version_t = detail::bjdata_version_t;
/// helper type for initializer lists of basic_json values /// helper type for initializer lists of basic_json values
using initializer_list_t = std::initializer_list<detail::json_ref<basic_json>>; using initializer_list_t = std::initializer_list<detail::json_ref<basic_json>>;
@@ -4352,27 +4354,30 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static std::vector<std::uint8_t> to_bjdata(const basic_json& j, static std::vector<std::uint8_t> to_bjdata(const basic_json& j,
const bool use_size = false, const bool use_size = false,
const bool use_type = false) const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
to_bjdata(j, result, use_size, use_type); to_bjdata(j, result, use_size, use_type, version);
return result; return result;
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false) const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
binary_writer<std::uint8_t>(o).write_ubjson(j, use_size, use_type, true, true); binary_writer<std::uint8_t>(o).write_ubjson(j, use_size, use_type, true, true, version);
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o, static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false) const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
binary_writer<char>(o).write_ubjson(j, use_size, use_type, true, true); binary_writer<char>(o).write_ubjson(j, use_size, use_type, true, true, version);
} }
/// @brief create a BSON serialization of a given JSON value /// @brief create a BSON serialization of a given JSON value

View File

@@ -11991,6 +11991,16 @@ class binary_reader
case 'Z': // null case 'Z': // null
return sax->null(); return sax->null();
case 'B': // byte
{
if (input_format != input_format_t::bjdata)
{
break;
}
std::uint8_t number{};
return get_number(input_format, number) && sax->number_unsigned(number);
}
case 'U': case 'U':
{ {
std::uint8_t number{}; std::uint8_t number{};
@@ -12191,7 +12201,7 @@ class binary_reader
return false; return false;
} }
if (size_and_type.second == 'C') if (size_and_type.second == 'C' || size_and_type.second == 'B')
{ {
size_and_type.second = 'U'; size_and_type.second = 'U';
} }
@@ -12213,6 +12223,13 @@ class binary_reader
return (sax->end_array() && sax->end_object()); return (sax->end_array() && sax->end_object());
} }
// If BJData type marker is 'B' decode as binary
if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B')
{
binary_t result;
return get_binary(input_format, size_and_type.first, result) && sax->binary(result);
}
if (size_and_type.first != npos) if (size_and_type.first != npos)
{ {
if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first)))
@@ -12686,6 +12703,7 @@ class binary_reader
#define JSON_BINARY_READER_MAKE_BJD_TYPES_MAP_ \ #define JSON_BINARY_READER_MAKE_BJD_TYPES_MAP_ \
make_array<bjd_type>( \ make_array<bjd_type>( \
bjd_type{'B', "byte"}, \
bjd_type{'C', "char"}, \ bjd_type{'C', "char"}, \
bjd_type{'D', "double"}, \ bjd_type{'D', "double"}, \
bjd_type{'I', "int16"}, \ bjd_type{'I', "int16"}, \
@@ -15646,6 +15664,13 @@ NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
{ {
/// how to encode BJData
enum class bjdata_version_t
{
draft2,
draft3,
};
/////////////////// ///////////////////
// binary writer // // binary writer //
/////////////////// ///////////////////
@@ -16353,11 +16378,14 @@ class binary_writer
@param[in] use_type whether to use '$' prefixes (optimized format) @param[in] use_type whether to use '$' prefixes (optimized format)
@param[in] add_prefix whether prefixes need to be used for this value @param[in] add_prefix whether prefixes need to be used for this value
@param[in] use_bjdata whether write in BJData format, default is false @param[in] use_bjdata whether write in BJData format, default is false
@param[in] bjdata_version which BJData version to use, default is draft2
*/ */
void write_ubjson(const BasicJsonType& j, const bool use_count, void write_ubjson(const BasicJsonType& j, const bool use_count,
const bool use_type, const bool add_prefix = true, const bool use_type, const bool add_prefix = true,
const bool use_bjdata = false) const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2)
{ {
const bool bjdata_draft3 = bjdata_version == bjdata_version_t::draft3;
switch (j.type()) switch (j.type())
{ {
case value_t::null: case value_t::null:
@@ -16447,7 +16475,7 @@ class binary_writer
for (const auto& el : *j.m_data.m_value.array) for (const auto& el : *j.m_data.m_value.array)
{ {
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata); write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
} }
if (!use_count) if (!use_count)
@@ -16465,11 +16493,11 @@ class binary_writer
oa->write_character(to_char_type('[')); oa->write_character(to_char_type('['));
} }
if (use_type && !j.m_data.m_value.binary->empty()) if (use_type && ((use_bjdata && bjdata_draft3) || !j.m_data.m_value.binary->empty()))
{ {
JSON_ASSERT(use_count); JSON_ASSERT(use_count);
oa->write_character(to_char_type('$')); oa->write_character(to_char_type('$'));
oa->write_character('U'); oa->write_character(use_bjdata && bjdata_draft3 ? 'B' : 'U');
} }
if (use_count) if (use_count)
@@ -16488,7 +16516,7 @@ class binary_writer
{ {
for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i) for (size_t i = 0; i < j.m_data.m_value.binary->size(); ++i)
{ {
oa->write_character(to_char_type('U')); oa->write_character(to_char_type((use_bjdata && bjdata_draft3) ? 'B' : 'U'));
oa->write_character(j.m_data.m_value.binary->data()[i]); oa->write_character(j.m_data.m_value.binary->data()[i]);
} }
} }
@@ -16505,7 +16533,7 @@ class binary_writer
{ {
if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end()) if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end())
{ {
if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata) if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata)
{ {
break; break;
} }
@@ -16549,7 +16577,7 @@ class binary_writer
oa->write_characters( oa->write_characters(
reinterpret_cast<const CharType*>(el.first.c_str()), reinterpret_cast<const CharType*>(el.first.c_str()),
el.first.size()); el.first.size());
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata); write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
} }
if (!use_count) if (!use_count)
@@ -17233,10 +17261,11 @@ class binary_writer
/*! /*!
@return false if the object is successfully converted to a bjdata ndarray, true if the type or size is invalid @return false if the object is successfully converted to a bjdata ndarray, true if the type or size is invalid
*/ */
bool write_bjdata_ndarray(const typename BasicJsonType::object_t& value, const bool use_count, const bool use_type) bool write_bjdata_ndarray(const typename BasicJsonType::object_t& value, const bool use_count, const bool use_type, const bjdata_version_t bjdata_version)
{ {
std::map<string_t, CharType> bjdtype = {{"uint8", 'U'}, {"int8", 'i'}, {"uint16", 'u'}, {"int16", 'I'}, std::map<string_t, CharType> bjdtype = {{"uint8", 'U'}, {"int8", 'i'}, {"uint16", 'u'}, {"int16", 'I'},
{"uint32", 'm'}, {"int32", 'l'}, {"uint64", 'M'}, {"int64", 'L'}, {"single", 'd'}, {"double", 'D'}, {"char", 'C'} {"uint32", 'm'}, {"int32", 'l'}, {"uint64", 'M'}, {"int64", 'L'}, {"single", 'd'}, {"double", 'D'},
{"char", 'C'}, {"byte", 'B'}
}; };
string_t key = "_ArrayType_"; string_t key = "_ArrayType_";
@@ -17266,10 +17295,10 @@ class binary_writer
oa->write_character('#'); oa->write_character('#');
key = "_ArraySize_"; key = "_ArraySize_";
write_ubjson(value.at(key), use_count, use_type, true, true); write_ubjson(value.at(key), use_count, use_type, true, true, bjdata_version);
key = "_ArrayData_"; key = "_ArrayData_";
if (dtype == 'U' || dtype == 'C') if (dtype == 'U' || dtype == 'C' || dtype == 'B')
{ {
for (const auto& el : value.at(key)) for (const auto& el : value.at(key))
{ {
@@ -20051,6 +20080,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
using error_handler_t = detail::error_handler_t; using error_handler_t = detail::error_handler_t;
/// how to treat CBOR tags /// how to treat CBOR tags
using cbor_tag_handler_t = detail::cbor_tag_handler_t; using cbor_tag_handler_t = detail::cbor_tag_handler_t;
/// how to encode BJData
using bjdata_version_t = detail::bjdata_version_t;
/// helper type for initializer lists of basic_json values /// helper type for initializer lists of basic_json values
using initializer_list_t = std::initializer_list<detail::json_ref<basic_json>>; using initializer_list_t = std::initializer_list<detail::json_ref<basic_json>>;
@@ -24232,27 +24263,30 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static std::vector<std::uint8_t> to_bjdata(const basic_json& j, static std::vector<std::uint8_t> to_bjdata(const basic_json& j,
const bool use_size = false, const bool use_size = false,
const bool use_type = false) const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
to_bjdata(j, result, use_size, use_type); to_bjdata(j, result, use_size, use_type, version);
return result; return result;
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_bjdata(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const bool use_size = false, const bool use_type = false) const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
binary_writer<std::uint8_t>(o).write_ubjson(j, use_size, use_type, true, true); binary_writer<std::uint8_t>(o).write_ubjson(j, use_size, use_type, true, true, version);
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/ /// @sa https://json.nlohmann.me/api/basic_json/to_bjdata/
static void to_bjdata(const basic_json& j, detail::output_adapter<char> o, static void to_bjdata(const basic_json& j, detail::output_adapter<char> o,
const bool use_size = false, const bool use_type = false) const bool use_size = false, const bool use_type = false,
const bjdata_version_t version = bjdata_version_t::draft2)
{ {
binary_writer<char>(o).write_ubjson(j, use_size, use_type, true, true); binary_writer<char>(o).write_ubjson(j, use_size, use_type, true, true, version);
} }
/// @brief create a BSON serialization of a given JSON value /// @brief create a BSON serialization of a given JSON value

View File

@@ -267,6 +267,34 @@ TEST_CASE("BJData")
} }
} }
SECTION("byte")
{
SECTION("0..255 (uint8)")
{
for (size_t i = 0; i <= 255; ++i)
{
CAPTURE(i)
// create JSON value with integer number (no byte type in JSON)
json j = -1;
j.get_ref<json::number_integer_t&>() = static_cast<json::number_integer_t>(i);
// check type
CHECK(j.is_number_integer());
// create byte vector
std::vector<uint8_t> const value
{
static_cast<uint8_t>('B'),
static_cast<uint8_t>(i),
};
// compare value
CHECK(json::from_bjdata(value) == j);
}
}
}
SECTION("number") SECTION("number")
{ {
SECTION("signed") SECTION("signed")
@@ -1501,6 +1529,14 @@ TEST_CASE("BJData")
SECTION("binary") SECTION("binary")
{ {
for (json::bjdata_version_t bjdata_version :
{
json::bjdata_version_t::draft2, json::bjdata_version_t::draft3
})
{
CAPTURE(bjdata_version)
const bool draft3 = (bjdata_version == json::bjdata_version_t::draft3);
SECTION("N = 0..127") SECTION("N = 0..127")
{ {
for (std::size_t N = 0; N <= 127; ++N) for (std::size_t N = 0; N <= 127; ++N)
@@ -1514,10 +1550,10 @@ TEST_CASE("BJData")
// create expected byte vector // create expected byte vector
std::vector<std::uint8_t> expected; std::vector<std::uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('[')); expected.push_back(static_cast<std::uint8_t>('['));
if (N != 0) if (draft3 || N != 0)
{ {
expected.push_back(static_cast<std::uint8_t>('$')); expected.push_back(static_cast<std::uint8_t>('$'));
expected.push_back(static_cast<std::uint8_t>('U')); expected.push_back(static_cast<std::uint8_t>(draft3 ? 'B' : 'U'));
} }
expected.push_back(static_cast<std::uint8_t>('#')); expected.push_back(static_cast<std::uint8_t>('#'));
expected.push_back(static_cast<std::uint8_t>('i')); expected.push_back(static_cast<std::uint8_t>('i'));
@@ -1528,9 +1564,9 @@ TEST_CASE("BJData")
} }
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, true); const auto result = json::to_bjdata(j, true, true, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
if (N == 0) if (!draft3 && N == 0)
{ {
CHECK(result.size() == N + 4); CHECK(result.size() == N + 4);
} }
@@ -1545,12 +1581,21 @@ TEST_CASE("BJData")
CHECK(result.back() != '\x00'); CHECK(result.back() != '\x00');
} }
if (draft3)
{
// roundtrip
CHECK(json::from_bjdata(result) == j);
CHECK(json::from_bjdata(result, true, false) == j);
}
else
{
// roundtrip only works to an array of numbers // roundtrip only works to an array of numbers
json j_out = s; json j_out = s;
CHECK(json::from_bjdata(result) == j_out); CHECK(json::from_bjdata(result) == j_out);
CHECK(json::from_bjdata(result, true, false) == j_out); CHECK(json::from_bjdata(result, true, false) == j_out);
} }
} }
}
SECTION("N = 128..255") SECTION("N = 128..255")
{ {
@@ -1566,7 +1611,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> expected; std::vector<uint8_t> expected;
expected.push_back(static_cast<std::uint8_t>('[')); expected.push_back(static_cast<std::uint8_t>('['));
expected.push_back(static_cast<std::uint8_t>('$')); expected.push_back(static_cast<std::uint8_t>('$'));
expected.push_back(static_cast<std::uint8_t>('U')); expected.push_back(static_cast<std::uint8_t>(draft3 ? 'B' : 'U'));
expected.push_back(static_cast<std::uint8_t>('#')); expected.push_back(static_cast<std::uint8_t>('#'));
expected.push_back(static_cast<std::uint8_t>('U')); expected.push_back(static_cast<std::uint8_t>('U'));
expected.push_back(static_cast<std::uint8_t>(N)); expected.push_back(static_cast<std::uint8_t>(N));
@@ -1576,18 +1621,27 @@ TEST_CASE("BJData")
} }
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, true); const auto result = json::to_bjdata(j, true, true, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 6); CHECK(result.size() == N + 6);
// check that no null byte is appended // check that no null byte is appended
CHECK(result.back() != '\x00'); CHECK(result.back() != '\x00');
if (draft3)
{
// roundtrip
CHECK(json::from_bjdata(result) == j);
CHECK(json::from_bjdata(result, true, false) == j);
}
else
{
// roundtrip only works to an array of numbers // roundtrip only works to an array of numbers
json j_out = s; json j_out = s;
CHECK(json::from_bjdata(result) == j_out); CHECK(json::from_bjdata(result) == j_out);
CHECK(json::from_bjdata(result, true, false) == j_out); CHECK(json::from_bjdata(result, true, false) == j_out);
} }
} }
}
SECTION("N = 256..32767") SECTION("N = 256..32767")
{ {
@@ -1606,25 +1660,34 @@ TEST_CASE("BJData")
std::vector<std::uint8_t> expected(N + 7, 'x'); std::vector<std::uint8_t> expected(N + 7, 'x');
expected[0] = '['; expected[0] = '[';
expected[1] = '$'; expected[1] = '$';
expected[2] = 'U'; expected[2] = draft3 ? 'B' : 'U';
expected[3] = '#'; expected[3] = '#';
expected[4] = 'I'; expected[4] = 'I';
expected[5] = static_cast<std::uint8_t>(N & 0xFF); expected[5] = static_cast<std::uint8_t>(N & 0xFF);
expected[6] = static_cast<std::uint8_t>((N >> 8) & 0xFF); expected[6] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, true); const auto result = json::to_bjdata(j, true, true, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 7); CHECK(result.size() == N + 7);
// check that no null byte is appended // check that no null byte is appended
CHECK(result.back() != '\x00'); CHECK(result.back() != '\x00');
if (draft3)
{
// roundtrip
CHECK(json::from_bjdata(result) == j);
CHECK(json::from_bjdata(result, true, false) == j);
}
else
{
// roundtrip only works to an array of numbers // roundtrip only works to an array of numbers
json j_out = s; json j_out = s;
CHECK(json::from_bjdata(result) == j_out); CHECK(json::from_bjdata(result) == j_out);
CHECK(json::from_bjdata(result, true, false) == j_out); CHECK(json::from_bjdata(result, true, false) == j_out);
} }
} }
}
SECTION("N = 32768..65535") SECTION("N = 32768..65535")
{ {
@@ -1643,25 +1706,34 @@ TEST_CASE("BJData")
std::vector<std::uint8_t> expected(N + 7, 'x'); std::vector<std::uint8_t> expected(N + 7, 'x');
expected[0] = '['; expected[0] = '[';
expected[1] = '$'; expected[1] = '$';
expected[2] = 'U'; expected[2] = draft3 ? 'B' : 'U';
expected[3] = '#'; expected[3] = '#';
expected[4] = 'u'; expected[4] = 'u';
expected[5] = static_cast<std::uint8_t>(N & 0xFF); expected[5] = static_cast<std::uint8_t>(N & 0xFF);
expected[6] = static_cast<std::uint8_t>((N >> 8) & 0xFF); expected[6] = static_cast<std::uint8_t>((N >> 8) & 0xFF);
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, true); const auto result = json::to_bjdata(j, true, true, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 7); CHECK(result.size() == N + 7);
// check that no null byte is appended // check that no null byte is appended
CHECK(result.back() != '\x00'); CHECK(result.back() != '\x00');
if (draft3)
{
// roundtrip
CHECK(json::from_bjdata(result) == j);
CHECK(json::from_bjdata(result, true, false) == j);
}
else
{
// roundtrip only works to an array of numbers // roundtrip only works to an array of numbers
json j_out = s; json j_out = s;
CHECK(json::from_bjdata(result) == j_out); CHECK(json::from_bjdata(result) == j_out);
CHECK(json::from_bjdata(result, true, false) == j_out); CHECK(json::from_bjdata(result, true, false) == j_out);
} }
} }
}
SECTION("N = 65536..2147483647") SECTION("N = 65536..2147483647")
{ {
@@ -1680,7 +1752,7 @@ TEST_CASE("BJData")
std::vector<std::uint8_t> expected(N + 9, 'x'); std::vector<std::uint8_t> expected(N + 9, 'x');
expected[0] = '['; expected[0] = '[';
expected[1] = '$'; expected[1] = '$';
expected[2] = 'U'; expected[2] = draft3 ? 'B' : 'U';
expected[3] = '#'; expected[3] = '#';
expected[4] = 'l'; expected[4] = 'l';
expected[5] = static_cast<std::uint8_t>(N & 0xFF); expected[5] = static_cast<std::uint8_t>(N & 0xFF);
@@ -1689,18 +1761,27 @@ TEST_CASE("BJData")
expected[8] = static_cast<std::uint8_t>((N >> 24) & 0xFF); expected[8] = static_cast<std::uint8_t>((N >> 24) & 0xFF);
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, true); const auto result = json::to_bjdata(j, true, true, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 9); CHECK(result.size() == N + 9);
// check that no null byte is appended // check that no null byte is appended
CHECK(result.back() != '\x00'); CHECK(result.back() != '\x00');
if (draft3)
{
// roundtrip
CHECK(json::from_bjdata(result) == j);
CHECK(json::from_bjdata(result, true, false) == j);
}
else
{
// roundtrip only works to an array of numbers // roundtrip only works to an array of numbers
json j_out = s; json j_out = s;
CHECK(json::from_bjdata(result) == j_out); CHECK(json::from_bjdata(result) == j_out);
CHECK(json::from_bjdata(result, true, false) == j_out); CHECK(json::from_bjdata(result, true, false) == j_out);
} }
} }
}
SECTION("Other Serializations") SECTION("Other Serializations")
{ {
@@ -1714,13 +1795,13 @@ TEST_CASE("BJData")
expected.push_back(static_cast<std::uint8_t>('[')); expected.push_back(static_cast<std::uint8_t>('['));
for (std::size_t i = 0; i < N; ++i) for (std::size_t i = 0; i < N; ++i)
{ {
expected.push_back(static_cast<std::uint8_t>('U')); expected.push_back(static_cast<std::uint8_t>(draft3 ? 'B' : 'U'));
expected.push_back(static_cast<std::uint8_t>(0x78)); expected.push_back(static_cast<std::uint8_t>(0x78));
} }
expected.push_back(static_cast<std::uint8_t>(']')); expected.push_back(static_cast<std::uint8_t>(']'));
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, false, false); const auto result = json::to_bjdata(j, false, false, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 12); CHECK(result.size() == N + 12);
// check that no null byte is appended // check that no null byte is appended
@@ -1742,12 +1823,12 @@ TEST_CASE("BJData")
for (size_t i = 0; i < N; ++i) for (size_t i = 0; i < N; ++i)
{ {
expected.push_back(static_cast<std::uint8_t>('U')); expected.push_back(static_cast<std::uint8_t>(draft3 ? 'B' : 'U'));
expected.push_back(static_cast<std::uint8_t>(0x78)); expected.push_back(static_cast<std::uint8_t>(0x78));
} }
// compare result + size // compare result + size
const auto result = json::to_bjdata(j, true, false); const auto result = json::to_bjdata(j, true, false, bjdata_version);
CHECK(result == expected); CHECK(result == expected);
CHECK(result.size() == N + 14); CHECK(result.size() == N + 14);
// check that no null byte is appended // check that no null byte is appended
@@ -1760,6 +1841,7 @@ TEST_CASE("BJData")
} }
} }
} }
}
SECTION("array") SECTION("array")
{ {
SECTION("empty") SECTION("empty")
@@ -2334,6 +2416,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_D = {'[', '#', 'i', 2, 'D', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 'D', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40}; std::vector<uint8_t> const v_D = {'[', '#', 'i', 2, 'D', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 'D', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40};
std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'}; std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '#', 'i', 2, 'C', 'a', 'C', 'a'}; std::vector<uint8_t> const v_C = {'[', '#', 'i', 2, 'C', 'a', 'C', 'a'};
std::vector<uint8_t> const v_B = {'[', '#', 'i', 2, 'B', 0xFF, 'B', 0xFF};
// check if vector is parsed correctly // check if vector is parsed correctly
CHECK(json::from_bjdata(v_TU) == json({true, true})); CHECK(json::from_bjdata(v_TU) == json({true, true}));
@@ -2351,6 +2434,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926})); CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_bjdata(v_S) == json({"a", "a"})); CHECK(json::from_bjdata(v_S) == json({"a", "a"}));
CHECK(json::from_bjdata(v_C) == json({"a", "a"})); CHECK(json::from_bjdata(v_C) == json({"a", "a"}));
CHECK(json::from_bjdata(v_B) == json({255, 255}));
// roundtrip: output should be optimized // roundtrip: output should be optimized
CHECK(json::to_bjdata(json::from_bjdata(v_T), true) == v_T); CHECK(json::to_bjdata(json::from_bjdata(v_T), true) == v_T);
@@ -2367,6 +2451,7 @@ TEST_CASE("BJData")
CHECK(json::to_bjdata(json::from_bjdata(v_D), true) == v_D); CHECK(json::to_bjdata(json::from_bjdata(v_D), true) == v_D);
CHECK(json::to_bjdata(json::from_bjdata(v_S), true) == v_S); CHECK(json::to_bjdata(json::from_bjdata(v_S), true) == v_S);
CHECK(json::to_bjdata(json::from_bjdata(v_C), true) == v_S); // char is serialized to string CHECK(json::to_bjdata(json::from_bjdata(v_C), true) == v_S); // char is serialized to string
CHECK(json::to_bjdata(json::from_bjdata(v_B), true) == v_U); // byte is serialized to uint8
} }
SECTION("optimized version (type and length)") SECTION("optimized version (type and length)")
@@ -2383,6 +2468,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', 'i', 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40}; std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', 'i', 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40};
std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'}; std::vector<uint8_t> const v_S = {'[', '#', 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', 'i', 2, 'a', 'a'}; std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', 'i', 2, 'a', 'a'};
std::vector<uint8_t> const v_B = {'[', '$', 'B', '#', 'i', 2, 0xFF, 0xFF};
// check if vector is parsed correctly // check if vector is parsed correctly
CHECK(json::from_bjdata(v_i) == json({127, 127})); CHECK(json::from_bjdata(v_i) == json({127, 127}));
@@ -2396,6 +2482,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926})); CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_bjdata(v_S) == json({"a", "a"})); CHECK(json::from_bjdata(v_S) == json({"a", "a"}));
CHECK(json::from_bjdata(v_C) == json({"a", "a"})); CHECK(json::from_bjdata(v_C) == json({"a", "a"}));
CHECK(json::from_bjdata(v_B) == json::binary(std::vector<uint8_t>({static_cast<uint8_t>(255), static_cast<uint8_t>(255)})));
// roundtrip: output should be optimized // roundtrip: output should be optimized
std::vector<uint8_t> const v_empty = {'[', '#', 'i', 0}; std::vector<uint8_t> const v_empty = {'[', '#', 'i', 0};
@@ -2410,6 +2497,8 @@ TEST_CASE("BJData")
CHECK(json::to_bjdata(json::from_bjdata(v_D), true, true) == v_D); CHECK(json::to_bjdata(json::from_bjdata(v_D), true, true) == v_D);
CHECK(json::to_bjdata(json::from_bjdata(v_S), true, true) == v_S); CHECK(json::to_bjdata(json::from_bjdata(v_S), true, true) == v_S);
CHECK(json::to_bjdata(json::from_bjdata(v_C), true, true) == v_S); // char is serialized to string CHECK(json::to_bjdata(json::from_bjdata(v_C), true, true) == v_S); // char is serialized to string
CHECK(json::to_bjdata(json::from_bjdata(v_B), true, true, json::bjdata_version_t::draft2) == v_U);
CHECK(json::to_bjdata(json::from_bjdata(v_B), true, true, json::bjdata_version_t::draft3) == v_B);
} }
SECTION("optimized ndarray (type and vector-size as optimized 1D array)") SECTION("optimized ndarray (type and vector-size as optimized 1D array)")
@@ -2428,6 +2517,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40}; std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40};
std::vector<uint8_t> const v_S = {'[', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'}; std::vector<uint8_t> const v_S = {'[', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 'a', 'a'}; std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 'a', 'a'};
std::vector<uint8_t> const v_B = {'[', '$', 'B', '#', '[', '$', 'i', '#', 'i', 2, 1, 2, 0xFF, 0xFF};
// check if vector is parsed correctly // check if vector is parsed correctly
CHECK(json::from_bjdata(v_0) == json::array()); CHECK(json::from_bjdata(v_0) == json::array());
@@ -2443,6 +2533,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926})); CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_bjdata(v_S) == json({"a", "a"})); CHECK(json::from_bjdata(v_S) == json({"a", "a"}));
CHECK(json::from_bjdata(v_C) == json({"a", "a"})); CHECK(json::from_bjdata(v_C) == json({"a", "a"}));
CHECK(json::from_bjdata(v_B) == json::binary(std::vector<uint8_t>({static_cast<uint8_t>(255), static_cast<uint8_t>(255)})));
} }
SECTION("optimized ndarray (type and vector-size ndarray with JData annotations)") SECTION("optimized ndarray (type and vector-size ndarray with JData annotations)")
@@ -2460,6 +2551,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_d = {'[', '$', 'd', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 0x00, 0x00, 0x80, 0x3F, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x40, 0x40, 0x00, 0x00, 0x80, 0x40, 0x00, 0x00, 0xA0, 0x40, 0x00, 0x00, 0xC0, 0x40}; std::vector<uint8_t> const v_d = {'[', '$', 'd', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 0x00, 0x00, 0x80, 0x3F, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x40, 0x40, 0x00, 0x00, 0x80, 0x40, 0x00, 0x00, 0xA0, 0x40, 0x00, 0x00, 0xC0, 0x40};
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF0, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x14, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x40}; std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF0, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x14, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x40};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 'a', 'b', 'c', 'd', 'e', 'f'}; std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 'a', 'b', 'c', 'd', 'e', 'f'};
std::vector<uint8_t> const v_B = {'[', '$', 'B', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06};
// check if vector is parsed correctly // check if vector is parsed correctly
CHECK(json::from_bjdata(v_e) == json({{"_ArrayData_", {254, 255}}, {"_ArraySize_", {2, 1}}, {"_ArrayType_", "uint8"}})); CHECK(json::from_bjdata(v_e) == json({{"_ArrayData_", {254, 255}}, {"_ArraySize_", {2, 1}}, {"_ArrayType_", "uint8"}}));
@@ -2475,6 +2567,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_d) == json({{"_ArrayData_", {1.f, 2.f, 3.f, 4.f, 5.f, 6.f}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "single"}})); CHECK(json::from_bjdata(v_d) == json({{"_ArrayData_", {1.f, 2.f, 3.f, 4.f, 5.f, 6.f}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "single"}}));
CHECK(json::from_bjdata(v_D) == json({{"_ArrayData_", {1., 2., 3., 4., 5., 6.}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "double"}})); CHECK(json::from_bjdata(v_D) == json({{"_ArrayData_", {1., 2., 3., 4., 5., 6.}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "double"}}));
CHECK(json::from_bjdata(v_C) == json({{"_ArrayData_", {'a', 'b', 'c', 'd', 'e', 'f'}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "char"}})); CHECK(json::from_bjdata(v_C) == json({{"_ArrayData_", {'a', 'b', 'c', 'd', 'e', 'f'}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "char"}}));
CHECK(json::from_bjdata(v_B) == json({{"_ArrayData_", {1, 2, 3, 4, 5, 6}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "byte"}}));
// roundtrip: output should be optimized // roundtrip: output should be optimized
CHECK(json::to_bjdata(json::from_bjdata(v_e), true, true) == v_e); CHECK(json::to_bjdata(json::from_bjdata(v_e), true, true) == v_e);
@@ -2489,6 +2582,7 @@ TEST_CASE("BJData")
CHECK(json::to_bjdata(json::from_bjdata(v_d), true, true) == v_d); CHECK(json::to_bjdata(json::from_bjdata(v_d), true, true) == v_d);
CHECK(json::to_bjdata(json::from_bjdata(v_D), true, true) == v_D); CHECK(json::to_bjdata(json::from_bjdata(v_D), true, true) == v_D);
CHECK(json::to_bjdata(json::from_bjdata(v_C), true, true) == v_C); CHECK(json::to_bjdata(json::from_bjdata(v_C), true, true) == v_C);
CHECK(json::to_bjdata(json::from_bjdata(v_B), true, true) == v_B);
} }
SECTION("optimized ndarray (type and vector-size as 1D array)") SECTION("optimized ndarray (type and vector-size as 1D array)")
@@ -2507,6 +2601,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', 'i', 1, 'i', 2, ']', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40}; std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', 'i', 1, 'i', 2, ']', 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40};
std::vector<uint8_t> const v_S = {'[', '#', '[', 'i', 1, 'i', 2, ']', 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'}; std::vector<uint8_t> const v_S = {'[', '#', '[', 'i', 1, 'i', 2, ']', 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', 'i', 1, 'i', 2, ']', 'a', 'a'}; std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', 'i', 1, 'i', 2, ']', 'a', 'a'};
std::vector<uint8_t> const v_B = {'[', '$', 'B', '#', '[', 'i', 1, 'i', 2, ']', 0xFF, 0xFF};
std::vector<uint8_t> const v_R = {'[', '#', '[', 'i', 2, ']', 'i', 6, 'U', 7}; std::vector<uint8_t> const v_R = {'[', '#', '[', 'i', 2, ']', 'i', 6, 'U', 7};
// check if vector is parsed correctly // check if vector is parsed correctly
@@ -2523,6 +2618,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926})); CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_bjdata(v_S) == json({"a", "a"})); CHECK(json::from_bjdata(v_S) == json({"a", "a"}));
CHECK(json::from_bjdata(v_C) == json({"a", "a"})); CHECK(json::from_bjdata(v_C) == json({"a", "a"}));
CHECK(json::from_bjdata(v_B) == json::binary(std::vector<uint8_t>({static_cast<uint8_t>(255), static_cast<uint8_t>(255)})));
CHECK(json::from_bjdata(v_R) == json({6, 7})); CHECK(json::from_bjdata(v_R) == json({6, 7}));
} }
@@ -2540,6 +2636,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40}; std::vector<uint8_t> const v_D = {'[', '$', 'D', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40, 0x4a, 0xd8, 0x12, 0x4d, 0xfb, 0x21, 0x09, 0x40};
std::vector<uint8_t> const v_S = {'[', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'}; std::vector<uint8_t> const v_S = {'[', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 'S', 'i', 1, 'a', 'S', 'i', 1, 'a'};
std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 'a', 'a'}; std::vector<uint8_t> const v_C = {'[', '$', 'C', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 'a', 'a'};
std::vector<uint8_t> const v_B = {'[', '$', 'B', '#', '[', '#', 'i', 2, 'i', 1, 'i', 2, 0xFF, 0xFF};
// check if vector is parsed correctly // check if vector is parsed correctly
CHECK(json::from_bjdata(v_i) == json({127, 127})); CHECK(json::from_bjdata(v_i) == json({127, 127}));
@@ -2553,6 +2650,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926})); CHECK(json::from_bjdata(v_D) == json({3.1415926, 3.1415926}));
CHECK(json::from_bjdata(v_S) == json({"a", "a"})); CHECK(json::from_bjdata(v_S) == json({"a", "a"}));
CHECK(json::from_bjdata(v_C) == json({"a", "a"})); CHECK(json::from_bjdata(v_C) == json({"a", "a"}));
CHECK(json::from_bjdata(v_B) == json::binary(std::vector<uint8_t>({static_cast<uint8_t>(255), static_cast<uint8_t>(255)})));
} }
SECTION("invalid ndarray annotations remains as object") SECTION("invalid ndarray annotations remains as object")
@@ -2594,6 +2692,17 @@ TEST_CASE("BJData")
} }
} }
SECTION("byte")
{
SECTION("parse bjdata markers in ubjson")
{
std::vector<uint8_t> const v = {'B', 1};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 1: syntax error while parsing UBJSON value: invalid byte: 0x42", json::parse_error&);
}
}
SECTION("strings") SECTION("strings")
{ {
SECTION("eof after S byte") SECTION("eof after S byte")
@@ -2803,6 +2912,10 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v0 = {'[', '#', 'T', ']'}; std::vector<uint8_t> const v0 = {'[', '#', 'T', ']'};
CHECK_THROWS_WITH(_ = json::from_bjdata(v0), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData size: expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x54"); CHECK_THROWS_WITH(_ = json::from_bjdata(v0), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData size: expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x54");
CHECK(json::from_bjdata(v0, true, false).is_discarded()); CHECK(json::from_bjdata(v0, true, false).is_discarded());
std::vector<uint8_t> const vB = {'[', '#', 'B', ']'};
CHECK_THROWS_WITH(_ = json::from_bjdata(vB), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing BJData size: expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x42");
CHECK(json::from_bjdata(v0, true, false).is_discarded());
} }
SECTION("parse bjdata markers as array size in ubjson") SECTION("parse bjdata markers as array size in ubjson")
@@ -2904,6 +3017,10 @@ TEST_CASE("BJData")
CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vU), "[json.exception.parse_error.110] parse error at byte 18: syntax error while parsing BJData number: unexpected end of input", json::parse_error&); CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vU), "[json.exception.parse_error.110] parse error at byte 18: syntax error while parsing BJData number: unexpected end of input", json::parse_error&);
CHECK(json::from_bjdata(vU, true, false).is_discarded()); CHECK(json::from_bjdata(vU, true, false).is_discarded());
std::vector<uint8_t> const vB = {'[', '$', 'B', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2, 3, 4, 5};
CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vU), "[json.exception.parse_error.110] parse error at byte 18: syntax error while parsing BJData number: unexpected end of input", json::parse_error&);
CHECK(json::from_bjdata(vU, true, false).is_discarded());
std::vector<uint8_t> const vT1 = {'[', '$', 'T', '#', '[', '$', 'i', '#', 'i', 2, 2, 3}; std::vector<uint8_t> const vT1 = {'[', '$', 'T', '#', '[', '$', 'i', '#', 'i', 2, 2, 3};
CHECK(json::from_bjdata(vT1, true, false).is_discarded()); CHECK(json::from_bjdata(vT1, true, false).is_discarded());
@@ -3197,6 +3314,21 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
CHECK(json::from_bjdata(v) == j); CHECK(json::from_bjdata(v) == j);
} }
SECTION("Byte Type")
{
const auto s = std::vector<std::uint8_t>(
{
static_cast<std::uint8_t>(222),
static_cast<std::uint8_t>(173),
static_cast<std::uint8_t>(190),
static_cast<std::uint8_t>(239)
});
json const j = {{"binary", json::binary(s)}, {"val", 123}};
std::vector<uint8_t> const v = {'{', 'i', 6, 'b', 'i', 'n', 'a', 'r', 'y', '[', '$', 'B', '#', 'i', 4, 222, 173, 190, 239, 'i', 3, 'v', 'a', 'l', 'i', 123, '}'};
//CHECK(json::to_bjdata(j) == v); // 123 value gets encoded as uint8
CHECK(json::from_bjdata(v) == j);
}
SECTION("String Type") SECTION("String Type")
{ {
SECTION("English") SECTION("English")
@@ -3448,7 +3580,7 @@ TEST_CASE("all BJData first bytes")
// these bytes will fail immediately with exception parse_error.112 // these bytes will fail immediately with exception parse_error.112
std::set<uint8_t> supported = std::set<uint8_t> supported =
{ {
'T', 'F', 'Z', 'U', 'i', 'I', 'l', 'L', 'd', 'D', 'C', 'S', '[', '{', 'N', 'H', 'u', 'm', 'M', 'h' 'T', 'F', 'Z', 'B', 'U', 'i', 'I', 'l', 'L', 'd', 'D', 'C', 'S', '[', '{', 'N', 'H', 'u', 'm', 'M', 'h'
}; };
for (auto i = 0; i < 256; ++i) for (auto i = 0; i < 256; ++i)