After !9950, respdiff's maximal disagreement percentage needs to be
adjusted as target disagreements between the tested version of the
"main" branch and the reference one jumped for the respdiff,
respdiff:asan, and respdiff:tsan jobs from on average 0.07% to 0.16% and
from 0.12% to 0.17% for the respdiff-third-party job.
In !9950, we concluded setting MAX_DISAGREEMENTS_PERCENTAGE to double
the average disagreement percentage works fine in the CI.
Merge branch 'mnowak/more-lenient-respdiff-limits' into 'main'
See merge request isc-projects/bind9!10293
After !9950, respdiff's maximal disagreement percentage needs to be
adjusted as target disagreements between the tested version of the
"main" branch and the reference one jumped for the respdiff,
respdiff:asan, and respdiff:tsan jobs from on average 0.07% to 0.16% and
from 0.12% to 0.17% for the respdiff-third-party job.
In !9950, we concluded setting MAX_DISAGREEMENTS_PERCENTAGE to double
the average disagreement percentage works fine in the CI.
Call `dns_adbname_ref` before calling `dns_resolver_createfetch` to
ensure `adbname->name` remains stable for the life of the fetch.
Closes#5239
Merge branch '5239-fix-adb-reference-counting' into 'main'
See merge request isc-projects/bind9!10290
When searching through a DNSKEY or KEY rrset for the key matching a particular algorithm and ID, it's a waste of time to convert every key into a `dst_key` object; it's faster to compute the key ID from the rdata, then do the full key conversion after determining that we've found the right key. This optimization was already used in the validator, but it's been refactored for code clarity, and is now also used in query.c and message.c.
Merge branch 'each-refactor-key-search' into 'main'
See merge request isc-projects/bind9!10258
when searching a DNSKEY or KEY rrset for the key that matches
a particular algorithm and ID, it's a waste of time to convert
every key into a dst_key object; it's faster to compute the key
ID by checksumming the region, and then only do the full key
conversion once we know we've found the correct key.
this optimization was already in use in the validator, but it's
been refactored for code clarity, and is now also used in query.c
and message.c.
dns_zonekey_iszonekey() was the only function defined in the
dns_zonekey module, and was only called from one place. it
makes more sense to group this with dns_dnssec functions.
This merge request resolves some performance regressions introduced
with the change from isc_symtab_t to isc_hashmap_t.
The key improvements are:
1. Using a faster hash function than both isc_hashmap_t and
isc_symtab_t. The previous implementation used SipHash, but the
hashflood resistance properties of SipHash are unneeded for config
parsing.
2. Shrinking the initial size of the isc_hashmap_t used inside
isc_symtab_t. Symtab is mainly used for config parsing, and the
when used that way it will have between 1 and 50 keys, but the
previous implementation initialized a map with 128 slots.
By initializing a smaller map, we speed up mallocs and optimize for
the typical case of few config keys.
3. Slight optimization of the string matching in the hashmap, so that
the tail is handled in a single load + comparison, instead of byte
by byte.
Of the three improvements, this is the least important.
Merge branch 'alessio/fxhash-symtab' into 'main'
See merge request isc-projects/bind9!10204
This merge request resolves some performance regressions introduced
with the change from isc_symtab_t to isc_hashmap_t.
The key improvements are:
1. Using a faster hash function than both isc_hashmap_t and
isc_symtab_t. The previous implementation used SipHash, but the
hashflood resistance properties of SipHash are unneeded for config
parsing.
2. Shrinking the initial size of the isc_hashmap_t used inside
isc_symtab_t. Symtab is mainly used for config parsing, and the
when used that way it will have between 1 and ~50 keys, but the
previous implementation initialized a map with 128 slots.
By initializing a smaller map, we speed up mallocs and optimize for
the typical case of few config keys.
3. Slight optimization of the string matching in the hashmap, so that
the tail is handled in a single load + comparison, instead of byte
by byte.
Of the three improvements, this is the least important.
The following small issues related to `dnssec-policy` have been fixed:
- In some cases the key manager inside BIND 9 could run every hour, while it could have run less often.
- While `CDS` and `CDNSKEY` records will be removed correctly from the zone when the corresponding `DS` record needs to be updated, the expected timing metadata when this will happen was never set.
- There were a couple of cases where the safety intervals are added inappropriately, delaying key rollovers longer than necessary.
- If you have identical `keys` in your `dnssec-policy`, they may be retired inappropriately. Note that having keys with identical properties is discouraged in all cases.
Closes#5242
Merge branch '5242-several-keymgr-issues' into 'main'
See merge request isc-projects/bind9!10251
The dnssec-keygen command for the ZSK generation for the zone
multisigner-model2.kasp was wrong (no ZSK was generated in the setup
script, but when 'named' is started, the missing ZSK was created
anyway by 'dnssec-policy'.
Only set the next time the keymgr should run if the value is non zero.
Otherwise we default back to one hour. This may happen if there is one
or more key with an unlimited lifetime.
The keymgr never set the expected timing metadata when CDS/CDNSKEY
records for the corresponding key will be removed from the zone. This
is not troublesome, as key states dictate when this happens, but with
the new pytest we use the timing metadata to determine if the CDS and/or
CDNSKEY for the given key needs to be published.
There are a couple of cases where the safety intervals are added
inappropriately:
1. When setting the PublishCDS/SyncPublish timing metadata, we don't
need to add the publish-safety value if we are calculating the time
when the zone is completely signed for the first time. This value
is for when the DNSKEY has been published and we add a safety
interval before considering the DNSKEY omnipresent.
2. The retire-safety value should only be added to ZSK rollovers if
there is an actual rollover happening, similar to adding the sign
delay.
3. The retire-safety value should only be added to KSK rollovers if
there is an actual rollover happening. We consider the new DS
omnipresent a bit later, so that we are forced to keep the old DS
a bit longer.
While converting the kasp system test to pytest, I encountered a small
bug in the keymgr code. We retire keys when there is more than one
key matching a 'keys' line from the dnssec-policy. But if there are
multiple identical 'keys' lines, as is the case for the test zone
'checkds-doubleksk.kasp', we retire one of the two keys that have the
same properties.
Fix this by checking if there are double matches. This is not fool proof
because there may be many keys for a few identical 'keys' lines, but it
is good enough for now. In practice it makes no sense to have a policy
that dictates multiple keys with identical properties.
Raw integer pointers were being used for the validator's nvalidations
and nfails values but the memory holding them could be freed before
they ceased to be used. Use reference counted counters instead.
Closes#5239
Merge branch '5239-use-counter-for-nvalidations-and-nfailss' into 'main'
See merge request isc-projects/bind9!10248
The fetch context that held these values could be freed while there
were still active pointers to the memory. Using a reference counted
pointer avoids this.
The check looks for logs that are not present, fails to make the
possible failure visible, and fails to bump the check enumerator:
I:checking that log-report-channel zones fail if '*._er/TXT' is missing (129)
grep: test.out4.129: No such file or directory
grep: test.out4.129: No such file or directory
I:checking that raw zone with bad class is handled (129)
The issue appeared in #3659.
Merge branch 'mnowak/checkzone-test-fix' into 'main'
See merge request isc-projects/bind9!10286
The check looks for logs that are not present, fails to make the
possible failure visible, and fails to bump the check enumerator:
I:checking that log-report-channel zones fail if '*._er/TXT' is missing (129)
grep: test.out4.129: No such file or directory
grep: test.out4.129: No such file or directory
I:checking that raw zone with bad class is handled (129)
Lines starting with A or NSEC are expected but not matched with the
OpenBSD grep. Extended regular expressions with direct use of
parentheses and the pipe symbol is more appropriate.
I:checking RRSIG query from cache (154)
I:failed
The issue appeared in #4805.
Merge branch 'mnowak/openbsd-grep-fix' into 'main'
See merge request isc-projects/bind9!10285
Lines starting with A or NSEC are expected but not matched with the
OpenBSD grep. Extended regular expressions with direct use of
parentheses and the pipe symbol is more appropriate.
I:checking RRSIG query from cache (154)
I:failed
When query responses timed out, the resolver could incorrectly increase the regular responses counters, even if no response was received. This has been fixed.
Closes#5193
Merge branch '5193-resolver-statistics-counters-fix' into 'main'
See merge request isc-projects/bind9!10227
When a response times out the fctx_cancelquery() function
incorrectly calculates it in the 'dns_resstatscounter_queryrtt5'
counter (i.e. >=1600 ms). To avoid this, the rctx_timedout()
function should make sure that 'rctx->finish' is NULL. And in order
to adjust the RTT values for the timed out server, 'rctx->no_response'
should be true. Update the rctx_timedout() function to make those
changes.
The resquery_response() function increases the response counter without
checking if the response was successful. Increase the counter only when
the result indicates success.
This branch started off as `michal/upforwd-asyncserver`. It quickly
turned out that the critical `asyncserver.py` change that was needed for
the `upforwd` system test was for the server to be able to read multiple
TCP queries on a single connection. As currently present in `main`,
`asyncserver.py` closes every client connection after servicing a single
query. Retaining that behavior would cause the `upforwd` system test to
fail and, in general, capturing all data sent by a client seems more
useful in tests than just closing connections quickly. `asyncserver.py`
can always be extended in the future (e.g. by adding a new
`ResponseAction` that the networking code would react to) to reinstate
the original behavior, if it turns out to be necessary.
While working on changing that particular `asyncserver.py` behavior, I
noticed a couple of other deficiencies in the TCP connection handling
code, so I started addressing them. One thing led to another and before
I noticed, enough changes were applied to be worth doing a separate
merge request, particularly given that the actual rewrite of
`upforwd/ans4/ans.pl` using `asyncserver.py` is trivial once the
required changes to `asyncserver.py` itself are applied.
Merge branch 'michal/asyncserver-tcp-improvements' into 'main'
See merge request isc-projects/bind9!10276
Instead of closing every incoming TCP connection after handling a single
query, continue receiving queries on each TCP connection until the
client disconnects itself. When coupled with response dropping, this
enables silently receiving all incoming data, simulating an unresponsive
server.
A TCP DNS client may send its queries in chunks, causing
StreamReader.read() to return less data than previously declared by the
client as the DNS message length; even the two-octet DNS message length
itself may be split up into two single-octet transmissions. Sending
data in chunks is valid client behavior that should not be treated as an
error. Add a new helper method for reading TCP data in a loop, properly
distinguishing between chunked queries and client disconnections. Use
the new method for reading all TCP data from clients.
A TCP peer may reset the connection at any point, but asyncserver.py
currently only handles connection resets when it is sending data to the
client. Handle connection resets during reading in the same way.
Add a helper class, Peer, which holds the <host, port> tuple of a
connection endpoint and gets pretty-printed when formatted as a string.
This enables passing instances of this new class directly to logging
functions, eliminating the need for the AsyncDnsServer._format_peer()
helper method.
The false positive rate is about 10-20 % when evaluating shotgun results
from a single run. Attempt to reduce the false positive rate by allowing
a re-run of failed jobs.
Merge branch 'nicki/ci-shotgun-reduce-false-positives' into 'main'
See merge request isc-projects/bind9!10271
The false positive rate is about 10-20 % when evaluating shotgun results
from a single run. Attempt to reduce the false positive rate by allowing
a re-run of failed jobs.
While there is a slight risk that barely noticable decreases in
performance might slip by more easily in MRs, they'd still likely pop up
during nightly or pre-release testing.
Also increase the tolerance threshold for DoH latency comparisons, as
those tests often experience increased jitter in the tail end latencies.
Replace custom DNS servers used in the "qmin" system test with new code
based on the isctest.asyncserver module. The revised code employs zone
files and a limited amount of custom logic, which massively improves
test readability and maintainability, extends logging, and fixes
non-compliant replies sent by some of the custom servers in response to
certain queries (e.g. AA=0 in authoritative empty non-terminal
responses, non-glue address records in ADDITIONAL section).
Merge branch 'michal/qmin-asyncserver' into 'main'
See merge request isc-projects/bind9!10195
The vulture tool seems to be unable to follow how the parent classes
defined in bin/tests/system/qmin/qmin_ans.py use mandatory properties
specified by child classes in bin/tests/system/qmin/ans*/ans.py. Make
the tool ignore not just ans.py servers, but also *_ans.py utility
modules above the ansX/ subdirectories to prevent false positives about
unused code from causing CI pipeline failures.
Some versions of the Hypothesis Python library - notably the one
included in stock OS repositories for Ubuntu 20.04 Focal Fossa - cause a
.hypothesis file to be created in a Python script's working directory
when the hypothesis module is present in its import chain. Ignore such
files by adding them to the list of expected test artifacts to prevent
pytest teardown checks from failing due to these files appearing in the
file system after running system tests.
Commit 6c010a5644324947c8c13b5600cd8d988ae7684f caused the PYTHONPATH
environment variable to be set for ans.py servers started using
start.pl. However, no system test has actually used the new
isctest.asyncserver module since that change was applied, so it has not
been noticed until now that including the source directory in PYTHONPATH
is only sufficient for in-tree builds. Include the build directory
instead of the source directory in the PYTHONPATH environment variable
set for ans.py servers started by start.pl so that they work correctly
for both in-tree and out-of-tree builds.
Replace custom DNS servers used in the "qmin" system test with new code
based on the isctest.asyncserver module. The revised code employs zone
files and a limited amount of custom logic, which massively improves
test readability and maintainability, extends logging, and fixes
non-compliant replies sent by some of the custom servers in response to
certain queries (e.g. AA=0 in authoritative empty non-terminal
responses, non-glue address records in ADDITIONAL section).
There was some code in dns_resolver unit meant to keep compatibility with BIND 8 breaking the DNS protocol. These should not be needed anymore.
Merge branch 'ondrej/resolver-bind-8-cleanup' into 'main'
See merge request isc-projects/bind9!10270
