Some of the Intel NIC drivers poll the transmit and receive rings only
when the interrupt status indicates that a completion ought to be
available.
When interrupt moderation is enabled, this can cause us to run out of
descriptors since both rings have only a small fill level. For the
virtual function driver, there may be no way to disable interrupt
moderation since this is at the discretion of the PF driver.
Polling the descriptor rings is a very cheap operation. Poll both
rings unconditionally, so that we continue to make progress even when
interrupts are not being delivered as expected.
With this change, the VMware-specific errata workaround (to force
unconditional polling of the receive ring) becomes redundant. Remove
this workaround.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
At some unknown point, the Intel 82599 PF driver has ceased to support
the use of "legacy" format receive descriptors by the VF driver, and
will unconditionally write back "advanced" format descriptors despite
the SRRCTL.DESCTYPE value indicating otherwise.
Add an abstraction to allow for parsing the length and status from an
advanced format descriptor, and use this in the intelxvf driver.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The HVM MMIO BAR contains no actual MMIO registers, and exists only to
provide a chunk of guest physical address space that can be mapped via
xenmem_add_to_physmap().
The existing code relies on virt_to_phys() to calculate the physical
frame numbers (PFNs) to be mapped. This fails on platforms such as
x86_64 BIOS where virt_to_phys() is undefined for MMIO space.
Fix by mapping the whole MMIO BAR once at probe time, allowing a
virtual address within the mapping to be used to determine the
underlying physical address.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The BIOS on some older AWS instance types (observed with c3.large in
eu-central-1) does not correctly preserve the upper halves of the
32-bit general-purpose registers. In particular, calling INT13,42 or
INT13,43 will zero the high half of %ebx. If the compiled code has a
live value in %ebx then this will result in undefined behaviour.
The defect appears to be AWS-specific: it does not exist in the
upstream Bochs BIOS code from which this BIOS is derived.
The iPXE disk log console is not performance-sensitive but must be as
reliable as possible, since it exists to be the debug log mechanism of
last resort. We therefore choose the path of maximum paranoia: we
work around this and other potential BIOS bugs by preserving all
non-output registers across all real-mode calls.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The error code (if any) from INT13,42 and INT13,43 is returned in %ah.
The code currently erroneously checks the value in %al, which happens
to work only because BIOSes will leave %al unmodified (i.e. containing
zero) and the operation is extremely unlikely to ever actually fail.
Fix by reading the error code from %ah instead of %al.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some older AWS instance types are listed as supporting both i386 and
x86_64. Ignore i386 when identifying a compatible image, since the
imported images are marked as being for x86_64 only.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Running tests across every instance type would be prohibitively
expensive and mostly redundant: it is unlikely that two instance types
that differ only in their vCPU or memory allocations will behave
differently.
Add a utility that can be used to verify that a predefined list of
instance types (matching those for which tests will be run) represents
a comprehensive subset of all instance types available within AWS.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Responsibility for validating the requested geometry currently lies
with the caller of fbcon_init(), which makes it non-obvious that (for
example) the pixel width is guaranteed to be large enough for the
requested width.
Validate all console configuration parameters against the pixel
geometry within fbcon_init(), since this is the function that performs
calculations based on them.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Wait for the full login response PDU (including any padding) to arrive
before closing and reopening the connection for a redirection.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Include a test that boots into the Fedora live installer kernel and
initramfs, with a small control script injected into the initramfs
that uploads a zero-length flag file to S3 and then powers down the
instance.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Support for 32-bit i386 is gradually fading away across the ecosystem.
It is reasonably common to encounter devices with 64-bit MMIO BARs in
systems that place these BARs above the 32-bit address space. These
devices cannot be used by a 32-bit build of iPXE.
The default binaries published in both the rolling release and the
tagged releases are already x86_64 rather than i386. The only
remaining trace of the 32-bit default is the unadorned "bin/" build
directory prefix, which is currently interpreted as being equivalent
to "bin-i386-pcbios/".
Switch the default build architecture from i386 to x86_64.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
In legacy mode when checking if MSI-X is enabled to calculate the
device-specific region offset, the code accesses the base of the MSI-X
capability structure instead of the Message Control register and, if
the Next Pointer value exceeds 0x80, the subsequent check of
PCI_MSIX_CTRL_ENABLE mask incorrectly evaluates to true.
This false positive forces an erroneous 24-byte base offset, shifting
subsequent configuration reads by 4 bytes. This manifests as a
mangled MAC address during device probe (e.g. 34:56:01:00:ff:ff).
Fix the read offset to target the Message Control register.
Reported-by: Kent Konishi <konishi.kento@fujitsu.com>
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add a utility that can be used to perform a fully automated test of a
list of AWS instance types.
Suitable regions and AMIs are identified automatically. Results are
reported by using iPXE to upload zero-length flag files to an S3
bucket using pre-signed URIs passed in the userdata script.
We use single-instance fleets rather than standalone instances. This
allows us to specify an expiry time after which any instances will be
terminated automatically, even if the test script itself crashes.
This initial version of the test script verifies the ability to
download a 1MB test file (created by the test itself and placed in the
same S3 bucket), and verifies that the interface survives being closed
and reopened.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The digest commands are useful for performing automated tests, and
cloud boot images are not size-constrained.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The imported AMI displays its commit ID as part of the startup banner,
but there is currently no sensible way to retrieve this information
without booting the image.
Add a "commit" tag to hold the commit ID from which an imported AMI
was built, to allow automated tests to unambiguously identify the
commit being tested.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The .note.GNU-stack section is an irritant that we include only to
silence spurious warnings from the linker.
GNU binutils 2.47 changes the type of this section from "progbits" to
the arguably more correct "note", which causes build failures due to
the existing explicit "progbits" declarations.
Remove the section type declaration, allowing the toolchain to set the
section type to whatever it will later insist upon having.
Reported-by: Russ Kubes <rkubes@users.noreply.github.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Experiments show that Windows peers will randomly and erroneously fail
retrieval protocol requests made over IPv6, reporting that the block
is not present (by setting SizeOfBlock=0 in the MSG_BLK response).
The exact same request made over IPv4 succeeds 100% of the time.
Record the most recently discovered peer separately for each discovery
socket, and attempt retrieval from the most recently discovered IPv4
peer before attempting retrieval from the most recently discovered
IPv6 peer. (In theory, an IPv6 address could be reported via the IPv4
discovery socket (or vice versa), but experiments show that Windows
will report only addresses within the discovery socket family.)
Clear the most recently discovered peer when the discovery socket is
closed, since this is now a natural point at which to do so.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The peer location strings (i.e. the peer address for HTTP POST, or the
base URI for HTTP GET or local filesystem) are theoretically unbounded
in size. Allocate these on the heap rather than on the stack.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Separate the profilers used to record retrieval protocol downloads
using HTTP POST (i.e. downloads from a real peer, or from a hosted
cache server) from retrieval protocol downloads using HTTP GET or
equivalent (i.e. most likely coming from a local filesystem cache).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
In theory, a PeerDist client should perform discovery for each
segment, and then attempt retrieval from any discovered peers.
In practice, the segment sizes chosen by the origin server are much
too small for this theoretically correct approach to be viable. With
the typically observed PeerDist v2 segment size of around 64kB, the
total time to retrieve a segment from a peer on a gigabit LAN is
around 0.6ms. This is almost two orders of magnitude less than the
average peer discovery response delay of ~30ms recommended by the
MS-PCCRD specification. This peer discovery response delay becomes
the dominant limiting factor in the overall download speed.
If a peer is already known to have one segment of a file, then there
is an extremely high probability that it will have other segments of
the same file. We therefore optimistically attempt to retrieve a new
segment from whichever peer happens to have most recently responded to
a discovery request for any other segment, while simultaneously
sending out a discovery request for the new segment. This allows the
overall download to progress without being limited by the peer
discovery response delay.
This introduces a new problem: we end up sending out a large volume of
discovery requests that will end up being ignored because the segment
has already been fully retrieved before any peer sends a discovery
response. Each discovery attempt generates around 1-10 network
packets (depending on the number of peers): this would be negligible
for sensibly sized segments of around 2MB but represents an increase
in traffic of up to 25% for the absurdly small 64kB segments typically
observed from a PeerDist v2 server.
Discovery requests are multicast XML documents, and impose a
non-negligible processing burden on each listening peer. With a 64kB
segment size, the amount of processing required to parse and respond
to a discovery request vastly exceeds the amount of processing
required to just send the segment via the retrieval protocol.
Fix by adding a delay before sending the initial discovery request for
each new segment, in order to elide the sending of almost all
discovery requests.
When discovery is redundant (i.e. when we will have optimistically
retrieved the whole segment before any discovery responses might
arrive), this initial delay is long enough to allow the pending
discovery request to be cancelled before it is ever sent.
When discovery is genuinely valuable (i.e. when our optimistic
assumption that the most recently discovered peer also has the new
segment turns out to be incorrect), this initial delay still allows
plenty of time for discovery to take place before we time out and
attempt an origin server request.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The temporary "user:password" string constructed for HTTP Basic
authentication is theoretically unbounded in size. Allocate this on
the heap rather than on the stack.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Profile the overall duration of each discovery attempt and the time
until the first peer response, to provide some visibility into the
overall discovery behaviour.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The emptiness or otherwise of the list of peers has not been a
meaningful test for whether or not discovery replies were received
ever since commit 9b6ad2d ("[peerdist] Assume that most recently
discovered peer can be reused").
Record the number of discovery replies received, and use this as the
test when deciding whether or not to reduce the discovery timeout.
Ignore any errors arising when reporting a discovered peer: the return
status from peerdisc_discovered() is relevant only when adding the
hosted cache server (since if a hosted cached server is defined then
no discovery will ever take place).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Separate the profilers used to record retrieval protocol downloads
from the profilers used to record raw block downloads from the origin
server.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
We currently send a Content-Length request header only when the
content length is non-zero. This breaks some use cases for a PUT
request, such as uploading a zero-length file to AWS S3.
Fix by choosing whether or not to include a Content-Length request
header based on the request method.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some strict PE parsers will refuse a debug directory that lies within
the file headers (rather than within a defined section), and will fail
to load the binary.
Place the debug directory at the start of the .rodata section for
iPXE, and at the start of the .data section for wimboot.
The debug directory irritatingly includes an RVA and a file offset,
neither of which can be calculated by the linker. Update these fields
at the time of running elf2efi, after having calculated the file
offsets for each section.
Debugged-by: Kansu Dincer <kdincer@microsoft.com>
Debugged-by: Christian Nilsson <nikize@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The MS-PCCRR specification defines an HTTP POST request/response pair
for retrieving an encrypted block from a peer (with the decryption
keys provided separately via the content information).
The HTTP POST request parameters serve only to identify the block: the
actual response body is effectively a static encrypted blob.
Define an additional (non-standard) retrieval protocol in which the
block is identified solely using the request URI, with the response
being the same content that would be returned as the HTTP POST
response body. This allows for encrypted blocks to be retrieved from
a static source such as AWS S3 or a local FAT filesystem, without
requiring a peer that can understand the retrieval protocol HTTP POST
request format.
Define the static request URI format as:
<base>/xx/xxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy-b.blk
where
- `xxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy` is the segment ID (HoHoDK)
as a lower-case hexadecimal string
- `xx` is the first two characters (i.e. the first byte) of the
segment ID
- `b` is the block index within the segment (which will always
be zero when using MS-PCCRC version 2 content information), as
an unpadded decimal string
- the extension `.blk` represents a block file
This path format is designed to allow for efficient storage in a local
FAT filesystem (by limiting both the number of directories and the
number of entries within each directory).
Allow a base cache directory to be specified via the ${peerpath}
setting, e.g.
set peerpath file:/peercache
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When the server does not want to use the key exchange algorithm
offered in our ClientHello, it may respond with a HelloRetryRequest
that specifies a new key exchange algorithm.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Handling a TLS version 1.3 HelloRetryRequest will require processing
two ServerHello messages, the second of which must not reset the
running transcript digest (and is not permitted to change the selected
cipher suite).
Move the responsibility for setting the protocol version alongside the
responsibility for setting the cipher suite in tls_select_cipher(),
and refuse to change either protocol version or cipher suite if they
have already been selected.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For a renegotiation, we currently rely on the previously selected
cipher suite to determine the length of the verification data. This
is the only context in which the recorded cipher suite does not
represent the selected cipher suite for the current connection.
Record the verification length separately as part of the verification
data structure, allowing the selected cipher suite to be cleared when
restarting the connection so that it unambiguously represents the
currently selected cipher suite at all times.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Generalise the mechanism used for storing a session ticket to allow
for storing copies of arbitrary TLS cursors.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
TLS currently uses free() instead of zfree() where the memory being
freed contains no secrets. The cost of using zfree() everywhere is
negligible (no code size increase, and used only in non-fast-path
operations), and avoids the need to reason about whether or not the
memory contains secrets.
Switch to using zfree() unconditionally throughout the TLS code.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
On receiving a HelloRetryRequest, the running transcript hash gets
replaced with a synthetic message of type "message_hash" containing
the current value of the running transcript hash (which at that point
covers only the initial ClientHello).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Using a CBC-mode cipher with a length that is not a multiple of the
block size will cause an uncontrollable out-of-bounds write of the
entire address space, resulting in a guaranteed crash. There is no
way to cause the CBC cipher to return before overwriting the entire
address space with pseudorandom data, and so this could only be used
as a denial-of-service attack.
Almost all cipher call sites already validate the length against the
block size. Add the relevant checks to the two call sites that do not
already do so.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Constructing exhaustive standalone tests for all TLS structures is
time-consuming and of limited value, since there would be no guarantee
that the construction logic used in the test case matched the
construction logic matched in the real TLS protocol engine.
We already exercise each failure path that can be triggered by genuine
runtime errors (e.g. malformed received data, or transmit data that is
too large to fit within the relevant length field). This leaves
untested the other major class of errors: an invalid structure
descriptor mapping (e.g. a missing field description) that would
render the structure impossible to parse or build.
Exercise the parser and builder for every structure type by creating
an all-zero descriptor (which is always permitted), sizing and
building the empty structure, and then parsing that structure. This
ensures that the parser and builder are both satisfied that the
descriptor mapping is valid, since the validity of the mapping is
independent of the data being built or parsed.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The recorded key exchange group is selected at initialisation and can
never be set to NULL, in the same way that the cipher suite can never
be set to NULL (and is therefore not checked at the points of use).
Remove the redundant null pointer check, since it simply serves to
confuse readers of the code.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
TLS version 1.3 defers the NewSessionTicket until after the handshake
has completed, and allows the server to send multiple NewSessionTicket
messages (e.g. to refresh short-lived tickets).
The default session ID generated in tls_save() will currently end up
with the same value on each invocation. While session IDs are unused
in TLS version 1.3, this could lead to confusing debug messages and
packet captures.
Add a function tls_random() to generate random nonces using the
channel ephemeral secret, a label, and a 64-bit counter, and this
function to generate unique session IDs.
Non-sequential record IVs are also generated using channel ephemeral
secrets. Simplify this code by using tls_random() instead of a direct
call to channel_ephemeral().
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The initial ClientHello is sent and digested before the key schedule
has been started (since the digest algorithm is not known until the
ServerHello arrives).
A server that sends a premature ServerHello with an all-zero nonce
before the ClientHello is sent can currently cause the key schedule's
"nonced" flag to become set. The flag will be reset when the key
schedule is started and so this is unclean but harmless.
Fix by ignoring any data that arrives before the digest algorithm has
been set (i.e. before the key schedule has been started), since the
null digest algorithm cannot meaningfully incorporate anything into a
running transcript digest.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
If the server switches from providing stateless session tickets to
providing stateful session IDs, then we will currently end up with a
stored session that retains the old session ticket along with the new
session ID. (This behaviour has been observed when connecting to an
Apache server that requires per-directory client authentication or
per-directory cipher suites: the server originally provides a
stateless session ticket but then, after renegotiation to request the
client certificate, provides a stateful session ID instead.)
Reset both the session ID and the session ticket whenever a new
session is saved (i.e. if the server provides either a new session ID
or a new session ticket), so that we do not retain stale credentials.
If no new session ID is provided, then create a new random session ID
(rather than leaving the session with an empty session ID) since we
rely on the echoed non-empty session ID to determine whether or not
the server is choosing to resume, even for stateless sessions.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Define structure descriptors and mappings for every data structure
currently sent by the TLS protocol engine, and use the standalone data
structure builder to remove a large amount of open-coded layout and
assembly logic.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For TLS version 1.1 (without explicit signature hash algorithm
identifiers), we currently use the cipher suite's public key algorithm
when verifying the ServerKeyExchange signature. This is indirectly
guaranteed to match the server certificate's own public key algorithm.
Use the server certificate's own public key algorithm directly, to be
consistent with the behaviour for client certificates. The signed
certificate is, by definition, the authoritative source for its own
choice of public key algorithm.
This leaves the public key algorithm aspect of the cipher suite as
being unused outside of debug messages. We do not enforce the
specification that the certificate's public key algorithm must match
the cipher suite's public key algorithm (if any) because doing so
serves no cryptographic purpose. For TLS version 1.2 and later, the
explicit signature hash algorithm identifiers override the cipher
suite, and in TLS version 1.3 the whole concept of a public key
algorithm is removed from the scope of the cipher suite.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Mirroring commit cbdb572 ("[crypto] Use certificate's own public key
algorithm for key matching"), the client CertificateVerify is also
currently constructed using the certificate's signature algorithm
(i.e. the public key algorithm of the issuer's key) rather than the
certificate's own public key algorithm.
Fix by using the certificate's own public key algorithm.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For TLS version 1.1, non-RSA signed digests use SHA-1 instead of
MD5+SHA1. Commit f095adb ("[tls] Use SHA-1 for TLS version 1.1 ECDSA
signatures") selected the correct digest algorithm for both server and
client authentication.
However, the key schedule currently refuses to generate client
CertificateVerify digests for any digest algorithm other than
MD5+SHA1, on the basis that only the MD5+SHA1 running transcript
digest value is available.
An MD5+SHA1 digest value is just the concatenation of an MD5 digest
value with a SHA-1 digest value, and so the SHA-1 digest value can be
provided for use with ECDSA client certificates.
Fix by special-casing the SHA-1 (and MD5) algorithms when generating a
signable digest value from the TLS version 1.1 key schedule.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When finding the certificate corresponding to a private key, the match
is currently performed using the certificate's signature algorithm
(i.e. the public key algorithm of the issuer's key) rather than the
certificate's own public key algorithm. This breaks key matching for
heterogenous certificate chains (e.g. an ECDSA client certificate
issued by an RSA intermediate certificate).
Fix by using the certificate's own public key algorithm.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The existing ad hoc code that builds TLS data structures generally
performs a single length calculation and a single allocation, and if
the allocation succeeds then the data structure will be filled in with
no possible further errors arising.
Allow this pattern to be replicated when using the generic builder, by
ensuring that an error in calculating the length of a contained data
structure (e.g. an extension) will automatically propagate that error
to the calculation of the length of the containing structure (e.g. a
ClientHello that contains the extension).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Provide a generic builder that can construct an arbitrary TLS data
structure based upon the same descriptors and binary-encoded mappings
as used for the generic parser.
The design allows for the caller to either provide data in advance or
to write through the descriptor pointers after building the data
structure. For example: this will allow the public key within a
ClientHello to be populated in situ within the key_share extension,
rather than requiring the caller to allocate and populate a temporary
buffer before building the ClientHello.
This necessitates removing the "const" from all pointers within the
descriptors, which is a worthwhile tradeoff for the sake of avoiding a
large number of temporary allocations.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
