The SOAP-over-UDP specification states that for each retransmission
the value of the message ID property must be the same. We currently
use a single (weakly) random message ID for all transmissions used to
discover a given segment.
Experiments suggest that Windows performs some deduplication of probe
messages and will respond to either the IPv4 or the IPv6 probe but not
to both, if they arrive with the same message ID in quick succession.
This undermines our attempt to prefer IPv4 peers: we can quite easily
end up receiving only IPv6 discovery responses.
Treat the IPv4 and IPv6 discovery probes as separate messages: use a
(trivially) different message ID for each address family but then
retain that message ID for each retransmission as per the spec.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
PeerDist individual block downloads may complete out of order, causing
the download progress indicator to occasionally jump backwards.
Use the block multiplexer's current segment index and the total number
of segments to construct the job progress indicator, since this is
guaranteed to increase monotonically.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Each PeerDist-encoded downloaded file typically results in several
thousand individual block downloads, each of which may end up being
attempted as a classic retrieval from a discovered peer, a retrieval
from an optimistically assumed peer, a retrieval from a hosted cache
server, a read from a local filesystem cache, and/or a raw range
request from the origin server.
The sheer volume of activity and the number of errors expected in
normal operation (e.g. for blocks missing from a particular peer) make
it difficult to use debug messages to gain insights into behaviour,
since per-event debug messages are either too rare to be informative
or so frequent that the overhead of printing the messages alters the
overall behaviour.
Gather error statistics separately for each category of individual
block download and for each discovery address family, along with the
most useful summary counts (e.g. for total blocks, total number of
discovery attempts, etc).
Estimate the total number of peers as being the highest number of
peers ever discovered for a segment within a single address family.
(We count per address family to avoid double-counting a single peer
that advertises itself using both IPv4 and IPv6 addresses.)
Report the number of discovery attempts independent of address family
(i.e. one discovery attempt means N transmitted packets, one sent via
each of the N supported address families), since that number is then
meaningfully comparable against the total number of blocks.
Report second and subsequent discovery attempts separately from
initial discovery attempts, so that the number of discovery attempts
can be compared against the number of blocks to determine how many
discoveries were elided completely.
Separate the discovery reply statistics by address family, not because
there is anything useful in knowing whether a particular error
happened for IPv4 versus IPv6, but because it avoids the need to
reason about whether or not a displayed result needs to be mentally
halved in order to be compared against the corresponding discovery
attempt statistic.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The peer count reported via the job progress message has been largely
meaningless ever since commit 9b6ad2d ("[peerdist] Assume that most
recently discovered peer can be reused").
Remove this obsolete and misleading mechanism for reporting peer
statistics.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
There are two checks that we perform to see whether a PCI device
exists at a given bus:dev.fn address. We check to see if the vendor
and device IDs read as either all-zeros or all-ones, and for a
non-zero function number we also check that the device is actually a
multi-function device by reading the header type from the device's
function zero.
For historical reasons, the multi-function header type check is
currently performed before the existence check. (The code originally
supported only a one-time linear enumeration of the PCI bus, and so
the header type for function zero was already known: checking the
header type first therefore used to allow the read of the vendor and
device IDs to be elided completely.)
On some UEFI systems (observed on an AWS a1.metal instance in
eu-central-1), attempting to read the vendor and device IDs for a
non-existent device will work as expected, but attempting to read the
header type will lock up the system. (No further diagnostics are
available: the a1.metal instance type has no serial port and does not
support taking a screenshot, and so the only available output is from
the iPXE disk log console.)
Work around this curious defect by swapping the order of the existence
and header type checks.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The top-level download.fedoraproject.org server seems to have been
reconfigured to be unable to redirect requests for individual files to
an appropriate mirror.
Work around this reconfiguration by using an explicit mirror.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The externally contributed commit 1d04900 ("[intel] Reset all virtual
function settings") seems to break the 82599 virtual function NICs as
found within some older AWS instance types (observed on a c3.large
instance in eu-central-1). If the transmit ring tail (VFTDT[0]) has a
non-zero value (i.e. if any packets have been transmitted), then
writing a zero to the transmit ring tail after the ring has been
disabled will permanently lock up the transmit datapath in a way that
survives even a VF reset.
Zeroing the head and tail registers when the interface is closed
appears to be completely redundant: they will be cleared anyway by the
virtual function reset (VFCTRL.RST), and they will also be explicitly
zeroed when the interface is reopened. The Linux ixgbevf driver does
not include any equivalent code to explicitly zero these registers
when the interface is closed.
Fix by leaving the head and tail registers untouched when the
interface is closed.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some of the Intel NIC drivers poll the transmit and receive rings only
when the interrupt status indicates that a completion ought to be
available.
When interrupt moderation is enabled, this can cause us to run out of
descriptors since both rings have only a small fill level. For the
virtual function driver, there may be no way to disable interrupt
moderation since this is at the discretion of the PF driver.
Polling the descriptor rings is a very cheap operation. Poll both
rings unconditionally, so that we continue to make progress even when
interrupts are not being delivered as expected.
With this change, the VMware-specific errata workaround (to force
unconditional polling of the receive ring) becomes redundant. Remove
this workaround.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
At some unknown point, the Intel 82599 PF driver has ceased to support
the use of "legacy" format receive descriptors by the VF driver, and
will unconditionally write back "advanced" format descriptors despite
the SRRCTL.DESCTYPE value indicating otherwise.
Add an abstraction to allow for parsing the length and status from an
advanced format descriptor, and use this in the intelxvf driver.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The HVM MMIO BAR contains no actual MMIO registers, and exists only to
provide a chunk of guest physical address space that can be mapped via
xenmem_add_to_physmap().
The existing code relies on virt_to_phys() to calculate the physical
frame numbers (PFNs) to be mapped. This fails on platforms such as
x86_64 BIOS where virt_to_phys() is undefined for MMIO space.
Fix by mapping the whole MMIO BAR once at probe time, allowing a
virtual address within the mapping to be used to determine the
underlying physical address.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The BIOS on some older AWS instance types (observed with c3.large in
eu-central-1) does not correctly preserve the upper halves of the
32-bit general-purpose registers. In particular, calling INT13,42 or
INT13,43 will zero the high half of %ebx. If the compiled code has a
live value in %ebx then this will result in undefined behaviour.
The defect appears to be AWS-specific: it does not exist in the
upstream Bochs BIOS code from which this BIOS is derived.
The iPXE disk log console is not performance-sensitive but must be as
reliable as possible, since it exists to be the debug log mechanism of
last resort. We therefore choose the path of maximum paranoia: we
work around this and other potential BIOS bugs by preserving all
non-output registers across all real-mode calls.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The error code (if any) from INT13,42 and INT13,43 is returned in %ah.
The code currently erroneously checks the value in %al, which happens
to work only because BIOSes will leave %al unmodified (i.e. containing
zero) and the operation is extremely unlikely to ever actually fail.
Fix by reading the error code from %ah instead of %al.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some older AWS instance types are listed as supporting both i386 and
x86_64. Ignore i386 when identifying a compatible image, since the
imported images are marked as being for x86_64 only.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Running tests across every instance type would be prohibitively
expensive and mostly redundant: it is unlikely that two instance types
that differ only in their vCPU or memory allocations will behave
differently.
Add a utility that can be used to verify that a predefined list of
instance types (matching those for which tests will be run) represents
a comprehensive subset of all instance types available within AWS.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Responsibility for validating the requested geometry currently lies
with the caller of fbcon_init(), which makes it non-obvious that (for
example) the pixel width is guaranteed to be large enough for the
requested width.
Validate all console configuration parameters against the pixel
geometry within fbcon_init(), since this is the function that performs
calculations based on them.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Wait for the full login response PDU (including any padding) to arrive
before closing and reopening the connection for a redirection.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Include a test that boots into the Fedora live installer kernel and
initramfs, with a small control script injected into the initramfs
that uploads a zero-length flag file to S3 and then powers down the
instance.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Support for 32-bit i386 is gradually fading away across the ecosystem.
It is reasonably common to encounter devices with 64-bit MMIO BARs in
systems that place these BARs above the 32-bit address space. These
devices cannot be used by a 32-bit build of iPXE.
The default binaries published in both the rolling release and the
tagged releases are already x86_64 rather than i386. The only
remaining trace of the 32-bit default is the unadorned "bin/" build
directory prefix, which is currently interpreted as being equivalent
to "bin-i386-pcbios/".
Switch the default build architecture from i386 to x86_64.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
In legacy mode when checking if MSI-X is enabled to calculate the
device-specific region offset, the code accesses the base of the MSI-X
capability structure instead of the Message Control register and, if
the Next Pointer value exceeds 0x80, the subsequent check of
PCI_MSIX_CTRL_ENABLE mask incorrectly evaluates to true.
This false positive forces an erroneous 24-byte base offset, shifting
subsequent configuration reads by 4 bytes. This manifests as a
mangled MAC address during device probe (e.g. 34:56:01:00:ff:ff).
Fix the read offset to target the Message Control register.
Reported-by: Kent Konishi <konishi.kento@fujitsu.com>
Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add a utility that can be used to perform a fully automated test of a
list of AWS instance types.
Suitable regions and AMIs are identified automatically. Results are
reported by using iPXE to upload zero-length flag files to an S3
bucket using pre-signed URIs passed in the userdata script.
We use single-instance fleets rather than standalone instances. This
allows us to specify an expiry time after which any instances will be
terminated automatically, even if the test script itself crashes.
This initial version of the test script verifies the ability to
download a 1MB test file (created by the test itself and placed in the
same S3 bucket), and verifies that the interface survives being closed
and reopened.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The digest commands are useful for performing automated tests, and
cloud boot images are not size-constrained.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The imported AMI displays its commit ID as part of the startup banner,
but there is currently no sensible way to retrieve this information
without booting the image.
Add a "commit" tag to hold the commit ID from which an imported AMI
was built, to allow automated tests to unambiguously identify the
commit being tested.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The .note.GNU-stack section is an irritant that we include only to
silence spurious warnings from the linker.
GNU binutils 2.47 changes the type of this section from "progbits" to
the arguably more correct "note", which causes build failures due to
the existing explicit "progbits" declarations.
Remove the section type declaration, allowing the toolchain to set the
section type to whatever it will later insist upon having.
Reported-by: Russ Kubes <rkubes@users.noreply.github.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Experiments show that Windows peers will randomly and erroneously fail
retrieval protocol requests made over IPv6, reporting that the block
is not present (by setting SizeOfBlock=0 in the MSG_BLK response).
The exact same request made over IPv4 succeeds 100% of the time.
Record the most recently discovered peer separately for each discovery
socket, and attempt retrieval from the most recently discovered IPv4
peer before attempting retrieval from the most recently discovered
IPv6 peer. (In theory, an IPv6 address could be reported via the IPv4
discovery socket (or vice versa), but experiments show that Windows
will report only addresses within the discovery socket family.)
Clear the most recently discovered peer when the discovery socket is
closed, since this is now a natural point at which to do so.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The peer location strings (i.e. the peer address for HTTP POST, or the
base URI for HTTP GET or local filesystem) are theoretically unbounded
in size. Allocate these on the heap rather than on the stack.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Separate the profilers used to record retrieval protocol downloads
using HTTP POST (i.e. downloads from a real peer, or from a hosted
cache server) from retrieval protocol downloads using HTTP GET or
equivalent (i.e. most likely coming from a local filesystem cache).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
In theory, a PeerDist client should perform discovery for each
segment, and then attempt retrieval from any discovered peers.
In practice, the segment sizes chosen by the origin server are much
too small for this theoretically correct approach to be viable. With
the typically observed PeerDist v2 segment size of around 64kB, the
total time to retrieve a segment from a peer on a gigabit LAN is
around 0.6ms. This is almost two orders of magnitude less than the
average peer discovery response delay of ~30ms recommended by the
MS-PCCRD specification. This peer discovery response delay becomes
the dominant limiting factor in the overall download speed.
If a peer is already known to have one segment of a file, then there
is an extremely high probability that it will have other segments of
the same file. We therefore optimistically attempt to retrieve a new
segment from whichever peer happens to have most recently responded to
a discovery request for any other segment, while simultaneously
sending out a discovery request for the new segment. This allows the
overall download to progress without being limited by the peer
discovery response delay.
This introduces a new problem: we end up sending out a large volume of
discovery requests that will end up being ignored because the segment
has already been fully retrieved before any peer sends a discovery
response. Each discovery attempt generates around 1-10 network
packets (depending on the number of peers): this would be negligible
for sensibly sized segments of around 2MB but represents an increase
in traffic of up to 25% for the absurdly small 64kB segments typically
observed from a PeerDist v2 server.
Discovery requests are multicast XML documents, and impose a
non-negligible processing burden on each listening peer. With a 64kB
segment size, the amount of processing required to parse and respond
to a discovery request vastly exceeds the amount of processing
required to just send the segment via the retrieval protocol.
Fix by adding a delay before sending the initial discovery request for
each new segment, in order to elide the sending of almost all
discovery requests.
When discovery is redundant (i.e. when we will have optimistically
retrieved the whole segment before any discovery responses might
arrive), this initial delay is long enough to allow the pending
discovery request to be cancelled before it is ever sent.
When discovery is genuinely valuable (i.e. when our optimistic
assumption that the most recently discovered peer also has the new
segment turns out to be incorrect), this initial delay still allows
plenty of time for discovery to take place before we time out and
attempt an origin server request.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The temporary "user:password" string constructed for HTTP Basic
authentication is theoretically unbounded in size. Allocate this on
the heap rather than on the stack.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Profile the overall duration of each discovery attempt and the time
until the first peer response, to provide some visibility into the
overall discovery behaviour.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The emptiness or otherwise of the list of peers has not been a
meaningful test for whether or not discovery replies were received
ever since commit 9b6ad2d ("[peerdist] Assume that most recently
discovered peer can be reused").
Record the number of discovery replies received, and use this as the
test when deciding whether or not to reduce the discovery timeout.
Ignore any errors arising when reporting a discovered peer: the return
status from peerdisc_discovered() is relevant only when adding the
hosted cache server (since if a hosted cached server is defined then
no discovery will ever take place).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Separate the profilers used to record retrieval protocol downloads
from the profilers used to record raw block downloads from the origin
server.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
We currently send a Content-Length request header only when the
content length is non-zero. This breaks some use cases for a PUT
request, such as uploading a zero-length file to AWS S3.
Fix by choosing whether or not to include a Content-Length request
header based on the request method.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some strict PE parsers will refuse a debug directory that lies within
the file headers (rather than within a defined section), and will fail
to load the binary.
Place the debug directory at the start of the .rodata section for
iPXE, and at the start of the .data section for wimboot.
The debug directory irritatingly includes an RVA and a file offset,
neither of which can be calculated by the linker. Update these fields
at the time of running elf2efi, after having calculated the file
offsets for each section.
Debugged-by: Kansu Dincer <kdincer@microsoft.com>
Debugged-by: Christian Nilsson <nikize@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The MS-PCCRR specification defines an HTTP POST request/response pair
for retrieving an encrypted block from a peer (with the decryption
keys provided separately via the content information).
The HTTP POST request parameters serve only to identify the block: the
actual response body is effectively a static encrypted blob.
Define an additional (non-standard) retrieval protocol in which the
block is identified solely using the request URI, with the response
being the same content that would be returned as the HTTP POST
response body. This allows for encrypted blocks to be retrieved from
a static source such as AWS S3 or a local FAT filesystem, without
requiring a peer that can understand the retrieval protocol HTTP POST
request format.
Define the static request URI format as:
<base>/xx/xxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy-b.blk
where
- `xxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy` is the segment ID (HoHoDK)
as a lower-case hexadecimal string
- `xx` is the first two characters (i.e. the first byte) of the
segment ID
- `b` is the block index within the segment (which will always
be zero when using MS-PCCRC version 2 content information), as
an unpadded decimal string
- the extension `.blk` represents a block file
This path format is designed to allow for efficient storage in a local
FAT filesystem (by limiting both the number of directories and the
number of entries within each directory).
Allow a base cache directory to be specified via the ${peerpath}
setting, e.g.
set peerpath file:/peercache
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When the server does not want to use the key exchange algorithm
offered in our ClientHello, it may respond with a HelloRetryRequest
that specifies a new key exchange algorithm.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Handling a TLS version 1.3 HelloRetryRequest will require processing
two ServerHello messages, the second of which must not reset the
running transcript digest (and is not permitted to change the selected
cipher suite).
Move the responsibility for setting the protocol version alongside the
responsibility for setting the cipher suite in tls_select_cipher(),
and refuse to change either protocol version or cipher suite if they
have already been selected.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
For a renegotiation, we currently rely on the previously selected
cipher suite to determine the length of the verification data. This
is the only context in which the recorded cipher suite does not
represent the selected cipher suite for the current connection.
Record the verification length separately as part of the verification
data structure, allowing the selected cipher suite to be cleared when
restarting the connection so that it unambiguously represents the
currently selected cipher suite at all times.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Generalise the mechanism used for storing a session ticket to allow
for storing copies of arbitrary TLS cursors.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
TLS currently uses free() instead of zfree() where the memory being
freed contains no secrets. The cost of using zfree() everywhere is
negligible (no code size increase, and used only in non-fast-path
operations), and avoids the need to reason about whether or not the
memory contains secrets.
Switch to using zfree() unconditionally throughout the TLS code.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
On receiving a HelloRetryRequest, the running transcript hash gets
replaced with a synthetic message of type "message_hash" containing
the current value of the running transcript hash (which at that point
covers only the initial ClientHello).
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Using a CBC-mode cipher with a length that is not a multiple of the
block size will cause an uncontrollable out-of-bounds write of the
entire address space, resulting in a guaranteed crash. There is no
way to cause the CBC cipher to return before overwriting the entire
address space with pseudorandom data, and so this could only be used
as a denial-of-service attack.
Almost all cipher call sites already validate the length against the
block size. Add the relevant checks to the two call sites that do not
already do so.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Constructing exhaustive standalone tests for all TLS structures is
time-consuming and of limited value, since there would be no guarantee
that the construction logic used in the test case matched the
construction logic matched in the real TLS protocol engine.
We already exercise each failure path that can be triggered by genuine
runtime errors (e.g. malformed received data, or transmit data that is
too large to fit within the relevant length field). This leaves
untested the other major class of errors: an invalid structure
descriptor mapping (e.g. a missing field description) that would
render the structure impossible to parse or build.
Exercise the parser and builder for every structure type by creating
an all-zero descriptor (which is always permitted), sizing and
building the empty structure, and then parsing that structure. This
ensures that the parser and builder are both satisfied that the
descriptor mapping is valid, since the validity of the mapping is
independent of the data being built or parsed.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The recorded key exchange group is selected at initialisation and can
never be set to NULL, in the same way that the cipher suite can never
be set to NULL (and is therefore not checked at the points of use).
Remove the redundant null pointer check, since it simply serves to
confuse readers of the code.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
TLS version 1.3 defers the NewSessionTicket until after the handshake
has completed, and allows the server to send multiple NewSessionTicket
messages (e.g. to refresh short-lived tickets).
The default session ID generated in tls_save() will currently end up
with the same value on each invocation. While session IDs are unused
in TLS version 1.3, this could lead to confusing debug messages and
packet captures.
Add a function tls_random() to generate random nonces using the
channel ephemeral secret, a label, and a 64-bit counter, and this
function to generate unique session IDs.
Non-sequential record IVs are also generated using channel ephemeral
secrets. Simplify this code by using tls_random() instead of a direct
call to channel_ephemeral().
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The initial ClientHello is sent and digested before the key schedule
has been started (since the digest algorithm is not known until the
ServerHello arrives).
A server that sends a premature ServerHello with an all-zero nonce
before the ClientHello is sent can currently cause the key schedule's
"nonced" flag to become set. The flag will be reset when the key
schedule is started and so this is unclean but harmless.
Fix by ignoring any data that arrives before the digest algorithm has
been set (i.e. before the key schedule has been started), since the
null digest algorithm cannot meaningfully incorporate anything into a
running transcript digest.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
If the server switches from providing stateless session tickets to
providing stateful session IDs, then we will currently end up with a
stored session that retains the old session ticket along with the new
session ID. (This behaviour has been observed when connecting to an
Apache server that requires per-directory client authentication or
per-directory cipher suites: the server originally provides a
stateless session ticket but then, after renegotiation to request the
client certificate, provides a stateful session ID instead.)
Reset both the session ID and the session ticket whenever a new
session is saved (i.e. if the server provides either a new session ID
or a new session ticket), so that we do not retain stale credentials.
If no new session ID is provided, then create a new random session ID
(rather than leaving the session with an empty session ID) since we
rely on the echoed non-empty session ID to determine whether or not
the server is choosing to resume, even for stateless sessions.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Define structure descriptors and mappings for every data structure
currently sent by the TLS protocol engine, and use the standalone data
structure builder to remove a large amount of open-coded layout and
assembly logic.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
