Add E2EE for embedded mode of Element Call (#3667)

* WIP refactor for removing m.call events * Always remember rtcsessions since we need to only have one instance * Fix tests * Fix import loop * Fix more cyclic imports & tests * Test session joining * Attempt to make tests happy * Always leave calls in the tests to clean up * comment + desperate attempt to work out what's failing * More test debugging * Okay, so these ones are fine? * Stop more timers and hopefully have happy tests * Test no rejoin * Test malformed m.call.member events * Test event emitting and also move some code to a more sensible place in the file * Test getActiveFoci() * Test event emitting (and also fix it) * Test membership updating & pruning on join * Test getOldestMembership() * Test member event renewal * Don't start the rtc manager until the client has synced Then we can initialise from the state once it's completed. * Fix type * Remove listeners added in constructor * Stop the client here too * Stop the client here also also * ARGH. Disable tests to work out which one is causing the exception * Disable everything * Re-jig to avoid setting listeners in the constructor and re-enable tests * No need to rename this anymore * argh, remove the right listener * Is it this test??? * Re-enable some tests * Try mocking getRooms to return something valid * Re-enable other tests * Give up trying to get the tests to work sensibly and deal with getRooms() returning nothing * Oops, don't enable the ones that were skipped before * One more try at the sensible way * Didn't work, go back to the hack way. * Log when we manage to send the member event update * Support `getOpenIdToken()` in embedded mode (#3676) * Call `sendContentLoaded()` (#3677) * Start MatrixRTC in embedded mode (#3679) * Reschedule the membership event check * Bump widget api version * Add mock for sendContentLoaded() * Embeded mode pre-requisites Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Embeded mode E2EE Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Encryption condition Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Revert "Embeded mode pre-requisites" This reverts commit 8cd73702052609c995ad754e31f85d0da0be4aa9. * Get back event type Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> fds Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Change embedded E2EE implementation Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * More log detail * Fix tests and also better assert because the tests were passing undefined which was considered fine because we were only checking for null. * Simplify updateCallMembershipEvent a bit * Split up updateCallMembershipEvent some more * Use `crypto.getRandomValues()` Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Rename to `membershipToUserAndDeviceId()` Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Better error Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add log line Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add comment Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Send call ID in enc events (also a small refactor) Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Revert making `joinRoomSession()` async Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Make `client` `private` again Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Just use `toString()` Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix `callId` check Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix map Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix map compare Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix emitting Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Explicit logging Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Refactor Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Make `updateEncryptionKeyEvent()` public Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Only update keys based on others Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix call order Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Improve logging Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Avoid races Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Revert "Avoid races" This reverts commit f65ed72d6e. * Add try-catch Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Make `updateEncryptionKeyEvent()` private Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Handle indices and throttling Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Fix merge mistakes Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Mort post-merge fixes Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Split out key generation from key sending And send all keys in a key event (changes the format of the key event) rather than just the one we just generated. * Remember and clear the timeout for the send key event So we don't schedule more key updates if one is already pending. Also don't update the last sent time when we didn't actually send the keys. * Make key event resends more robust * Attempt to make tests pass * crypto wasn't defined at all * Hopefully get interface right * Fix key format on the wire to base64 * Add comment * More standard method order * Rename encryptMedia The js-sdk doesn't do media and therefore doesn't do media encryption * Stop logging encryption keys now * Use regular base64 It's not going in a URL, so no need * Re-add base64url randomstring was using it. Also give it a test. * Add tests for randomstring * Switch between either browser or node crypto Let's see if this will work... * Obviously crypto has already solved this * Some tests for MatrixRTCSession key stuff * Test keys object contents * Change keys event format To move away from m. keys * Test key event retries * Test onCallEncryption * Test event sending & spam prevention * Test event cancelation * Test onCallEncryption called * Some errors didn't have data * Fix binary key comparison & add log line * Fix compare function with undefined values * Remove more key logging * Check content.keys is an array * Check key index & key * Better function name * Tests too --------- Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> Co-authored-by: David Baker <dave@matrix.org> Co-authored-by: David Baker <dbkr@users.noreply.github.com>
2025-11-25 05:23:13 +03:00 · 2023-10-31 17:01:46 +01:00
parent 370dd6a0eb
commit bf81c4bfeb
11 changed files with 682 additions and 20 deletions
--- a/src/matrixrtc/MatrixRTCSession.ts
+++ b/src/matrixrtc/MatrixRTCSession.ts
@@ -22,12 +22,24 @@ import { MatrixClient } from "../client";
 import { EventType } from "../@types/event";
 import { CallMembership, CallMembershipData } from "./CallMembership";
 import { Focus } from "./focus";
-import { MatrixEvent } from "../matrix";
-import { randomString } from "../randomstring";
+import { MatrixError, MatrixEvent } from "../matrix";
+import { randomString, secureRandomBase64Url } from "../randomstring";
+import { EncryptionKeysEventContent } from "./types";
+import { decodeBase64, encodeUnpaddedBase64 } from "../base64";
+import { isNumber } from "../utils";

 const MEMBERSHIP_EXPIRY_TIME = 60 * 60 * 1000;
 const MEMBER_EVENT_CHECK_PERIOD = 2 * 60 * 1000; // How often we check to see if we need to re-send our member event
 const CALL_MEMBER_EVENT_RETRY_DELAY_MIN = 3000;
+const UPDATE_ENCRYPTION_KEY_THROTTLE = 3000;
+
+const getParticipantId = (userId: string, deviceId: string): string => `${userId}:${deviceId}`;
+const getParticipantIdFromMembership = (m: CallMembership): string => getParticipantId(m.sender!, m.deviceId);
+
+function keysEqual(a: Uint8Array, b: Uint8Array): boolean {
+    if (a === b) return true;
+    return a && b && a.length === b.length && a.every((x, i) => x === b[i]);
+}

 export enum MatrixRTCSessionEvent {
    // A member joined, left, or updated a property of their membership.
@@ -36,6 +48,8 @@ export enum MatrixRTCSessionEvent {
    // separate from MembershipsChanged, ie. independent of whether our member event
    // has succesfully gone through.
    JoinStateChanged = "join_state_changed",
+    // The key used to encrypt media has changed
+    EncryptionKeyChanged = "encryption_key_changed",
 }

 export type MatrixRTCSessionEventHandlerMap = {
@@ -44,6 +58,11 @@ export type MatrixRTCSessionEventHandlerMap = {
        newMemberships: CallMembership[],
    ) => void;
    [MatrixRTCSessionEvent.JoinStateChanged]: (isJoined: boolean) => void;
+    [MatrixRTCSessionEvent.EncryptionKeyChanged]: (
+        key: Uint8Array,
+        encryptionKeyIndex: number,
+        participantId: string,
+    ) => void;
 };

 /**
@@ -65,12 +84,18 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M

    private memberEventTimeout?: ReturnType<typeof setTimeout>;
    private expiryTimeout?: ReturnType<typeof setTimeout>;
+    private keysEventUpdateTimeout?: ReturnType<typeof setTimeout>;

    private activeFoci: Focus[] | undefined;

    private updateCallMembershipRunning = false;
    private needCallMembershipUpdate = false;

+    private manageMediaKeys = false;
+    // userId:deviceId => array of keys
+    private encryptionKeys = new Map<string, Array<Uint8Array>>();
+    private lastEncryptionKeyUpdateRequest?: number;
+
    /**
     * Returns all the call memberships for a room, oldest first
     */
@@ -175,18 +200,28 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M
     * This will not subscribe to updates: remember to call subscribe() separately if
     * desired.
     * This method will return immediately and the session will be joined in the background.
+     *
+     * @param activeFoci - The list of foci to set as currently active in the call member event
+     * @param manageMediaKeys - If true, generate and share a a media key for this participant,
+     *                          and emit MatrixRTCSessionEvent.EncryptionKeyChanged when
+     *                          media keys for other participants become available.
     */
-    public joinRoomSession(activeFoci: Focus[]): void {
+    public joinRoomSession(activeFoci: Focus[], manageMediaKeys?: boolean): void {
        if (this.isJoined()) {
            logger.info(`Already joined to session in room ${this.room.roomId}: ignoring join call`);
            return;
        }

-        logger.info(`Joining call session in room ${this.room.roomId}`);
+        logger.info(`Joining call session in room ${this.room.roomId} with manageMediaKeys=${manageMediaKeys}`);
        this.activeFoci = activeFoci;
        this.relativeExpiry = MEMBERSHIP_EXPIRY_TIME;
+        this.manageMediaKeys = manageMediaKeys ?? false;
        this.membershipId = randomString(5);
        this.emit(MatrixRTCSessionEvent.JoinStateChanged, true);
+        if (manageMediaKeys) {
+            this.makeNewSenderKey();
+            this.requestKeyEventSend();
+        }
        // We don't wait for this, mostly because it may fail and schedule a retry, so this
        // function returning doesn't really mean anything at all.
        this.triggerCallMembershipEventUpdate();
@@ -207,9 +242,21 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M
            return new Promise((resolve) => resolve(false));
        }

+        const userId = this.client.getUserId();
+        const deviceId = this.client.getDeviceId();
+
+        if (!userId) throw new Error("No userId");
+        if (!deviceId) throw new Error("No deviceId");
+
+        // clear our encryption keys as we're done with them now (we'll
+        // make new keys if we rejoin). We leave keys for other participants
+        // as they may still be using the same ones.
+        this.encryptionKeys.set(getParticipantId(userId, deviceId), []);
+
        logger.info(`Leaving call session in room ${this.room.roomId}`);
        this.relativeExpiry = undefined;
        this.activeFoci = undefined;
+        this.manageMediaKeys = false;
        this.membershipId = undefined;
        this.emit(MatrixRTCSessionEvent.JoinStateChanged, false);

@@ -228,6 +275,142 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M
        });
    }

+    public getKeysForParticipant(userId: string, deviceId: string): Array<Uint8Array> | undefined {
+        return this.encryptionKeys.get(getParticipantId(userId, deviceId));
+    }
+
+    /**
+     * A map of keys used to encrypt and decrypt (we are using a symmetric
+     * cipher) given participant's media. This also includes our own key
+     */
+    public getEncryptionKeys(): IterableIterator<[string, Array<Uint8Array>]> {
+        return this.encryptionKeys.entries();
+    }
+
+    private getNewEncryptionKeyIndex(): number {
+        const userId = this.client.getUserId();
+        const deviceId = this.client.getDeviceId();
+
+        if (!userId) throw new Error("No userId!");
+        if (!deviceId) throw new Error("No deviceId!");
+
+        return (this.getKeysForParticipant(userId, deviceId)?.length ?? 0) % 16;
+    }
+
+    private setEncryptionKey(
+        userId: string,
+        deviceId: string,
+        encryptionKeyIndex: number,
+        encryptionKeyString: string,
+    ): void {
+        const keyBin = decodeBase64(encryptionKeyString);
+
+        const participantId = getParticipantId(userId, deviceId);
+        const encryptionKeys = this.encryptionKeys.get(participantId) ?? [];
+
+        if (keysEqual(encryptionKeys[encryptionKeyIndex], keyBin)) return;
+
+        encryptionKeys[encryptionKeyIndex] = keyBin;
+        this.encryptionKeys.set(participantId, encryptionKeys);
+        this.emit(MatrixRTCSessionEvent.EncryptionKeyChanged, keyBin, encryptionKeyIndex, participantId);
+    }
+
+    /**
+     * Generate a new sender key and add it at the next available index
+     */
+    private makeNewSenderKey(): void {
+        const userId = this.client.getUserId();
+        const deviceId = this.client.getDeviceId();
+
+        if (!userId) throw new Error("No userId");
+        if (!deviceId) throw new Error("No deviceId");
+
+        const encryptionKey = secureRandomBase64Url(16);
+        const encryptionKeyIndex = this.getNewEncryptionKeyIndex();
+        logger.info("Generated new key at index " + encryptionKeyIndex);
+        this.setEncryptionKey(userId, deviceId, encryptionKeyIndex, encryptionKey);
+    }
+
+    /**
+     * Requests that we resend our keys to the room. May send a keys event immediately
+     * or queue for alter if one has already been sent recently.
+     */
+    private requestKeyEventSend(): void {
+        if (!this.manageMediaKeys) return;
+
+        if (
+            this.lastEncryptionKeyUpdateRequest &&
+            this.lastEncryptionKeyUpdateRequest + UPDATE_ENCRYPTION_KEY_THROTTLE > Date.now()
+        ) {
+            logger.info("Last encryption key event sent too recently: postponing");
+            if (this.keysEventUpdateTimeout === undefined) {
+                this.keysEventUpdateTimeout = setTimeout(this.sendEncryptionKeysEvent, UPDATE_ENCRYPTION_KEY_THROTTLE);
+            }
+            return;
+        }
+
+        this.sendEncryptionKeysEvent();
+    }
+
+    /**
+     * Re-sends the encryption keys room event
+     */
+    private sendEncryptionKeysEvent = async (): Promise<void> => {
+        if (this.keysEventUpdateTimeout !== undefined) {
+            clearTimeout(this.keysEventUpdateTimeout);
+            this.keysEventUpdateTimeout = undefined;
+        }
+        this.lastEncryptionKeyUpdateRequest = Date.now();
+
+        logger.info("Sending encryption keys event");
+
+        if (!this.isJoined()) return;
+
+        const userId = this.client.getUserId();
+        const deviceId = this.client.getDeviceId();
+
+        if (!userId) throw new Error("No userId");
+        if (!deviceId) throw new Error("No deviceId");
+
+        const myKeys = this.getKeysForParticipant(userId, deviceId);
+
+        if (!myKeys) {
+            logger.warn("Tried to send encryption keys event but no keys found!");
+            return;
+        }
+
+        try {
+            await this.client.sendEvent(this.room.roomId, EventType.CallEncryptionKeysPrefix, {
+                keys: myKeys.map((key, index) => {
+                    return {
+                        index,
+                        key: encodeUnpaddedBase64(key),
+                    };
+                }),
+                device_id: deviceId,
+                call_id: "",
+            } as EncryptionKeysEventContent);
+
+            logger.debug(
+                `Embedded-E2EE-LOG updateEncryptionKeyEvent participantId=${userId}:${deviceId} numSent=${myKeys.length}`,
+            );
+        } catch (error) {
+            const matrixError = error as MatrixError;
+            if (matrixError.event) {
+                // cancel the pending event: we'll just generate a new one with our latest
+                // keys when we resend
+                this.client.cancelPendingEvent(matrixError.event);
+            }
+            if (this.keysEventUpdateTimeout === undefined) {
+                const resendDelay = matrixError.data?.retry_after_ms ?? 5000;
+                logger.warn(`Failed to send m.call.encryption_key, retrying in ${resendDelay}`, error);
+                this.keysEventUpdateTimeout = setTimeout(this.sendEncryptionKeysEvent, resendDelay);
+            } else {
+                logger.info("Not scheduling key resend as another re-send is already pending");
+            }
+        }
+    };
+
    /**
     * Sets a timer for the soonest membership expiry
     */
@@ -254,6 +437,64 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M
        return this.memberships[0];
    }

+    public onCallEncryption = (event: MatrixEvent): void => {
+        const userId = event.getSender();
+        const content = event.getContent<EncryptionKeysEventContent>();
+
+        const deviceId = content["device_id"];
+        const callId = content["call_id"];
+
+        if (!userId) {
+            logger.warn(`Received m.call.encryption_keys with no userId: callId=${callId}`);
+            return;
+        }
+
+        // We currently only handle callId = ""
+        if (callId !== "") {
+            logger.warn(
+                `Received m.call.encryption_keys with unsupported callId: userId=${userId}, deviceId=${deviceId}, callId=${callId}`,
+            );
+            return;
+        }
+
+        if (!Array.isArray(content.keys)) {
+            logger.warn(`Received m.call.encryption_keys where keys wasn't an array: callId=${callId}`);
+            return;
+        }
+
+        for (const key of content.keys) {
+            if (!key.key || !isNumber(key.index)) {
+                logger.warn(`Received m.call.encryption_keys with invalid entry: callId=${callId}`);
+                continue;
+            }
+
+            const encryptionKey = key.key;
+            const encryptionKeyIndex = key.index;
+
+            if (
+                !encryptionKey ||
+                encryptionKeyIndex === undefined ||
+                encryptionKeyIndex === null ||
+                callId === undefined ||
+                callId === null ||
+                typeof deviceId !== "string" ||
+                typeof callId !== "string" ||
+                typeof encryptionKey !== "string" ||
+                typeof encryptionKeyIndex !== "number"
+            ) {
+                logger.warn(
+                    `Malformed call encryption_key: userId=${userId}, deviceId=${deviceId}, encryptionKeyIndex=${encryptionKeyIndex} callId=${callId}`,
+                );
+            } else {
+                logger.debug(
+                    `Embedded-E2EE-LOG onCallEncryption userId=${userId}:${deviceId} encryptionKeyIndex=${encryptionKeyIndex}`,
+                    this.encryptionKeys,
+                );
+                this.setEncryptionKey(userId, deviceId, encryptionKeyIndex, encryptionKey);
+            }
+        }
+    };
+
    public onMembershipUpdate = (): void => {
        const oldMemberships = this.memberships;
        this.memberships = MatrixRTCSession.callMembershipsForRoom(this.room);
@@ -267,6 +508,24 @@ export class MatrixRTCSession extends TypedEventEmitter<MatrixRTCSessionEvent, M
            this.emit(MatrixRTCSessionEvent.MembershipsChanged, oldMemberships, this.memberships);
        }

+        const isMyMembership = (m: CallMembership): boolean =>
+            m.sender === this.client.getUserId() && m.deviceId === this.client.getDeviceId();
+        const callMembersChanged =
+            oldMemberships
+                .filter((m) => !isMyMembership(m))
+                .map(getParticipantIdFromMembership)
+                .sort()
+                .join() !==
+            this.memberships
+                .filter((m) => !isMyMembership(m))
+                .map(getParticipantIdFromMembership)
+                .sort()
+                .join();
+
+        if (callMembersChanged && this.isJoined()) {
+            this.requestKeyEventSend();
+        }
+
        this.setExpiryTimer();
    };