Run Playwright tests against Dendrite & Pinecone periodically (#28888)

* Switch to TestContainers for manging services in Playwright

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Flip fixture dependency order

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Remove mas dep

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Update matrix-authentication-service in Playwright tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* delint

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix SMTP port

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Comments

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Strip ansi from playwright logs to make them more readable

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Actually do the update

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Remove access to homeserver.config.baseUrl field in favour of homeserver.baseUrl

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Use sane default_server_config and specify server.invalid in the specific tests which demand it

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix mas run

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* break cycle

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* typo

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* prettier

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Wire up basics of dendriteHomeserver

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Run Playwright tests against Dendrite & Pinecone periodically

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix playwright-image-updates.yaml workflow

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Add `X-Run-All-Tests` label for running all tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Ignore failing tests in stale-screenshot-reporter.ts to avoid confusing errors

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Specify Synapse ui_auth.session_timeout only on tests which require it

As Dendrite lacks this configuration option

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* React to MatrixEvent sender/target sentinels being updated for rendering state events

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Remove test code

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* React to sentinel changes in EventListSummary

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Docs

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Avoid reusing user1234

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix stale-screenshot-reporter.ts

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Clean up public rooms between tests on reused homeserver

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Deflake spotlight when homeserver is reused

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Deflake more tests using existing username

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Clean mailhog between tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix more flakes

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix missing _request

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix playwright flaky tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Wipe mailhog between test runs

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Delint

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* delint

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Deflake more tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix flaky tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix flaky tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix mas config

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix another flaky test

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix playwright flakes due to floating promises

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Fix flaky playwright tests

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Iterate

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

* Update services.ts

---------

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>

.github/workflows/end-to-end-tests.yaml

@@ -114,6 +114,8 @@ jobs:
                     - Chrome
                     - Firefox
                     - WebKit
+                    - Dendrite
+                    - Pinecone
                 runAllTests:
                     - ${{ github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'X-Run-All-Tests') }}
                 # Skip the Firefox & Safari runs unless this was a cron trigger or PR has X-Run-All-Tests label
@@ -122,6 +124,10 @@ jobs:
                       project: Firefox
                     - runAllTests: false
                       project: WebKit
+                    - runAllTests: false
+                      project: Dendrite
+                    - runAllTests: false
+                      project: Pinecone
         steps:
             - uses: actions/checkout@v4
               with:

docs/playwright.md

@@ -66,11 +66,11 @@ as is typical for Playwright tests. Likewise, tests live in `playwright/e2e`.
 of Synapse/Dendrite. These servers are what Element-web runs against in the tests.
 
 Synapse can be launched with different configurations in order to test element
-in different configurations. You can specify `synapseConfigOptions` as such:
+in different configurations. You can specify `synapseConfig` as such:
 
 ```typescript
 test.use({
-    synapseConfigOptions: {
+    synapseConfig: {
         // The config options to pass to the Synapse instance
     },
 });

playwright.config.ts

@@ -8,19 +8,25 @@ Please see LICENSE files in the repository root for full details.
 
 import { defineConfig, devices } from "@playwright/test";
 
+import { Options } from "./playwright/services";
+
 const baseURL = process.env["BASE_URL"] ?? "http://localhost:8080";
 
-export default defineConfig({
+const chromeProject = {
-    projects: [
-        {
-            name: "Chrome",
-            use: {
     ...devices["Desktop Chrome"],
     channel: "chromium",
     permissions: ["clipboard-write", "clipboard-read", "microphone"],
     launchOptions: {
         args: ["--use-fake-ui-for-media-stream", "--use-fake-device-for-media-stream", "--mute-audio"],
     },
+};
+
+export default defineConfig<Options>({
+    projects: [
+        {
+            name: "Chrome",
+            use: {
+                ...chromeProject,
             },
         },
         {
@@ -48,6 +54,22 @@ export default defineConfig({
             },
             ignoreSnapshots: true,
         },
+        {
+            name: "Dendrite",
+            use: {
+                ...chromeProject,
+                homeserverType: "dendrite",
+            },
+            ignoreSnapshots: true,
+        },
+        {
+            name: "Pinecone",
+            use: {
+                ...chromeProject,
+                homeserverType: "pinecone",
+            },
+            ignoreSnapshots: true,
+        },
     ],
     use: {
         viewport: { width: 1280, height: 720 },

playwright/e2e/create-room/create-room.spec.ts

@@ -27,7 +27,7 @@ test.describe("Create Room", () => {
         // Submit
         await dialog.getByRole("button", { name: "Create room" }).click();
 
-        await expect(page).toHaveURL(/\/#\/room\/#test-room-1:localhost/);
+        await expect(page).toHaveURL(new RegExp(`/#/room/#test-room-1:${user.homeServer}`));
         const header = page.locator(".mx_RoomHeader");
         await expect(header).toContainText(name);
     });

playwright/e2e/crypto/backups.spec.ts

@@ -9,6 +9,7 @@ Please see LICENSE files in the repository root for full details.
 import { type Page } from "@playwright/test";
 
 import { test, expect } from "../../element-web-test";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 async function expectBackupVersionToBe(page: Page, version: string) {
     await expect(page.locator(".mx_SecureBackupPanel_statusList tr:nth-child(5) td")).toHaveText(
@@ -19,6 +20,7 @@ async function expectBackupVersionToBe(page: Page, version: string) {
 }
 
 test.describe("Backups", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
     test.use({
         displayName: "Hanako",
     });

playwright/e2e/crypto/complete-security.spec.ts

@@ -8,8 +8,10 @@ Please see LICENSE files in the repository root for full details.
 
 import { test, expect } from "../../element-web-test";
 import { logIntoElement } from "./utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Complete security", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
     test.use({
         displayName: "Jeff",
     });

playwright/e2e/crypto/crypto.spec.ts

@@ -11,6 +11,7 @@ import { expect, test } from "../../element-web-test";
 import { autoJoin, copyAndContinue, createSharedRoomWithUser, enableKeyBackup, verify } from "./utils";
 import { Bot } from "../../pages/bot";
 import { ElementAppPage } from "../../pages/ElementAppPage";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 const checkDMRoom = async (page: Page) => {
     const body = page.locator(".mx_RoomView_body");
@@ -67,6 +68,7 @@ const bobJoin = async (page: Page, bob: Bot) => {
 };
 
 test.describe("Cryptography", function () {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
     test.use({
         displayName: "Alice",
         botCreateOpts: {

playwright/e2e/crypto/decryption-failure-messages.spec.ts

@@ -28,6 +28,8 @@ test.describe("Cryptography", function () {
     });
 
     test.describe("decryption failure messages", () => {
+        test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
+
         test("should handle device-relative historical messages", async ({
             homeserver,
             page,

playwright/e2e/crypto/device-verification.spec.ts

@@ -219,9 +219,10 @@ test.describe("Device verification", { tag: "@no-webkit" }, () => {
         /* And we're all done! */
         const infoDialog = page.locator(".mx_InfoDialog");
         await infoDialog.getByRole("button", { name: "They match" }).click();
-        await expect(
+        // We don't assert the full string as the device name is unset on Synapse but set to the user ID on Dendrite
-            infoDialog.getByText(`You've successfully verified (${aliceBotClient.credentials.deviceId})!`),
+        await expect(infoDialog.getByText(`You've successfully verified`)).toContainText(
-        ).toBeVisible();
+            `(${aliceBotClient.credentials.deviceId})`,
+        );
         await infoDialog.getByRole("button", { name: "Got it" }).click();
     });
 });

playwright/e2e/crypto/logout.spec.ts

@@ -8,8 +8,10 @@ Please see LICENSE files in the repository root for full details.
 
 import { test, expect } from "../../element-web-test";
 import { createRoom, enableKeyBackup, logIntoElement, sendMessageInCurrentRoom } from "./utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Logout tests", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
     test.beforeEach(async ({ page, homeserver, credentials }) => {
         await logIntoElement(page, credentials);
     });

playwright/e2e/editing/editing.spec.ts

@@ -12,6 +12,7 @@ import type { EventType, IContent, ISendEventResponse, MsgType, Visibility } fro
 import { expect, test } from "../../element-web-test";
 import { ElementAppPage } from "../../pages/ElementAppPage";
 import { SettingLevel } from "../../../src/settings/SettingLevel";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 async function sendEvent(app: ElementAppPage, roomId: string): Promise<ISendEventResponse> {
     return app.client.sendEvent(roomId, null, "m.room.message" as EventType, {
@@ -31,6 +32,8 @@ function mkPadding(n: number): IContent {
 }
 
 test.describe("Editing", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3488");
+
     // Edit "Message"
     const editLastMessage = async (page: Page, edit: string) => {
         const eventTile = page.locator(".mx_RoomView_MessageList .mx_EventTile_last");

playwright/e2e/knock/create-knock-room.spec.ts

@@ -9,8 +9,10 @@ Please see LICENSE files in the repository root for full details.
 import { test, expect } from "../../element-web-test";
 import { waitForRoom } from "../utils";
 import { Filter } from "../../pages/Spotlight";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Create Knock Room", () => {
+    test.skip(isDendrite, "Dendrite does not have support for knocking");
     test.use({
         displayName: "Alice",
         labsFlags: ["feature_ask_to_join"],

playwright/e2e/knock/knock-into-room.spec.ts

@@ -13,8 +13,10 @@ import { type Visibility } from "matrix-js-sdk/src/matrix";
 import { test, expect } from "../../element-web-test";
 import { waitForRoom } from "../utils";
 import { Filter } from "../../pages/Spotlight";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Knock Into Room", () => {
+    test.skip(isDendrite, "Dendrite does not have support for knocking");
     test.use({
         displayName: "Alice",
         labsFlags: ["feature_ask_to_join"],

playwright/e2e/knock/manage-knocks.spec.ts

@@ -10,8 +10,10 @@ Please see LICENSE files in the repository root for full details.
 
 import { test, expect } from "../../element-web-test";
 import { waitForRoom } from "../utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Manage Knocks", () => {
+    test.skip(isDendrite, "Dendrite does not have support for knocking");
     test.use({
         displayName: "Alice",
         labsFlags: ["feature_ask_to_join"],

playwright/e2e/lazy-loading/lazy-loading.spec.ts

@@ -10,8 +10,12 @@ import { Bot } from "../../pages/bot";
 import type { Locator, Page } from "@playwright/test";
 import type { ElementAppPage } from "../../pages/ElementAppPage";
 import { test, expect } from "../../element-web-test";
+import { Credentials } from "../../plugins/homeserver";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Lazy Loading", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3488");
+
     const charlies: Bot[] = [];
 
     test.use({
@@ -35,12 +39,18 @@ test.describe("Lazy Loading", () => {
     });
 
     const name = "Lazy Loading Test";
-    const alias = "#lltest:localhost";
     const charlyMsg1 = "hi bob!";
     const charlyMsg2 = "how's it going??";
     let roomId: string;
 
-    async function setupRoomWithBobAliceAndCharlies(page: Page, app: ElementAppPage, bob: Bot, charlies: Bot[]) {
+    async function setupRoomWithBobAliceAndCharlies(
+        page: Page,
+        app: ElementAppPage,
+        user: Credentials,
+        bob: Bot,
+        charlies: Bot[],
+    ) {
+        const alias = `#lltest:${user.homeServer}`;
         const visibility = await page.evaluate(() => (window as any).matrixcs.Visibility.Public);
         roomId = await bob.createRoom({
             name,
@@ -95,7 +105,13 @@ test.describe("Lazy Loading", () => {
         }
     }
 
-    async function joinCharliesWhileAliceIsOffline(page: Page, app: ElementAppPage, charlies: Bot[]) {
+    async function joinCharliesWhileAliceIsOffline(
+        page: Page,
+        app: ElementAppPage,
+        user: Credentials,
+        charlies: Bot[],
+    ) {
+        const alias = `#lltest:${user.homeServer}`;
         await app.client.network.goOffline();
         for (const charly of charlies) {
             await charly.joinRoom(alias);
@@ -107,19 +123,19 @@ test.describe("Lazy Loading", () => {
         await app.client.waitForNextSync();
     }
 
-    test("should handle lazy loading properly even when offline", async ({ page, app, bot }) => {
+    test("should handle lazy loading properly even when offline", async ({ page, app, bot, user }) => {
         test.slow();
         const charly1to5 = charlies.slice(0, 5);
         const charly6to10 = charlies.slice(5);
 
         // Set up room with alice, bob & charlies 1-5
-        await setupRoomWithBobAliceAndCharlies(page, app, bot, charly1to5);
+        await setupRoomWithBobAliceAndCharlies(page, app, user, bot, charly1to5);
         // Alice should see 2 messages from every charly with the correct display name
         await checkPaginatedDisplayNames(app, charly1to5);
 
         await openMemberlist(app);
         await checkMemberList(page, charly1to5);
-        await joinCharliesWhileAliceIsOffline(page, app, charly6to10);
+        await joinCharliesWhileAliceIsOffline(page, app, user, charly6to10);
         await checkMemberList(page, charly6to10);
 
         for (const charly of charlies) {

playwright/e2e/login/login-consent.spec.ts

@@ -12,6 +12,7 @@ import { expect, test } from "../../element-web-test";
 import { selectHomeserver } from "../utils";
 import { Credentials, HomeserverInstance } from "../../plugins/homeserver";
 import { consentHomeserver } from "../../plugins/homeserver/synapse/consentHomeserver.ts";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 // This test requires fixed credentials for the device signing keys below to work
 const username = "user1234";
@@ -113,6 +114,8 @@ test.use({
 
 test.describe("Login", () => {
     test.describe("Password login", () => {
+        test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
+
         test("Loads the welcome page by default; then logs in with an existing account and lands on the home screen", async ({
             credentials,
             page,

playwright/e2e/oidc/oidc-native.spec.ts

@@ -9,12 +9,10 @@ Please see LICENSE files in the repository root for full details.
 import { test, expect } from "../../element-web-test.ts";
 import { registerAccountMas } from ".";
 import { ElementAppPage } from "../../pages/ElementAppPage.ts";
-import { isDendrite } from "../../plugins/homeserver/dendrite";
 import { masHomeserver } from "../../plugins/homeserver/synapse/masHomeserver.ts";
 
 test.use(masHomeserver);
 test.describe("OIDC Native", { tag: ["@no-firefox", "@no-webkit"] }, () => {
-    test.skip(isDendrite, "does not yet support MAS");
     test.slow(); // trace recording takes a while here
 
     test("can register the oauth2 client and an account", async ({

playwright/e2e/one-to-one-chat/one-to-one-chat.spec.ts

@@ -9,12 +9,15 @@ Please see LICENSE files in the repository root for full details.
 
 import { test as base, expect } from "../../element-web-test";
 import { Credentials } from "../../plugins/homeserver";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 const test = base.extend<{
     user2?: Credentials;
 }>({});
 
 test.describe("1:1 chat room", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3492");
+
     test.use({
         displayName: "Jeff",
         user2: async ({ homeserver }, use, testInfo) => {

playwright/e2e/permalinks/permalinks.spec.ts

@@ -31,7 +31,7 @@ test.describe("permalinks", () => {
         await charlotte.prepareClient();
 
         // We don't use a bot for danielle as we want a stable MXID.
-        const danielleId = "@danielle:localhost";
+        const danielleId = `@danielle:${user.homeServer}`;
 
         const room1Id = await app.client.createRoom({ name: room1Name });
         const room2Id = await app.client.createRoom({ name: room2Name });

playwright/e2e/polls/polls.spec.ts

@@ -11,8 +11,11 @@ import { Bot } from "../../pages/bot";
 import { SettingLevel } from "../../../src/settings/SettingLevel";
 import { Layout } from "../../../src/settings/enums/Layout";
 import type { Locator, Page } from "@playwright/test";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Polls", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3492");
+
     type CreatePollOptions = {
         title: string;
         options: string[];

playwright/e2e/read-receipts/editing-messages-in-threads.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("editing messages", () => {
         test.describe("in threads", () => {
             test("An edit of a threaded message makes the room unread", async ({

playwright/e2e/read-receipts/editing-messages-main-timeline.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("editing messages", () => {
         test.describe("in the main timeline", () => {
             test("Editing a message leaves a room read", async ({ roomAlpha: room1, roomBeta: room2, util, msg }) => {

playwright/e2e/read-receipts/editing-messages-thread-roots.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("editing messages", () => {
         test.describe("thread roots", () => {
             test("An edit of a thread root leaves the room read", async ({

playwright/e2e/read-receipts/high-level.spec.ts

@@ -9,8 +9,10 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { customEvent, many, test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
     test.slow();
 
     test.describe("Ignored events", () => {

playwright/e2e/read-receipts/new-messages-in-threads.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { many, test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("new messages", () => {
         test.describe("in threads", () => {
             test("Receiving a message makes a room unread", async ({

playwright/e2e/read-receipts/new-messages-main-timeline.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { many, test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("new messages", () => {
         test.describe("in the main timeline", () => {
             test("Receiving a message makes a room unread", async ({

playwright/e2e/read-receipts/new-messages-thread-roots.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { many, test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("new messages", () => {
         test.describe("thread roots", () => {
             test("Reading a thread root does not mark the thread as read", async ({

playwright/e2e/read-receipts/reactions-in-threads.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test, expect } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("reactions", () => {
         test.describe("in threads", () => {
             test("A reaction to a threaded message does not make the room unread", async ({

playwright/e2e/read-receipts/reactions-main-timeline.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("reactions", () => {
         test.describe("in the main timeline", () => {
             test("Receiving a reaction to a message does not make a room unread", async ({

playwright/e2e/read-receipts/reactions-thread-roots.spec.ts

@@ -9,8 +9,10 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
     test.describe("reactions", () => {
         test.describe("thread roots", () => {
             test("A reaction to a thread root does not make the room unread", async ({

playwright/e2e/read-receipts/read-receipts.spec.ts

@@ -12,8 +12,10 @@ import { expect } from "../../element-web-test";
 import { ElementAppPage } from "../../pages/ElementAppPage";
 import { Bot } from "../../pages/bot";
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
     test.use({
         displayName: "Mae",
         botCreateOpts: { displayName: "Other User" },

playwright/e2e/read-receipts/redactions-in-threads.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("redactions", () => {
         test.describe("in threads", () => {
             test("Redacting the threaded message pointed to by my receipt leaves the room read", async ({

playwright/e2e/read-receipts/redactions-main-timeline.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("redactions", () => {
         test.describe("in the main timeline", () => {
             test("Redacting the message pointed to by my receipt leaves the room read", async ({

playwright/e2e/read-receipts/redactions-thread-roots.spec.ts

@@ -9,8 +9,11 @@ Please see LICENSE files in the repository root for full details.
 /* See readme.md for tips on writing these tests. */
 
 import { test } from ".";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Read receipts", { tag: "@mergequeue" }, () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.describe("redactions", () => {
         test.describe("thread roots", () => {
             test("Redacting a thread root after it was read leaves the room read", async ({

playwright/e2e/register/register.spec.ts

@@ -8,6 +8,7 @@ Please see LICENSE files in the repository root for full details.
 
 import { test, expect } from "../../element-web-test";
 import { consentHomeserver } from "../../plugins/homeserver/synapse/consentHomeserver.ts";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.use(consentHomeserver);
 test.use({
@@ -23,6 +24,8 @@ test.use({
 });
 
 test.describe("Registration", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
+
     test.beforeEach(async ({ page }) => {
         await page.goto("/#/register");
     });

playwright/e2e/right-panel/file-panel.spec.ts

@@ -10,6 +10,7 @@ import { Download, type Page } from "@playwright/test";
 
 import { test, expect } from "../../element-web-test";
 import { viewRoomSummaryByName } from "./utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 const ROOM_NAME = "Test room";
 const NAME = "Alice";
@@ -181,6 +182,8 @@ test.describe("FilePanel", () => {
     });
 
     test.describe("download", () => {
+        test.skip(isDendrite, "due to a Dendrite sending Content-Disposition inline");
+
         test("should download an image via the link on the panel", async ({ page, context }) => {
             // Upload an image file
             await uploadFile(page, "playwright/sample-files/riot.png");

playwright/e2e/right-panel/right-panel.spec.ts

@@ -38,7 +38,10 @@ test.describe("RightPanel", () => {
     });
 
     test.describe("in rooms", () => {
-        test("should handle long room address and long room name", { tag: "@screenshot" }, async ({ page, app }) => {
+        test(
+            "should handle long room address and long room name",
+            { tag: "@screenshot" },
+            async ({ page, app, user }) => {
                 await app.client.createRoom({ name: ROOM_NAME_LONG });
                 await viewRoomSummaryByName(page, app, ROOM_NAME_LONG);
 
@@ -49,7 +52,7 @@ test.describe("RightPanel", () => {
                 await localAddresses.getByRole("textbox").fill(ROOM_ADDRESS_LONG);
                 await expect(page.getByText("This address is available to use")).toBeVisible();
                 await localAddresses.getByRole("button", { name: "Add" }).click();
-            await expect(localAddresses.getByText(`#${ROOM_ADDRESS_LONG}:localhost`)).toHaveClass(
+                await expect(localAddresses.getByText(`#${ROOM_ADDRESS_LONG}:${user.homeServer}`)).toHaveClass(
                     "mx_EditableItem_item",
                 );
 
@@ -61,7 +64,8 @@ test.describe("RightPanel", () => {
                 await app.toggleRoomInfoPanel();
 
                 await expect(page.locator(".mx_RightPanel")).toMatchScreenshot("with-name-and-address.png");
-        });
+            },
+        );
 
         test("should handle clicking add widgets", async ({ page, app }) => {
             await viewRoomSummaryByName(page, app, ROOM_NAME);

playwright/e2e/room-directory/room-directory.spec.ts

@@ -10,6 +10,7 @@ import type { Preset, Visibility } from "matrix-js-sdk/src/matrix";
 import { test, expect } from "../../element-web-test";
 
 test.describe("Room Directory", () => {
+    test.skip(({ homeserverType }) => homeserverType === "pinecone", "Pinecone's /publicRooms API takes forever");
     test.use({
         displayName: "Ray",
         botCreateOpts: { displayName: "Paul" },
@@ -32,14 +33,14 @@ test.describe("Room Directory", () => {
             await localAddresses.getByRole("textbox").fill("gaming");
             await expect(page.getByText("This address is available to use")).toBeVisible();
             await localAddresses.getByRole("button", { name: "Add" }).click();
-            await expect(localAddresses.getByText("#gaming:localhost")).toHaveClass("mx_EditableItem_item");
+            await expect(localAddresses.getByText(`#gaming:${user.homeServer}`)).toHaveClass("mx_EditableItem_item");
 
             // Publish into the public rooms directory
             const publishedAddresses = page.locator(".mx_SettingsFieldset", { hasText: "Published Addresses" });
-            await expect(publishedAddresses.locator("#canonicalAlias")).toHaveValue("#gaming:localhost");
+            await expect(publishedAddresses.locator("#canonicalAlias")).toHaveValue(`#gaming:${user.homeServer}`);
             const checkbox = publishedAddresses
                 .locator(".mx_SettingsFlag", {
-                    hasText: "Publish this room to the public in localhost's room directory?",
+                    hasText: `Publish this room to the public in ${user.homeServer}'s room directory?`,
                 })
                 .getByRole("switch");
             await checkbox.check();
@@ -87,7 +88,7 @@ test.describe("Room Directory", () => {
                 .getByRole("button", { name: "Join" })
                 .click();
 
-            await expect(page).toHaveURL("/#/room/#test1234:localhost");
+            await expect(page).toHaveURL(`/#/room/#test1234:${user.homeServer}`);
         },
     );
 });

playwright/e2e/room_options/marked_unread.spec.ts

@@ -7,10 +7,13 @@ Please see LICENSE files in the repository root for full details.
 */
 
 import { test, expect } from "../../element-web-test";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 const TEST_ROOM_NAME = "The mark unread test room";
 
 test.describe("Mark as Unread", () => {
+    test.skip(isDendrite, "due to Dendrite bug https://github.com/element-hq/dendrite/issues/2970");
+
     test.use({
         displayName: "Tom",
         botCreateOpts: {

playwright/e2e/settings/general-room-settings-tab.spec.ts

@@ -36,7 +36,7 @@ test.describe("General room settings tab", () => {
         await expect(settings.getByText("Show more")).toBeVisible();
     });
 
-    test("long address should not cause dialog to overflow", { tag: "@no-webkit" }, async ({ page, app }) => {
+    test("long address should not cause dialog to overflow", { tag: "@no-webkit" }, async ({ page, app, user }) => {
         const settings = await app.settings.openRoomSettings("General");
         // 1. Set the room-address to be a really long string
         const longString = "abcasdhjasjhdaj1jh1asdhasjdhajsdhjavhjksd".repeat(4);
@@ -45,7 +45,7 @@ test.describe("General room settings tab", () => {
         await settings.locator("#roomAliases").getByText("Add", { exact: true }).click();
 
         // 2. wait for the new setting to apply ...
-        await expect(settings.locator("#canonicalAlias")).toHaveValue(`#${longString}:localhost`);
+        await expect(settings.locator("#canonicalAlias")).toHaveValue(`#${longString}:${user.homeServer}`);
 
         // 3. Check if the dialog overflows
         const dialogBoundingBox = await page.locator(".mx_Dialog").boundingBox();

playwright/e2e/sliding-sync/sliding-sync.spec.ts

@@ -69,6 +69,11 @@ const test = base.extend<{
 });
 
 test.describe("Sliding Sync", () => {
+    test.skip(
+        ({ homeserverType }) => homeserverType === "pinecone",
+        "due to a bug in Pinecone https://github.com/element-hq/dendrite/issues/3490",
+    );
+
     const checkOrder = async (wantOrder: string[], page: Page) => {
         await expect(page.getByRole("group", { name: "Rooms" }).locator(".mx_RoomTile_title")).toHaveText(wantOrder);
     };

playwright/e2e/spaces/spaces.spec.ts

@@ -10,6 +10,7 @@ import type { Locator, Page } from "@playwright/test";
 import { test, expect } from "../../element-web-test";
 import type { Preset, ICreateRoomOpts } from "matrix-js-sdk/src/matrix";
 import { ElementAppPage } from "../../pages/ElementAppPage";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 async function openSpaceCreateMenu(page: Page): Promise<Locator> {
     await page.getByRole("button", { name: "Create a space" }).click();
@@ -50,6 +51,7 @@ function spaceChildInitialState(roomId: string): ICreateRoomOpts["initial_state"
 }
 
 test.describe("Spaces", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3488");
     test.use({
         displayName: "Sue",
         botCreateOpts: { displayName: "BotBob" },
@@ -82,7 +84,7 @@ test.describe("Spaces", () => {
 
             // Copy matrix.to link
             await page.getByRole("button", { name: "Share invite link" }).click();
-            expect(await app.getClipboardText()).toEqual("https://matrix.to/#/#lets-have-a-riot:localhost");
+            expect(await app.getClipboardText()).toEqual(`https://matrix.to/#/#lets-have-a-riot:${user.homeServer}`);
 
             // Go to space home
             await page.getByRole("button", { name: "Go to my first room" }).click();
@@ -169,13 +171,13 @@ test.describe("Spaces", () => {
             room_alias_name: "space",
         });
 
-        const menu = await openSpaceContextMenu(page, app, "#space:localhost");
+        const menu = await openSpaceContextMenu(page, app, `#space:${user.homeServer}`);
         await menu.getByRole("menuitem", { name: "Invite" }).click();
 
         const shareDialog = page.locator(".mx_SpacePublicShare");
         // Copy link first
         await shareDialog.getByRole("button", { name: "Share invite link" }).click();
-        expect(await app.getClipboardText()).toEqual("https://matrix.to/#/#space:localhost");
+        expect(await app.getClipboardText()).toEqual(`https://matrix.to/#/#space:${user.homeServer}`);
         // Start Matrix invite flow
         await shareDialog.getByRole("button", { name: "Invite people" }).click();
 

playwright/e2e/spaces/threads-activity-centre/threadsActivityCentre.spec.ts

@@ -8,8 +8,14 @@
 
 import { expect, test } from ".";
 import { CommandOrControl } from "../../utils";
+import { isDendrite } from "../../../plugins/homeserver/dendrite";
 
 test.describe("Threads Activity Centre", { tag: "@no-firefox" }, () => {
+    test.skip(
+        isDendrite,
+        "due to Dendrite lacking full threads support https://github.com/element-hq/dendrite/issues/3283",
+    );
+
     test.use({
         displayName: "Alice",
         botCreateOpts: { displayName: "Other User" },

playwright/e2e/spotlight/spotlight.spec.ts

@@ -12,6 +12,7 @@ import { Filter } from "../../pages/Spotlight";
 import { Bot } from "../../pages/bot";
 import type { Locator, Page } from "@playwright/test";
 import type { ElementAppPage } from "../../pages/ElementAppPage";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 function roomHeaderName(page: Page): Locator {
     return page.locator(".mx_RoomHeader_heading");
@@ -89,6 +90,7 @@ const test = base.extend<{
 });
 
 test.describe("Spotlight", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3488");
     test.use({
         displayName: "Jim",
     });

playwright/e2e/threads/threads.spec.ts

@@ -8,8 +8,10 @@ Please see LICENSE files in the repository root for full details.
 import { SettingLevel } from "../../../src/settings/SettingLevel";
 import { Layout } from "../../../src/settings/enums/Layout";
 import { test, expect } from "../../element-web-test";
+import { isDendrite } from "../../plugins/homeserver/dendrite";
 
 test.describe("Threads", () => {
+    test.skip(isDendrite, "due to a Dendrite bug https://github.com/element-hq/dendrite/issues/3489");
     test.use({
         displayName: "Tom",
         botCreateOpts: {

playwright/e2e/widgets/stickers.spec.ts

@@ -88,7 +88,7 @@ async function sendStickerFromPicker(page: Page) {
     await expect(page.locator(".mx_AppTileFullWidth#stickers")).not.toBeVisible();
 }
 
-async function expectTimelineSticker(page: Page, roomId: string, contentUri: string) {
+async function expectTimelineSticker(page: Page, serverName: string, roomId: string, contentUri: string) {
     const contentId = contentUri.split("/").slice(-1)[0];
     // Make sure it's in the right room
     await expect(page.locator(".mx_EventTile_sticker > a")).toHaveAttribute("href", new RegExp(`/${roomId}/`));
@@ -98,7 +98,7 @@ async function expectTimelineSticker(page: Page, roomId: string, contentUri: str
     // download URL.
     await expect(page.locator(`img[alt="${STICKER_NAME}"]`)).toHaveAttribute(
         "src",
-        new RegExp(`/localhost/${contentId}`),
+        new RegExp(`/${serverName}/${contentId}`),
     );
 }
 
@@ -156,7 +156,7 @@ test.describe("Stickers", { tag: ["@no-firefox", "@no-webkit"] }, () => {
         await expect(page).toHaveURL(`/#/room/${room.roomId}`);
         await openStickerPicker(app);
         await sendStickerFromPicker(page);
-        await expectTimelineSticker(page, room.roomId, contentUri);
+        await expectTimelineSticker(page, user.homeServer, room.roomId, contentUri);
 
         // Ensure that when we switch to a different room that the sticker
         // goes to the right place
@@ -164,7 +164,7 @@ test.describe("Stickers", { tag: ["@no-firefox", "@no-webkit"] }, () => {
         await expect(page).toHaveURL(`/#/room/${roomId2}`);
         await openStickerPicker(app);
         await sendStickerFromPicker(page);
-        await expectTimelineSticker(page, roomId2, contentUri);
+        await expectTimelineSticker(page, user.homeServer, roomId2, contentUri);
     });
 
     test("should handle a sticker picker widget missing creatorUserId", async ({
@@ -183,7 +183,7 @@ test.describe("Stickers", { tag: ["@no-firefox", "@no-webkit"] }, () => {
         await expect(page).toHaveURL(`/#/room/${room.roomId}`);
         await openStickerPicker(app);
         await sendStickerFromPicker(page);
-        await expectTimelineSticker(page, room.roomId, contentUri);
+        await expectTimelineSticker(page, user.homeServer, room.roomId, contentUri);
     });
 
     test("should render invalid mimetype as a file", async ({ webserver, page, app, user, room }) => {

playwright/flaky-reporter.ts

@@ -28,6 +28,8 @@ class FlakyReporter implements Reporter {
     private flakes = new Map<string, TestCase[]>();
 
     public onTestEnd(test: TestCase): void {
+        // Ignores flakes on Dendrite and Pinecone as they have their own flakes we do not track
+        if (["Dendrite", "Pinecone"].includes(test.parent.project()?.name)) return;
         const title = `${test.location.file.split("playwright/e2e/")[1]}: ${test.title}`;
         if (test.outcome() === "flaky") {
             if (!this.flakes.has(title)) {

playwright/plugins/homeserver/dendrite/index.ts

@@ -6,34 +6,8 @@ SPDX-License-Identifier: AGPL-3.0-only OR GPL-3.0-only OR LicenseRef-Element-Com
 Please see LICENSE files in the repository root for full details.
 */
 
-import { DendriteContainer, PineconeContainer } from "../../../testcontainers/dendrite.ts";
+import { Options } from "../../../services.ts";
-import { Fixtures } from "../../../element-web-test.ts";
 
-export const dendriteHomeserver: Fixtures = {
+export const isDendrite = ({ homeserverType }: Options): boolean => {
-    _homeserver: [
+    return homeserverType === "dendrite" || homeserverType === "pinecone";
-        // eslint-disable-next-line no-empty-pattern
-        async ({}, use) => {
-            const container =
-                process.env["PLAYWRIGHT_HOMESERVER"] === "dendrite" ? new DendriteContainer() : new PineconeContainer();
-            await use(container);
-        },
-        { scope: "worker" },
-    ],
-    homeserver: [
-        async ({ logger, network, _homeserver: homeserver }, use) => {
-            const container = await homeserver
-                .withNetwork(network)
-                .withNetworkAliases("homeserver")
-                .withLogConsumer(logger.getConsumer("dendrite"))
-                .start();
-
-            await use(container);
-            await container.stop();
-        },
-        { scope: "worker" },
-    ],
 };
-
-export function isDendrite(): boolean {
-    return process.env["PLAYWRIGHT_HOMESERVER"] === "dendrite" || process.env["PLAYWRIGHT_HOMESERVER"] === "pinecone";
-}

playwright/plugins/homeserver/index.ts

@@ -46,3 +46,5 @@ export interface Credentials {
     displayName?: string;
     username: string; // the localpart of the userId
 }
+
+export type HomeserverType = "synapse" | "dendrite" | "pinecone";

playwright/plugins/homeserver/synapse/consentHomeserver.ts

@@ -54,4 +54,9 @@ export const consentHomeserver: Fixtures = {
         },
         { scope: "worker" },
     ],
+
+    context: async ({ homeserverType, context }, use, testInfo) => {
+        testInfo.skip(homeserverType !== "synapse", "does not yet support MAS");
+        await use(context);
+    },
 };

playwright/plugins/homeserver/synapse/emailHomeserver.ts

@@ -26,4 +26,9 @@ export const emailHomeserver: Fixtures = {
         },
         { scope: "worker" },
     ],
+
+    context: async ({ homeserverType, context }, use, testInfo) => {
+        testInfo.skip(homeserverType !== "synapse", "does not yet support MAS");
+        await use(context);
+    },
 };

playwright/plugins/homeserver/synapse/legacyOAuthHomeserver.ts

@@ -21,7 +21,8 @@ export const legacyOAuthHomeserver: Fixtures = {
         },
         { scope: "worker" },
     ],
-    context: async ({ context, oAuthServer }, use, testInfo) => {
+    context: async ({ homeserverType, context, oAuthServer }, use, testInfo) => {
+        testInfo.skip(homeserverType !== "synapse", "does not yet support OIDC");
         oAuthServer.onTestStarted(testInfo);
         await use(context);
     },

playwright/plugins/homeserver/synapse/masHomeserver.ts

@@ -79,4 +79,9 @@ export const masHomeserver: Fixtures = {
             default_server_config: wellKnown,
         });
     },
+
+    context: async ({ homeserverType, context }, use, testInfo) => {
+        testInfo.skip(homeserverType !== "synapse", "does not yet support MAS");
+        await use(context);
+    },
 };

playwright/services.ts

@@ -16,6 +16,8 @@ import { StartedMatrixAuthenticationServiceContainer } from "./testcontainers/ma
 import { HomeserverContainer, StartedHomeserverContainer } from "./testcontainers/HomeserverContainer.ts";
 import { MailhogContainer, StartedMailhogContainer } from "./testcontainers/mailhog.ts";
 import { OAuthServer } from "./plugins/oauth_server";
+import { DendriteContainer, PineconeContainer } from "./testcontainers/dendrite.ts";
+import { HomeserverType } from "./plugins/homeserver";
 
 export interface TestFixtures {
     mailhogClient: mailhog.API;
@@ -37,7 +39,9 @@ export interface Services {
     oAuthServer?: OAuthServer;
 }
 
-export interface Options {}
+export interface Options {
+    homeserverType: HomeserverType;
+}
 
 export const test = base.extend<TestFixtures, Services & Options>({
     logger: [
@@ -104,21 +108,35 @@ export const test = base.extend<TestFixtures, Services & Options>({
     },
 
     synapseConfig: [{}, { scope: "worker" }],
+    homeserverType: ["synapse", { option: true, scope: "worker" }],
     _homeserver: [
-        // eslint-disable-next-line no-empty-pattern
+        async ({ homeserverType }, use) => {
-        async ({}, use) => {
+            let container: HomeserverContainer<any>;
-            const container = new SynapseContainer();
+            switch (homeserverType) {
+                case "synapse":
+                    container = new SynapseContainer();
+                    break;
+                case "dendrite":
+                    container = new DendriteContainer();
+                    break;
+                case "pinecone":
+                    container = new PineconeContainer();
+                    break;
+            }
+
             await use(container);
         },
         { scope: "worker" },
     ],
     homeserver: [
-        async ({ logger, network, _homeserver: homeserver, synapseConfig, mas }, use) => {
+        async ({ homeserverType, logger, network, _homeserver: homeserver, synapseConfig, mas }, use) => {
+            if (homeserver instanceof SynapseContainer) {
+                homeserver.withConfig(synapseConfig);
+            }
             const container = await homeserver
                 .withNetwork(network)
                 .withNetworkAliases("homeserver")
-                .withLogConsumer(logger.getConsumer("synapse"))
+                .withLogConsumer(logger.getConsumer(homeserverType))
-                .withConfig(synapseConfig)
                 .withMatrixAuthenticationService(mas)
                 .start();
 
@@ -137,7 +155,11 @@ export const test = base.extend<TestFixtures, Services & Options>({
         { scope: "worker" },
     ],
 
-    context: async ({ logger, context, request, homeserver, mailhogClient }, use, testInfo) => {
+    context: async ({ homeserverType, synapseConfig, logger, context, request, homeserver }, use, testInfo) => {
+        testInfo.skip(
+            !(homeserver instanceof SynapseContainer) && Object.keys(synapseConfig).length > 0,
+            `Test specifies Synapse config options so is unsupported with ${homeserverType}`,
+        );
         homeserver.setRequest(request);
         await logger.onTestStarted(context);
         await use(context);

playwright/testcontainers/dendrite.ts

@@ -236,8 +236,9 @@ export class DendriteContainer extends GenericContainer implements HomeserverCon
         return this;
     }
 
+    // Dendrite does not support MAS at this time
     public withMatrixAuthenticationService(mas?: StartedMatrixAuthenticationServiceContainer): this {
-        throw new Error("Dendrite does not support MAS.");
+        return this;
     }
 
     public override async start(): Promise<StartedDendriteContainer> {
@@ -264,4 +265,10 @@ export class PineconeContainer extends DendriteContainer {
 }
 
 // Surprisingly, Dendrite implements the same register user Synapse Admin API, so we can just extend it
-export class StartedDendriteContainer extends StartedSynapseContainer {}
+export class StartedDendriteContainer extends StartedSynapseContainer {
+    protected async deletePublicRooms(): Promise<void> {
+        // Dendrite does not support admin users managing the room directory
+        // https://github.com/element-hq/dendrite/blob/main/clientapi/routing/directory.go#L365
+        return;
+    }
+}