mirror of
https://github.com/ossrs/srs.git
synced 2026-10-24 09:59:20 +03:00
Adds `srs-integ`, a black-box integration suite for the SRS media server, and fixes the SRS bugs it found: over 120 of them, in every protocol. The suite tests SRS from outside the process, only through what a client, an operator or a callback server can see, so a rewritten server could pass the same cases. The suite now has about 1,450 cases (1,353 main and 99 timer), with no known bugs left: every bug a case found is fixed in this PR. It also removes some unused features, adds env support for a few more directives, and changes a few hook and API fields. ## srs-integ suite `cmd/integ` is a Go module of its own, `srs-integ`, separate from the root `srsx` module. It imports nothing from the repository. - **Run:** `make integ` makes the test media, then runs the cases in two `go test` processes at once: the main cases and the `TestTimer_` cases. `make integ-unit` runs the unit tests of the clients and analyzers, and `bin/srs-integ` gives `probe`, `publish` and `prepare` from the command line. - **Server adapter:** each server is `objs/srs -e`, configured only by env vars, on ports from `SRS_INTEG_PORTS` (14000-14999 in the Port Plan). Most cases share a Common server, split into 4 shards. Cases that need other settings run on variant servers, and groups share these where they can. - **Case classes:** Fast (≤100ms), Normal (≤500ms) and Timer (≤10s, only in `TestTimer_` groups). A case that runs over its limit fails. `KnownBug` marks a case that fails because of an SRS bug, and the case fails once the bug is fixed, so the mark can't go stale. - **Clients and analyzers written for the suite:** RTMP and RTMPS (both handshakes, any chunk size, enhanced RTMP, malformed input), FLV, TS, AAC/MP3, MP4, HLS (TS, fMP4, AES-128, SAMPLE-AES) and DASH. Each one fills a shared stream report: codecs, picture size, sequence headers, keyframes, timestamps and A/V sync. - **Clients built on libraries:** WebRTC WHIP/WHEP on pion (UDP and TCP, NACK/RTX/TWCC recorder, raw DTLS sessions), SRT on gosrt, RTSP on gortsplib, and a GB28181 PS-over-RTP publisher with no SIP. - **Test media:** generated from `doc/source.flv` by `make integ-prepare` and never committed. The files cover H.264, HEVC, VP9, AV1, AAC, Opus, MP3 and G.711, in FLV, MP4 and TS. A manifest keeps the media from being made again when nothing changed. - **Feature coverage:** the HTTP API, HTTPS API and exporter, reload, heartbeat, RTMP, RTMPS, HTTP-FLV, HTTP-TS, HTTP AAC and MP3, HTTPS, HLS, DASH, WebRTC, SRT, RTSP, GB28181, the MPEG-TS over UDP caster, every HTTP hook, security rules, forward, edge, the origin cluster, DVR, static files and VOD, signals, the command line and env settings. - **Conversion matrix:** each source (RTMP/RTMPS, WHIP, SRT, GB28181) is played by each player (RTMP, HTTP-FLV, HTTP-TS, MP3, AAC, HLS ts and fMP4, DASH, WHEP, SRT, RTSP) with each codec it can carry. The README has the matrix. - **Scale:** `-integ.scale=N` runs each case N times. A run of about 1,000 cases passes in about 11s. ## SRS bug fixes Every fix has a utest and, where a client can see it, an srs-integ case that failed before the fix. The main ones by area: - **HLS:** fMP4: - The default init file is now listed by its dir. - `hls_entry_prefix` now applies to fMP4 URIs. - MP3 streams now get an audio track in the init file. - Each publish now starts with a discontinuity. - `hls_keys` now encrypts the samples with cbcs. - A segment that fails to open is now dropped. Before, it crashed SRS. Other HLS fixes: key URIs, AES padding, HLS starts again after `hls_dispose`, the ratios, a TS republish, session names, and the `hls_ctx` of another stream's session is now checked as a new session. - **DASH:** the config, the MPD (window, update period, HEVC and MP3 codecs), init files of MP3 and of one-track streams, and dispose. - **WebRTC:** - Keyframes with an SEI, and Sender Reports and their mapping. - Opus timing, G.711, and WHEP SSRCs. - A refused WHIP publisher now frees its stream. - The start and hooks, and WebRTC over TCP. - **SRT:** only the configured latencies are set, as ffmpeg does. Also fixed: HEVC headers and CTS, frame counts, and the listener. - **RTSP:** a republish to a kept player, PLAY without SETUP, unknown methods, closes, DESCRIBE/TEARDOWN, request bodies and `on_stop`. - **GB28181:** - SSRC parsing. - HEVC to RTMP is no longer sent in legacy FLV tags. - A crash on a second media connection. - Latency with no audio, sliced frames, CTS, session lifetime, and the muxer and caster output. - **RTMP and HTTP:** - Timestamps that jump back without time correction no longer stop a player. - The republish param and the publish timeout after a pause. - The no-origin close and the RTMPS port. - HTTP ranges, including MP4 suffix ranges and VOD 416. - Redirects, mounts, unmounts and URL `..`. - CORS, API auth paths, JSONP 400 and 403 answers. - Static files and the request echo. - **Hooks:** - The heartbeat URL's query is now sent. - `on_play` clients and client types, and `on_dvr`. - A client that `on_connect` refused no longer gets `on_close`. - **Other:** DVR (MP4 info frame, audio rate, audio cut), listen ports above 65535, the env vhost check, the UDP caster, process stats, the tcmalloc rate, the encoder loop, the edge failover, STUN, coroutine joins, and the resource manager's fast-id cache (WebRTC and GB lookups never used it). ## Behavior changes - **Hooks:** - `on_publish`, `on_unpublish`, `on_play` and `on_stop` have a new `type` field after `client_id`, such as `rtmp-play` or `hls-play`, the same value as `/api/v1/clients`. - Every protocol now sends `param` with its leading `?`. - HTTP players send their own `http(s)://` tcUrl, and WHIP/WHEP clients send `webrtc://<host>/<app>`. - **API:** `/api/v1/features` adds `srt`, `rtsp`, `gb28181`, `rtc`, `simulator` and `sanitizer`. The `/api/v1/` index lists only the URLs SRS serves. - **Env support:** - The vhost security rules. - The edge protocol. - The origin cluster and coworkers. - The DVR apply filter. - **Timers in seconds:** `hls_dispose`, `dash_dispose`, `dvr_duration`, `queue_length`, `stun_timeout` and the heartbeat `interval` now accept fractions such as `0.5`. A fraction used to be cut to whole seconds, so `0.5` read as 0. - **Removed features:** a config or env var that names one of these, on or off, now makes SRS refuse to start (and `-t` fail) with `<name> is removed`: - HDS, including `--hds=on`. - The FLV POST caster and `asprocess`. - `query_latest_version` and `first_wait_for_qlv`. - The vhost `bandcheck`, `refer` and `atc_auto`. - Edge `token_traverse`. Check clients with the edge's HTTP hooks instead. ## Tests, CI and skills - **CI:** a new `integration-test` job builds `trunk/Dockerfile.test` (now with `--simulator=on`) and runs srs-integ. - It caches the test media. - It runs the main cases and the timer cases as separate steps, each retried on its own. - Limits are scaled by 4 with `-integ.limit-scale=4`. - **Unit tests:** more stable. They use random seeds, the test servers no longer share ports or leak fds, and a gtest listener fails a test that leaves a handler on a global resource manager. - **Skills:** - The `srs-develop` Gate and Full tiers run `make integ`. - `integration-tests.md` explains how to add cases. - The new `internal-audit-for-srs` skill audits each commit blind before review. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
279 lines
10 KiB
Go
279 lines
10 KiB
Go
// Copyright (c) 2026 Winlin
|
|
//
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
package cases
|
|
|
|
import (
|
|
"encoding/json"
|
|
"net/http"
|
|
"regexp"
|
|
"strconv"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
|
|
"srs-integ/internal/check"
|
|
"srs-integ/internal/media"
|
|
"srs-integ/internal/server"
|
|
)
|
|
|
|
// The RAW API's codes: the API is off, or a reload is not allowed; and an rpc
|
|
// it doesn't serve.
|
|
const (
|
|
rawDisabledCode = 1061
|
|
rawUnknownCode = 1060
|
|
)
|
|
|
|
// The reload states SRS reports: none yet, and a reload that failed to parse
|
|
// its config file, which an env-only server never has. 1023 is SRS's code
|
|
// for an invalid config.
|
|
const (
|
|
reloadStateInit = 0
|
|
reloadStateParsing = 10
|
|
reloadConfigErr = 1023
|
|
)
|
|
|
|
// rawUnservedRPCs are the rpc values the RAW API doesn't serve: none, empty,
|
|
// the query and update SRS turned off for good, a case it doesn't match and
|
|
// an unknown one.
|
|
var rawUnservedRPCs = []string{"", "?rpc=", "?rpc=query", "?rpc=update", "?rpc=RAW", "?rpc=none"}
|
|
|
|
// reloadIDPattern is a reload id, 7 random letters and digits.
|
|
var reloadIDPattern = regexp.MustCompile(`^[a-z0-9]{7}$`)
|
|
|
|
// rawPath is the RAW API's path with a query.
|
|
func rawPath(q string) string {
|
|
return "/api/v1/raw" + q
|
|
}
|
|
|
|
// checkRawCode checks that a RAW API request answers 200 with only a code.
|
|
func checkRawCode(c *check.C, srs *server.Server, q string, code int) {
|
|
c.Helper()
|
|
r := apiFetch(c, srs, rawPath(q))
|
|
c.CheckEqual(r.status, http.StatusOK, q+" status")
|
|
c.CheckEqual(r.header.Get("Content-Type"), "application/json", q+" content type")
|
|
c.CheckEqual(string(r.body), `{"code":`+strconv.Itoa(code)+`}`, q+" body")
|
|
}
|
|
|
|
// checkRawConfig checks the answer of rpc=raw: the API's own settings and
|
|
// the RAW API's, of which query and update are always off.
|
|
func checkRawConfig(c *check.C, srs *server.Server, enabled, allowReload bool) {
|
|
c.Helper()
|
|
v := apiObject(c, srs, rawPath("?rpc=raw"))
|
|
checkFields(c, v, []string{"code", "http_api"}, "rpc=raw")
|
|
api := object(c, v, "http_api")
|
|
checkFields(c, api, []string{"enabled", "listen", "crossdomain", "raw_api"}, "http_api")
|
|
c.CheckEqual(api["enabled"], true, "http_api enabled")
|
|
c.CheckEqual(api["listen"], strconv.Itoa(srs.Ports[server.HTTPAPI]), "http_api listen")
|
|
c.CheckEqual(api["crossdomain"], true, "http_api crossdomain")
|
|
raw := object(c, api, "raw_api")
|
|
checkFields(c, raw, []string{"enabled", "allow_reload", "allow_query", "allow_update"}, "raw_api")
|
|
c.CheckEqual(raw["enabled"], enabled, "raw_api enabled")
|
|
c.CheckEqual(raw["allow_reload"], allowReload, "raw_api allow_reload")
|
|
c.CheckEqual(raw["allow_query"], false, "raw_api allow_query")
|
|
c.CheckEqual(raw["allow_update"], false, "raw_api allow_update")
|
|
}
|
|
|
|
// reloadStatus is the answer of rpc=reload-fetch: the last reload's error
|
|
// code and message, its state and its id.
|
|
type reloadStatus struct {
|
|
Err int `json:"err"`
|
|
Msg string `json:"msg"`
|
|
State int `json:"state"`
|
|
Rid string `json:"rid"`
|
|
}
|
|
|
|
// reloadFetch gets the status of the last reload.
|
|
func reloadFetch(c *check.C, srs *server.Server) reloadStatus {
|
|
c.Helper()
|
|
v := apiObject(c, srs, rawPath("?rpc=reload-fetch"))
|
|
checkFields(c, v, []string{"code", "data"}, "rpc=reload-fetch")
|
|
data := object(c, v, "data")
|
|
checkFields(c, data, []string{"err", "msg", "state", "rid"}, "reload-fetch data")
|
|
b, err := json.Marshal(data)
|
|
c.Must(err, "encode reload-fetch data")
|
|
var st reloadStatus
|
|
c.Must(json.Unmarshal(b, &st), "decode reload-fetch data")
|
|
c.Check(reloadIDPattern.MatchString(st.Rid), "reload id %q, want 7 letters and digits", st.Rid)
|
|
return st
|
|
}
|
|
|
|
// reload asks for a reload by rpc=reload, which SRS answers at once, then
|
|
// waits until a reload with another id than old ran, and returns its status.
|
|
// SRS reloads in its main cycle, once a second.
|
|
func reload(c *check.C, srs *server.Server, old string) reloadStatus {
|
|
c.Helper()
|
|
checkRawCode(c, srs, "?rpc=reload", 0)
|
|
return waitReload(c, srs, old)
|
|
}
|
|
|
|
// waitReload waits until a reload with another id than old ran, and returns
|
|
// its status. SRS reloads in its main cycle, once a second.
|
|
func waitReload(c *check.C, srs *server.Server, old string) reloadStatus {
|
|
c.Helper()
|
|
var st reloadStatus
|
|
if !until(2*time.Second, func() bool {
|
|
st = reloadFetch(c, srs)
|
|
return st.Rid != old
|
|
}) {
|
|
c.Fatalf("no reload after 2s: the reload id is still %v", old)
|
|
}
|
|
return st
|
|
}
|
|
|
|
// checkReloadFailed checks the status of a reload of an env-only server: it
|
|
// failed to parse a config file, since there is none.
|
|
func checkReloadFailed(c *check.C, st reloadStatus) {
|
|
c.Helper()
|
|
c.CheckEqual(st.State, reloadStateParsing, "reload state")
|
|
c.CheckEqual(st.Err, reloadConfigErr, "reload err")
|
|
c.Check(st.Msg != "" && st.Msg != "Success", "reload msg %q, want the parse error", st.Msg)
|
|
}
|
|
|
|
// apiRaw adds the cases of the RAW API off, as on Common: rpc=raw still
|
|
// answers, every other rpc is refused.
|
|
func apiRaw(t *testing.T) {
|
|
// RawOffQuery reads the RAW API's settings, which rpc=raw answers even
|
|
// with the API off.
|
|
check.Case(t, "RawOffQuery", check.Fast, func(c *check.C) {
|
|
checkRawConfig(c, server.Common(c), false, false)
|
|
})
|
|
|
|
// RawOffRefused asks for a reload, its status, and each rpc the API
|
|
// doesn't serve: each is refused as the API being off.
|
|
check.Case(t, "RawOffRefused", check.Normal, func(c *check.C) {
|
|
srs := server.Common(c)
|
|
for _, q := range append([]string{"?rpc=reload", "?rpc=reload-fetch"}, rawUnservedRPCs...) {
|
|
checkRawCode(c, srs, q, rawDisabledCode)
|
|
}
|
|
})
|
|
|
|
// RawJsonp reads rpc=raw with a JSONP callback: the same answer wrapped
|
|
// in it.
|
|
check.Case(t, "RawJsonp", check.Fast, func(c *check.C) {
|
|
srs := server.Common(c)
|
|
plain := apiFetch(c, srs, rawPath("?rpc=raw"))
|
|
r := apiFetch(c, srs, rawPath("?rpc=raw&callback=cb"))
|
|
c.CheckEqual(r.status, http.StatusOK, "status")
|
|
c.CheckEqual(r.header.Get("Content-Type"), "text/javascript", "content type")
|
|
c.CheckEqual(string(r.body), "cb("+string(plain.body)+")", "body")
|
|
})
|
|
}
|
|
|
|
// TestApi_RawApi checks the RAW API on with reload off, on the shared
|
|
// RemuxOff, whose other groups never use the API. They publish segments at
|
|
// once, which can hold an API answer past the Fast limit, so the cases are
|
|
// Normal.
|
|
func TestApi_RawApi(t *testing.T) {
|
|
t.Parallel()
|
|
srs := server.Shared(t, "RemuxOff", sharedRemuxOff())
|
|
|
|
check.Case(t, "RawQuery", check.Normal, func(c *check.C) {
|
|
checkRawConfig(c, srs, true, false)
|
|
})
|
|
|
|
// ReloadNotAllowed asks for a reload, which is refused with the code of
|
|
// the API being off.
|
|
check.Case(t, "ReloadNotAllowed", check.Normal, func(c *check.C) {
|
|
checkRawCode(c, srs, "?rpc=reload", rawDisabledCode)
|
|
})
|
|
|
|
// ReloadFetchBeforeReload reads the status of a server that never
|
|
// reloaded: no error, no state, and an id that stays the same.
|
|
check.Case(t, "ReloadFetchBeforeReload", check.Normal, func(c *check.C) {
|
|
st := reloadFetch(c, srs)
|
|
c.CheckEqual(st, reloadStatus{Err: 0, Msg: "Success", State: reloadStateInit, Rid: st.Rid}, "reload status")
|
|
c.CheckEqual(reloadFetch(c, srs).Rid, st.Rid, "the reload id of a second fetch")
|
|
})
|
|
|
|
// UnknownRpc asks for each rpc the API doesn't serve.
|
|
check.Case(t, "UnknownRpc", check.Normal, func(c *check.C) {
|
|
for _, q := range rawUnservedRPCs {
|
|
checkRawCode(c, srs, q, rawUnknownCode)
|
|
}
|
|
})
|
|
}
|
|
|
|
// reloadMu makes the reloads of SegmentsKeep, by the RAW API or by SIGHUP,
|
|
// run one at a time, since SRS keeps the status of the last reload only.
|
|
var reloadMu sync.Mutex
|
|
|
|
// lockReload holds reloadMu, out of the case time, until the case ends.
|
|
func lockReload(c *check.C) {
|
|
c.Untimed(func() {
|
|
reloadMu.Lock()
|
|
c.Cleanup(reloadMu.Unlock)
|
|
})
|
|
}
|
|
|
|
// TestTimer_Reload_RawApi reloads by the RAW API, on the shared timer server
|
|
// SegmentsKeep with reload allowed. The server is env-only, so each reload
|
|
// fails to parse a config file and changes nothing for the other groups;
|
|
// SRS keeps the status of the last reload only, so the cases run one at a
|
|
// time, with those of TestTimer_Reload_Sighup.
|
|
func TestTimer_Reload_RawApi(t *testing.T) {
|
|
t.Parallel()
|
|
srs := server.Shared(t, "SegmentsKeep", sharedSegmentsKeep())
|
|
lock := lockReload
|
|
|
|
check.Case(t, "RawQuery", check.Normal, func(c *check.C) {
|
|
checkRawConfig(c, srs, true, true)
|
|
})
|
|
|
|
// ReloadWithoutFile reloads: SRS answers code 0 at once, and reloads in
|
|
// its next cycle, within about 1s, where it fails to parse the config
|
|
// file an env-only server doesn't have.
|
|
check.Case(t, "ReloadWithoutFile", check.Timer, func(c *check.C) {
|
|
lock(c)
|
|
before := reloadFetch(c, srs)
|
|
start := time.Now()
|
|
st := reload(c, srs, before.Rid)
|
|
took := time.Since(start)
|
|
checkReloadFailed(c, st)
|
|
c.Check(took <= 1500*time.Millisecond, "the reload ran after %v, want within the 1s cycle", took)
|
|
c.CheckEqual(reloadFetch(c, srs), st, "the status of a second fetch")
|
|
})
|
|
|
|
// EachReloadNewId reloads twice, one after the other: each reload has an
|
|
// id of its own and the same failure.
|
|
check.Case(t, "EachReloadNewId", check.Timer, func(c *check.C) {
|
|
lock(c)
|
|
first := reload(c, srs, reloadFetch(c, srs).Rid)
|
|
second := reload(c, srs, first.Rid)
|
|
checkReloadFailed(c, first)
|
|
checkReloadFailed(c, second)
|
|
c.Check(first.Rid != second.Rid, "both reloads have the id %v", first.Rid)
|
|
})
|
|
|
|
// StreamPlaysAcrossReload reloads while a stream plays: the publisher
|
|
// stays, the player gets media after the reload, and the RAW API's
|
|
// settings are the same.
|
|
check.Case(t, "StreamPlaysAcrossReload", check.Timer, func(c *check.C) {
|
|
lock(c)
|
|
stream := streamName(c)
|
|
_, pubID, streamID := publishRTMP(c, srs, stream, media.H264HighAAC)
|
|
player, _ := playRTMPOf(c, srs, stream, nil)
|
|
var last atomic.Int64
|
|
player.SetDeadline(time.Time{})
|
|
go func() {
|
|
for {
|
|
if _, err := player.ReadTag(); err != nil {
|
|
return
|
|
}
|
|
last.Store(time.Now().UnixNano())
|
|
}
|
|
}()
|
|
raw := apiFetch(c, srs, rawPath("?rpc=raw"))
|
|
|
|
checkReloadFailed(c, reload(c, srs, reloadFetch(c, srs).Rid))
|
|
done := time.Now().UnixNano()
|
|
c.Check(until(500*time.Millisecond, func() bool { return last.Load() > done }), "the player got no media in 500ms after the reload")
|
|
pub := object(c, apiStreamOf(c, srs, streamID), "publish")
|
|
c.CheckEqual(pub["active"], true, "stream publish active")
|
|
c.CheckEqual(pub["cid"], pubID, "stream publish cid, the publisher")
|
|
c.CheckEqual(string(apiFetch(c, srs, rawPath("?rpc=raw")).body), string(raw.body), "rpc=raw after the reload")
|
|
})
|
|
}
|