### Before
- Only `totalCount - keepMin` pipelines were queried from the DB, causing the diff never to be evaluated
- The missing count then caused the later subsetting to be effectively non-functional, as it would include almost all pipelines due to the missing batch filtered out earlier
### After
- `totalCount`pipelines are queried from the DB in batches
- The sorting and filtering is done after the loop
- More helpful debug logs
Reviewed-on: https://codeberg.org/crowci/crow/pulls/79
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
follow-up #10 (which was a git hickup)
## Behaviour
Log purging is attempted in an asynchronous process before a new pipeline of a specific repo is started.
It does so by
1. Getting all existing pipelines
2. Filtering by `keepMin` and `keepDuration` settings
3. Calling `LogDelete` for all remaining pipelines
Deleting only logs instead of the full pipeline (which `crow-cli pipeline purge` does) is preferred to keep historic pipeline information. Storing this in the DB is just a single line and doesn't contain much content (in contrast to logs).
## Defaults
- No minimum count is kept (`CROW_DEFAULT_LOGS_PIPELINES_KEEP_MIN`)
- All pipelines of the last 90 days (per repo) are kept (`CROW_DEFAULT_LOGS_KEEP_DURATION`)
## Todo
- [x] implement purge call during pipeline start
- [x] add settings to DB column and repo settings
- [x] tests
- [x] think about defaults
- [x] Currently the purge happens on all pipelines in scope, including ones which have already been cleared. To avoid these unnecessary calls, which also will add up for repos with many pipelines, an indicator is needed which allows filtering these pipelines out.
fix#9
Co-authored-by: crowci-bot <admin@crowci.dev>
Reviewed-on: https://codeberg.org/crowci/crow/pulls/34
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
- Use title and message in error message
- Errors: don't auto-remove popup, force click
- Show API method and endpoint in error message title
- Use black/white bg and notification type color only on the side
- Return errors as JSON instead of strings. Include the api endpoint in the error message.
- improve error messages when missing access token
- improve pipeling config load error messages
- Error messages are shown for 9s (instead of 3)
- remove custom `notifyError` function
- Increase popup width from 300px to 340px
Co-authored-by: crowci-bot <admin@crowci.dev>
Co-authored-by: Anbraten <6918444+anbraten@users.noreply.github.com>
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
Reviewed-on: https://codeberg.org/crowci/crow/pulls/56
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
- `mockery` can now be run in one call instead of multiple ones
- Use the recommended "packages" approach which will be the new default moving forward
- Enforce consistent naming of mock files
- Remove warnings when invoking `mockery`
- Get rid of *many* linter warnings 🎉️
Reviewed-on: https://codeberg.org/crowci/crow/pulls/38
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
remove some old environment and add all missing options to set the whole build-in environment on `cli exec` via flags
---
*Sponsored by Kithara Software GmbH*
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
This is the first step towards support for multiple forges (#138). It
inserts a forge using the currently existing env varaibles into db and
uses this forge from db later on in all places of the code.
closes#621
addresses #138
# TODO
- [x] add forges table
- [x] add id of forge to repo
- [x] use forge of repo
- [x] add forge from env vars to db if not exists
- [x] migrate repo.ForgeID to the newly generated forge
- [x] support cache with forge from repo
- [x] maybe add forge loading cache? (use LRU cache for forges, I expect
users to have less than 10 forges normally)
---------
Co-authored-by: qwerty287 <80460567+qwerty287@users.noreply.github.com>
We can not just update some records for steps, as we want the pipeline
engine as single source of truth but not manage the state.
And the server should only manage the state but not how pipelines work.
We can match the pipeline but neither workflows or steps 1:1, so we
"update" them as a whole by deleting existing workflow and step data and
insert the new info from engine.
close #3494
close #3472
---------
*Sponsored by Kithara Software GmbH*
---------
Co-authored-by: Robert Kaussow <xoxys@rknet.org>
This stores workflows and steps to DB even if it is not yet approved and
thus blocked.
I'm not really happy with this, because even though it is stored, it
must parse the pipeline again and set back the original UUID. If you
have any ideas how to fix/improve this just comment.
In addition, this allows to view step list and side panel for approved
pipelines, https://github.com/woodpecker-ci/woodpecker/pull/2345 is
partially not longer necessary.
Closes https://github.com/woodpecker-ci/woodpecker/issues/895
---------
Co-authored-by: 6543 <6543@obermui.de>
