### Before
- Only `totalCount - keepMin` pipelines were queried from the DB, causing the diff never to be evaluated
- The missing count then caused the later subsetting to be effectively non-functional, as it would include almost all pipelines due to the missing batch filtered out earlier
### After
- `totalCount`pipelines are queried from the DB in batches
- The sorting and filtering is done after the loop
- More helpful debug logs
Reviewed-on: https://codeberg.org/crowci/crow/pulls/79
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
follow-up #10 (which was a git hickup)
## Behaviour
Log purging is attempted in an asynchronous process before a new pipeline of a specific repo is started.
It does so by
1. Getting all existing pipelines
2. Filtering by `keepMin` and `keepDuration` settings
3. Calling `LogDelete` for all remaining pipelines
Deleting only logs instead of the full pipeline (which `crow-cli pipeline purge` does) is preferred to keep historic pipeline information. Storing this in the DB is just a single line and doesn't contain much content (in contrast to logs).
## Defaults
- No minimum count is kept (`CROW_DEFAULT_LOGS_PIPELINES_KEEP_MIN`)
- All pipelines of the last 90 days (per repo) are kept (`CROW_DEFAULT_LOGS_KEEP_DURATION`)
## Todo
- [x] implement purge call during pipeline start
- [x] add settings to DB column and repo settings
- [x] tests
- [x] think about defaults
- [x] Currently the purge happens on all pipelines in scope, including ones which have already been cleared. To avoid these unnecessary calls, which also will add up for repos with many pipelines, an indicator is needed which allows filtering these pipelines out.
fix#9
Co-authored-by: crowci-bot <admin@crowci.dev>
Reviewed-on: https://codeberg.org/crowci/crow/pulls/34
Co-authored-by: pat-s <patrick.schratz@gmail.com>
Co-committed-by: pat-s <patrick.schratz@gmail.com>
