Commit Graph
249 Commits
Author SHA1 Message Date
Sylvestre Ledru a15203bb8c tests: wc: ensure -c counts all of a file when descriptors run out
Reported at https://github.com/uutils/coreutils/issues/15742

* tests/wc/page-size.sh: Add a test case.

Link: https://github.com/coreutils/coreutils/pull/446
2026-11-07 12:35:10 +00:00
Collin Funk c2dd4d5d34 tests: wc: simplify construction of the input file
* tests/wc/page-size.sh: Prefer 'truncate' redirecting 'head /dev/zero'.
Suggested by Andreas Schwab.
2026-10-08 17:49:14 -07:00
Pádraig Brady f2d996bd1b tests: avoid NULs in test log
* tests/wc/page-size.sh: Remove NULs from logs,
also change from using Heredocs which are overkill.
2026-10-08 13:14:44 +01:00
Collin Funk 77e7674add tests: wc: add a test for the fix in the previous commit
* tests/wc/page-size.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-10-08 00:39:50 -07:00
Pádraig Brady 3aae63af9a wc: optimize ASCII and UTF8 character counting
Generate ASCII and UTF-8 data:
  $ yes $(yes eeeaae | head -n10K | paste -s -d,) | head -n10K > ll.in
  $ yes $(yes éééááé | head -n9 | paste -s -d,) | head -n1M > mb.in

ASCII is 14x faster:
  $ time src/wc-before -m < ll.in
  real	0m1.801s
  $ time src/wc-after -m < ll.in
  real	0m0.123s

UTF-8 is 7x faster:
  $ time src/wc-before -m < mb.in
  real	0m1.399s
  $ time src/wc-after -m < mb.in
  real	0m0.189s

* src/wc.c (wc): Use the u8_count() routine like cut(1),
to optimize counting valid UTF-8 characters.
* tests/wc/wc.pl: Add a test case.
* NEWS: Mention the improvement.
2026-10-03 15:47:15 +01:00
Pádraig Brady 5ef76b9ac8 wc: fix handling of multi-byte chars spanning buffers
Commit v9.4-43-ga6064bb86 introduced this edge case
in handling multi-byte characters spanning the input buffer.

* src/wc.c (wc): Ensure we include the bytes previously read
but as yet unprocessed.
* tests/wc/mb-non-utf8.sh: Add a new test for miscounted GB118030 chars.
* tests/local.mk: Reference the new test.
* tests/wc/wc.pl: Add a new check for word count with a
UTF-8 space spanning a buffer boundary.
* NEWS: Mention the bug fix.
2026-10-03 15:47:15 +01:00
Collin Funk 2db599e8b0 tests: wc: test -m in single byte and multi byte locales
* tests/wc/wc.pl ($mb_locale, $single_byte_locale): New variables.
(@mbTests): Add some test cases.
2026-07-22 18:52:55 -07:00
aizu-m 6a850b9aac wc: fix out-of-bounds read for single-byte wide characters
In legacy CJK locales a single input byte can decode to a wide character
above UCHAR_MAX, e.g. SHIFT-JIS bytes 0xA1..0xDF become U+FF61..U+FF9F.
wc then indexed its 256-entry wc_isprint/wc_isspace tables with that
value and read past their end.
* src/wc.c (wc): Take the byte-indexed fast path only when the decoded
character fits in a byte; wider values fall through to c32width and
c32isspace.
* tests/wc/wc-sjis.sh: New test.
* tests/local.mk (all_tests): Add it.
* NEWS: Mention the fix.

Link: https://github.com/coreutils/coreutils/pull/298
2026-06-19 15:39:10 +01:00
Collin Funk 236df1fa6a tests: wc,du: add additional --files0-from test cases
* tests/wc/wc-files0-from.pl ($limits): New variable.
(@Tests): Prefer the error strings from getlimits over writing them by
hand. Add test cases for --files0-from listing missing files and
duplicate files.
* tests/du/files0-from.pl ($limits): New variable.
(@Tests): Prefer the error strings from getlimits over writing them by
hand. Add test cases for --files0-from listing missing files. Add tests
for --files0-from listing duplicate files with and without the -l option
also in use.
2026-02-28 18:36:34 -08:00
Collin Funk 6a3dde5dd2 wc: add aarch64 Neon optimization for wc -l
Here is an example of the performance improvement:

    $ yes abcdefghijklmnopqrstuvwxyz | head -n 100000000 > input
    $ time ./src/wc-prev -l < input
    100000000

    real	0m0.793s
    user	0m0.630s
    sys	0m0.162s
    $ time ./src/wc -l < input
    100000000

    real	0m0.230s
    user	0m0.065s
    sys	0m0.164s

* NEWS: Mention the performance improvement.
* gnulib: Update to the latest commit.
* configure.ac: Check the the necessary intrinsics and functions.
* src/local.mk (noinst_LIBRARIES) [USE_NEON_WC_LINECOUNT]: Add
src/libwc_neon.a.
(src_libwc_neon_a_SOURCES, wc_neon_ldadd, src_libwc_neon_a_CFLAGS)
[USE_NEON_WC_LINECOUNT]: New variables.
(src_wc_LDADD) [USE_NEON_WC_LINECOUNT]: Add $(wc_neon_ldadd).
* src/wc.c [USE_NEON_WC_LINECOUNT]: Include sys/auxv.h and asm/hwcap.h.
(neon_supported) [USE_NEON_WC_LINECOUNT]: New function.
(wc_lines) [USE_NEON_WC_LINECOUNT]: Use neon_supported and
wc_lines_neon.
* src/wc.h (wc_lines_neon): Add declaration.
* src/wc_neon.c: New file.
* doc/coreutils.texi (Hardware Acceleration): Document the "-ASIMD"
hwcap and the variable used in ./configure to override detection of Neon
instructions.
* tests/wc/wc-cpu.sh: Also add "-ASIMD" to disable the use of Neon
instructions.
2026-02-18 20:09:44 -08:00
Collin Funk 73d3a49f45 maint: run 'make update-copyright' 2026-01-01 10:56:16 -08:00
Collin Funk 8dd89f55d5 tests: wc: protect against a hang on GNU/Hurd
Reported by Bruno Haible in
<https://lists.gnu.org/r/coreutils/2025-11/msg00051.html>.

* tests/wc/wc-total.sh: Skip a test that would exhaust memory on
GNU/Hurd.
2025-11-06 16:52:03 -08:00
Mathieu Bordere 67e9068c5f wc: add AVX512 function for line counting
* configure.ac: Add detection of AVX512 intrinsics for wc.
* src/local.mk: Build AVX512 wc libraries.
* src/wc.c: Add runtime detection of AVX512 intrinsics and call
appropriate function when detected.
* src/wc.h (wc_lines_avx512): Declare function.
* tests/wc/wc-cpu.sh: Add a test that disables AVX512 intrinsics.
* src/wc_avx512.c: New file containing the wc -l implementation using
AVX512. The logic and code is reused from the AVX2 implementation with
slight adaptations. Replaced __builtin_popcount by __builtin_popcountll
and the combination of _mm256_cmpeq_epi8 and _mm256_movemask_epi8 by a
single call to _mm512_cmpeq_epi8_mask.
* NEWS: Mention the improvement.
2025-09-30 14:09:37 +01:00
Pádraig Brady 0cc3e1bfb6 tests: wc: fix hardware acceleration disabling test
* tests/wc/wc-cpu.sh: The message is only printed with wc -l.
Reported by Mathieu Borderé.
2025-09-24 15:53:34 +01:00
Pádraig Brady 7bd3839cda cksum,wc: support disabling hardware acceleration at runtime
This is useful to give better test coverage at least,
and may be useful for users to tune their environment.

* bootstrap.conf: Reference the cpu-supports gnulib module.
* src/cksum.c: Use cpu_supports() rather than __builtin_cpu_supports().
* src/wc.c: Likewise.
* tests/cksum/cksum.sh: Adjust to testing all implementations.
* tests/wc/wc-cpu.sh: A new test to do likewise.
* tests/local.mk: Reference the new wc test.
2025-09-14 13:43:49 +01:00
Collin Funk c9a30d6781 maint: use consistent references to standard files in messages
* cfg.mk (sc_standard_outputs): Add a grep command for source files.
* src/du.c (main): Use standard input instead of stdin, standard output
instead of stdout, and standard error instead of stderr in messages.
* src/nohup.c (main): Likewise.
* src/sort.c (main): Likewise.
* src/split.c (main): Likewise.
* src/stdbuf.c (main): Likewise.
* src/wc.c (main): Likewise.
* tests/du/files0-from.pl (@Tests): Adjust test case to new messages.
* tests/sort/sort-files0-from.pl: Likewise.
* tests/wc/wc-files0-from.pl: Likewise.
2025-08-05 18:46:04 -07:00
Pádraig Brady 28b176085f maint: update all copyright year number ranges
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...

* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
2025-01-01 09:33:08 +00:00
Aearil feda512e97 wc: fix -w with breaking space over UCHAR_MAX
* src/wc.c (wc): Fix regression introduced in commit v9.4-48-gf40c6b5cf.
* tests/wc/wc-nbsh.sh: Add test cases for "standard" spaces.
Fixes https://bugs.gnu.org/69369
2024-02-26 17:33:02 +00:00
Pádraig Brady a966dcdb69 maint: update all copyright year number ranges
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...

* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
2024-01-01 13:27:23 +00:00
Paul Eggert 31076e8689 wc: fix word count bug
* bootstrap.conf (gnulib_modules): Remove c32isprint.
* src/wc.c (wc): Consider all non-white-space characters
to be word constituents, even if they are not printable.
POSIX requires this, and it is what BSD does.
Partly do this by simplifying the check for a word,
by counting word starts rather than word ends.
* tests/wc/wc.pl: Test for the bug.
2023-09-23 00:28:27 -07:00
Sylvestre Ledru 15925d0e5b tests: move tests to a directory per utility
* cfg.mk: Adjust syntax check exclusion paths.
* tests/local.mk: Adjust for renamed tests.
2023-06-19 13:12:37 +01:00
Jim Meyering d6ec836387 move wc tests from own subdir into a single script
* configure.ac (AC_CONFIG_FILES): Remove wc/Makefile from the list.
* tests/Makefile.am (SUBDIRS): Remove wc from the list.
(TESTS): Add misc/wc.
* tests/misc/wc: New file, derived from tests/wc/Tests.pm.
* tests/wc/Test.pm: Remove file.
2008-04-27 21:31:33 +02:00
Jim Meyering 6e3e819f4b Remove all .cvsignore files from version control. 2007-08-25 00:27:11 +02:00
Jim Meyering 71aa3ea880 Update all copyright notices to use the newer form. 2007-07-23 14:35:58 +02:00
Jim Meyering 33342c1a0a Change "version 2" to "version 3" in all copyright notices. 2007-07-10 13:35:56 +02:00
Jim Meyering 893ac688cc Remove the generated tests/*/Makefile.am files from version control.
* tests/cut/Makefile.am: git-remove this generated file.
* tests/head/Makefile.am: Likewise.
* tests/join/Makefile.am: Likewise.
* tests/pr/Makefile.am: Likewise.
* tests/sort/Makefile.am: Likewise.
* tests/tac/Makefile.am: Likewise.
* tests/tail/Makefile.am: Likewise.
* tests/test/Makefile.am: Likewise.
* tests/tr/Makefile.am: Likewise.
* tests/uniq/Makefile.am: Likewise.
* tests/wc/Makefile.am: Likewise.
* .cvsignore, .gitignore: Ignore these generated files.

* src/.cvsignore, src/.gitignore: Add chcon here, ...
* .cvsignore, .gitignore: ... not here.
2007-05-13 18:23:41 +02:00
Jim Meyering 9c88531fb0 Put CU_TEST_NAME in the environment for each test run by "make check".
* Transform all Makefile.am files so that when running "make check",
CU_TEST_NAME is set to the name of the test.  This is so that when I
run valgrind-enabled (--log-file-qualifier=CU_TEST_NAME) "make check"
on the entire package it is more convenient to map a leak or error
found in a valgrind log file back to the offending test.
Use this command:
(echo tests/Makefile.am.in; find tests -name Makefile.am) \
 |xargs perl -pi -e '/^(\s*)PATH=...VG_PATH_PREFIX/ and ' \
   -e 'print $1,q|CU_TEST_NAME=`basename $(abs_srcdir)`,$$tst |,"\\\n"'
2007-01-13 11:32:19 +01:00
Paul Eggert 6d124bf98a * aclocal.m4, config.hin, configure:
Remove from CVS, since ./bootstrap generates them automatically.
* .cvsignore: Add INSTALL, Makefile.in, *.cache, *.lineno, *.log.
Remove more-specific entries.  This catches files like configure.lineno.
* man/.cvsignore: Add Makefile.in.
* src/.cvsignore: Add Makefile.in.
Remove .version, dir.c, install, mvdir, stamp-v, vdir.c, version.c.

For .cvsignore file under the tests directory:
Add Makefile.in.  Sort entries if necessary.  Remove *.I, *.E,
*.X, *.O, *-tests, build-script, mk-script if they're never
created in this directory.

* build-aux/.cvsignore: New file.
* doc/.cvsignore: Add Makefile.in, coreutils.html, coreutils.pdf,
coreutils.ps, coreutils.tps.  Remove coreutils.cm (dunno what it
is, but the makefile doesn't mention it).  Remove coreutils.info
as it is subsumed by coreutils.info*.
* lib/.cvsignore: Add Makefile.in, getdate.tab.h.
Remove stat.c, sysexit.h.
* m4/.cvsignore: Remove Makefile, dev-ino.m4, glibc2.m4, intdiv0.m4,
inttypes-h.m4, inttypes-pri.m4, jm-glibc-io.m4, lcmessage.m4,
lock.m4, printf-posix.m4, same-inode.m4, size_max.m4,
uintmax_t.m4, ulonglong.m4, visibility.m4, xsize.m4.  Many of
these are put in automatically by ../bootstrap, but that's a
bootstrap bug that I plan to fix shortly.
* po/.cvsignore: More ../bootstrap-related fixes, plus remove old
cruft.  Add *.po, LINGUAS, Makevars.  Remove *.cat, *.msg,
cat-id-dbl.c, messages.mo, stamp-cat-id.
2006-08-22 18:50:24 +00:00
Paul Eggert 0f7a044109 Remove from CVS, since the bootstrap script generates them automatically. 2006-08-21 07:26:38 +00:00
Jim Meyering 03ffc923e9 . 2006-08-19 19:34:16 +00:00
Jim Meyering 9b3bb934bc Some of my 2006-07-03 changes to tests/*/Makefile.am were being
backed out due to updates provoked by the copyright changes.
* tests/Makefile.am.in (PATH): Prepend $(VG_PATH_PREFIX), so that
it propagates to the derived Makefile.am files.
($(srcdir)/Makefile.am): Mark generated .am files as read-only,
so we don't mistakenly edit them again.
* tests/cut/Makefile.am: Regenerate.
* tests/head/Makefile.am: Likewise.
* tests/join/Makefile.am: Likewise.
* tests/pr/Makefile.am: Likewise.
* tests/sort/Makefile.am: Likewise.
* tests/tac/Makefile.am: Likewise.
* tests/tail/Makefile.am: Likewise.
* tests/test/Makefile.am: Likewise.
* tests/tr/Makefile.am: Likewise.
* tests/uniq/Makefile.am: Likewise.
* tests/wc/Makefile.am: Likewise.
2006-08-19 16:02:11 +00:00
Paul Eggert 218fa5c61b Add/fix copyright notices and adjust to latest GNU FDL. 2006-08-17 19:58:17 +00:00
Jim Meyering e5b0d0aa85 . 2006-08-15 12:05:34 +00:00
Paul Eggert ade68b234a Regenerate. 2006-08-09 23:46:10 +00:00
Paul Eggert acaeed035b Regenerate. 2006-08-09 20:24:38 +00:00
Paul Eggert a58ac00556 Regenerate. 2006-07-17 03:52:18 +00:00
Paul Eggert 914c8674a5 Regenerate. 2006-07-09 17:20:26 +00:00
Jim Meyering ae87c86b4a * tests/Makefile.am, tests/*/Makefile.am: (TESTS_ENVIRONMENT):
Add $VG_PATH_PREFIX as a prefix to $PATH
2006-07-03 12:55:25 +00:00
Jim Meyering 9980147451 . 2006-05-06 15:52:56 +00:00
Jim Meyering a35fc49153 . 2006-03-26 12:33:28 +00:00
Jim Meyering de33c6c49d . 2006-03-12 22:47:35 +00:00
Jim Meyering 3f7e0c2173 . 2006-02-20 13:00:54 +00:00
Jim Meyering 7dabfc7e73 . 2006-01-12 08:40:34 +00:00
Jim Meyering 084903a2b1 . 2006-01-04 21:06:39 +00:00
Jim Meyering 1e1938032b . 2005-12-21 10:23:51 +00:00
Jim Meyering c99e1d2cb6 . 2005-12-17 10:48:30 +00:00
Jim Meyering 049a807bc6 . 2005-12-15 14:47:36 +00:00
Jim Meyering b3e300393e . 2005-12-03 23:19:58 +00:00
Jim Meyering 66878248f6 . 2005-11-22 15:33:39 +00:00
Jim Meyering 708c4e6682 . 2005-10-23 16:00:59 +00:00