Token vs. rule

Perl 6 distinguishes between "regex", "token", and "rule".

A regex corresponds to what we know of as a regular expression in Perl 5 (except regexes have a new syntax -- see S05).

Tokens and rules are regexes that don't backtrack by default.

This makes them especially useful for language parsers, where backtracking is generally not needed.