re2

Ruby bindings to re2, an "efficient, principled regular expression library".

C++

re2 - safer regular expressions in Ruby

Ruby bindings to RE2, a “fast, safe, thread-friendly alternative to
backtracking regular expression engines like those used in PCRE, Perl, and
Python”.

Current version: 2.14.0
Bundled RE2 version: libre2.11 (2024-07-02)

RE2('h.*o').full_match?("hello")    #=> true
RE2('e').full_match?("hello")       #=> false
RE2('h.*o').partial_match?("hello") #=> true
RE2('e').partial_match?("hello")    #=> true
RE2('(\w+):(\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

Why RE2?
Usage
Requirements
Thanks
Contact
License
- Dependencies

Why RE2?

While recent
versions of
Ruby have improved defences against regular expression denial of service
(ReDoS) attacks, it is still possible for
users to craft malicious patterns that take a long time to process by using
syntactic features such as back-references, lookaheads and possessive
quantifiers. RE2 aims to
eliminate ReDoS by design:

Safety is RE2’s raison d’être.

RE2 was designed and implemented with an explicit goal of being able to
handle regular expressions from untrusted users without risk. One of its
primary guarantees is that the match time is linear in the length of the
input string. It was also written with production concerns in mind: the
parser, the compiler and the execution engines limit their memory usage by
working within a configurable budget – failing gracefully when exhausted –
and they avoid stack overflow by eschewing recursion.

— Why RE2?

Usage

Install re2 as a dependency:

# In your Gemfile
gem "re2"

# Or without Bundler
gem install re2

Include in your code:

require "re2"

Full API documentation automatically generated from the latest version is
available at https://mudge.name/re2/.

While re2 uses the same naming scheme as Ruby’s built-in regular expression
library (with Regexp and
MatchData), its API is slightly
different:

Compiling regular expressions

[!WARNING]
RE2’s regular expression syntax differs from PCRE and Ruby’s built-in
Regexp library, see the
official syntax page for more
details.

The core class is RE2::Regexp which
takes a regular expression as a string and compiles it internally into an RE2
object. A global function RE2 is available to concisely compile a new
RE2::Regexp:

re = RE2('(\w+):(\d+)')
#=> #<RE2::Regexp /(\w+):(\d+)/>
re.ok? #=> true

re = RE2('abc)def')
re.ok?   #=> false
re.error #=> "missing ): abc(def"

[!TIP]
Note the use of single quotes when passing the regular expression as
a string to RE2 so that the backslashes aren’t interpreted as escapes.

When compiling a regular expression, an optional second argument can be used to change RE2’s default options, e.g. stop logging syntax and execution errors to stderr with log_errors:

RE2('abc)def', log_errors: false)

See the API documentation for RE2::Regexp#initialize for all the available options.

Matching interface

There are two main methods for matching: RE2::Regexp#full_match? requires the regular expression to match the entire input text, and RE2::Regexp#partial_match? looks for a match for a substring of the input text, returning a boolean to indicate whether a match was successful or not.

RE2('h.*o').full_match?("hello")    #=> true
RE2('e').full_match?("hello")       #=> false

RE2('h.*o').partial_match?("hello") #=> true
RE2('e').partial_match?("hello")    #=> true

Submatch extraction

[!TIP]
Only extract the number of submatches you need as performance is improved
with fewer submatches (with the best performance when avoiding submatch
extraction altogether).

Both matching methods have a second form that can extract submatches as RE2::MatchData objects: RE2::Regexp#full_match and RE2::Regexp#partial_match.

m = RE2('(\w+):(\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

m[0] #=> "ruby:1234"
m[1] #=> "ruby"
m[2] #=> "1234"

m = RE2('(\w+):(\d+)').full_match("r")
#=> nil

RE2::MatchData supports retrieving submatches by numeric index or by name if present in the regular expression:

m = RE2('(?P<word>\w+):(?P<number>\d+)').full_match("ruby:1234")
#=> #<RE2::MatchData "ruby:1234" 1:"ruby" 2:"1234">

m["word"]   #=> "ruby"
m["number"] #=> "1234"

They can also be used with Ruby’s pattern matching:

case RE2('(\w+):(\d+)').full_match("ruby:1234")
in [word, number]
  puts "Word: #{word}, Number: #{number}"
else
  puts "No match"
end
# Word: ruby, Number: 1234

case RE2('(?P<word>\w+):(?P<number>\d+)').full_match("ruby:1234")
in word:, number:
  puts "Word: #{word}, Number: #{number}"
else
  puts "No match"
end
# Word: ruby, Number: 1234

By default, both full_match and partial_match will extract all submatches into the RE2::MatchData based on the number of capturing groups in the regular expression. This can be changed by passing an optional second argument when matching:

m = RE2('(\w+):(\d+)').full_match("ruby:1234", submatches: 1)
=> #<RE2::MatchData "ruby:1234" 1:"ruby">

[!WARNING]
If the regular expression has no capturing groups or you pass submatches: 0, the matching method will behave like its full_match? or
partial_match? form and only return true or false rather than
RE2::MatchData.

Scanning text incrementally

If you want to repeatedly match regular expressions from the start of some input text, you can use RE2::Regexp#scan to return an Enumerable RE2::Scanner object which will lazily consume matches as you iterate over it:

scanner = RE2('(\w+)').scan(" one two three 4")
scanner.each do |match|
  puts match.inspect
end
# ["one"]
# ["two"]
# ["three"]
# ["4"]

Searching simultaneously

RE2::Set represents a collection of
regular expressions that can be searched for simultaneously. Calling
RE2::Set#add with
a regular expression will return the integer index at which it is stored within
the set. After all patterns have been added, the set can be compiled using
RE2::Set#compile,
and then
RE2::Set#match
will return an array containing the indices of all the patterns that matched.

set = RE2::Set.new
set.add("abc")         #=> 0
set.add("def")         #=> 1
set.add("ghi")         #=> 2
set.compile            #=> true
set.match("abcdefghi") #=> [0, 1, 2]
set.match("ghidefabc") #=> [2, 1, 0]

Encoding

[!WARNING]
Note RE2 only supports UTF-8 and ISO-8859-1 encoding so strings will be
returned in UTF-8 by default or ISO-8859-1 if the :utf8 option for the
RE2::Regexp is set to false (any other encoding’s behaviour is undefined).

For backward compatibility: re2 won’t automatically convert string inputs to
the right encoding so this is the responsibility of the caller, e.g.

# By default, RE2 will process patterns and text as UTF-8
RE2(non_utf8_pattern.encode("UTF-8")).match(non_utf8_text.encode("UTF-8"))

# If the :utf8 option is false, RE2 will process patterns and text as ISO-8859-1
RE2(non_latin1_pattern.encode("ISO-8859-1"), utf8: false).match(non_latin1_text.encode("ISO-8859-1"))

Requirements

This gem requires the following to run:

Ruby 2.6 to 3.3

It supports the following RE2 ABI versions:

libre2.0 (prior to release 2020-03-02) to libre2.11 (2023-07-01 to 2024-07-02)

Native gems

Where possible, a pre-compiled native gem will be provided for the following platforms:

Linux
- aarch64-linux and arm-linux (requires glibc 2.29+)
- x86-linux and x86_64-linux (requires glibc 2.17+)
- musl-based systems such as Alpine are supported as long as a glibc-compatible library is installed
macOS x86_64-darwin and arm64-darwin
Windows x64-mingw32 and x64-mingw-ucrt

Verifying the gems

SHA256 checksums are included in the release notes for each version and can be checked with sha256sum, e.g.

$ gem fetch re2 -v 2.14.0
Fetching re2-2.14.0-arm64-darwin.gem
Downloaded re2-2.14.0-arm64-darwin
$ sha256sum re2-2.14.0-arm64-darwin.gem
3c922d54a44ac88499f6391bc2f9740559381deaf7f4e49eef5634cf32efc2ce  re2-2.14.0-arm64-darwin.gem

GPG signatures are attached to each release (the assets ending in .sig) and can be verified if you import our signing key 0x39AC3530070E0F75 (or fetch it from a public keyserver, e.g. gpg --keyserver keyserver.ubuntu.com --recv-key 0x39AC3530070E0F75):

$ gpg --verify re2-2.14.0-arm64-darwin.gem.sig re2-2.14.0-arm64-darwin.gem
gpg: Signature made Fri  2 Aug 12:39:12 2024 BST
gpg:                using RSA key 702609D9C790F45B577D7BEC39AC3530070E0F75
gpg: Good signature from "Paul Mucur <[email protected]>" [unknown]
gpg:                 aka "Paul Mucur <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7026 09D9 C790 F45B 577D  7BEC 39AC 3530 070E 0F75

The fingerprint should be as shown above or you can independently verify it with the ones shown in the footer of https://mudge.name.

Installing the `ruby` platform gem

[!WARNING]
We strongly recommend using the native gems where possible to avoid the need
for compiling the C++ extension and its dependencies which will take longer
and be less reliable.

If you wish to compile the gem, you will need to explicitly install the ruby platform gem:

# In your Gemfile with Bundler 2.3.18+
gem "re2", force_ruby_platform: true

# With Bundler 2.1+
bundle config set force_ruby_platform true

# With older versions of Bundler
bundle config force_ruby_platform true

# Without Bundler
gem install re2 --platform=ruby

You will need a full compiler toolchain for compiling Ruby C extensions (see
Nokogiri’s “The Compiler
Toolchain”)
plus the toolchain required for compiling the vendored version of RE2 and its
dependency Abseil which includes CMake, a compiler
with C++14 support such as clang 3.4 or
gcc 5 and a recent version of
pkg-config. On
Windows, you’ll also need pkgconf 2.1.0+ to avoid undefined reference
errors when attempting to
compile Abseil.

Using system libraries

If you already have RE2 installed, you can instruct the gem not to use its own vendored version:

gem install re2 --platform=ruby -- --enable-system-libraries

# If RE2 is not installed in /usr/local, /usr, or /opt/homebrew:
gem install re2 --platform=ruby -- --enable-system-libraries --with-re2-dir=/path/to/re2/prefix

Alternatively, you can set the RE2_USE_SYSTEM_LIBRARIES environment variable instead of passing --enable-system-libraries to the gem command.

Thanks

Thanks to Jason Woods who contributed the
original implementations of RE2::MatchData#begin and RE2::MatchData#end.
Thanks to Stefano Rivera who first contributed
C++11 support.
Thanks to Stan Hu for reporting a bug with empty
patterns and RE2::Regexp#scan, contributing support for libre2.11
(2023-07-01) and for vendoring RE2 and abseil and compiling native gems in
2.0.
Thanks to Sebastian Reitenbach for reporting
the deprecation and removal of the utf8 encoding option in RE2.
Thanks to Sergio Medina for reporting a bug when
using RE2::Scanner#scan with an invalid regular expression.
Thanks to Pritam Baral for contributing the
initial support for RE2::Set.
Thanks to Mike Dalessio for reviewing the
precompilation of native gems in 2.0.
Thanks to Peter Zhu for
ruby_memcheck and helping find
the memory leaks fixed in 2.1.3.
Thanks to Jean Boussier for contributing the
switch to Ruby’s TypedData API and the resulting garbage collection
improvements in 2.4.0.
Thanks to Manuel Jacob for reporting a bug
when passing strings with null bytes.

Contact

All issues and suggestions should go to GitHub Issues.

License

This library is licensed under the BSD 3-Clause License, see LICENSE.txt.

Dependencies

The source code of RE2 is distributed in the ruby platform gem. This code is licensed under the BSD 3-Clause License, see LICENSE-DEPENDENCIES.txt.

The source code of Abseil is distributed in the ruby platform gem. This code is licensed under the Apache License 2.0, see LICENSE-DEPENDENCIES.txt.

re2