Pattern-Based Text Extraction and Standardization with Lookup Tables

Extracts information from text using lookup tables of regular expressions. Each text entry is compared against all patterns, and all matching patterns and their corresponding substrings are returned. If a text entry matches multiple patterns, multiple rows are generated to capture each match. This approach enables comprehensive pattern coverage when processing large or complex text datasets.

regextable

Description

regextable extracts regular-expression-based pattern matches from a vector of text using a lookup table of regular expressions. It requires two inputs:

1. data: A vector of text to search (typically a data frame with a text column)
2. regex_table: A lookup table (a data frame with a column of strings or regular expressions to search for, typically called pattern)

For each matching substring, regextable::extract returns

• the row number of data
• the pattern
• the matched substring
• Optionally, other columns in data or regex_table

Installation

devtools::install_github("judgelord/regextable")

library(regextable)

Data

The examples below use the example regex lookup table members and example data cr2007_03_01 from the legislators package, which are also included in this package for illustration.

data("members")
head(members)
#> # A tibble: 6 × 9
#>   congress chamber   bioname                         pattern       icpsr state_abbrev district_code first_name last_name
#>      <dbl> <chr>     <chr>                           <chr>         <dbl> <chr>                <dbl> <chr>      <chr>    
#> 1      110 President BUSH, George Walker             "george bush… 99910 USA                      0 George     BUSH     
#> 2      110 House     BONNER, Jr., Josiah Robins (Jo) "josiah bonn… 20300 AL                       1 Josiah     BONNER   
#> 3      110 House     ROGERS, Mike Dennis             "mike rogers… 20301 AL                       3 Mike       ROGERS   
#> 4      110 House     DAVIS, Artur                    "artur davis… 20302 AL                       7 Artur      DAVIS    
#> 5      110 House     CRAMER, Robert E. (Bud), Jr.    "robert cram… 29100 AL                       5 Robert     CRAMER   
#> 6      110 House     EVERETT, Robert Terry           "robert ever… 29300 AL                       2 Robert     EVERETT

data("cr2007_03_01")
head(cr2007_03_01)
#> # A tibble: 6 × 5
#>   date       text                                header                                                    url   url_txt
#>   <date>     <chr>                               <chr>                                                     <chr> <chr>  
#> 1 2007-03-01 HON. SAM GRAVES;Mr. GRAVES          RECOGNIZING JARRETT MUCK FOR ACHIEVING THE RANK OF EAGLE… http… https:…
#> 2 2007-03-01 HON. MARK UDALL;Mr. UDALL           INTRODUCING A CONCURRENT RESOLUTION HONORING THE 50TH AN… http… https:…
#> 3 2007-03-01 HON. JAMES R. LANGEVIN;Mr. LANGEVIN BIOSURVEILLANCE ENHANCEMENT ACT OF 2007; Congressional R… http… https:…
#> 4 2007-03-01 HON. JIM COSTA;Mr. COSTA            A TRIBUTE TO THE LIFE OF MRS. VERNA DUTY; Congressional … http… https:…
#> 5 2007-03-01 HON. SAM GRAVES;Mr. GRAVES          RECOGNIZING JARRETT MUCK FOR ACHIEVING THE RANK OF EAGLE… http… https:…
#> 6 2007-03-01 HON. SANFORD D. BISHOP;Mr. BISHOP   IN HONOR OF SYNOVUS BEING NAMED ONE OF THE BEST COMPANIE… http… https:…

Text cleaning

Before matching, by default, clean_text() is applied to standardize text for better matching in messy text. It converts text to lowercase, removes excess punctuation, replaces line breaks and dashes with spaces, and collapses multiple spaces into a single space. Text cleaning is applied only during matching and does not modify the original input data. Users can disable this behavior by setting do_clean_text = FALSE.

text <- "  HELLO---WORLD  "
cleaned_text <- clean_text(text)
print(cleaned_text)
#> [1] "hello world"

Extract regex-based matches from text

Description

extract() performs regex-based matching on a text column using a pattern lookup table. All patterns that match each row are returned, along with the corresponding pattern and optional metadata from the pattern table. If multiple patterns match the same text, multiple rows are returned, one per match.

Required Parameters

• data: A data frame or character vector containing the text to search.
• regex_table: A regex lookup table with at least one pattern column.

Optional Parameters

• col_name: (default "text") Column name in the data frame containing text to search through.
• pattern_col: (default "pattern") Name of the regex pattern column in regex_table.
• data_return_cols: (default NULL) Vector of additional columns from data to include in the output.
• regex_return_cols: (default NULL) Vector of additional columns from regex_table to include in the output.
• date_col: (default NULL) Column in data containing dates for filtering.
• date_start: (default NULL) Start date for filtering rows.
• date_end: (default NULL) End date for filtering rows.
• remove_acronyms: (default FALSE) If TRUE, removes all-uppercase patterns from regex_table.
• do_clean_text: (default TRUE) If TRUE, cleans text before matching.
• verbose: (default TRUE) If TRUE, displays progress messages.
• cl: (default NULL) A cluster object or integer specifying child processes for parallel evaluation (ignored on Windows).

Returns

A data frame with one row per match, including:

• row_id: the internal row number of the text in the input data

• Optional columns from the input data (if data_return_cols specified)

• Optional columns from the regex table (if regex_return_cols specified)

• pattern: the regex pattern matched

• match: the substring matched in the text

• pattern, the first regex pattern matched in each row

• row_id, the row number of the text

• Additional columns from data specified in data_return_cols

• Additional columns from regex_table specified in regex_return_cols

Basic Usage

The simplest use of extract() with only the required arguments and returned columns specified. This finds all matches in the text column using the provided regex table.

#Extract patterns using only required arguments
result <- extract(
  data = cr2007_03_01,
  regex_table = members,
  data_return_cols = c("text"),
  regex_return_cols = c("icpsr") 
)

head(result)
#> # A tibble: 6 × 5
#>   row_id text                                icpsr pattern                                                         match
#>    <int> <chr>                               <dbl> <chr>                                                           <chr>
#> 1      1 HON. SAM GRAVES;Mr. GRAVES          20124 "samuel graves|\\bs graves|sam graves|(^|senator |representati… SAM …
#> 2      2 HON. MARK UDALL;Mr. UDALL           29906 "mark udall|\\bm udall|mark e udall|\\bna udall|(^|senator |re… MARK…
#> 3      3 HON. JAMES R. LANGEVIN;Mr. LANGEVIN 20136 "james langevin|\\bj langevin|james r langevin|jim langevin|ji… jame…
#> 4      4 HON. JIM COSTA;Mr. COSTA            20501 "jim costa|\\bj costa|james costa|(^|senator |representative )… JIM …
#> 5      5 HON. SAM GRAVES;Mr. GRAVES          20124 "samuel graves|\\bs graves|sam graves|(^|senator |representati… SAM …
#> 6      6 HON. SANFORD D. BISHOP;Mr. BISHOP   29339 "sanford bishop|sanford dixon bishop|\\bs bishop|sanford d bis… sanf…

Advanced Usage

Shows how to use optional arguments for more control, such as filtering by date ranges and removing acronyms. This is useful when you want to narrow matches, disable text cleaning, control returned columns, or suppress messages.

# Advanced usage with optional filters
result_advanced <- extract(
  data = cr2007_03_01,
  regex_table = members,
  date_col = "date",               
  date_start = "2007-01-01",
  date_end = "2007-12-31",
  remove_acronyms = TRUE,
  data_return_cols = c("text"),
  regex_return_cols = c("icpsr")
)

head(result_advanced)
#> # A tibble: 6 × 5
#>   row_id text                                icpsr pattern                                                         match
#>    <int> <chr>                               <dbl> <chr>                                                           <chr>
#> 1      1 HON. SAM GRAVES;Mr. GRAVES          20124 "samuel graves|\\bs graves|sam graves|(^|senator |representati… SAM …
#> 2      2 HON. MARK UDALL;Mr. UDALL           29906 "mark udall|\\bm udall|mark e udall|\\bna udall|(^|senator |re… MARK…
#> 3      3 HON. JAMES R. LANGEVIN;Mr. LANGEVIN 20136 "james langevin|\\bj langevin|james r langevin|jim langevin|ji… jame…
#> 4      4 HON. JIM COSTA;Mr. COSTA            20501 "jim costa|\\bj costa|james costa|(^|senator |representative )… JIM …
#> 5      5 HON. SAM GRAVES;Mr. GRAVES          20124 "samuel graves|\\bs graves|sam graves|(^|senator |representati… SAM …
#> 6      6 HON. SANFORD D. BISHOP;Mr. BISHOP   29339 "sanford bishop|sanford dixon bishop|\\bs bishop|sanford d bis… sanf…

Future Development

• Add support for typo_table to correct known text errors before matching.
• Improve strict matching rules for patterns that may need more inclusive or more restrictive word boundaries.
• Enable user-defined ID systems (e.g., corporations, campaigns) and control whether text is returned with matches.