Unstructured Data & Natural Language Processing
Keith Dillon
Spring 2020
Topic 2: Handling Text
This topic:¶
- Character encodings, unicode
- Regular expressions
- Tokenization, segmentation, & stemming
- Approximate sequence matching
Reading:
- https://www.oreilly.com/library/view/fluent-python/9781491946237/ch04.html
- J&M Chapter 2, "Regular Expressions, Text Normalization, and Edit Distance"
- "The Algorithm Design Manual," Chapter 8, Steven Skiena, 1997.
- http://www.langmead-lab.org/teaching-materials/
Text Processing Levels¶
- Character
- Words
- Sentences / multiple words
- Paragraphs / multiple sentences
- Document
- Corpus / multiple documents
Source: Taming Text, p 9
Character¶
- Character encodings
- Case (upper and lower)
- Punctuation
- Numbers
Words¶
- Word segmentation: dividing text into words. Fairly easy for English and other languages that use whitespace; much harder for languages like Chinese and Japanese.
- Stemming: the process of shortening a word to its base or root form.
- Abbreviations, acronyms, and spelling. All help understand words.
Sentences¶
- Sentence boundary detection is a well-understood problem in English, but is still not perfect.
- Phrase detection. San Francisco and quick red fox are examples of phrases.
- Parsing: breaking sentences down into subject-verb and other relation- ships often yields useful information about words and their relation- ships to each other.
- Combining the definitions of words and their relationships to each other to determine the meaning of a sentence.
Paragraphs¶
At this level, processing becomes more difficult in an effort to find deeper understanding of an author’s intent.
For example, algorithms for summarization often require being able to identify which sentences are more important than others.
Document¶
Similar to the paragraph level, understanding the meaning of a document often requires knowledge that goes beyond what’s contained in the actual document.
Authors often expect readers to have a certain background or possess certain reading skills.
Corpus¶
At this level, people want to quickly find items of interest as well as group related documents and read summaries of those documents.
Applications that can aggregate and organize facts and opinions and find relationships are particularly useful.
I. Character Encodings¶
Character Encodings - map character to binary¶
- ASCII - 7 bits
- char - 8-bit - a-z,A-Z,0-9,...
- multi-byte encodings for other languages
ascii(38)
str(38)
chr(38)
Unicode - we all hate it¶
Unicode - "code points"¶
lookup table of unique numbers denoting every possible character
Encoding still needed -- various standards
- UTF-8 (dominant standard),16 - variable-length
- UTF-32 - 4-byte
chr(2^30+1) # for python 3
Mojibake¶
Incorrect, unreadable characters shown when computer software fails to show text correctly.
It is a result of text being decoded using an unintended character encoding.
Very common in Japanese websites, hence the name:
文字 (moji) "character" + 化け (bake) "transform"
Good News: It is systematic (find the right encoding) and easy to fix
Example¶
# manually load text without specifying encoding
with open('shakespeare_all.txt') as f:
shakespeare = f.read()
shakespeare[:50]
FTFY module¶
shakespeare[:50]
import ftfy
ftfy.fix_text(shakespeare[:50])
help(ftfy)
What about the line breaks?
with open('shakespeare_all.txt') as f:
shakespeare = f.read().splitlines()
shakespeare[:10]
# Munging text
shakespeare = [ftfy.fix_text(line.strip()) for line in shakespeare if line]
shakespeare[:10]
Unicode Sandwich¶
best practice for handling text
"bytes should be decoded to str as early as possible on input (e.g., when opening a file for reading). The “meat” of the sandwich is the business logic of your program, where text handling is done exclusively on str objects. You should never be encoding or decoding in the middle of other processing. On output, the str are encoded to bytes as late as possible. Most web frameworks work like that, and we rarely touch bytes when using them."
https://www.oreilly.com/library/view/fluent-python/9781491946237/ch04.html
Summary¶
- Unicode is a necessary evil
- Alway try to keep it Unicode
- Be explicit about encodings
- If you see mojibake, don't ╯°□°）╯︵ ┻━┻. ftfy
II. String Processing and Regular Expressions¶
Strings¶
Array (list) of characters
a = "Hello"
b = "World"
print(a+b)
print(a+' '+b+'.')
from string import *
whos
Student Activity¶
Let's make a simple password generator function!
Your one-liner should return something like this:
'kZmuSUVeVC'
'mGEsuIfl91'
'FEFsWwAgLM'
import random
import string
n = 10
random = ''.join((random.choice(string.ascii_letters + string.digits) for n in range(n)))
Regular Expressions ("regex")¶
Used in grep, awk, ed, perl, ...
Regular expression is a pattern matching language, the "RE language".
A Domain Specific Language (DSL). Powerful (but limited language). E.g. SQL, Markdown.
from re import *
whos
Regex string functions¶
- findall() - Returns a list containing all matches
- search() - Returns a Match object if there is a match anywhere in the string
- split() - Returns a list where the string has been split at each match
- sub() - Replaces one or many matches with a string
- match() apply the pattern at the start of the string
Motivating example¶
Write a regex to match common misspellings of calendar: "calendar", "calandar", or "celender"
# Let's explore how to do this
# Patterns to match
dat0 = ["calendar", "calandar", "celender"]
# Patterns to not match
dat1 = ["foo", "cal", "calli", "calaaaandar"]
# Interleave them
string = " ".join([item
for pair in zip(dat0, dat1)
for item in pair])
string
# You match it with literals
literal1 = 'calendar'
literal2 = 'calandar'
literal3 = 'celender'
patterns = "|".join([literal1, literal2, literal3])
patterns
import re
print(re.findall(patterns, string))
... a better way¶
Let's write it with regex language
# A little bit of meta-programming (strings all the way down!)
sub_pattern = '[ae]'
pattern2 = sub_pattern.join(["c","l","nd","r"])
print(pattern2)
print(string)
re.findall(pattern2, string)
Regex Terms¶
- target string: This term describes the string that we will be searching, that is, the string in which we want to find our match or search pattern.
- search expression: The pattern we use to find what we want. Most commonly called the regular expression.
- literal: A literal is any character we use in a search or matching expression, for example, to find ind in windows the ind is a literal string - each character plays a part in the search, it is literally the string we want to find.
- metacharacter: A metacharacter is one or more special characters that have a unique meaning and are NOT used as literals in the search expression. For example "." means any character.
- escape sequence: An escape sequence is a way of indicating that we want to use one of our metacharacters as a literal.
function(search_expression, target_string)¶
pick function based on goal (find all matches, replace matches, find first match, ...)
form search expression to account for variations in target we allow. E.g. possible misspellings.
Metacharacters¶
special characters that have a unique meaning
Escape sequence "\"¶
A way of indicating that we want to use one of our metacharacters as a literal.
In a regular expression an escape sequence is metacharacter \ (backslash) in front of the metacharacter that we want to use as a literal.
Ex: If we want to find \file in the target string c:\file then we would need to use the search expression \\file (each \ we want to search for as a literal (there are 2) is preceded by an escape sequence ).
Special Escape Sequences¶
\A - specified characters at beginning of string. Ex: "\AThe"
\b - specified characters at beginning or end of a word. Ex: r"\bain" r"ain\b"
\B - specified characters present but NOT at beginning (or end) of a word. Ex: r"\Bain" r"ain\B"
\d - string contains digits (numbers from 0-9)
\D - string DOES NOT contain digits
\s - string contains a white space character
\S - string DOES NOT contain a white space character
\w - string contains any word characters (characters from a to Z, digits from 0-9, and the underscore _ character)
\W - string DOES NOT contain any word characters
\Z - specified characters are at the end of the string
Set¶
a set of characters inside a pair of square brackets [] with a special meaning:
[arn] one of the specified characters (a, r, or n) are present
[a-n] any lower case character, alphabetically between a and n
[^arn] any character EXCEPT a, r, and n
[0123] any of the specified digits (0, 1, 2, or 3) are present
[0-9] any digit between 0 and 9
[0-5][0-9] any two-digit numbers from 00 and 59
[a-zA-Z] any character alphabetically between a and z, lower case OR upper case
[+] In sets, +, *, ., |, (), $,{} has no special meaning, so [+] means any + character in the string
target_string = '415-805-1888'
pattern1 = '[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]'
pattern2 = '\d\d\d-\d\d\d-\d\d\d\d'
pattern3 = '\d{3}-\d{3}-\d{4}'
print(re.search(pattern1,target_string))
print(re.search(pattern2,target_string))
print(re.search(pattern3,target_string))
Great!
\d{3}-\d{3}-\d{4} uses Quantifiers.
Quantifiers: allow you to specify how many times the preceding expression should match.
{} is extact quantifier.
print(re.findall('x?','xxxy'))
print(re.findall('x+','xxxy'))
Capturing groups¶
Problem: You have odd line breaks in your text.
text = 'Long-\nterm problems with short-\nterm solutions.'
print(text)
Solution: Write a regex to find the "dash with line break" and replace it with just a line break.
import re
# 1st Attempt
text = 'Long-\nterm problems with short-\nterm solutions.'
re.sub('(\w+)-\n(\w+)', r'-', text)
Not right!
We need capturing groups!
Caputuring groups allow you to apply regex operators to the groups that have been matched by regex.
For for example, if you wanted to list all the image files in a folder. You could then use a pattern such as ^(IMG\d+\.png)$ to capture and extract the full filename, but if you only wanted to capture the filename without the extension, you could use the pattern ^(IMG\d+)\.png$ which only captures the part before the period.
re.sub('(\w+)-\n(\w+)', r'\1-\2', text)
The parentheses around the word characters (specified by \w) means that any matching text should be captured into a group.
The '\1' and '\2' specifiers refer to the text in the first and second captured groups.
"Long" and "term" are the first and second captured groups for the first match.
"short" and "term" are the first and second captured groups for the next match.
NOTE: 1-based indexing
Regex: Connection to statistical concepts¶
False positives (Type I error): Matching strings that we should not have matched
False negatives (Type II error): Not matching strings that we should have matched
Reducing the error rate for a task often involves two antagonistic efforts:
- Minimizing false positives
- Minimizing false negatives
In a perfect world, you would be able to minimize both but in reality you often have to trade one for the other.
III. Tokenization, segmentation, & stemming¶
By The End Of This section You Should Be Able To:¶
- Code two different ways of segmenting sentences
- Recognize there are different tokenizers
- List the difference among stemmers
- Explain the difference between stemming and lemmatization
4 ways to show mastery ¶
- Define the technical term in Plain English
- Calculate by hand
- Code by hand (i.e., above the API)
- Code by using the best library (i.e., below the API)
1. Plain English definition of sentence segmentation:¶
Sentence segmentation is dividing a stream of language into component sentences.
Sentences can be defined as a set of words that is complete in itself, typically containing a subject and predicate. Sentences typically are statements, questions, exclamations, or commands.
Sentence segmentation is typically done using punctuation, particularly the full stop character "." as a reasonable approximation.
However there are complications because full stop character is also used to end abbreviations, which may or may not also terminate a sentence.
For example, Dr. Evil.
2. Sentence segmentation by hand¶
A Confederacy Of Dunces
By John Kennedy Toole
A green hunting cap squeezed the top of the fleshy balloon of a head. The green earflaps, full of large ears and uncut hair and the fine bristles that grew in the ears themselves, stuck out on either side like turn signals indicating two directions at once. Full, pursed lips protruded beneath the bushy black moustache and, at their corners, sank into little folds filled with disapproval and potato chip crumbs. In the shadow under the green visor of the cap Ignatius J. Reilly’s supercilious blue and yellow eyes looked down upon the other people waiting under the clock at the D.H. Holmes department store, studying the crowd of people for signs of bad taste in dress.
sentence_1 = A green hunting cap squeezed the top of the fleshy balloon of a head.
sentence_2 = The green earflaps, full of large ears and uncut hair and the fine bristles that grew in the ears themselves, stuck out on either side like turn signals indicating two directions at once.
sentence_3 = Full, pursed lips protruded beneath the bushy black moustache and, at their corners, sank into little folds filled with disapproval and potato chip crumbs.
sentence_4 = In the shadow under the green visor of the cap Ignatius J. Reilly’s supercilious blue and yellow eyes looked down upon the other people waiting under the clock at the D.H. Holmes department store, studying the crowd of people for signs of bad taste in dress.
3. Code by hand (i.e., above the API)¶
text = """A green hunting cap squeezed the top of the fleshy balloon of a head. The green earflaps, full of large ears and uncut hair and the fine bristles that grew in the ears themselves, stuck out on either side like turn signals indicating two directions at once. Full, pursed lips protruded beneath the bushy black moustache and, at their corners, sank into little folds filled with disapproval and potato chip crumbs. In the shadow under the green visor of the cap Ignatius J. Reilly’s supercilious blue and yellow eyes looked down upon the other people waiting under the clock at the D.H. Holmes department store, studying the crowd of people for signs of bad taste in dress. """
import re
pattern = "|".join(['!', # end with "!"
'\?', # end with "?"
'\.\D', # end with "." and the full stop is not followed by a number
'\.\s']) # end with "." and the full stop is followed by a whitespace
re.split(pattern, text)
3. Code by hand (i.e., above the API): Version 2.0¶
pattern = "(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s"
re.split(pattern, text)
4. Code by using the best library (i.e., below the API)¶
from textblob import TextBlob
text = """A green hunting cap squeezed the top of the fleshy balloon of a head. The green earflaps, full of large ears and uncut hair and the fine bristles that grew in the ears themselves, stuck out on either side like turn signals indicating two directions at once. Full, pursed lips protruded beneath the bushy black moustache and, at their corners, sank into little folds filled with disapproval and potato chip crumbs. In the shadow under the green visor of the cap Ignatius J. Reilly’s supercilious blue and yellow eyes looked down upon the other people waiting under the clock at the D.H. Holmes department store, studying the crowd of people for signs of bad taste in dress. """
blob = TextBlob(text)
blob.sentences
Tokenization¶
Toenization: Breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens
The simplest way to tokenize is to split on white space
sentence1 = 'Sky is blue and trees are green'
sentence1.split(' ')
Sometimes you might also want to deal with abbreviations, hypenations, puntuations and other characters.
In those cases, you would want to use regex.
However, going through a sentence multiple times can be slow to run if the corpus is long
import re
sentence2 = 'This state-of-the-art technology is cool, isn\'t it?'
sentence2 = re.sub('-', ' ', sentence2)
sentence2 = re.sub('[,|.|?]', '', sentence2)
sentence2 = re.sub('n\'t', ' not', sentence2)
sentence2_tokens = re.split('\s+', sentence2)
print(sentence2_tokens)
In this case, there are 11 tokens and the size of the vocabulary is 10
print('Number of tokens:', len(sentence2_tokens))
print('Number of vocabulary:', len(set(sentence2_tokens)))
Most text manipulation packages have tokenize functions¶
from nltk.tokenize import word_tokenize
text1 = "It's true that the chicken was the best bamboozler in the known multiverse."
tokens = word_tokenize(text1)
print(tokens)
import nltk
help(nltk.tokenize)
Morphemes¶
A morpheme is the smallest unit of language that has meaning.
There are two types of morpheme:
- stems
- affixes (suffixes, prefixes, infixes, and circumfixes)
"believe" is a stem.
"un" and "able" are affixes.
What we want to do in NLP preprocessing is get the stem a stem by eliminating the affixes from a token.
Stemming¶
Stemming usually refers to a crude heuristic process that chops off the ends of words.
Ex: automates, automating and automatic could be stemmed to automat
from nltk import stem
porter = stem.porter.PorterStemmer()
porter.stem("cars")
porter.stem("octopi")
porter.stem("am")
There are 3 types of commonly used stemmers, and each consists of slightly different rules for systematically replacing affixes in tokens. In general, Lancaster stemmer stems the most aggresively, i.e. removing the most suffix from the tokens, followed by Snowball and Porter
Porter Stemmer:
- Most commonly used stemmer and the most gentle stemmers
- The most computationally intensive of the algorithms (Though not by a very significant margin)
- The oldest stemming algorithm in existence
Snowball Stemmer:
- Universally regarded as an improvement over the Porter Stemmer
- Slightly faster computation time than the Porter Stemmer
Lancaster Stemmer:
- Very aggressive stemming algorithm
- With Porter and Snowball Stemmers, the stemmed representations are usually fairly intuitive to a reader
- With Lancaster Stemmer, shorter tokens that are stemmed will become totally obfuscated
- The fastest algorithm and will reduce the vocabulary
- However, if one desires more distinction between tokens, Lancaster Stemmer is not recommended
from nltk import stem
tokens = ['player', 'playa', 'playas', 'pleyaz']
# Define Porter Stemmer
porter = stem.porter.PorterStemmer()
# Define Snowball Stemmer
snowball = stem.snowball.EnglishStemmer()
# Define Lancaster Stemmer
lancaster = stem.lancaster.LancasterStemmer()
print('Porter Stemmer:', [porter.stem(i) for i in tokens])
print('Snowball Stemmer:', [snowball.stem(i) for i in tokens])
print('Lancaster Stemmer:', [lancaster.stem(i) for i in tokens])
Lemmatization¶
Lemmatization aims to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
This is doing things properly with the use of a vocabulary and morphological analysis of words.
Check for understanding¶
How are stemming and lemmatization similar?
The goal of both stemming and lemmatization is to reduce inflectional forms and derivationally-related forms of a token to a common base form.
from textblob import Word
w = Word("cars")
w.lemmatize()
Word("octopi").lemmatize()
Word("am").lemmatize()
from textblob import Word
w = Word("litter")
w.definitions
Summary¶
- Tokenization separates words in a sentence
- You would normalize or process the sentence during tokenization to obtain sensible tokens
- These normalizations include:
- Replacing special characters with spaces such as
,.-=!using regex - Lowercasing
- Stemming to remove the suffix of tokens to make tokens more uniform
- Replacing special characters with spaces such as
- There are three types of commonly used stemmers. They are Porter, Snowball and Lancaster
- Lancaster is the fastest and also the most aggressive stemmer and Snowball is a good balance between speed and quality of stemming
IV. Approximate Sequence Matching¶
Exact Matching¶
Fing places where pattern $P$ is found within text $T$.
What python functions do this?
Also very important problem. Not trivial for massive datasets.
Alignment - compare $P$ to same-length substring of $T$ at some starting point.
How many calculations will this take in the most naive approach possible?
Improving on the Naive Exact-matching algorithm¶
Naive approach: test all possible alignments for match.
Ideas for improvement:
- stop comparing given alignment at first mismatch.
- use result of prior alignments to shorten or skip subsequent alignments.
The Boyer-Moore algorithm¶
Approximate matching: Motivational problems¶
- Matching Regular Expressions to text efficiently
- Biological sequence alignment between different species
- Matching noisy trajectories through space
- Clustering sequences into a few groups of most similar classes
- Applying k Nearest Neighbors classification to sequences
Matching time-series with varying timescales¶
How similar are these two curves, assuming we ignore varying timescales?
Example: we wish to determine location of hiker given altitude measurements during hike.
Note the amount of warping is often not the distance metric. We first "warp" the pattern, then compute distance some other way, e.g. LMS. Final distance as the closest LMS possible over all acceptable warpings.
Dynamic Time Warping
II. Dynamic Programming Review¶
Fibonacci sequence¶
\begin{align} f(n) &= f(n-1) + f(n-2) \\ f(0) &= 0 \\ f(1) &= 1 \\ \end{align}Recursive calculation¶
Inefficient due to repeatedly calculating same terms
def fib_recursive(n):
if n == 0: return 0
if n == 1: return 1
return fib_recursive(n-1) + fib_recursive(n-2)
Each term requires two additional terms be calculated. Exponential time.
DP calculation¶
Intelligently plan terms to calculate and store ("cache"), e.g. in a table.
Each term requires one term be calculated.
DP Caching¶
Always plan out the data strcture and calculation order.
- need to make sure you have sufficient space
- need to choose optimal strategy to fill in
The data structure to fill in for Fibonacci is trivial:
Optimal order is to start at bottom and work up to $n$, so always have what you need for next term.
What are caches?¶
Caches are storage for information to be used in the near future in a more accessible form.
The difference between dynamic programming and recursion¶
1) Direction
Recursion: Starts at the end/largest and divides into subproblems.
DP: Starts at the smallest and builds up a solution.
2) Amount of computation:
During recursion, the same sub-problems are solved multiple times.
DP is basically a memorization technique which uses a table to store the results of sub-problem so that if same sub-problem is encountered again in future, it could directly return the result instead of re-calculating it.
DP = apply common sense to do recursive problems more efficiently.
def fib_recursive(n):
if n == 0: return 0
if n == 1: return 1
return fib_recursive(n-1) + fib_recursive(n-2)
def fib_dp(n):
fib_seq = [0, 1]
for i in range(2,n+1):
fib_seq.append(fib_seq[i-1] + fib_seq[i-2])
return fib_seq[n]
Let's compare the runtime
%timeit -n4 fib_recursive(30)
%timeit -n4 fib_dp(100)
Important Point¶
The hardest parts of dynamic programming is:
- Recognizing when to use it
- Picking the right data structure for caching
Generator approach¶
def fib():
a, b = 0, 1
while True:
a, b = b, a+b
yield a
f = fib()
for i in range(10):
print(next(f))
from itertools import islice
help(islice)
n = 100
next(islice(fib(), n-1, n))
%timeit -n4 next(islice(fib(), n-1, n))
Memoization¶
A technique used in computing to speed up programs. This is accomplished by memorizing the calculation results of processed input such as the results of function calls.
If the same input or a function call with the same parameters is used, the previously stored results can be used again and unnecessary calculation are avoided.
In many cases a simple array is used for storing the results, but lots of other structures can be used as well, such as associative arrays, called hashes in Perl or dictionaries in Python.
Source: http://www.amazon.com/Algorithm-Design-Manual-Steve-Skiena/dp/0387948600
def fib_recursive(n):
if n == 0: return 0
if n == 1: return 1
return fib_recursive(n-1) + fib_recursive(n-2)
def fib_dp(n):
cache = [0,1]
for i in range(2,n+1):
cache.append(cache[i-1]+cache[i-2])
return cache[n]
def memoize(f):
memo = {}
def helper(x):
if x not in memo:
memo[x] = f(x)
return memo[x]
return helper
fib_recursive_memoized = memoize(fib_recursive)
fib_recursive_memoized(9)
%timeit fib_recursive(9)
%timeit fib_recursive_memoized(9)
Memoization in Python (LRU cache)¶
LRU = Least Recently Used
from functools import lru_cache
@lru_cache()
def fib_recursive(n):
"Calculate nth Fibonacci number using recursion"
if n == 0: return 0
if n == 1: return 1
return fib_recursive(n-1) + fib_recursive(n-2)
%timeit fib_recursive(n)
Memoization in Python (joblib)¶
Exercise: Change making problem.¶
The objective is to determine the smallest number of currency of a particular denomination required to make change for a given amount.
For example, if the denomination of the currency are \$1 and \\$2 and it was required to make change for \$3 then we would use \\$1 + \$2 i.e. 2 pieces of currency.
However if the amount was \$4 then we would could either use \\$1+\$1+\\$1+\$1 or \\$1+\$1+\\$2 or \$2+\\$2 and the minimum number of currency would 2 (\$2+\\$2).
Solution: dynamic programming (DP).¶
The minimum number of coins required to make change for \$P is the number of coins required to make change for the amount \\$P-x plus 1 (+1 because we need another coin to get us from \$P-x to P).
These can be illustrated mathematically as:
Let us assume that we have $n$ currency of distinct denomination. Where the denomination of the currency $i$ is $v_i$. We can sort the currency according to denomination values such that $v_1<v_2<v_3<..<v_n$
Let us use $C(p)$ to denote the minimum number of currency required to make change for $ \$p$
Using the principles of recursion $C(p)=min_i C(p-v_i)+1$
For example, assume we want to make 5, and $v_1=1, v_2=2, v_3=3$.
Therefore $C(5) = min(C(5-1)+1, C(5-2)+1, C(5-3)+1)$ $\Longrightarrow min(C(4)+1, C(3)+1, C(2)+1)$
Exercise: Compute a polynomial over a grid of points¶
How can we use redundancy here?
DP for RE matching¶
https://sites.google.com/site/jennyshelloworld/company-blog/regular-expression-matching---dp
Things get tricky when using wildcards ".", "?", "*"
Mini-summary¶
In a recursive approach, your function recursively calls itself on smaller subproblems as needed, until the calculation is done. Potentially performing many redundant calculations naively.
With memoization, you form a cache of these recursive calls, and check if the same one is called again before recalculating. Still potential danger since do not plan memory needs or usage.
With Dynamic Programming, you generally follow some plan to produce the cache of results for smaller subproblems.
III. Edit Distance¶
Edit Distance between two strings¶
Minimum number of operations needed to convert one string into other
Ex. typo correction: how do we decide what "scool" was supposed to be?
Consider possibilities with lowest edit distance. "school" or "cool".
Hamming distance - operations consist only of substitutions of character (i.e. count differences)
Levenshtein distance - operations are the removal, insertion, or substitution of a character
"Fuzzy String Matching"
def hammingDistance(x, y):
''' Return Hamming distance between x and y '''
assert len(x) == len(y)
nmm = 0
for i in range(0, len(x)):
if x[i] != y[i]:
nmm += 1
return nmm
hammingDistance('brown', 'blown')
hammingDistance('cringe', 'orange')
Levenshtein distance (between strings $P$ and $T$)¶
Special case where the operations are the insertion, deletion, or substitution of a character
- Insertion – Insert a single character into pattern $P$ to help it match text $T$ , such as changing “ago” to “agog.”
- Deletion – Delete a single character from pattern $P$ to help it match text $T$ , such as changing “hour” to “our.”
- Substitution – Replace a single character from pattern $P$ with a different character in text $T$ , such as changing “shot” to “spot.”
Count the minimum number needed to convert $P$ into $T$.
Interchangably called "edit distance".
Exercise: What are the Hamming and Edit distances?¶
\begin{align} T: \text{"The quick brown fox"} \\ P: \text{"The quick grown fox"} \\ \end{align}Exercise: What are the Hamming and Edit distances?¶
Comprehension check: give three different ways to transform $P$ into $T$ (not necessarily fewest operations)
Edit distance - Divide and conquer¶
How do we use simpler comparisons to perform more complex ones?
Use substring match results to compute
Consider by starting from first character and building up
The DP matrix¶
Initialization¶
Computation¶
Final edit distance is lower right corner¶
Comparison of entirety of both strings
So the net result tells us the minimum number of operations is 2. But what are they?
Traceback: the minimum "edit"¶
- Diagonal = Match (M) or Substitution (S) - depending if distance increased while following diagonal or remained constant
- Vertical = Deletion (D)
- Horizontal = Insertion (I)
Traceback: the minimum "edit"¶
- Diagonal = Match (M) or Substitution (S) - depending if distance increased while following diagonal or remained constant
- Vertical = Deletion (D)
- Horizontal = Insertion (I)
see http://www.cs.jhu.edu/~langmea/resources/lecture_notes/dp_and_edit_dist.pdf for more details
Note: use of (I) vs. (D) here depends which of the two strings you are editing to become the other
Note II: an error in backtrace line in picture, can you find it?
Traceback is a form of diff¶
Storing just diffs reduces storage amount.
For example, git
Seek lowest difference on bottom row, endpoint of substring in $T$.
def edDistRecursive(x, y):
if len(x) == 0: return len(y)
if len(y) == 0: return len(x)
delt = 1 if x[-1] != y[-1] else 0
vert = edDistRecursive(x[:-1], y) + 1
horz = edDistRecursive(x, y[:-1]) + 1
diag = edDistRecursive(x[:-1], y[:-1]) + delt
return min(diag, vert, horz)
edDistRecursive('Shakespeare', 'shake spear') # this takes a while!
Memoized version¶
def edDistRecursiveMemo(x, y, memo=None):
''' A version of edDistRecursive with memoization. For each x, y we see, we
record result from edDistRecursiveMemo(x, y). In the future, we retrieve
recorded result rather than re-run the function. '''
if memo is None: memo = {}
if len(x) == 0: return len(y)
if len(y) == 0: return len(x)
if (len(x), len(y)) in memo:
return memo[(len(x), len(y))]
delt = 1 if x[-1] != y[-1] else 0
diag = edDistRecursiveMemo(x[:-1], y[:-1], memo) + delt
vert = edDistRecursiveMemo(x[:-1], y, memo) + 1
horz = edDistRecursiveMemo(x, y[:-1], memo) + 1
ans = min(diag, vert, horz)
memo[(len(x), len(y))] = ans
return ans
edDistRecursiveMemo('Shakespeare', 'shake spear') # this is very fast
DP version¶
from numpy import zeros
def edDistDp(x, y):
""" Calculate edit distance between sequences x and y using
matrix dynamic programming. Return distance. """
D = zeros((len(x)+1, len(y)+1), dtype=int)
D[0, 1:] = range(1, len(y)+1)
D[1:, 0] = range(1, len(x)+1)
for i in range(1, len(x)+1):
for j in range(1, len(y)+1):
delt = 1 if x[i-1] != y[j-1] else 0
D[i, j] = min(D[i-1, j-1]+delt, D[i-1, j]+1, D[i, j-1]+1)
return D[len(x), len(y)]
edDistDp('Shakespeare', 'shake spear')
NLTK version¶
from nltk.metrics.distance import edit_distance
edit_distance('intention', 'execution')
edDistDp('intention', 'execution')
Levenshtein Demo¶
Check for Understanding¶
What are the givens for DP?
- A map
- A goal
What is the output of DP?
Best "path" from any "location" - meaning what?
How is the value calculated in DP?
It is the min/max of:
- The previous max
- The current value
How does DP generate the best "path"?
Start at goal and calculate the value function for every step all the path back to start
Mini-Summary (take 2?)¶
- Dynamic programming is a improved version of recrusion
- Dynamic programming uses caches to save compute, storing the results of previous calculations in a systematic way for later use
- Backtrace is storing the diff for later use.
Further Improvement/Optimization¶
Many more specialized algorithms exist in the fields DP is used.
DP is a category of methods, not a single method.
Can you think of ways to perform these tasks faster?
Plan B. Be even more clever about intermediate calculations. E.g. compute table in way more likely to find optimal sooner (before filling entire table).
Plan A. "good" approximations (hopefully). E.g. computing a match based on only short substrings, not entire string.
IV. Dynamic Time Warping¶
Matching time series: Dynamic Time Warping¶
Generally similar to edit distance algorithm, except we compute a function $d()$ at each comparison
import numpy as np
# define whatever distance metric you want
def d(x,y):
return abs(x-y)
# the DP DWT distance algorithm
def DTWDistance(A, B):
n = len(A)
m = len(B)
DTW = np.zeros((n,m))
for i in range(0,n):
for j in range(0,m):
DTW[i, j] = np.inf
DTW[0, 0] = 0
for i in range(1,n):
for j in range(1,m):
cost = d(A[i], B[j])
DTW[i, j] = cost + min((DTW[i-1, j ], # insertion
DTW[i , j-1], # deletion
DTW[i-1, j-1])) # match
print(DTW)
return DTW[n-1, m-1]
DTWDistance([1,2,3],[1,3,3,3,3])
V. Summary¶
Pre-filtering, Pruning, etc.¶
When performing a slow search algorithm over a large dataset, start with fast algorithm with poor FPR to reject obvious mismatches.
Ex. BLAST (sequence alignment) in bioinformatics, followed by slow accurate structure alignment technique.
General result for optimal ordering of filters based on efficiency and speed: K. Dillon and Y.-P. Wang, “On efficient meta-filtering of big data,” in Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th
Hints¶
The sooner you start to code, the longer the program will take.
– Roy Carlson, University of Wisconsin
- You will write 2 lines of code that will be ~140 characters. Like most computer science challenges, it will require more thinking that writing.
If you can’t write it down in English, you can’t code it. – Peter Halpern
- Practice the algorithm on your white desk table several times before trying to code.