Character N-grams
Imagine you are trying to identify a mystery sender by looking at their unique handwriting style. You focus on the specific ways they connect their letters, such as how they loop their lowercase 'g' or how they slant their 't' strokes. In the digital world, we apply this same logic to anonymous texts using a method called character n-grams. By breaking down a document into tiny, overlapping sequences of letters, we can capture the rhythmic fingerprint of an author. This mathematical approach allows us to see patterns that are invisible to the naked eye.
The Logic of Sequential Letter Patterns
When we analyze text at the character level, we move beyond simple word counts or vocabulary lists. A character n-gram is simply a contiguous sequence of n characters taken from a given piece of text. If you have the word 'apple' and look for 3-character sequences, you get 'app', 'ppl', and 'ple'. These small pieces act like building blocks for the entire document. By counting how often each sequence appears, we create a numerical profile of the writer. This profile reflects their personal habits, such as their preference for certain letter combinations or their tendency to use specific punctuation marks in unique clusters.
Think of this process like a chef who creates a signature spice blend for their famous soup. Even if you do not know the exact ingredients, you can identify the chef by the specific ratio of salt, pepper, and garlic in every bowl. The character n-grams are the individual pinches of spice that define the flavor of the writing. Just as a chef might consistently add a pinch of cumin after the garlic, a writer might consistently use the sequence 'ing' after certain verbs. These small, repetitive habits form a consistent pattern that remains stable across different works by the same person.
Implementing N-Gram Counting for Text Analysis
To turn these patterns into usable data, we must count the frequency of every possible sequence within a text. We typically use a sliding window that moves one character at a time across the entire document. This systematic approach ensures that we capture every single transition between letters, spaces, and punctuation. The resulting data set is often large, but it provides a very precise map of the author's stylistic choices. Because we are looking at characters rather than whole words, this method works well even when the author uses a varied vocabulary or changes topics frequently.
| N-Gram Size | Focus Area | Primary Use Case |
|---|---|---|
| 1-gram | Characters | General frequency |
| 2-gram | Bigrams | Local transitions |
| 3-gram | Trigrams | Complex patterns |
We can organize these n-grams to see how they influence the overall style of a text. Consider the following observations about how these counts reveal hidden traits:
- High frequency of specific bigrams often reveals the underlying phonetic structure of the writer’s native language.
- Consistent trigram usage highlights the author’s preference for common suffixes or specific grammatical structures that appear repeatedly.
- Punctuation-inclusive n-grams capture the rhythm of a writer, showing how they pause or emphasize ideas within their sentences.
By comparing these frequency distributions between two different documents, we can calculate a mathematical distance between them. If the distributions are nearly identical, it suggests that the same person likely wrote both texts. This remains one of the most reliable ways to link anonymous content to a known source, as it is very difficult for a writer to consciously alter every single character sequence in their work. The math provides an objective truth that bypasses the subjective guesses we might make when reading the text ourselves.
Character n-grams reveal an author's unique writing signature by quantifying the frequency of specific, recurring letter sequences within their text.
The next Station introduces vector spaces, which determine how we compare these frequency counts to find the closest stylistic matches.