Live briefing
Everything about Lexical analysis
Conversion of character sequences into token sequences in computer science
Lexical tokenization is conversion of a text into meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Lexical analysisA 5-minute orientation
- Skim a recent paper: Lexical Profile of Newspapers Revisited: A Corpus-Based AnalysisInfluential work since 2022
- Pick up a classic book: Prosodic constituency in the lexiconThe most-reprinted book on the subject
- Read the landmark paper: Lexical Frequency Profiles: A Monte Carlo AnalysisCited 62 times
- Go sideways: Lex (software)A closely related topic
- Go sideways: RE/flexA closely related topic
📖 Definitions
Noun The conversion of a stream of characters to a stream of meaningful tokens; normally to simplify parsing.
🏆 Landmark papers (most cited)
Lexical Frequency Profiles: A Monte Carlo Analysis
A lexical analysis of environmental sound categories.
Lexical Analysis of US Political Speeches
Lexical Cohesion Analysis of Political Speech
Lexical data augmentation for sentiment analysis
The cost of lexical analysis
Corpus Analysis and Lexical Pragmatics: An Overview
Farsi lexical analysis and stop word list
🆕 Influential since 2022
📚 Books worth reading
Prosodic constituency in the lexiconSharon Inkelas · 1990
Sonority constraints on prosodic structureDraga Zec · 1994
Emergence of PhonologyTamar Keren-Portnoy, Marilyn M. Vihman · 2013
Metaanalysis of psychoanalysisDr. Andrej Poleev · 2016
Lexical cohesion and corpus linguisticsJohn Flowerdew, Michaela Mahlberg · 2009- 📕Etude et implantation de la coordination en vue de l'analyse automatique du français écrit dans le cadre de la grammaire lexicale fonctionnelleCaroline Brun · 2000
- 📕Corpus perspectives on patterns of lexisHilde Hasselgård · 2013
- 📕Analisi del discorsoEmilio D'Agostino · 1992
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Lex (software) Lex is a computer program that generates lexical analyzers ("scanners" or "lexers"). It is commonly used with the yacc parser generator and is the standard…
- RE/flex RE/flex (or RE-flex) is a computer program that generates lexical analyzers also known as "scanners" or "lexers". Lexical analysis is the process of converting…
- Syntax (programming languages) The syntax of computer source code is code structured and ordered restricted to computer language rules. Like a natural language, a computer language (i…
- Flex (lexical analyzer generator) Flex (fast lexical analyzer generator) is a free and open-source software alternative to lex. It is a computer program that generates lexical analyzers…
- Lexer hack In computer programming, the lexer hack is a solution to parsing a context-sensitive grammar such as C, where classifying a sequence of characters as a…
- Syntax error A syntax error is a mismatch in the syntax of data input to a computer system that requires a specific syntax. For source code in a programming language…
- String literal A string literal or anonymous string is a literal for a string value in source code. Commonly, a programming language includes a string literal code construct…
- Integer literal In computer science, an integer literal is a kind of literal for an integer whose value is directly represented in source code. For example, in the assignment…