Add server-side tokenizer #185

Merged
lars.winzer merged 16 commits from feat/tokenizer into main 2026-03-15 13:52:51 +01:00

16 Commits

Author SHA1 Message Date
Lars Simon Winzer 08dc94b4b0 Style: Comply with Spotless
Created issue #2 regarding this issue
2026-03-15 13:38:59 +01:00
Lars Simon Winzer ca11c135f1 Style: Bracket after 'if' and record in one-line 2026-03-15 13:28:06 +01:00
Lars Simon Winzer 60f59e03d4 Add: TokenClassifier to perform more extensive syntax verification and classify tokens 2026-03-15 13:26:59 +01:00
Lars Simon Winzer 1e794337b5 Remove: 'NEWLINE' member from TokenType enum as commands dont span over multiple lines 2026-03-15 13:08:23 +01:00
Lars Simon Winzer 3509a967a6 Add: Tokenizer does no longer classify tokens and instead returns 'RawToken's 2026-03-15 12:21:56 +01:00
Lars Simon Winzer 4c0e6aeb36 Delete: Falsly added file for rudimentary quick testing 2026-03-15 12:20:37 +01:00
Lars Simon Winzer 3e138ab84e Add: State to use RawToken 2026-03-15 12:11:59 +01:00
Lars Simon Winzer 45fd696fec Add: RawToken and RawTokenType for token representation 2026-03-15 12:08:32 +01:00
Lars Simon Winzer b898357d76 Merge branch 'main' into 'feat/tokenizer' 2026-03-14 18:24:03 +01:00
Lars Simon Winzer f3c34dfa7f Move: Tokenizer related tests into server server directory 2026-03-14 17:14:37 +01:00
Lars Simon Winzer 2af3fe7c35 Move: Tokenizer related files into server directory 2026-03-14 17:13:18 +01:00
Lars Simon Winzer 14c18df0eb Test: Tokenizer exception 2026-03-14 16:37:11 +01:00
Lars Simon Winzer 208a284204 Test: Tokenizer state 2026-03-14 16:33:34 +01:00
Lars Simon Winzer ef2a00268f Add: Tokenizer to tokenize provided string into tokens and perform syntax check 2026-03-14 16:13:35 +01:00
Lars Simon Winzer d403908341 Add: TokenizerException thrown during tokenization 2026-03-14 16:12:50 +01:00
Lars Simon Winzer fa54fd75c1 Add: Tokenizer state and token representation 2026-03-14 15:32:40 +01:00