University-of-Basel-Studentprojects/Programmierprojekt
Archived
1
0
Fork 0
Code Issues 17 Pull Requests Actions Wiki Activity

Add server-side tokenizer #185

Merged
lars.winzer merged 16 commits from feat/tokenizer into main 2026-03-15 13:52:51 +01:00
Conversation 1 Commits 16 Files Changed 10
+407
lars.winzer commented 2026-03-15 13:50:42 +01:00 (Migrated from git.scicore.unibas.ch)
Copy Link
Copy Source

Added the server-side tokenizer, consisting of two stages.

  1. Tokenizer: Tokenize the given String into RawTokens. Check for basic syntax violations like unescaped strings.
  2. TokenizerClassifier: Classifies the RawTokens of the Tokenizer and gives them meaning (COMMAND, KEY, VALUE). Applies further checks regarding that a seperator has both an KEY and a VALUE.

Additionally the State and the TokenizerException are covered by unit tests.

However, the Tokenizer and TokenizerClassifier are currently NOT covered (see #4).

Added the server-side tokenizer, consisting of two stages. 1. Tokenizer: Tokenize the given String into RawTokens. Check for basic syntax violations like unescaped strings. 2. TokenizerClassifier: Classifies the RawTokens of the Tokenizer and gives them meaning (COMMAND, KEY, VALUE). Applies further checks regarding that a seperator has both an KEY and a VALUE. Additionally the State and the TokenizerException are covered by unit tests. However, the Tokenizer and TokenizerClassifier are currently NOT covered (see #4).
lars.winzer commented 2026-03-15 13:52:51 +01:00 (Migrated from git.scicore.unibas.ch)
Copy Link
Copy Source

mentioned in commit 61f0a23bfa

mentioned in commit 61f0a23bfa859ece47947dc2b439beb0ea16cd4d
lars.winzer (Migrated from git.scicore.unibas.ch) merged commit 61f0a23bfa into main 2026-03-15 13:52:51 +01:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
achievement
Tasks related to milestone achievements
 bug
Issues related to programming errors on both client and server
 ci
Issues related to problems with the gitlab ci (incorrect or missing jobs)
 client
Issues (To-Dos and tasks) related to the client
 commands
Issues and tasks related to commands and thier infrastructure (CommandHandler, CommandParser, ...)
 documentation
Issues related to the documentation (for new or unclear, incorrect or incomplete entries)
 enhancement
Issues related to enhancements (also known as feature-requests)
 game-engine
Issues related to the game engine (incorrect state transitions or move validations)
 linting
Issues related to linting i.e. false detections, too generous linting, ...
 messages
Issues and tasks related to the messages and chats
 network
Issues and tasks related to the network (SessionReader / SessionWriter, Parser, ResponseBody, ...)
 outreach
Issues and tasks related to the outreach website and everything directly related to it
 server
Issues (To-Dos and tasks) related to the server
 task
Issues outlining and discussing a general task or implementation
 templates
Issues related to GitLab templates
 ui
Issues and tasks related to the client-side ui
 unit-test
Issues related to missing or already planned unit tests
 user
Issues and tasks related to the user management
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
lars-winzer (Lars Winzer)
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: University-of-Basel-Studentprojects/Programmierprojekt#185
Delete Branch "feat/tokenizer"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?