Trying to create a lexer for kind-of-classic 80's BASIC

[Stoomkracht]

CVBasic, it's short manual: https://github.com/nanochess/CVBasic/bl ... manual.txt
A compiler for a few similar '80's consoles.

I want to display the labels and procedures separately in the syntax view and obviously have autocomplete.

Format
mylabel:

and

myprocedure: PROCEDURE
optional RETURN
END

I'm crying in the corner, because I can't make it work. I have problems with the END for example, because it also shows up in END IF (so space have a function in keywords). Any expert look and initial version would be greatly appreciated.

[main Alexey]

here is a solution how to avoid conflict END vs END IF.

make new 'parser' rule (ie regex) to cacth END IF as a sigle token. regex will be like:
(?-i)END\x20IF\b

it helped?

[main Alexey]

mylabel:
and
myprocedure: PROCEDURE

you want to catch them with different 'parser' rules, right? you can.
read about 'regex assertions'. it is regex feature. it helps here.
with the 'assertion' the rule for
myprocedure: PROCEDURE
will be something like this:

(?-i)[a-zA-Z_]\w*:(?=\x20PROCEDURE\b)

then put this rule above the 2nd rule, which is only
[a-z_]\w*:
(case insensitive by default).

play with my regex here: https://regex101.com/r/vBPzaM/1

[Stoomkracht]

Under rules i created ProcedureStart and ProcedureEnd rule. But when I double click on the procedure in the syntax tree, it keeps selecting till END IF, instead of the real end of procedure (END...newline)

I want labels and procedures separately listed in the tree view (group formatting string).

Here (incomplete) code to test lexer:

Code: Select all

title_screen:
	MODE 0
	DEFINE CHAR 128,89,dungeon_title_char
	DEFINE COLOR 128,89,dungeon_title_color

	SCREEN dungeon_title_pattern,0,0,32,8,32
	
		'
	' Build the random map
	'
generate_maze:
	PLAY OFF

	CLS
	IF level = 1 THEN BORDER 4
	IF level = 2 THEN BORDER 12
	IF level = 3 THEN BORDER 6
	IF level = 4 THEN BORDER 5
	IF level = 5 THEN BORDER 13
	
wait_for_read:	PROCEDURE
	FOR d = 0 TO 60
		WAIT
	NEXT d
	END

	'
	' Battle monster
	' c = Monster (1-3)
	'
battle_monster:		PROCEDURE
	monster_type = c + level
	monster_hp = monster_type * monster_type * 2

	DO
		damage = RANDOM(current_weapon * 10 + 1)

		GOSUB clear_status
		IF damage = 0 THEN
			PRINT AT 546,"You miss!!!"
			sound_effect = EFFECT_MISS
			sound_state = 0
		ELSE
			IF damage > monster_hp THEN damage = monster_hp
			PRINT AT 546,"You hit for ",<>damage
			PRINT AT 578,"damage"
			sound_effect = EFFECT_HIT
			sound_state = 0
		END IF

		GOSUB wait_for_read

		monster_hp = monster_hp - damage
		IF monster_hp = 0 THEN EXIT DO

		damage = RANDOM(monster_type + 1)
		IF current_armor > damage THEN
			damage = 0
		ELSE
			damage = damage - current_armor
		END IF

		GOSUB clear_status

		IF damage > #hp THEN damage = #hp

		IF damage = 0 THEN
			PRINT AT 546,"Monster miss"
			sound_effect = EFFECT_MISS
			sound_state = 0
		ELSE
			PRINT AT 546,"Monster hits you"
			PRINT AT 578,"for ",<>damage," damage!"
			sound_effect = EFFECT_HIT
			sound_state = 0
		END IF
		#hp = #hp - damage
		GOSUB update_stats

		GOSUB wait_for_read

		IF #hp = 0 THEN EXIT DO

	LOOP UNTIL #hp < 1 OR monster_hp < 1

	IF #hp = 0 THEN
		GOSUB clear_status
		PRINT AT 546,"You've been killed!"
	ELSE
		GOSUB clear_status
		PRINT AT 546,"Monster killed!!!"
		c = RANDOM(50)
		IF c THEN PRINT AT 578,<>c," gold found"
		#gold = #gold + c
	END IF
	GOSUB wait_for_read

	END

next_rand:	PROCEDURE
	#seed = (#seed * 139 + 5) % 191
	c = #seed % 4
	END

I tried creating parser rule END[\x20\t]*$ and reference it in Rules tab, but it doesn't do anything. If I just add END to Keys list, it matches also END IF. I do not know how to make it match END...endline and only END...endline.

Keyword list:

Code: Select all

ABS
AMM RET
AND
ASM
ASM INCLUDE
BANK
BANK ROM
BANK SELECT
BITMAP
BORDER
CALL
CASE
CASE ELSE
CLS
CONST
CONT
CONT.BUTTON
CONT.BUTTON2
CONT.DOWN
CONT.KEY
CONT.LEFT
CONT.RIGHT
CONT.UP
CONT1
CONT1.BUTTON
CONT1.BUTTON2
CONT1.DOWN
CONT1.KEY
CONT1.LEFT
CONT1.RIGHT
CONT1.UP
CONT2
CONT2.BUTTON
CONT2.BUTTON2
CONT2.DOWN
CONT2.KEY
CONT2.LEFT
CONT2.RIGHT
CONT2.UP
DATA
DATA BYTE
DATA VARPTR
DEF FN
DEFINE CHAR
DEFINE CHAR PLETTER
DEFINE COLOR
DEFINE COLOR PLETTER
DEFINE SPRITE
DEFINE SPRITE PLETTER
DEFINE VRAM
DEFINE VRAM PLETTER
DEFINE VRAM READ
DIM
DO
DO UNTIL
DO WHILE
ELSE
ELSEIF
END
END IF
END SELECT
EXIT DO
EXIT FOR
EXIT WHILE
FAST
FOR 
FRAMENTSC
GOSUB
GOTO
IF
INCLUDE
INP
LEN
LOOP
LOOP UNTIL
LOOP WHILE
MODE
MUSIC.PLAYING
NEXT
NOT
ON
ON FRAME GOSUB
OPTION EXPLICIT
OPTION EXPLICIT OFF
OPTION EXPLICIT ON
OR
OUT
PALETTE
PALETTE LOAD
PALETTE LOAD VARPTR
PEEK
PLAY
PLAY FULL
PLAY FULL NO DRUMS
PLAY NONE
PLAY OFF
PLAY SIMPLE
PLAY SIMPLE NO DRUMS
POKE
POS
PRINT
PRINT AT
PRINT CHR$
RANDOM
READ
READ BYTE
RESTORE
RETURN
SCREEN
SCREEN DISABLE
SCREEN ENABLE
SCROLL
SELECT CASE
SGN
SIGNED
SOUND
SPRITE
SPRITE FLICKER OFF
SPRITE FLICKER ON
STEP
THEN
TO
UNSIGNED
USR
VARPTR
VDP
VDP.STATUS
VPEEK
VPOKE
WAIT
WEND
WHILE
XOR

[main Alexey]

I made the good start-point. the lexer, not very full, but
- folds procedure blocks
- folds IF blocks; handles THEN when some text follows THEN on the same line

what you need to do
- adjust file extension
- adjust Number rule for floating-numbers
- adjust String rule for escape-chars
- add all other keywords (I added only 10-20)

To install this lexer, open zip file in Cud/SynWrite, 'File / Open'.

[main Alexey]

note how to add keywords.
you have combined keywords, with space. how to add them?
solution:
add regex rule which finds all words.

for 4 words
SPRITE
SPRITE FLICKER OFF
SPRITE FLICKER ON
STEP

regex will be:

Code: Select all

(?-i)(SPRITE\x20+(FLICKER\x20+(ON|OFF))|SPRITE|STEP)\b

[Stoomkracht]

Thanks. Still have issues.. If I try to add a rule for just a label (without the PROCEDURE keyword), it will list things double in the syntax tree, because it matches both label AND procedure. If I play with regex that match only labels and one for procedures in Parser, I can't get it to work in Conditions.

Label regex:
^[\x20\t]*\w+:[\x20\t]*$

Procedure regex:
^[\x20\t]*\w+:[\x20\t]*PROCEDURE[\x20\t]*$

[main Alexey]

known problem: double items in code-tree. try this: mark both rules with '[x] Cancel next rules' checkbox in the 'Lexer properties / Rules / Properties' tab.

if you have other problems, attach your .lcf file.

don't forget to add
(?-i)
at the begin of PROCEDURE regex.

don't forget that your 2 regex'es don't allow trailing comments!

[main Alexey]

I ask you to publish your lexer (zip file with .lcf / .cuda-lexmap / install.inf) for all people.

[Stoomkracht]

Couldn't make a satisfying version and because too frustrated with trial and error and navigating various tabs and somehow regex masking not working,, while it does work in regex search. I am giving it another look.