3 j\=@sddgZddlZddlZGdddeZGdddeZGdd d eZGd ddeZGd d d eZ Gd dde Z GdddZ Gddde ZGdddeZGdddedZdS)LexerLexerStateChangeNc@seZdZdZddZdS)LexErrora Exception raised if an invalid character is encountered and no default error handler function is defined. The .text attribute of the exception contains all remaining untokenized text. The .error_index is the index location of the error. cCs|f|_||_||_dS)N)argstext error_index)selfmessagerrr =/u1/h0/jkinne/public_html/cs420-s2019/code/sly-0.4/sly/lex.py__init__.szLexError.__init__N)__name__ __module__ __qualname____doc__r r r r r r'src@seZdZdZdS) PatternErrorzn Exception raised if there's some kind of problem with the specified regex patterns in the lexer. N)r rrrr r r r r3src@seZdZdZdS)LexerBuildErrorzN Exception raised if there's some sort of problem building the lexer. N)r rrrr r r r r:src@seZdZdZdddZdS)rz9 Exception raised to force a lexing state change NcCs||_||_dS)N)newstatetok)rrrr r r r DszLexerStateChange.__init__)N)r rrrr r r r r r@sc@seZdZdZd ZddZdS) Tokenz+ Representation of a single token. typevaluelinenoindexc Cs&d|jd|jd|jd|jd S)Nz Token(type=z, value=z , lineno=z, index=))rrrr)rr r r __repr__MszToken.__repr__N)rrrr)r rrr __slots__rr r r r rHsrcs2eZdZedfdd ZddZddZZS) TokenStrNcstj||}||_||_|S)N)super__new__keyremap)clsrr r!r) __class__r r rQszTokenStr.__new__cCs|jdk r||j|j|f<dS)N)r!r )rr rr r r __setitem__Ys zTokenStr.__setitem__cCs |jdk r|j|j|j|f<dS)N)r!r )rr r r r __delitem__^s zTokenStr.__delitem__)NN)r rr staticmethodrr$r% __classcell__r r )r#r rPsrc@seZdZddZdS)_BeforecCs||_||_dS)N)rpattern)rrr)r r r r csz_Before.__init__N)r rrr r r r r r(bsr(cs@eZdZdZddZfddZfddZfdd ZZS) LexerMetaDictzZ Special dictionary that prohibits duplicate definitions in lexer specifications. cCsi|_g|_i|_dS)N)beforedeleter!)rr r r r kszLexerMetaDict.__init__cst|trt|||j}t|tr>|j|j|<t|j||j}||krt|t r||}t|trt |rt||_nt d|dt j ||dS)NzName z redefined) isinstancestrrr!r(rr+r)propertycallableAttributeErrorrr$)rr rZprior)r#r r r$ps    zLexerMetaDict.__setitem__cs.|jj|||kr|jrn tj|SdS)N)r,appendisupperrr%)rr )r#r r r%s zLexerMetaDict.__delitem__csH||kr8|jddjr8|dddkr8t|||jStj|SdS)Nignore__)splitr3rr!r __getitem__)rr )r#r r r9s*zLexerMetaDict.__getitem__) r rrrr r$r%r9r'r r )r#r r*gs   r*cs,eZdZdZeddZfddZZS) LexerMetaz/ Metaclass for collecting lexing rules cCs"t}dd}||d<t|d<|S)Ncs|f|fdd}|S)Ncs:djddD}t|dr0|d|j|_n||_|S)N|css|]}d|dVqdS)(rNr ).0Zpatr r r szELexerMeta.__prepare__.._..decorate..r))joinhasattrr))funcr))patternsr r decorates  z2LexerMeta.__prepare__.._..decorater )r)ZextrarCr )rBr r6s  z LexerMeta.__prepare__.._r6r+)r*r()metanamebasesdr6r r r __prepare__s  zLexerMeta.__prepare__cs^|d=|d=dd|jD}tj||||}t||_|j|_|j|_|j |_ |j |S)Nr6r+cSs,i|]$\}}t|trt|n|t|qSr )r-rr.)r=r valr r r sz%LexerMeta.__new__..) itemsrrdict _attributesr!_remapr+_beforer,_delete_build)rDZclsnamerFZ attributesZcls_attributesr")r#r r rs  zLexerMeta.__new__)r rrr classmethodrHrr'r r )r#r r:s r:c@seZdZeZeZdZdZeZ eZ iZ eZ iZ iZiZdZdZeddZeddZdd Zd d Zd d ZdddZddZdS)rrNcsg}x$jD]}t|tr |j|jq Wt|}x6jjD]&\}}|jksj|j dsjt |drsz(Lexer._collect_rules..>rTrU) __bases__r-r:extend_rulesrLrMrK _token_names startswithr@r0rrrrOinsertr2r.)r"ZrulesbaseZexistingr rnr+r )r"r _collect_ruless4         zLexer._collect_rulescCs2dt|krt|jd|jt|jB|_t|j|_t|j|_t|j |_ x<|j j D].\\}}}||j kr~i|j |<||j ||<q^Wt}x |j j D]}|j |j qW|t|j}|rdj|}t|d|jg}x|jD]\} } | jdr"| dd} |jj| t| tr4| } nt| rR| |j| <t| d} d | d | d } y|jj| |j} Wn4tk r}ztd | |WYdd}~XnX| jd rtd| d|j| qW|sdS|jjdj||j|_t|jtstdt dd|j!Ds.tddS)z Build the lexer object from the collected tokens and regular expressions. Validate the rules to make sure they look sane. tokensz) class does not define a tokens attributez, z not included in token(s)r4Nr)z(?P<>rzInvalid regex for token rSzRegex for token z matches empty inputr;z!ignore specifier must be a stringcss|]}t|tVqdS)N)r-r.)r=Zlitr r r r>MszLexer._build..z%literals must be specified as strings)"varsrrrZsetr`_ignored_tokensrL _token_funcs _remappingrNrKvaluesupdater?r_rYr[addr-r.r0getattr regex_modulecompilereflags Exceptionrmatchr2 _master_rerTallrU)r"r rIZnewtokZ remapped_toksrGZ undefinedZmissingZpartsZtoknamerr)partZcpater r r rQsV              " z Lexer._buildcCs,t|tstd|jr"|j|||_dS)z) Begin a new lexer state z!state must be a subclass of LexerN)r-r:AssertionError_Lexer__set_stater#)rr"r r r beginPs z Lexer.begincCs.|jdkrg|_|jjt||j|dS)z7 Push a new lexer state onto the stack N)_Lexer__state_stackr2rrw)rr"r r r push_stateYs zLexer.push_statecCs|j|jjdS)z2 Pop a lexer state from the stack N)rwrxpop)rr r r pop_statebszLexer.pop_stater5c#sdfdd}||_|t|||_zdx\y||krf|d7}wNWntk r|dSXt}||_||_j||}|r0|j}|j |_ |j |_|jkrވ|jj |j |j|_|jkr||_||_|j||}|j}|j}|sqN|jkr(qN|VqN||kr`|||_ |j |_|d7}|VqN||_||_d|_||d|_ |j |}|dk r|V|j}|j}qNWWd||_||_||_XdS)Ncs(|j|j|j|j|j|jdS)N)rerqrTrfrUrg)r")_ignorere _literalsrqrgrfr r _set_stateks z"Lexer.tokenize.._set_stater5ZERROR)rvrr IndexErrorrrrrpendgrouprZ lastgroupgeterror)rrrrr~rmr )r|rer}rqrgrfr tokenizehsf           zLexer.tokenizecCs(td|jdd|j|j|jdS)NzIllegal character rz at index )rrr)rtr r r rsz Lexer.error)r5r)r rrrdr`rUrTrnrerlrZrfrergrPrNrxrvrRr_rQrwryr{rrr r r r rs( < J   N) metaclass)__all__rcopyrorrrrobjectrr.rr(rLr*rr:rr r r r "s ((