This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Summary: | Tokens embedding gets lost after modification of the top level token under some circumstances | ||
---|---|---|---|
Product: | editor | Reporter: | Marek Fukala <mfukala> |
Component: | Lexer | Assignee: | Miloslav Metelka <mmetelka> |
Status: | VERIFIED FIXED | ||
Severity: | blocker | CC: | jskrivanek, mschovanek, pjiricka |
Priority: | P2 | ||
Version: | 6.x | ||
Hardware: | All | ||
OS: | All | ||
Issue Type: | DEFECT | Exception Reporter: | |
Bug Depends on: | |||
Bug Blocks: | 100225 | ||
Attachments: |
The JSP file to reproduce
NPE and AIOOBE stack trace |
Created attachment 41184 [details]
The JSP file to reproduce
The lookahead of the big tokens spans the whole next token plus one character. The incremental algorithm correctly relexes the subsequent token as well. Consider a similar situation with a language having the following tokens: "+" PLUS token "-" MINUS token "+-+" PMP token When given an input "+--" it would be lexed as PLUS(lookahead=2) MINUS(lookahead=1) MINUS(lookahead=0) Now if initial "+" would get removed i.e. the input would be "--" then if only the PLUS token would get removed then it would become MINUS(lookahead=1) MINUS(lookahead=0) However by batch scanning the list would be MINUS(lookahead=0) MINUS(lookahead=0) Allowing the lookaheads of the incremental setup become different from the batch lexing setup depending on the order of modifications and their place is dangerous since the correctness of incremental modifications could not be easily verified and therefore the present algorithm prohibits that by relexing an extra token till the end of the modification token's lookahead. Therefore the desired fix is to check that the second relexed token is the same like the original one and only report the change of the first token. I will fix it ASAP. It blocks high visible P1 issue #100225, increasing priority to P1. Really -> P1 After discussion with Pavel B. we agreed that is not necessary to fix the issue #100225 for M9, so decreasing this back to P2. As you like, but the fix should be safe. The reproducing case should be fixed and when running with -J-Dorg.netbeans.lib.lexer.TokenHierarchyOperation.level=FINE also shows just one removed and added token. I have added a test as well. Checking in src/org/netbeans/lib/lexer/inc/TokenListUpdater.java; /cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListUpdater.java,v <-- TokenListUpdater.java new revision: 1.10; previous revision: 1.9 done Checking in src/org/netbeans/lib/lexer/inc/TokenListChange.java; /cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListChange.java,v <-- TokenListChange.java new revision: 1.9; previous revision: 1.8 done Checking in src/org/netbeans/lib/lexer/LAState.java; /cvs/lexer/src/org/netbeans/lib/lexer/LAState.java,v <-- LAState.java new revision: 1.4; previous revision: 1.3 done RCS file: /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java,v done Checking in test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/test/inc/TokenListUpdaterExtraTest.java,v <-- TokenListUpdaterExtraTest.java initial revision: 1.1 done RCS file: /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java,v done Checking in test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java; /cvs/lexer/test/unit/src/org/netbeans/lib/lexer/lang/TestSaveTokensInLATokenId.java,v <-- TestSaveTokensInLATokenId.java initial revision: 1.1 Sorry, my browser did not refresh the priority field correctly so changing back to P2. Reopening because lexer throws NPEs and AIOOBEs while testing JSF or Struts support in web applications (see attached stack traces). Created attachment 41811 [details]
NPE and AIOOBE stack trace
Sorry, I did not realize that there can be removed token(s) without any added ones e.g. if a sinle-char removed token had no lookahead. Fixed. Checking in TokenListUpdater.java; /cvs/lexer/src/org/netbeans/lib/lexer/inc/TokenListUpdater.java,v <-- TokenListUpdater.java new revision: 1.11; previous revision: 1.10 *** Issue 102746 has been marked as a duplicate of this issue. *** OK, tests are passing without exceptions since build 20070427-1216. *** Issue 103088 has been marked as a duplicate of this issue. *** |
To reproduce open the attached JSP file, wait for a few seconds until the javascript and css sections gets colored, and try to put some spaces inside the javascript code on line 10. The javascript coloring gets lost just after the modification, it is created again after a while when the parser finishes, but the coloring is broken meanwhile and code completion has also problem with this behaviour. From what I remembed and understood when we debugged the problem on my box, the problem is in the bigger lookahead of JSPLexer in the jsp close tag than is the token length which causes the jsp token being relexed and thus the embedding-preserving code is doesn't work. I also am attaching the debug messages: <<<<<<<<<<<<<<<<<< LEXER CHANGE START ------------------ ROOT CHANGE: "text/x-jsp", index=11, offset=52 R[0]: E["text/x-jsp/text/html"] IHC(28887505): DefT(TEXT, <null-text>, 52, 147) IHC(15149474) R[1]: DefT(ENDTAG, <null-text>, 199, 13) IHC(6189862) A[0]: DefT(TEXT, "\n <p>\n text\n </p>\n \n <script type="text/javascript">\n function hello() {\n abst x \n } \n </script>\n \n", 52, 148) IHC(13229344) A[1]: DefT(ENDTAG, "</jsp:useBean", 200, 13) IHC(13925107) EVENT: modOffset=163, insertedLength=1, Affected(52,213) >>>>>>>>>>>>>>>>>> LEXER CHANGE END (TokenHierarchy Check OK) ------------------