Antlr4 – 输入中没有可行的替代方案

huangapple
go评论133阅读模式
英文:

Antlr4 - no viable alternative at input

问题

我正在尝试创建一个简单的HOCON解析器（从现有的JSON解析器开始）。

语法定义如下：

  1. /** 参考自Terence Parr的《The Definitive ANTLR 4 Reference》 */
  3. // 派生自http://json.org
  4. grammar HOCON;
  6. hocon
  7.    : value
  8.    | pair
  9.    ;
  11. obj
  12.    : object_begin pair (','? pair)* object_end
  13.    | object_begin object_end
  14.    ;
  16. pair
  17.    : STRING KV? value {fmt.Println("pairstr",$STRING.GetText())}
  18.    | KEY KV? value {fmt.Println("pairkey",$KEY.GetText())}
  19.    ;
  21. array
  22.    : array_begin value (',' value)* array_end
  23.    | array_begin array_end
  24.    ;
  26. value
  27.    : STRING {fmt.Println($STRING.GetText())}
  28.    | REFERENCE {fmt.Println($REFERENCE.GetText())}
  29.    | RAWSTRING {fmt.Println($RAWSTRING.GetText())}
  30.    | NUMBER {fmt.Println($NUMBER.GetText())}
  31.    | obj
  32.    | array
  33.    | 'true'
  34.    | 'false'
  35.    | 'null'
  36.    ;
  38. COMMENT
  39.    : '#' ~( '\r' | '\n' )* -> skip
  40.    ;
  42. STRING
  43.    : '"' (ESC | ~ ["\\])* '"'
  44.    | '\'' (ESC | ~ ['\\])* '\''
  45.    ;
  47. RAWSTRING
  48.    : (ESC | ALPHANUM)+
  49.    ;
  51. KEY
  52.    : ( '.' | ALPHANUM | '-')+
  53.    ;
  55. REFERENCE
  56.    : '${' (ALPHANUM|'.')+ '}'
  57.    ;
  59. fragment ESC
  60.    : '\\' (["\\/bfnrt] | UNICODE)
  61.    ;
  64. fragment UNICODE
  65.    : 'u' HEX HEX HEX HEX
  66.    ;
  68. fragment ALPHANUM
  69.    : [0-9a-zA-Z]
  70.    ;
  72. fragment HEX
  73.    : [0-9a-fA-F]
  74.    ;
  76. KV
  77.    : [=:]
  78.    ;
  80. array_begin
  81.    : '[' { fmt.Println("BEGIN [") }
  82.    ;
  84. array_end
  85.    : ']' { fmt.Println("] END") }
  86.    ;
  88. object_begin
  89.    : '{' { fmt.Println("OBJ {") }
  90.    ;
  92. object_end
  93.    : '}' { fmt.Println("} OBJ") }
  94.    ;
  96. NUMBER
  97.    : '-'? INT '.' [0-9]+ EXP? | '-'? INT EXP | '-'? INT
  98.    ;
  100. fragment INT
  101.    : '0' | [1-9] [0-9]*
  102.    ;
  104. // no leading zeros
  106. fragment EXP
  107.    : [Ee] [+\-]? INT
  108.    ;
  110. // \- since - means "range" inside [...]
  112. WS
  113.    : [ \t\n\r]+ -> skip
  114.    ;

错误是：

  1. 第2行第2列，输入“{journal”没有可行的替代项
  2. pairkey akka.persistence

导致错误的示例输入是：

  1. akka.persistence {
  2.   journal {
  3.     # Absolute path to the journal plugin configuration entry used by
  4.     # persistent actor or view by default.
  5.     # Persistent actor or view can override `journalPluginId` method
  6.     # in order to rely on a different journal plugin.
  7.     plugin = ""
  8.   }
  9. }

然而，如果我将其更新为使用带引号的字符串：

  1. akka.persistence {
  2.   'journal' {
  3.     # Absolute path to the journal plugin configuration entry used by
  4.     # persistent actor or view by default.
  5.     # Persistent actor or view can override `journalPluginId` method
  6.     # in order to rely on a different journal plugin.
  7.     'plugin' = ""
  8.   }
  9. }

一切都按预期工作。

看起来我在KEY定义中漏掉了一些东西，但我无法确定具体是什么。

用于测试的Go代码如下：

  1. package main
  3. import (
  4. 	"github.com/antlr/antlr4/runtime/Go/antlr"
  5. 	"go-hocon/parser"
  6. )
  8. func main() {
  9. 	is, _ := antlr.NewFileStream("test/simple1.conf")
  11. 	lex := parser.NewHOCONLexer(is)
  12. 	p := parser.NewHOCONParser(antlr.NewCommonTokenStream(lex, 0))
  13. 	p.BuildParseTrees = true
  14. 	p.Hocon()
  15. }
英文:

I am trying to create a simple HOCON parser (started from the existing JSON one).

The grammar is defined as:

  1. /** Taken from "The Definitive ANTLR 4 Reference" by Terence Parr */
  3. // Derived from http://json.org
  4. grammar HOCON;
  6. hocon
  7.    : value
  8.    | pair
  9.    ;
  11. obj
  12.    : object_begin pair (','? pair)* object_end
  13.    | object_begin object_end
  14.    ;
  16. pair
  17.    : STRING KV? value {fmt.Println("pairstr",$STRING.GetText())}
  18.    | KEY KV? value {fmt.Println("pairkey",$KEY.GetText())}
  19.    ;
  21. array
  22.    : array_begin value (',' value)* array_end
  23.    | array_begin array_end
  24.    ;
  26. value
  27.    : STRING {fmt.Println($STRING.GetText())}
  28.    | REFERENCE {fmt.Println($REFERENCE.GetText())}
  29.    | RAWSTRING {fmt.Println($RAWSTRING.GetText())}
  30.    | NUMBER {fmt.Println($NUMBER.GetText())}
  31.    | obj
  32.    | array
  33.    | 'true'
  34.    | 'false'
  35.    | 'null'
  36.    ;
  38. COMMENT
  39.    : '#' ~( '\r' | '\n' )* -> skip
  40.    ;
  42. STRING
  43.    : '"' (ESC | ~ ["\\])* '"'
  44.    | '\'' (ESC | ~ ['\\])* '\''
  45.    ;
  47. RAWSTRING
  48.    : (ESC | ALPHANUM)+
  49.    ;
  51. KEY
  52.    : ( '.' | ALPHANUM | '-')+
  53.    ;
  55. REFERENCE
  56.    : '${' (ALPHANUM|'.')+ '}'
  57.    ;
  59. fragment ESC
  60.    : '\\' (["\\/bfnrt] | UNICODE)
  61.    ;
  64. fragment UNICODE
  65.    : 'u' HEX HEX HEX HEX
  66.    ;
  68. fragment ALPHANUM
  69.    : [0-9a-zA-Z]
  70.    ;
  72. fragment HEX
  73.    : [0-9a-fA-F]
  74.    ;
  76. KV
  77.    : [=:]
  78.    ;
  80. array_begin
  81.    : '[' { fmt.Println("BEGIN [") }
  82.    ;
  84. array_end
  85.    : ']' { fmt.Println("] END") }
  86.    ;
  88. object_begin
  89.    : '{' { fmt.Println("OBJ {") }
  90.    ;
  92. object_end
  93.    : '}' { fmt.Println("} OBJ") }
  94.    ;
  96. NUMBER
  97.    : '-'? INT '.' [0-9] + EXP? | '-'? INT EXP | '-'? INT
  98.    ;
  100. fragment INT
  101.    : '0' | [1-9] [0-9]*
  102.    ;
  104. // no leading zeros
  106. fragment EXP
  107.    : [Ee] [+\-]? INT
  108.    ;
  110. // \- since - means "range" inside [...]
  112. WS
  113.    : [ \t\n\r] + -> skip
  114.    ;

the error is:

  1. line 2:2 no viable alternative at input '{journal'
  2. pairkey akka.persistence

the sample input that gives the error is:

  1. akka.persistence {
  2.   journal {
  3.     # Absolute path to the journal plugin configuration entry used by
  4.     # persistent actor or view by default.
  5.     # Persistent actor or view can override `journalPluginId` method
  6.     # in order to rely on a different journal plugin.
  7.     plugin = ""
  8.   }
  9. }

however if I will update it to use quoted strings:

  1. akka.persistence {
  2.   'journal' {
  3.     # Absolute path to the journal plugin configuration entry used by
  4.     # persistent actor or view by default.
  5.     # Persistent actor or view can override `journalPluginId` method
  6.     # in order to rely on a different journal plugin.
  7.     'plugin' = ""
  8.   }
  9. }

everything works as expected.

Looks like I miss something in the KEY definition, but I can't really find out what exactly.

The Go code to test it out is:

  1. package main
  3. import (
  4. 	"github.com/antlr/antlr4/runtime/Go/antlr"
  5. 	"go-hocon/parser"
  6. )
  8. func main() {
  9. 	is, _ := antlr.NewFileStream("test/simple1.conf")
  11. 	lex := parser.NewHOCONLexer(is)
  12. 	p := parser.NewHOCONParser(antlr.NewCommonTokenStream(lex, 0))
  13. 	p.BuildParseTrees = true
  14. 	p.Hocon()
  15. }

答案1

得分: 1

你的第一个输入将_journal_识别为RAWSTRING。

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:28='journal',<RAWSTRING>,2:2]
  4. [@3,30:30='{',<'{'>,2:10]
  5. [@4,277:282='plugin',<RAWSTRING>,7:4]
  6. [@5,284:284='=',<KV>,7:11]
  7. [@6,286:287='""',<STRING>,7:13]
  8. [@7,292:292='}',<'}'>,8:2]
  9. [@8,295:295='}',<'}'>,9:0]
  10. [@9,298:297='<EOF>',<EOF>,10:0]
  11. 第2行第2列的输入“{journal”没有可行的替代项。

另一方面，“'journal'”被识别为字符串，但是它有单引号，显然你不想要这些单引号：

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:30=''journal'',<STRING>,2:2]  <-- 现在它是一个隐式字符串标记
  4. [@3,32:32='{',<'{'>,2:12]
  5. [@4,279:284='plugin',<RAWSTRING>,7:4]
  6. [@5,286:286='=',<KV>,7:11]
  7. [@6,288:289='""',<STRING>,7:13]
  8. [@7,294:294='}',<'}'>,8:2]
  9. [@8,297:297='}',<'}'>,9:0]
  10. [@9,300:299='<EOF>',<EOF>,10:0]
  11. 第7行第4列的输入“{plugin”没有可行的替代项。
  12. 第8行第2列的输入“}”与期望的{'true'，'false'，'null'，'['，'{'，STRING，RAWSTRING，REFERENCE，KV，NUMBER}不匹配。

为什么会这样呢？因为词法分析器规则按照以下方式绑定：

  1. 首先匹配最长的输入。
  2. 匹配隐式标记（如“journal”）。
  3. 如果输入长度相等，则根据词法分析器规则的顺序进行匹配。

在你的情况下，将'journal'放在那里_使其匹配为隐式标记_，所以它似乎工作正常。但是只有因为有了那些单引号，这使得它根据上述规则2进行匹配。如果没有引号，这两个标记将被匹配为RAWSTRING，这不符合规则

  1. pair
  2.    : STRING KV? value //{fmt.Println("pairstr",$STRING.GetText())}

因此会出现错误。

如何修复？我反转了词法分析器规则：

  1. RAWSTRING
  2.    : (ESC | ALPHANUM)+
  3.    ;
  4.  
  5. STRING
  6.    : ''' (ESC | ~ ['\\])* '''
  7.    | '\'' (ESC | ~ ['\\])* '\''
  8.    ;

并修改了pair：

  1. pair
  2.    : RAWSTRING KV? value //{fmt.Println("pairstr",$STRING.GetText())}

现在它可以正确解析：

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:28='journal',<RAWSTRING>,2:2]
  4. [@3,30:30='{',<'{'>,2:10]
  5. [@4,277:282='plugin',<RAWSTRING>,7:4]
  6. [@5,284:284='=',<KV>,7:11]
  7. [@6,286:287='""',<STRING>,7:13]
  8. [@7,292:292='}',<'}'>,8:2]
  9. [@8,295:295='}',<'}'>,9:0]
  10. [@9,298:297='<EOF>',<EOF>,10:0]
英文:

Your first input makes journal lex as a RAWSTRING.

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:28='journal',<RAWSTRING>,2:2]
  4. [@3,30:30='{',<'{'>,2:10]
  5. [@4,277:282='plugin',<RAWSTRING>,7:4]
  6. [@5,284:284='=',<KV>,7:11]
  7. [@6,286:287='""',<STRING>,7:13]
  8. [@7,292:292='}',<'}'>,8:2]
  9. [@8,295:295='}',<'}'>,9:0]
  10. [@9,298:297='<EOF>',<EOF>,10:0]
  11. line 2:2 no viable alternative at input '{journal'

On the other hand, 'journal' lexes as a string, but has those single quotes which you clearly don't want:

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:30=''journal'',<STRING>,2:2]  <-- now it's a string implicit token
  4. [@3,32:32='{',<'{'>,2:12]
  5. [@4,279:284='plugin',<RAWSTRING>,7:4]
  6. [@5,286:286='=',<KV>,7:11]
  7. [@6,288:289='""',<STRING>,7:13]
  8. [@7,294:294='}',<'}'>,8:2]
  9. [@8,297:297='}',<'}'>,9:0]
  10. [@9,300:299='<EOF>',<EOF>,10:0]
  11. line 7:4 no viable alternative at input '{plugin'
  12. line 8:2 mismatched input '}' expecting {'true', 'false', 'null', '[', '{', STRING, RAWSTRING, REFERENCE, KV, NUMBER}

Why? Because lexer rules bind in the following way:

  1. Match longest input first.
  2. Match implicit tokens (like 'journal')
  3. If length of input match is equal, match based on the order of the lexer rules.

In your case, putting 'journal' makes it match as an implicit token, so it seems to work okay. But only because of those single quotes, which makes it match per rule 2 above Without the quotes, these two tokens are being matched as RAWSTRING, which doesn't fit the rule

  1. pair
  2.    : STRING KV? value //{fmt.Println("pairstr",$STRING.GetText())}

Hence the error.

How to fix? Well, I reversed the lexer rules:

  1. RAWSTRING
  2.    : (ESC | ALPHANUM)+
  3.    ;
  4.  
  5. STRING
  6.    : '"' (ESC | ~ ["\\])* '"'
  7.    | '\'' (ESC | ~ ['\\])* '\''
  8.    ;

And changed pair:

  1. pair
  2.    : RAWSTRING KV? value //{fmt.Println("pairstr",$STRING.GetText())}

Now it parses fine:

  1. [@0,0:15='akka.persistence',<KEY>,1:0]
  2. [@1,17:17='{',<'{'>,1:17]
  3. [@2,22:28='journal',<RAWSTRING>,2:2]
  4. [@3,30:30='{',<'{'>,2:10]
  5. [@4,277:282='plugin',<RAWSTRING>,7:4]
  6. [@5,284:284='=',<KV>,7:11]
  7. [@6,286:287='""',<STRING>,7:13]
  8. [@7,292:292='}',<'}'>,8:2]
  9. [@8,295:295='}',<'}'>,9:0]
  10. [@9,298:297='<EOF>',<EOF>,10:0]

huangapple
  • 本文由 发表于 2017年7月7日 10:37:21
  • 转载请务必保留本文链接：https://go.coder-hub.com/44961694.html
  • antlr4
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定