JavaCC creating custom Token class

I'm working on a school assignment for my compiler and interpreters course and our current task is to create a scanner and a set of tokens using JavaCC. I have a pretty solid understanding of how JavaCC works but my problem is finding resources online to help me out when I get stuck. I am working on creating a custom Token class, let's call it NewToken.Java. I know that the base Token class has an image variable and a kind variable but I want to implement my own variable "value". Furthermore I want to figure out how I can assign this value. I want the value variable to hold the literal value of what I scan, for example, my NewToken is being matched to the following

< IDENTIFIER:(< LETTER >)+ ( < LETTER > | < DIGIT >)* >
< #LETTER:["a" - "z"] >
< #DIGIT: ["0" - "9"] >

so something along the lines of Name123Name would get caught and when it does I want to store the string "Name123Name" into the 'value' variable of my NewToken object. I hope this makes sense, I am still new to JavaCC and may be calling things by there wrong name here.

 public NewToken(){}
  public NewToken(int kind){
    this(kind,null);
  }
  public NewToken(int kind, String image){
    this.kind=kind;
    this.image=image;
    this.value=image;
  }
  public String toString(){
    return image;
  }
  public static Token newToken(int ofKind, String image){
    switch(ofKind){
      default : return new Token(ofKind, image);
    }
  }
  public static Token newToken(int ofKind){
    return newToken(ofKind, null);
  }
}

Above is part of my code for the NewToken class, I have it extending Token and implementing java.io.serializable. I created by using the code generated for Token.java. I also have my variable declarations and my getValue() function which are not listed here to save space. I'm not looking for anyone to do my work for me I just need some guidance on how I would get this working, thank you in advance.

First off, I think the newToken routine should return objects of type NewToken rather than Token.

  public static Token newToken(int ofKind, String image){
    return new NewToken(ofKind, image);
  }
  public static Token newToken(int ofKind){
    return new NewToken(ofKind, null);
  } 

(I don’t think you need that second method. But, I’m not completely sure, so I’ll leave it.)

It’s a bit unclear to me how you want value to differ from image, but I’m going to assume that you can compute the desired value for value from the image and the kind. And I’ll further assume that you have implemented this function as a static method.

private static String computeValue(int kind, String image) {...}

Delete the first two constructors and the remaining one should be:

  private NewToken(int kind, String image){
    this.kind = kind;
    this.image = image;
    this.value = computeValue( kind, image );
  }

The answer that Professor Norvell is giving you is based on using a very old, obsolete version of JavaCC. The way he's suggesting you go about things is probably about the best way of doing it, if you were going to use the legacy JavaCC.

However, the most advanced version of JavaCC is JavaCC 21 and it handles this sort of use case straight out of the box in a very clean, elegant manner. See here for information on this specifically.

As you can see, you can put annotations in your grammar file that cause the various Token subclasses to be generated and used.

Also, JavaCC 21 has code injection that allows you to inject code directly into the any generated files, including the Token subclasses. That feature is not at all present in legacy JavaCC either. But using that, you could just inject your computeValue method right into the appropriate Token subclass.

INJECT NewToken :
{
   private static String computeValue(int kind, String image) {...}
}

You put that in your grammar and the computeValue method just gets inserted into the generated NewToken.java file.

By the way, there is an article about JavaCC 21 that appeared recently on dzone.com.