What's the equivalent of appengine:run for the Java 11 Cloud SDK?

tl;dr: How can I run this project locally, in a way that Datastore will work? (Zip download link here.)

I'm migrating a Java 8 project that used App Engine and Datastore over to Java 11.

With Java 8, I used the Cloud SDK-based App Engine plugin to run the server locally using mvn appengine:run and to deploy to the live server using mvn appengine:deploy.

I followed this guide which told me to delete the appengine-web.xml file and use app.yaml instead..

To deploy to the live server, I can still use mvn appengine:deploy and this works fine, with and without Datastore.

To deploy locally, I run mvn package exec:java. This works fine for running a basic server without Datastore, but if I add some example Datastore code, then I get this error:

java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder.

I know this is because I don't have a local App Engine environment running. But if I try to run the local server using mvn appengine:run then I get this error complaining about the missing appengine-web.xml file:

[ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:run (default-cli) on project datastore-hello-world:
Failed to run devappserver: java.nio.file.NoSuchFileException:
C:\Users\kevin\Documents\GitHub\HappyCoding\examples\google-cloud\google-cloud-example-projects\datastore-hello-world\target\datastore-hello-world-1\WEB-INF\appengine-web.xml -> [Help 1]

So I'm stuck between needing to run mvn appengine:run which requires appengine-web.xml, and needing to update to Java 11, which says to use app.yaml instead of appengine-web.xml.

The Java 11 App Engine docs only say this about running locally:

> To test your application's functionality before deploying, run your application in your local environment with the development tools that you usually use.

The above GitHub and zip links contain the whole project (5 total files), but in the interest of including my code directly in the question, here are the files that matter the most:

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>io.happycoding</groupId>
  <artifactId>datastore-hello-world</artifactId>
  <version>1</version>

  <properties>
    <!-- App Engine currently supports Java 11 -->
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jetty.version>9.4.31.v20200723</jetty.version>

    <!-- Project-specific properties -->
    <mainClass>io.happycoding.ServerMain</mainClass>
    <googleCloudProjectId>YOUR_PROJECT_ID_HERE</googleCloudProjectId>
  </properties>

  <dependencies>
    <!-- Java Servlets API -->
    <dependency>
      <groupId>javax.servlet</groupId>
      <artifactId>javax.servlet-api</artifactId>
      <version>4.0.1</version>
    </dependency>

    <!-- Jetty -->
    <dependency>
      <groupId>org.eclipse.jetty</groupId>
      <artifactId>jetty-server</artifactId>
      <version>${jetty.version}</version>
    </dependency>
    <dependency>
      <groupId>org.eclipse.jetty</groupId>
      <artifactId>jetty-annotations</artifactId>
      <version>${jetty.version}</version>
    </dependency>

    <!-- Datastore -->
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-datastore</artifactId>
      <version>1.104.0</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <!-- Copy static resources like html files into the output jar file. -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>2.7</version>
        <executions>
          <execution>
            <id>copy-web-resources</id>
            <phase>compile</phase>
            <goals><goal>copy-resources</goal></goals>
            <configuration>
              <outputDirectory>
                ${project.build.directory}/classes/META-INF/resources
              </outputDirectory>
              <resources>
                <resource><directory>./src/main/webapp</directory></resource>
              </resources>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <!-- Package everything into a single executable jar file. -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.4</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals><goal>shade</goal></goals>
            <configuration>
              <createDependencyReducedPom>false</createDependencyReducedPom>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>${mainClass}</mainClass>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <!-- Exec plugin for deploying the local server. -->
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <goals>
              <goal>java</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <mainClass>${mainClass}</mainClass>
        </configuration>
      </plugin>

      <!-- App Engine plugin for deploying to the live site. -->
      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>appengine-maven-plugin</artifactId>
        <version>2.2.0</version>
        <configuration>
          <projectId>${googleCloudProjectId}</projectId>
          <version>1</version>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

ServerMain.java

package io.happycoding;

import java.net.URL;
import org.eclipse.jetty.annotations.AnnotationConfiguration;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.handler.DefaultHandler;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.DefaultServlet;
import org.eclipse.jetty.webapp.Configuration;
import org.eclipse.jetty.webapp.WebAppContext;
import org.eclipse.jetty.webapp.WebInfConfiguration;

/**
 * Starts up the server, including a DefaultServlet that handles static files,
 * and any servlet classes annotated with the @WebServlet annotation.
 */
public class ServerMain {

  public static void main(String[] args) throws Exception {

    // Create a server that listens on port 8080.
    Server server = new Server(8080);
    WebAppContext webAppContext = new WebAppContext();
    server.setHandler(webAppContext);

    // Load static content from inside the jar file.
    URL webAppDir =
        ServerMain.class.getClassLoader().getResource("META-INF/resources");
    webAppContext.setResourceBase(webAppDir.toURI().toString());

    // Enable annotations so the server sees classes annotated with @WebServlet.
    webAppContext.setConfigurations(new Configuration[]{ 
      new AnnotationConfiguration(),
      new WebInfConfiguration(), 
    });

    // Look for annotations in the classes directory (dev server) and in the
    // jar file (live server)
    webAppContext.setAttribute(
        "org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern", 
        ".*/target/classes/|.*\\.jar");

    // Handle static resources, e.g. html files.
    webAppContext.addServlet(DefaultServlet.class, "/");

    // Start the server! 🚀
    server.start();
    System.out.println("Server started!");

    // Keep the main thread alive while the server is running.
    server.join();
  }
}

HelloWorldServlet.java

package io.happycoding.servlets;

import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.cloud.datastore.Entity;
import com.google.cloud.datastore.Key;
import java.io.IOException;

import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@WebServlet("/hello")
public class HelloWorldServlet extends HttpServlet {

  @Override
  public void doGet(HttpServletRequest request, HttpServletResponse response)
      throws IOException {

    Datastore datastore = DatastoreOptions.getDefaultInstance().getService();

    String kind = "Task";
    // The name/ID for the new entity
    String name = "sampletask1";
    // The Cloud Datastore key for the new entity
    Key taskKey = datastore.newKeyFactory().setKind(kind).newKey(name);

    // Prepares the new entity
    Entity task = Entity.newBuilder(taskKey)
        .set("description", "Buy milk")
        .build();

    // Saves the entity
    datastore.put(task);

    System.out.printf("Saved %s: %s%n", task.getKey().getName(), task.getString("description"));

    //Retrieve entity
    Entity retrieved = datastore.get(taskKey);

    response.setContentType("text/html;");
    response.getWriter().println("<h1>Hello world!</h1>");
    response.getWriter().println("Retrieved: " + taskKey.getName() + ": " + retrieved.getString("description"));
  }
}

How can I run a Java 11 App Engine server locally so that Datastore works?

Based on guillaume blaquiere's suggestion in their comment, I tried following this guide for manually running Datastore locally.

I ran gcloud beta emulators datastore start in one command line, which seemed to run fine, and then I ran $(gcloud beta emulators datastore env-init) in another command line, and I got this error:

ERROR: (gcloud.beta.emulators.datastore.env-init)
Unable to find env.yaml in the data_dir 
[/tmp/tmp.zolNZC8fnr/emulators/datastore]. 
Please ensure you have started the appropriate emulator.

I went back to the first command line where I ran Datastore, and I noticed this line in the output:

Storage: /tmp/tmp.sxkNeaNHxo/emulators/datastore/WEB-INF/appengine-generated/local_db.bin

I copied that directory path into the data-dir argument:

$(gcloud beta emulators datastore env-init --data-dir=/tmp/tmp.sxkNeaNHxo/emulators/datastore)

That command seemed to succeed (with no output) so I ran my local server: mvn package exec:java

Finally, this seems to work locally.

To summarize:

Step 1: Run this command: gcloud beta emulators datastore start

Step 2: In the output, find the line that looks like this:

Storage: /tmp/tmp.sxkNeaNHxo/emulators/datastore/WEB-INF/appengine-generated/local_db.bin

Copy the /tmp/tmp.YOUR_PATH_HERE/emulators/datastore path.

Step 3: In a different command line, run this command, pasting in the path you just copied:

$(gcloud beta emulators datastore env-init --data-dir=/tmp/tmp.YOUR_PATH_HERE/emulators/datastore)

Step 4: In that second command line, run your local server: mvn package exec:java

This feels pretty convoluted (and undocumented), so I'm still very open to suggestions for how to streamline this. But for now, this at least works.

You can keep it all running locally in Java 11 in the same way it all worked for Java 8.
But you need to keep the Legacy Bundled Services:
https://cloud.google.com/appengine/docs/standard/java-gen2/services/access#java-11

• So you still have a appengine-web.xml which gcloud coverts to an app.yaml in the staging directory
• You need to add <app-engine-apis>true</app-engine-apis> to your appengine-web.xml file.
• I also needed to ensure I had the latest maven gcloud, appengine SDK, and other dependencies in my pom.xml file.