Not using IsolationActivity via AndroidBenchmarkRunner

I'm trying to try out Android's Jetpack Benchmark library and cannot seem to get past this error. Have tried everything I can think of, would appreciate any help making this simple example work.

java.lang.AssertionError: ERRORS (not suppressed): ACTIVITY-MISSING
(Suppressed errors: DEBUGGABLE EMULATOR UNLOCKED)

WARNING: Not using IsolationActivity via AndroidBenchmarkRunner
AndroidBenchmarkRunner should be used to isolate benchmarks from interference
from other visible apps. To fix this, add the following to your module-level
build.gradle:
android.defaultConfig.testInstrumentationRunner
= "androidx.benchmark.junit4.AndroidBenchmarkRunner"

I have followed the sample guide here, and my defaultConfig contains:

testInstrumentationRunner 'androidx.benchmark.junit4.AndroidBenchmarkRunner'
// Suppressing errors I can ignore while I'm testing. Unfortunately, suppressing 
// the ACTIVITY-MISSING error causes runtime crashes
testInstrumentationRunnerArgument 'androidx.benchmark.suppressErrors', 'EMULATOR,DEBUGGABLE,UNLOCKED'

and my dependencies contain:

    androidTestImplementation 'androidx.annotation:annotation:1.1.0'
    androidTestImplementation 'androidx.test:runner:1.2.0'
    androidTestImplementation 'androidx.test:core:1.2.0'
    androidTestImplementation 'androidx.test:rules:1.2.0'
    androidTestImplementation 'org.hamcrest:hamcrest-library:1.3'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.benchmark:benchmark-junit4:1.0.0'

My benchmark code is as follows:

import android.util.Log;

import androidx.benchmark.BenchmarkState;
import androidx.benchmark.junit4.BenchmarkRule;
import androidx.test.ext.junit.runners.AndroidJUnit4;

import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(AndroidJUnit4.class)
public class BenchmarkTest {

    @Rule
    public BenchmarkRule benchmarkRule = new BenchmarkRule();

    @Test
    public void log() {
        final BenchmarkState state = benchmarkRule.getState();
        while (state.keepRunning()) {
            Log.d("LogBenchmark", "the cost of writing this log method will be measured");
        }
    }
}

I faced the same issue when I followed the quick start guide. However, I got it to work by adding a new benchmark module and picking up the dependencies from the sample app.

Steps to add the benchmark module:

1. File > New > New Module

2. Select Benchmark Module

Dependencies:

    implementation fileTree(dir: 'libs', include: ['*.jar'])

    androidTestImplementation project("<enter your project>")
    androidTestImplementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"

    androidTestImplementation 'junit:junit:4.12'

    androidTestImplementation 'androidx.test:runner:1.2.0'
    androidTestImplementation 'androidx.test:rules:1.2.0'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'

    androidTestImplementation 'androidx.appcompat:appcompat:1.1.0'
    androidTestImplementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    androidTestImplementation 'androidx.cardview:cardview:1.0.0'
    androidTestImplementation 'androidx.recyclerview:recyclerview:1.0.0'
    androidTestImplementation 'androidx.benchmark:benchmark-junit4:1.1.0-alpha01'

 buildTypes {
        debug {
            minifyEnabled true
            proguardFiles getDefaultProguardFile("proguard-android-optimize.txt"), "benchmark-proguard-rules.pro"
        }
        release {
            isDefault = true
            isDefault = true
            minifyEnabled true
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'benchmark-proguard-rules.pro'
            signingConfig signingConfigs.debug
        }
    }

Be sure to define the release signing config, since creating a benchmark module does not add it by default.

I had this same problem and was only able to solve it after checking out their sample app. It looks like they forgot to mention the pro guard rules. Unfortunately this kind of thing happens a lot so check the sample apps.

I believe if you're missing this it might be why you get that error:

-keepattributes *Annotation*
-keepclasseswithmembers @org.junit.runner.RunWith public class *

Another funny thing I am seeing is that they do not use the instrumentation runner in their project.

testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner"

https://github.com/android/performance-samples/blob/master/BenchmarkSample/benchmark/benchmark-proguard-rules.pro

https://developer.android.com/studio/profile/benchmark

>To set up benchmarking for regular benchmarking rather than one-off benchmarking, you isolate benchmarks into their own module.

So, make sure that you put the benchmark in a module library (It should apply plugin 'com.android.library', not 'com.android.application')

Also, put the plugin:

apply plugin: 'androidx.benchmark'

If you are using a Chinese phone to run tests, try to disable background restrictions for the app com.example.banckmark.test and enable the Auto-Start checkbox. This may also work for devices from other manufacturers. See https://dontkillmyapp.com/