SpringBoot项目使用jmh进行接口基准测试

JMH（Java Microbenchmark Harness）是一个 Java 工具，用于构建、运行和分析用 Java 和其他针对 JVM 的语言编写的 纳米/微米/毫/宏观 基准测试，而且是由Java虚拟机团队开发的。简单说，就是用来测量代码运行性能。

• JMH官网：https://openjdk.org/projects/code-tools/jmh/
• JMH源码下载：https://github.com/openjdk/jmh
• JMH示例代码：https://github.com/openjdk/jmh/tree/1.36/jmh-samples/src/main/java/org/openjdk/jmh/samples

JMeter可能是最常用的性能测试工具。它既支持图形界面，也支持命令行，属于黑盒测试的范畴，对非开发人员比较友好，上手也非常容易。

图形界面一般用于编写、调试测试用例，而实际的性能测试建议还是在命令行下运行。

很多场景下JMeter和JMH都可以做性能测试，但是对于严格意义上的基准测试来说，只有JMH才适合。JMeter的测试结果精度相对JVM较低、所以JMeter不适合于类级别的基准测试，更适合于对精度要求不高、耗时相对较长的操作。

• JMeter测试精度差： JMeter自身框架比较重，举个例子：使用JMH测试一个方法，平均耗时0.01ms，而使用JMeter测试的结果平均耗时20ms，相差200倍。
• JMeter内置很多采样器：JMeter内置了支持多种网络协议的采样器，可以在不写Java代码的情况下实现很多复杂的测试。JMeter支持集群的方式运行，方便模拟多用户、高并发压力测试。

JMeter适合一些相对耗时的集成功能测试，如API接口的测试。JMH适合于类或者方法的单元测试。

JMH非常好的一点是我们非常灵活： 我们可以手动来编写测试的代码。

比如

@Benchmark
public void test() {
    this.controller.getRandomPoet();
}

准备工作

导入依赖

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.22</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.22</version>
    <scope>provided</scope>
</dependency>

部分注解/字段含义

注解	说明
@BenchmarkMode	JMH进行Benchmark时使用的模式。包括Throughput, AverageTime, SampleTime和SingleShotTime
@OutputTimeUnit	输出的时间单位
@Iteration	JMH测试的最小单位。一次iteration代表的是一秒，JMH在这一秒内不断调用需要Benchmark的方法。
@WarmUp	预热行为。在实际进行Benchmark前先进行预热，让Benchmark结果更接近真实情况。
@State	定义类实例的生命周期，类似于Spring Bean的Scope。包括Scope.Thread, Scope.Benchmark和Scope.Group
@Fork	进行fork的次数。如果fork数是2的话，则JMH会fork出两个进程来进行测试。
@Measurement	提供真正的测试阶段参数。指定迭代的次数，每次迭代的运行时间和每次迭代测试调用的数量。
@Setup	在执行benchmark之前被执行，主要用于初始化。
@TearDown	在所有benchmark执行结束以后执行，主要用于资源的回收等。
@Benchmark	表示该方法需要进行benchmark的对象。
@Param	用来指定某项参数的多种情况。特别适合用来测试一个函数在不同的参数输入的情况下的性能。

BenchMark基准测试

首先给出 SortBenchMarkTest , 表示我们需要测试的代码

@State(Scope.Thread)
public class SortBenchMarkTest {
    private List<String> list;

    @Setup
    public void setup() {
        list = new ArrayList<>();
        for (int i = 0; i < 10000; i++) {
            list.add(UUID.randomUUID().toString());
        }
    }

    @TearDown
    public void tearDown() {
        list = null;
    }

    @Benchmark
    public void testSort() {
        Collections.sort(list);
    }
}

接着创建 MyBenchmarkTest , 是来执行测试的代码

public class MyBenchmarkTest {
    @Test
    public void testMyBenchmark() throws Exception {
        Options options = new OptionsBuilder()
            .include(SortBenchMarkTest.class.getSimpleName())
            .forks(1)
            .threads(1)
            .warmupIterations(5)
            .measurementIterations(5)
            .mode(Mode.AverageTime)
            .build();
        new Runner(options).run();
    }
}

我们也可以把测试的结果保存到文件中, 并且通过平台进行分析:

private Options getOptions() {
    String resultFilePrefix = this.getClass().getSimpleName() + "jmh-result";
    ResultFormatType resultsFileOutputType = ResultFormatType.JSON;
    return new OptionsBuilder()
        .include("\\." + this.getClass().getSimpleName() + "\\.")
        .shouldDoGC(true)
        .shouldFailOnError(true)
        .forks(1)
        .threads(1) // 测试线程数
        .warmupIterations(0) // 预热迭代次数
        .measurementIterations(1) // 测试迭代次数
        .timeUnit(TimeUnit.MICROSECONDS)
        .resultFormat(resultsFileOutputType)
        .result(buildResultsFileName(resultFilePrefix, resultsFileOutputType))
        .shouldFailOnError(true)
        .jvmArgs("-server")
        .build();
}

private static String buildResultsFileName(String resultFilePrefix, ResultFormatType resultType) {
    String suffix = switch (resultType) {
        case CSV -> ".csv";
        case SCSV -> ".scsv";
        case LATEX -> ".tex";
        case JSON -> ".json";
        default -> ".json";
    };
    return String.format("../target/%s%s", resultFilePrefix, suffix);
}

测试结果 (SortStringBenchmarkTestjmh-result.json) 如下:

[
    {
        "jmhVersion" : "1.22",
        "benchmark" : "com.dhx.apiinterface.jmh.SortStringBenchmarkTest.testSort",
        "mode" : "avgt",
        "threads" : 1,
        "forks" : 1,
        "jvm" : "D:\\jdk17_\\bin\\java.exe",
        "jvmArgs" : [
            "-server"
        ],
        "jdkVersion" : "17.0.6",
        "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
        "vmVersion" : "17.0.6+9-LTS-190",
        "warmupIterations" : 0,
        "warmupTime" : "10 s",
        "warmupBatchSize" : 1,
        "measurementIterations" : 1,
        "measurementTime" : "10 s",
        "measurementBatchSize" : 1,
        "primaryMetric" : {
            "score" : 36.30124459659404,
            "scoreError" : "NaN",
            "scoreConfidence" : [
                "NaN",
                "NaN"
            ],
            "scorePercentiles" : {
                "0.0" : 36.30124459659404,
                "50.0" : 36.30124459659404,
                "90.0" : 36.30124459659404,
                "95.0" : 36.30124459659404,
                "99.0" : 36.30124459659404,
                "99.9" : 36.30124459659404,
                "99.99" : 36.30124459659404,
                "99.999" : 36.30124459659404,
                "99.9999" : 36.30124459659404,
                "100.0" : 36.30124459659404
            },
            "scoreUnit" : "us/op",
            "rawData" : [
                [
                    36.30124459659404
                ]
            ]
        },
        "secondaryMetrics" : {
        }
    }
]

其实可以写到一个类里面:

@State(Scope.Benchmark)
@BenchmarkMode(Mode.AverageTime)
@Timeout(time = 120)
public class SortStringBenchmarkTest {

    private List<String> list;

    @Setup
    public void setup() {
        list = new ArrayList<>();
        for (int i = 0; i < 10000; i++) {
            list.add(UUID.randomUUID().toString());
        }
    }

    @TearDown
    public void tearDown() {
        list = null;
    }

    @Benchmark
    public void testSort() {
        Collections.sort(list);
    }

    @Test
    public void testMyBenchmark() throws Exception {
        new Runner(getOptions()).run();
    }

    private Options getOptions() {
        String resultFilePrefix = this.getClass().getSimpleName() + "jmh-result";
        ResultFormatType resultsFileOutputType = ResultFormatType.JSON;
        return new OptionsBuilder()
            .include("\\." + this.getClass().getSimpleName() + "\\.")
            .shouldDoGC(true)
            .shouldFailOnError(true)
            .forks(1)
            .threads(1) // 测试线程数
            .warmupIterations(0) // 预热迭代次数
            .measurementIterations(1) // 测试迭代次数
            .timeUnit(TimeUnit.MICROSECONDS)
            .resultFormat(resultsFileOutputType)
            .result(buildResultsFileName(resultFilePrefix, resultsFileOutputType))
            .shouldFailOnError(true)
            .jvmArgs("-server")
            .build();
    }

    private static String buildResultsFileName(String resultFilePrefix, ResultFormatType resultType) {
        String suffix = switch (resultType) {
            case CSV -> ".csv";
            case SCSV -> ".scsv";
            case LATEX -> ".tex";
            case JSON -> ".json";
            default -> ".json";
        };
        return String.format("target/%s%s", resultFilePrefix, suffix);
    }
}

查看target下的测试数据文件

访问 https://jmh.morethan.io/# 进行测试结果分析:

抽离公共代码

上面的代码, 我们可以抽离出一个抽象类:

public abstract class BaseBenchMarkTest {
    protected Options getOptions() {
        String resultFilePrefix = this.getClass().getSimpleName() + "jmh-result";
        ResultFormatType resultsFileOutputType = ResultFormatType.JSON;
        return new OptionsBuilder()
            .include("\\." + this.getClass().getSimpleName() + "\\.")
            .shouldDoGC(true)
            .shouldFailOnError(true)
            .forks(1)
            .threads(1) // 测试线程数
            .warmupIterations(5) // 预热迭代次数
            .measurementIterations(10) // 测试迭代次数
            .timeUnit(TimeUnit.MICROSECONDS)
            .resultFormat(resultsFileOutputType)
            .result(buildResultsFileName(resultFilePrefix, resultsFileOutputType))
            .shouldFailOnError(true)
            .jvmArgs("-server")
            .build();
    }

    private static String buildResultsFileName(String resultFilePrefix, ResultFormatType resultType) {
        String suffix = switch (resultType) {
            case CSV -> ".csv";
            case SCSV -> ".scsv";
            case LATEX -> ".tex";
            case JSON -> ".json";
            default -> ".json";
        };
        return String.format("target/%s%s", resultFilePrefix, suffix);
    }
}

修改之后的测试代码如下:

public class SortStringBenchmarkTest extends BaseBenchMarkTest {

    private List<String> list;
    @Setup
    public void setup() {
        list = new ArrayList<>();
        for (int i = 0; i < 10000; i++) {
            list.add(UUID.randomUUID().toString());
        }
    }
    @TearDown
    public void tearDown() {
        list = null;
    }

    @Benchmark
    public void testSort() {
        Collections.sort(list);
    }


    @Test
    public void testMyBenchmark() throws Exception {
        new Runner(getOptions()).run();
    }

}

SpringBoot中使用BenchMark进行测试

对于SpringBoot中的测试, 关键在于需要用到Bean, 此时需要我们手动获取上下文信息并getBean()

获取上下文 :

ConfigurableApplicationContext context;

@Setup
public void setup() {
    context = SpringApplication.run(ApiInterfaceApplication.class);
}

@TearDown
public void tearDown() {
    // 在测试结束后进行清理工作，比如关闭Spring上下文
    context.close();
}

获取Bean:

this.controller = this.context.getBean(Version1Controller.class);

进行测试:

public class RandomPoetBenchmarkTest extends SpringBaseBenchMarkTest {
    private Version1Controller controller;

    @Benchmark
    public void test() {
        this.controller.getRandomPoet();
    }

    @BeforeEach
    public void getBean() {
        this.controller = this.context.getBean(Version1Controller.class);
    }

    @Test
    public void executeJmhRunner() throws RunnerException, IOException {
        new Runner(getOptions()).run();
    }

}

测试信息

# JMH version: 1.22
# VM version: JDK 17.0.6, Java HotSpot(TM) 64-Bit Server VM, 17.0.6+9-LTS-190
# VM invoker: D:\jdk17_\bin\java.exe
# VM options: -Dvisualvm.id=441431370386300 -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:C:\software\IntelliJ IDEA 2022.2.3\lib\idea_rt.jar=12472:C:\software\IntelliJ IDEA 2022.2.3\bin -Dfile.encoding=UTF-8
# Warmup: 5 iterations, 10 s each
# Measurement: 20 iterations, 10 s each
# Timeout: 120 s per iteration
# Threads: 4 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: com.dhx.apiinterface.jmh.NewV1ApiBenchmarkTest.test

# Run progress: 0.00% complete, ETA 00:04:10
# Fork: 1 of 1
# Warmup Iteration   1: 1.257 ±(99.9%) 0.074 ms/op
# Warmup Iteration   2: 1.277 ±(99.9%) 0.071 ms/op
# Warmup Iteration   3: 1.266 ±(99.9%) 0.053 ms/op
# Warmup Iteration   4: 1.228 ±(99.9%) 0.055 ms/op
# Warmup Iteration   5: 1.259 ±(99.9%) 0.031 ms/op
Iteration   1: 1.392 ±(99.9%) 0.055 ms/op
Iteration   2: 1.248 ±(99.9%) 0.068 ms/op
Iteration   3: 1.158 ±(99.9%) 0.088 ms/op
Iteration   4: [2024-01-03 23:15:15.583] [] INFO  o.a.d.r.t.n.NettyServerHandler ==>  [DUBBO] The connection of /100.122.217.6:56510 -> /100.72.207.93:20880 is established., dubbo version: 3.1.6, current host: 192.168.166.1
1.100 ±(99.9%) 0.042 ms/op
Iteration   5: 1.096 ±(99.9%) 0.046 ms/op
Iteration   6: 1.193 ±(99.9%) 0.038 ms/op
Iteration   7: 1.454 ±(99.9%) 0.058 ms/op
Iteration   8: 1.329 ±(99.9%) 0.037 ms/op
Iteration   9: 1.212 ±(99.9%) 0.075 ms/op
Iteration  10: 1.242 ±(99.9%) 0.047 ms/op
Iteration  11: 1.300 ±(99.9%) 0.074 ms/op
Iteration  12: 1.193 ±(99.9%) 0.058 ms/op
Iteration  13: 1.264 ±(99.9%) 0.033 ms/op
Iteration  14: 1.228 ±(99.9%) 0.074 ms/op
Iteration  15: 1.394 ±(99.9%) 0.036 ms/op
Iteration  16: 1.413 ±(99.9%) 0.079 ms/op
Iteration  17: 1.184 ±(99.9%) 0.052 ms/op
Iteration  18: 1.170 ±(99.9%) 0.044 ms/op
Iteration  19: 1.306 ±(99.9%) 0.074 ms/op

Result "com.dhx.apiinterface.jmh.NewV1ApiBenchmarkTest.test":
  1.257 ±(99.9%) 0.088 ms/op [Average]
  (min, avg, max) = (1.096, 1.257, 1.454), stdev = 0.101
  CI (99.9%): [1.169, 1.345] (assumes normal distribution)

# Run complete. Total time: 00:04:51

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                   Mode  Cnt  Score   Error  Units
NewV1ApiBenchmarkTest.test  avgt   20  1.257 ± 0.088  ms/op

Reference

• https://blog.csdn.net/qq_24598601/article/details/130163751
• https://blog.csdn.net/qq_24598601/article/details/130163751
• https://www.cnblogs.com/Kevin-ZhangCG/p/14950999.html