Web Framework Performance Test
环境
- 5.6.16-1-MANJARO
- Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz
- 8G内存
- rustc 1.44.1 (2020-06-17)
- Python 3.8.3
- SIEGE 4.0.5
测试命令参数
起先我采用如下参数测试tide，但是发现并发数设置成为1150,重复次数设置为5,就会出现下面的错误：
$ siege -c 1150 -r 5 http://192.168.1.100:8080
[error] descriptor table full sock.c:402: Too many open files
如果把重复次数降低为4,并发数最高可以设置为1180，服务器能够正常响应出统计结果,再增加并发数就会出现上面的错误了。 所以我最终将并发数设置为1000，然后自由调整重复次数，使之不报错正常输出结果。
TIDE
- tide = "0.11.0"
- async-std = "1.6.0"
如下将重复数量设置为5，观察CPU大约50%，结果如下，系统无报错：
$ siege -c 1000 -r 5 http://192.168.1.100:8080
{ "transactions": 5000,
"availability": 100.00,
"elapsed_time": 4.16,
"data_transferred": 0.05,
"response_time": 0.39,
"transaction_rate": 1201.92,
"throughput": 0.01,
"concurrency": 471.26,
"successful_transactions": 5000,
"failed_transactions": 0,
"longest_transaction": 3.24,
"shortest_transaction": 0.00
}
如下将重复数量设置为10，观察CPU大约70%，结果如下，系统无报错：
$ siege -c 1000 -r 10 http://192.168.1.100:8080
{ "transactions": 10000,
"availability": 100.00,
"elapsed_time": 6.02,
"data_transferred": 0.10,
"response_time": 0.31,
"transaction_rate": 1661.13,
"throughput": 0.02,
"concurrency": 513.53,
"successful_transactions": 10000,
"failed_transactions": 0,
"longest_transaction": 3.26,
"shortest_transaction": 0.00
}
如下将重复数量设置为15，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 15 http://192.168.1.100:8080
{ "transactions": 15000,
"availability": 100.00,
"elapsed_time": 6.62,
"data_transferred": 0.16,
"response_time": 0.24,
"transaction_rate": 2265.86,
"throughput": 0.02,
"concurrency": 540.57,
"successful_transactions": 15000,
"failed_transactions": 0,
"longest_transaction": 3.64,
"shortest_transaction": 0.00
}
如下将重复数量设置为20，观察CPU大约90%以上，结果如下，系统无报错：
$ siege -c 1000 -r 20 http://192.168.1.100:8080
{ "transactions": 20000,
"availability": 100.00,
"elapsed_time": 12.79,
"data_transferred": 0.21,
"response_time": 0.36,
"transaction_rate": 1563.72,
"throughput": 0.02,
"concurrency": 561.40,
"successful_transactions": 20000,
"failed_transactions": 0,
"longest_transaction": 3.62,
"shortest_transaction": 0.00
}
如下将重复数量设置为25，观察CPU大约90%以上，结果如下，系统无报错：
$ siege -c 1000 -r 25 http://192.168.1.100:8080
{ "transactions": 25000,
"availability": 100.00,
"elapsed_time": 19.78,
"data_transferred": 0.26,
"response_time": 0.48,
"transaction_rate": 1263.90,
"throughput": 0.01,
"concurrency": 611.58,
"successful_transactions": 25000,
"failed_transactions": 0,
"longest_transaction": 4.13,
"shortest_transaction": 0.00
}
如下将重复数量设置为30，观察CPU大约100%，结果如下，系统有报错：
$ siege -c 1000 -r 30 http://192.168.1.100:8080
...
[error] socket: 37758720 address is unavailable.: Cannot assign requested address
[error] socket: -1428867328 address is unavailable.: Cannot assign requested address
...
{ "transactions": 28235,
"availability": 95.77,
"elapsed_time": 27.66,
"data_transferred": 0.30,
"response_time": 0.61,
"transaction_rate": 1020.79,
"throughput": 0.01,
"concurrency": 625.06,
"successful_transactions": 28235,
"failed_transactions": 1248,
"longest_transaction": 4.90,
"shortest_transaction": 0.00
}
FLASK
如下将重复数量设置为5，观察CPU不足50%，结果如下，系统无报错：
$ siege -c 1000 -r 5 http://192.168.1.100:5000
{ "transactions": 5000,
"availability": 100.00,
"elapsed_time": 14.66,
"data_transferred": 0.06,
"response_time": 0.87,
"transaction_rate": 341.06,
"throughput": 0.00,
"concurrency": 298.16,
"successful_transactions": 5000,
"failed_transactions": 0,
"longest_transaction": 14.47,
"shortest_transaction": 0.00
}
如下将重复数量设置为10，观察CPU约为50%，结果如下，系统无报错：
$ siege -c 1000 -r 10 http://192.168.1.100:5000
{ "transactions": 10000,
"availability": 100.00,
"elapsed_time": 28.01,
"data_transferred": 0.12,
"response_time": 0.85,
"transaction_rate": 357.02,
"throughput": 0.00,
"concurrency": 304.87,
"successful_transactions": 10000,
"failed_transactions": 0,
"longest_transaction": 27.67,
"shortest_transaction": 0.00
}
如下将重复数量设置为15，观察CPU约为50%上下，结果如下，系统有报错：
$ siege -c 1000 -r 15 http://192.168.1.100:5000
...
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
...
{ "transactions": 14984,
"availability": 99.89,
"elapsed_time": 57.53,
"data_transferred": 0.19,
"response_time": 1.20,
"transaction_rate": 260.46,
"throughput": 0.00,
"concurrency": 313.80,
"successful_transactions": 14984,
"failed_transactions": 16,
"longest_transaction": 55.05,
"shortest_transaction": 0.00
}
如下将重复数量设置为30，观察CPU约为50%上下，结果如下，系统有报错：
$ siege -c 1000 -r 30 http://192.168.1.100:5000
...
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
...
{ "transactions": 29710,
"availability": 99.03,
"elapsed_time": 119.20,
"data_transferred": 0.37,
"response_time": 0.68,
"transaction_rate": 249.24,
"throughput": 0.00,
"concurrency": 170.37,
"successful_transactions": 29710,
"failed_transactions": 290,
"longest_transaction": 57.13,
"shortest_transaction": 0.00
}
ACTIX-WEB（后续简称ACTIX)
- actix-web = "2"
- actix-rt = "1.0"
如下将重复数量设置为10，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 10 http://127.0.0.1:8000
{ "transactions": 10000,
"availability": 100.00,
"elapsed_time": 2.72,
"data_transferred": 0.11,
"response_time": 0.22,
"transaction_rate": 3676.47,
"throughput": 0.04,
"concurrency": 815.28,
"successful_transactions": 10000,
"failed_transactions": 0,
"longest_transaction": 1.32,
"shortest_transaction": 0.00
}
如下将重复数量设置为20，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 20 http://127.0.0.1:8000
{ "transactions": 20000,
"availability": 100.00,
"elapsed_time": 4.91,
"data_transferred": 0.23,
"response_time": 0.20,
"transaction_rate": 4073.32,
"throughput": 0.05,
"concurrency": 815.21,
"successful_transactions": 20000,
"failed_transactions": 0,
"longest_transaction": 1.05,
"shortest_transaction": 0.00
}
如下将重复数量设置为30，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 30 http://127.0.0.1:8000
{ "transactions": 30000,
"availability": 100.00,
"elapsed_time": 7.66,
"data_transferred": 0.34,
"response_time": 0.22,
"transaction_rate": 3916.45,
"throughput": 0.04,
"concurrency": 878.44,
"successful_transactions": 30000,
"failed_transactions": 0,
"longest_transaction": 1.21,
"shortest_transaction": 0.00
}
如下将重复数量设置为40，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 40 http://127.0.0.1:8000
{ "transactions": 40000,
"availability": 100.00,
"elapsed_time": 10.32,
"data_transferred": 0.46,
"response_time": 0.23,
"transaction_rate": 3875.97,
"throughput": 0.04,
"concurrency": 882.39,
"successful_transactions": 40000,
"failed_transactions": 0,
"longest_transaction": 1.28,
"shortest_transaction": 0.00
}
如下将重复数量设置为100，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 100 http://127.0.0.1:8000
{ "transactions": 100000,
"availability": 100.00,
"elapsed_time": 24.92,
"data_transferred": 1.14,
"response_time": 0.23,
"transaction_rate": 4012.84,
"throughput": 0.05,
"concurrency": 910.48,
"successful_transactions": 100000,
"failed_transactions": 0,
"longest_transaction": 1.09,
"shortest_transaction": 0.00
}
如下将重复数量设置为200，观察CPU大约90%，结果如下，系统无报错：
$ siege -c 1000 -r 200 http://127.0.0.1:8000
{ "transactions": 200000,
"availability": 100.00,
"elapsed_time": 49.66,
"data_transferred": 2.29,
"response_time": 0.23,
"transaction_rate": 4027.39,
"throughput": 0.05,
"concurrency": 932.71,
"successful_transactions": 200000,
"failed_transactions": 0,
"longest_transaction": 1.30,
"shortest_transaction": 0.00
}
我觉得再测下去意义不大了，就是一个等待时间问题，从指标上看，重复数量越多，结果越好，估计是开始和结束的开销被稀释了。
WARP
- warp = "0.2"
如下将重复数量设置为10，观察CPU大约50%，结果如下，系统无报错：
$ siege -c 1000 -r 10 http://127.0.0.1:3030/hello/warp
{ "transactions": 10000,
"availability": 100.00,
"elapsed_time": 2.67,
"data_transferred": 0.11,
"response_time": 0.23,
"transaction_rate": 3745.32,
"throughput": 0.04,
"concurrency": 872.75,
"successful_transactions": 10000,
"failed_transactions": 0,
"longest_transaction": 1.31,
"shortest_transaction": 0.00
}
如下将重复数量设置为30，观察CPU大约50%，结果如下，系统无报错：
$ siege -c 1000 -r 30 http://127.0.0.1:3030/hello/warp
{ "transactions": 30000,
"availability": 100.00,
"elapsed_time": 7.68,
"data_transferred": 0.34,
"response_time": 0.24,
"transaction_rate": 3906.25,
"throughput": 0.04,
"concurrency": 926.62,
"successful_transactions": 30000,
"failed_transactions": 0,
"longest_transaction": 1.64,
"shortest_transaction": 0.00
}
如下将重复数量设置为200，观察CPU大约50%，结果如下，系统无报错：
$ siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp
{ "transactions": 200000,
"availability": 100.00,
"elapsed_time": 51.50,
"data_transferred": 2.29,
"response_time": 0.26,
"transaction_rate": 3883.50,
"throughput": 0.04,
"concurrency": 993.57,
"successful_transactions": 200000,
"failed_transactions": 0,
"longest_transaction": 0.73,
"shortest_transaction": 0.00
}
总结
由于我主要关注的是几个框架之间的差别，所以只用了siege一个压测工具，以后有机会再使用apache benchmark或者社员建议的vegeta试试看。
只看测试指标，WARP的表现似乎最好，它的CPU利用率明显比ACTIX低，而同时其它指标与ACTIX不相上下，甚至更好，但不明白为什么WARP的整体测试耗时（elapsed_time)比ACTIX要多几秒钟。
WARP与ACTIX表现都比较稳定，压测不出错，而且指标上下浮动不大。TIDE重复数量到30以后就开始系统报错了，而且多次测试结果相差较大，可能正如其开发者所说，项目仍然不建议被用于生产，还不太成熟吧。类似的FLASK测试重复数量到15以后就开始系统报错，并且多次测试结果上下浮动较大（上下各50%，限于篇幅，这个结果没有在文章中体现）
1
共 4 条评论, 1 页
评论区
写评论顶！
actix貌似会泄漏内存, 多压压会不会炸
另外回答一下社员的一些疑问：
昨天actix-web测试总耗时比warp高一倍的原因找到了，就是ulimit默认为1024导致的，将其改为2000,actix-web的指标就变好了，下面是ulimit -n 2000之后的测试结果：
$ siege -c 1000 -r 200 http://127.0.0.1:8000/actix
{ "transactions": 200000, "availability": 100.00, "elapsed_time": 52.88, "data_transferred": 2.29, "response_time": 0.25, "transaction_rate": 3782.15, "throughput": 0.04, "concurrency": 942.51, "successful_transactions": 200000, "failed_transactions": 0, "longest_transaction": 1.45, "shortest_transaction": 0.00 } $ siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp
{ "transactions": 200000, "availability": 100.00, "elapsed_time": 51.64, "data_transferred": 2.29, "response_time": 0.25, "transaction_rate": 3872.97, "throughput": 0.04, "concurrency": 986.77, "successful_transactions": 200000, "failed_transactions": 0, "longest_transaction": 1.52, "shortest_transaction": 0.00 }
下面是actix的CPU利用率与内在使用统计，第三列是CPU（单位：100%），第四列是内存（单位：100%） $ ps aux |grep actix yw 3338 84.8 0.3 351272 28940 pts/2 Sl+ 09:13 1:36 ./of-actix yw 3598 122 5.7 10290008 452776 pts/3 Sl+ 09:14 0:53 siege -c 1000 -r 200 http://127.0.0.1:8000/actix
下面是warp的CPU利用率与内在使用统计，第三列是CPU（单位：100%），第四列是内存（单位：100%） $ ps aux |grep warp yw 4973 66.6 0.0 10352 7784 pts/2 R+ 09:18 0:14 ./of-warp yw 4974 117 6.1 10309324 479468 pts/3 Sl+ 09:18 0:17 siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp
这次统计可以看出，actix和warp在CPU利用率上都可以比测试工具更低，warp比actix的CPU利用率仍然低一些，特别是warp比actix明显省内存。