< 返回版块

bluephoen1x 发表于 2020-06-28 22:39

Tags:性能,warp,actix,tide,flask

Web Framework Performance Test

环境

  • 5.6.16-1-MANJARO
  • Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz
  • 8G内存
  • rustc 1.44.1 (2020-06-17)
  • Python 3.8.3
  • SIEGE 4.0.5

测试命令参数

起先我采用如下参数测试tide,但是发现并发数设置成为1150,重复次数设置为5,就会出现下面的错误:

$ siege -c 1150 -r 5 http://192.168.1.100:8080  
[error] descriptor table full sock.c:402: Too many open files

如果把重复次数降低为4,并发数最高可以设置为1180,服务器能够正常响应出统计结果,再增加并发数就会出现上面的错误了。 所以我最终将并发数设置为1000,然后自由调整重复次数,使之不报错正常输出结果。

TIDE

  • tide = "0.11.0"
  • async-std = "1.6.0"

如下将重复数量设置为5,观察CPU大约50%,结果如下,系统无报错:

$ siege -c 1000 -r 5 http://192.168.1.100:8080

{	"transactions":			        5000,
	"availability":			      100.00,
	"elapsed_time":			        4.16,
	"data_transferred":		        0.05,
	"response_time":		        0.39,
	"transaction_rate":		     1201.92,
	"throughput":			        0.01,
	"concurrency":			      471.26,
	"successful_transactions":	        5000,
	"failed_transactions":		           0,
	"longest_transaction":		        3.24,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为10,观察CPU大约70%,结果如下,系统无报错:

$ siege -c 1000 -r 10 http://192.168.1.100:8080

{	"transactions":			       10000,
	"availability":			      100.00,
	"elapsed_time":			        6.02,
	"data_transferred":		        0.10,
	"response_time":		        0.31,
	"transaction_rate":		     1661.13,
	"throughput":			        0.02,
	"concurrency":			      513.53,
	"successful_transactions":	       10000,
	"failed_transactions":		           0,
	"longest_transaction":		        3.26,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为15,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 15 http://192.168.1.100:8080

{	"transactions":			       15000,
	"availability":			      100.00,
	"elapsed_time":			        6.62,
	"data_transferred":		        0.16,
	"response_time":		        0.24,
	"transaction_rate":		     2265.86,
	"throughput":			        0.02,
	"concurrency":			      540.57,
	"successful_transactions":	       15000,
	"failed_transactions":		           0,
	"longest_transaction":		        3.64,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为20,观察CPU大约90%以上,结果如下,系统无报错:

$ siege -c 1000 -r 20 http://192.168.1.100:8080

{	"transactions":			       20000,
	"availability":			      100.00,
	"elapsed_time":			       12.79,
	"data_transferred":		        0.21,
	"response_time":		        0.36,
	"transaction_rate":		     1563.72,
	"throughput":			        0.02,
	"concurrency":			      561.40,
	"successful_transactions":	       20000,
	"failed_transactions":		           0,
	"longest_transaction":		        3.62,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为25,观察CPU大约90%以上,结果如下,系统无报错:

$ siege -c 1000 -r 25 http://192.168.1.100:8080

{	"transactions":			       25000,
	"availability":			      100.00,
	"elapsed_time":			       19.78,
	"data_transferred":		        0.26,
	"response_time":		        0.48,
	"transaction_rate":		     1263.90,
	"throughput":			        0.01,
	"concurrency":			      611.58,
	"successful_transactions":	       25000,
	"failed_transactions":		           0,
	"longest_transaction":		        4.13,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为30,观察CPU大约100%,结果如下,系统有报错:

$ siege -c 1000 -r 30 http://192.168.1.100:8080
...
[error] socket: 37758720 address is unavailable.: Cannot assign requested address
[error] socket: -1428867328 address is unavailable.: Cannot assign requested address
...
{	"transactions":			       28235,
	"availability":			       95.77,
	"elapsed_time":			       27.66,
	"data_transferred":		        0.30,
	"response_time":		        0.61,
	"transaction_rate":		     1020.79,
	"throughput":			        0.01,
	"concurrency":			      625.06,
	"successful_transactions":	       28235,
	"failed_transactions":		        1248,
	"longest_transaction":		        4.90,
	"shortest_transaction":		        0.00
}

FLASK

如下将重复数量设置为5,观察CPU不足50%,结果如下,系统无报错:

$ siege -c 1000 -r 5 http://192.168.1.100:5000

{	"transactions":			        5000,
	"availability":			      100.00,
	"elapsed_time":			       14.66,
	"data_transferred":		        0.06,
	"response_time":		        0.87,
	"transaction_rate":		      341.06,
	"throughput":			        0.00,
	"concurrency":			      298.16,
	"successful_transactions":	        5000,
	"failed_transactions":		           0,
	"longest_transaction":		       14.47,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为10,观察CPU约为50%,结果如下,系统无报错:

$ siege -c 1000 -r 10 http://192.168.1.100:5000

{	"transactions":			       10000,
	"availability":			      100.00,
	"elapsed_time":			       28.01,
	"data_transferred":		        0.12,
	"response_time":		        0.85,
	"transaction_rate":		      357.02,
	"throughput":			        0.00,
	"concurrency":			      304.87,
	"successful_transactions":	       10000,
	"failed_transactions":		           0,
	"longest_transaction":		       27.67,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为15,观察CPU约为50%上下,结果如下,系统有报错:

$ siege -c 1000 -r 15 http://192.168.1.100:5000
...
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
...

{	"transactions":			       14984,
	"availability":			       99.89,
	"elapsed_time":			       57.53,
	"data_transferred":		        0.19,
	"response_time":		        1.20,
	"transaction_rate":		      260.46,
	"throughput":			        0.00,
	"concurrency":			      313.80,
	"successful_transactions":	       14984,
	"failed_transactions":		          16,
	"longest_transaction":		       55.05,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为30,观察CPU约为50%上下,结果如下,系统有报错:

$ siege -c 1000 -r 30 http://192.168.1.100:5000
...
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:635: Connection reset by peer
...
{	"transactions":			       29710,
	"availability":			       99.03,
	"elapsed_time":			      119.20,
	"data_transferred":		        0.37,
	"response_time":		        0.68,
	"transaction_rate":		      249.24,
	"throughput":			        0.00,
	"concurrency":			      170.37,
	"successful_transactions":	       29710,
	"failed_transactions":		         290,
	"longest_transaction":		       57.13,
	"shortest_transaction":		        0.00
}

ACTIX-WEB(后续简称ACTIX)

  • actix-web = "2"
  • actix-rt = "1.0"

如下将重复数量设置为10,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 10 http://127.0.0.1:8000

{	"transactions":			       10000,
	"availability":			      100.00,
	"elapsed_time":			        2.72,
	"data_transferred":		        0.11,
	"response_time":		        0.22,
	"transaction_rate":		     3676.47,
	"throughput":			        0.04,
	"concurrency":			      815.28,
	"successful_transactions":	       10000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.32,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为20,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 20 http://127.0.0.1:8000

{	"transactions":			       20000,
	"availability":			      100.00,
	"elapsed_time":			        4.91,
	"data_transferred":		        0.23,
	"response_time":		        0.20,
	"transaction_rate":		     4073.32,
	"throughput":			        0.05,
	"concurrency":			      815.21,
	"successful_transactions":	       20000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.05,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为30,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 30 http://127.0.0.1:8000

{	"transactions":			       30000,
	"availability":			      100.00,
	"elapsed_time":			        7.66,
	"data_transferred":		        0.34,
	"response_time":		        0.22,
	"transaction_rate":		     3916.45,
	"throughput":			        0.04,
	"concurrency":			      878.44,
	"successful_transactions":	       30000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.21,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为40,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 40 http://127.0.0.1:8000

{	"transactions":			       40000,
	"availability":			      100.00,
	"elapsed_time":			       10.32,
	"data_transferred":		        0.46,
	"response_time":		        0.23,
	"transaction_rate":		     3875.97,
	"throughput":			        0.04,
	"concurrency":			      882.39,
	"successful_transactions":	       40000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.28,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为100,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 100 http://127.0.0.1:8000

{	"transactions":			      100000,
	"availability":			      100.00,
	"elapsed_time":			       24.92,
	"data_transferred":		        1.14,
	"response_time":		        0.23,
	"transaction_rate":		     4012.84,
	"throughput":			        0.05,
	"concurrency":			      910.48,
	"successful_transactions":	      100000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.09,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为200,观察CPU大约90%,结果如下,系统无报错:

$ siege -c 1000 -r 200 http://127.0.0.1:8000

{	"transactions":			      200000,
	"availability":			      100.00,
	"elapsed_time":			       49.66,
	"data_transferred":		        2.29,
	"response_time":		        0.23,
	"transaction_rate":		     4027.39,
	"throughput":			        0.05,
	"concurrency":			      932.71,
	"successful_transactions":	      200000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.30,
	"shortest_transaction":		        0.00
}

我觉得再测下去意义不大了,就是一个等待时间问题,从指标上看,重复数量越多,结果越好,估计是开始和结束的开销被稀释了。

WARP

  • warp = "0.2"

如下将重复数量设置为10,观察CPU大约50%,结果如下,系统无报错:

$ siege -c 1000 -r 10 http://127.0.0.1:3030/hello/warp

{	"transactions":			       10000,
	"availability":			      100.00,
	"elapsed_time":			        2.67,
	"data_transferred":		        0.11,
	"response_time":		        0.23,
	"transaction_rate":		     3745.32,
	"throughput":			        0.04,
	"concurrency":			      872.75,
	"successful_transactions":	       10000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.31,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为30,观察CPU大约50%,结果如下,系统无报错:

$ siege -c 1000 -r 30 http://127.0.0.1:3030/hello/warp

{	"transactions":			       30000,
	"availability":			      100.00,
	"elapsed_time":			        7.68,
	"data_transferred":		        0.34,
	"response_time":		        0.24,
	"transaction_rate":		     3906.25,
	"throughput":			        0.04,
	"concurrency":			      926.62,
	"successful_transactions":	       30000,
	"failed_transactions":		           0,
	"longest_transaction":		        1.64,
	"shortest_transaction":		        0.00
}

如下将重复数量设置为200,观察CPU大约50%,结果如下,系统无报错:

$ siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp

{	"transactions":			      200000,
	"availability":			      100.00,
	"elapsed_time":			       51.50,
	"data_transferred":		        2.29,
	"response_time":		        0.26,
	"transaction_rate":		     3883.50,
	"throughput":			        0.04,
	"concurrency":			      993.57,
	"successful_transactions":	      200000,
	"failed_transactions":		           0,
	"longest_transaction":		        0.73,
	"shortest_transaction":		        0.00
}

总结

由于我主要关注的是几个框架之间的差别,所以只用了siege一个压测工具,以后有机会再使用apache benchmark或者社员建议的vegeta试试看。

只看测试指标,WARP的表现似乎最好,它的CPU利用率明显比ACTIX低,而同时其它指标与ACTIX不相上下,甚至更好,但不明白为什么WARP的整体测试耗时(elapsed_time)比ACTIX要多几秒钟。

WARP与ACTIX表现都比较稳定,压测不出错,而且指标上下浮动不大。TIDE重复数量到30以后就开始系统报错了,而且多次测试结果相差较大,可能正如其开发者所说,项目仍然不建议被用于生产,还不太成熟吧。类似的FLASK测试重复数量到15以后就开始系统报错,并且多次测试结果上下浮动较大(上下各50%,限于篇幅,这个结果没有在文章中体现)

评论区

写评论
shaitao 2020-06-29 15:56

actix貌似会泄漏内存, 多压压会不会炸

作者 bluephoen1x 2020-06-29 09:46

另外回答一下社员的一些疑问:

  1. http 1.1协议吗? 是
  2. 都是生产模式/release下吗? 是生产/release模式
  3. 日志都关闭了吗? 关了
  4. 都设置了相同的进程/worker数吗(尤其是Flask同学)? Flask同学是单线/进程,因为在不依赖于其它工具的情况下,Flask开启多线/进程,结果更差。我测试的目的只在于纯框架,所以依赖其它工具的结果就算更好,也不是我关注的焦点。而Rust系的框架我也不知道算单线/进程还是多线/进程,他们都是异步框架,这个我没学明白。
  5. 都预热过吗? 这个我不懂,没有特殊处理,想必是没有预热吧。从同样环境多次测试结果看,rust系的框架结果上下浮动不大,是否说明rust系框架不需要预热,而flask多次测试结果上下偏差很大(可以达到50%),估计是需要您所说的预热吧。
  6. 内网带宽是否一致(比如有没有人突然看视频破解Wi-Fi密码什么的导致路由器抖动)? 本机测试。
  7. 测试机和服务器是否有其他进程影响(i3处理器软件开多点可能来个信息都能卡一下)? 都是本机测试,我把能关的程序都关了,我感觉对测试影响不大。
  8. 是否有经过虚拟机、docker,是否经过虚拟网卡? 无虚拟机,无docker,无虚拟网卡。
作者 bluephoen1x 2020-06-29 09:35

昨天actix-web测试总耗时比warp高一倍的原因找到了,就是ulimit默认为1024导致的,将其改为2000,actix-web的指标就变好了,下面是ulimit -n 2000之后的测试结果:

$ siege -c 1000 -r 200 http://127.0.0.1:8000/actix

{ "transactions": 200000, "availability": 100.00, "elapsed_time": 52.88, "data_transferred": 2.29, "response_time": 0.25, "transaction_rate": 3782.15, "throughput": 0.04, "concurrency": 942.51, "successful_transactions": 200000, "failed_transactions": 0, "longest_transaction": 1.45, "shortest_transaction": 0.00 } $ siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp

{ "transactions": 200000, "availability": 100.00, "elapsed_time": 51.64, "data_transferred": 2.29, "response_time": 0.25, "transaction_rate": 3872.97, "throughput": 0.04, "concurrency": 986.77, "successful_transactions": 200000, "failed_transactions": 0, "longest_transaction": 1.52, "shortest_transaction": 0.00 }

下面是actix的CPU利用率与内在使用统计,第三列是CPU(单位:100%),第四列是内存(单位:100%) $ ps aux |grep actix yw 3338 84.8 0.3 351272 28940 pts/2 Sl+ 09:13 1:36 ./of-actix yw 3598 122 5.7 10290008 452776 pts/3 Sl+ 09:14 0:53 siege -c 1000 -r 200 http://127.0.0.1:8000/actix

下面是warp的CPU利用率与内在使用统计,第三列是CPU(单位:100%),第四列是内存(单位:100%) $ ps aux |grep warp yw 4973 66.6 0.0 10352 7784 pts/2 R+ 09:18 0:14 ./of-warp yw 4974 117 6.1 10309324 479468 pts/3 Sl+ 09:18 0:17 siege -c 1000 -r 200 http://127.0.0.1:3030/hello/warp

这次统计可以看出,actix和warp在CPU利用率上都可以比测试工具更低,warp比actix的CPU利用率仍然低一些,特别是warp比actix明显省内存。

1 共 3 条评论, 1 页